It seems like I go through this sort of thing every couple of months. Some kind of email retrieval issue crops up, typically involving SSL. Luckily, I take the time to jot it down so I can refer back in case it’s a repeat. Today’s problem turned out to be something new, so it now gets its own place here on the blog.
So, as I said fetchmail wasn’t retrieving mail from one of my email accounts. It wasn’t likely a problem with fetchmail per say, as the first hint regarding the problem was a cryptic error message in the log file:
Server certificate verification error: unable to get local issuer certificate
fetchmail: This means that the root signing certificate (issued for
/C=US/ST=New Hampshire/L=Hanover/O=Dartmouth College/OU=Alumni
Relations/CN=yertle2.dartmouth.org) is not in the trusted CA certificate
locations, or that c_rehash needs to be run on the certificate directory. For
details, please see the documentation of --sslcertpath and --sslcertfile in
the manual page.
140057131230888:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:1166: fetchmail: SSL connection failed.
Well, it wasn’t entirely cryptic. It looked like something might have gone sideways with my root certificate for this particular email server. I run a debian/testing setup and the root certificate is not part of the standard ca-certificates package. So my initial investigation revolved around verifying the current certificate. The short of the story there is, I had no obvious errors with the root certificate.
Unfortunately, this made the error very cryptic: complaining that the root certificate isn’t where it’s supposed to be when, by all appearances it was right there. I even tried using fetchmail‘s sslcertfile option, with no luck.
So I decided to use the openssl command line tools to try and debug the problem. The following command:
openssl s_client -connect yertle2.dartmouth.org:995
yielded the following output:
CONNECTED(00000003)
depth=0 C = US, ST = New Hampshire, L = Hanover, O = Dartmouth College, OU = Alumni Relations, CN = yertle2.dartmouth.org
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = New Hampshire, L = Hanover, O = Dartmouth College, OU = Alumni Relations, CN = yertle2.dartmouth.org
verify error:num=27:certificate not trusted
verify return:1
depth=0 C = US, ST = New Hampshire, L = Hanover, O = Dartmouth College, OU = Alumni Relations, CN = yertle2.dartmouth.org
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/C=US/ST=New Hampshire/L=Hanover/O=Dartmouth College/OU=Alumni Relations/CN=yertle2.dartmouth.org
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3
---
Server certificate
-----BEGIN CERTIFICATE-----
.
.
.
-----END CERTIFICATE-----
subject=/C=US/ST=New Hampshire/L=Hanover/O=Dartmouth College/OU=Alumni Relations/CN=yertle2.dartmouth.org
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3
---
No client certificate CA names sent
---
SSL handshake has read 1781 bytes and written 518 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : AES256-SHA
Session-ID: 3CDD3CFE066DBAF78E0251EF5D726E6C2DE4DDB45EB5817EA147E0FB6BB6F84C
Session-ID-ctx:
Master-Key: 2BCA7C13F22CB5E1F4253BECBC0F30291781B1AE64EEE9CEC68B686EC90094F91946AD0FB45BEA0E91E7D295E5AA7472
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1350090920
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
---
The error code that caught my eye was the one that reads verify. I did some googling on that and the beginning of a solution began to take shape. In particular, this page proved very informative.
error:num=21:unable to verify the first certificate
The problem here is something called an ‘intermediate certificate.’ I wasn’t familiar with them previous to this experience because they are typically handled under the hood. Apparently, browser’s use them all the time and will automatically find and install them. The thing about intermediate certificates is they provide better security because of the probabilities involved in duplicating multiple, chained certificates.
So the problem I was running into was that my server was missing the intermediate certificate and, unlike a browser, fetchmail won’t find and install them. Fortunately, the above output also contains the name of that certificate: Digicert High Assurance CA-3. A little bit more googling quickly revealed a download area where I could download the file via a secure connection with the browser. I did so, but wasn’t able to use it as is because the certificate file was in a binary format. To convert it to a PEM file format, I had to use the following command:
openssl -in file.crt -inform DER -out file.crt -outform PEM
Now I had a proper certificate file that ca-certificates would recognize. I installed it in the same directory as the root certificate it was chained to, in this case /usr/local/share/ca-certificates/ and then updated the SSL database (using the update-ca-certificates command on the command-line). After verifying that the new file had been properly handled, I tested it once again using openssl.
I got the same output as above, which was disconcerting considering everything seemed to be setup appropriately. Some more investigation revealed the problem was that I needed to specify the path with certificate chain in the openssl command. So the command that revealed everything worked again was:
openssl s_client -connect yertle2.dartmouth.org:995 -CApath /etc/ssl/certs
The path option could just as easily be the exact directory with the certificate chain. The verify return code came back as 0, meaning success.
Several minutes later, fetchmail was retrieving emails again from the server.