I noticed yesterday that I was not receiving mail from one of my mail accounts. I’m a fetchmail user, so I immediately went to the log file. I won’t reproduce the exact message, but essentially fetchmail couldn’t validate the ssl certificate from the server because it didn’t think I had the appropriate root certificate.
And just like that, I was off on another expedition into the wonderful land of computer administrivia.
The problem, for me, was that I did, in fact, have the root certificate. What’s more, that certificate had been working just fine as of a couple of weeks ago. Being a debian user, I have ca-certificates installed and I recalled an update to that package recently. So my first check was to see if the ca-certificates package was still recognizing the appropriate root ca.
It turned out I had that certificate installed redundantly: once in the /usr/share/ca-certificates directory and again in the /usr/local/share/ca-certificates directory where custom certificates are supposed to reside. While it appeared that both instances were being recognized, I decided that the redundancy was unnecessary and possibly messing things up. So, just to make a change and see if it fixed things, I eliminated the redundant entry in the main /usr/share/ directory branch.
No dice.
My next thought was perhaps the certificate had expired. To test this theory, I pointed the fetchmailrc right to the certificate using the sslcertfile option. That brought things back to life, so now I knew my certificate was valid.
My conclusion was that, for some reason, the ca-certficate package was only half-recognizing the certificate file. While inspecting the /etc/ssl/certs/ directory, I finally noticed that my custom certificate did not have any of the symbolic links that looked like hashes. There was only a pem symbolic link. So I tried a number of different command line incantations involving update-ca-certificates and dpkg-reconfigure ca-certificates but nothing brought me a resolution.
So I turned to google to see if anyone else had run into this problem. No such luck. But I did find one final ray of hope: ca-certificates is persnickety about the PEM file format of the certificate files. It didn’t make complete sense to me that the file format was the issue, since it sure looked like a PEM formatted certificate file. It had even been working up until the most recent update to the package!
But without any other good looking options, I used the following command-line incantation on the certificate file in question:
openssl x509 -in certfile.crt -out newcertfile.crt -outform PEM
Lo and behold, the file size changed. It got smaller. As it happens, it got 1 byte smaller for each line in the file. Subsequent dickering with diff revealed that it’s likely a line termination issue. So I installed the new file and ran the update-ca-certificates command. Lo and behold, the hashed symbolic links for my custom certificate appeared. More importantly, fetchmail once again could retrieve mail from that server.
And that’s how I fixed my custom certificate issue with ca-certificates.