STARTTLS negotiation errors affecting mail

I just corrected a problem that was causing mail to back up on one of my home Linux machines that was trying to send mail to my mail server. I’m concerned that it may have a wider impact on email soon.

Starting June 25, my home Linux machine started reporting errors and queuing mail to my mail server. The error log on my home machine reported:

Jun 25 07:35:11 kernel sm-mta[5232]: STARTTLS=client: 5232:error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small:s3_clnt.c:3339:

I did a little searching, and found this page that proposed a possible solution, then updated itself to report that it didn’t work. On my mail server, I created a new dhparams file and configured sendmail to refer to it (as described in the article, except I already had defined cipher suites so I didn’t do that part).  I rebuilt the config and restarted sendmail, and it looks like everything is working. For me.

My bigger concern is this: I was lucky; the system encountering the error was sending to a server under my own control. But email servers typically talk across domains, and if the error happens sending to someone else’s email server, the options aren’t very pretty. One option is to turn off STARTTLS, but that is the opposite of the intent of the openssl change. Another might be to revert the openssl update.

Apparently a 512-bit DH key is a pretty common default. The change requires a 768 bit minimum key size, so the question is what to do when the key is too small. Perhaps changing the default and then giving more time before making it a requirement would minimize the breakage.