In modifying stunnel to work with the OpenSSL FIPS-certified module, I found out that private keys cannot be in the default OpenSSL format; they must be in PKCS8 format (due to the MD5 algorithm being disabled in FIPS mode).
Talking to Dr. Henson of the OpenSSL group I was able to convert my private keys from the default format to PKCS8, but I found that stunnel was not able to handle these keys. I got the following error output (note that server.pem contains a PKCS8 private key and a PKCS7 public cert): 2006.06.08 17:49:38 LOG7[1120:616]: Certificate: server.pem 2006.06.08 17:49:38 LOG7[1120:616]: Key file: server.pem 2006.06.08 17:49:42 LOG3[1120:616]: error stack: 140B3009 : error:140B3009:SSL routines:SSL_CTX_use_RSAPrivateKey_file:PEM lib 2006.06.08 17:49:42 LOG3[1120:616]: error stack: 906700D : error:0906700D:PEM routines:PEM_ASN1_read_bio:ASN1 lib 2006.06.08 17:49:42 LOG3[1120:616]: error stack: 2306A075 : error:2306A075:PKCS12 routines:PKCS12_DECRYPT_D2I:pkcs12 pbe crypt error 2006.06.08 17:49:42 LOG3[1120:616]: error stack: 23077073 : error:23077073:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 algor cipherinit error 2006.06.08 17:49:42 LOG3[1120:616]: SSL_CTX_use_RSAPrivateKey_file: 6074079: error:06074079:digital envelope routines:EVP_PBE_CipherInit:unknown pbe algorithm
2006.06.08 17:49:42 LOG3[1120:616]: Server is down
Continuing with Dr. Henson, he informed me that this ssl error output meant that "the PBE table has not been initialized in the application". I corrected this by adding a call to OpenSSL_add_all_algorithms() into ssl_init() in ssl.c. This fixed my error with the PEM files.
I am posting this so that this bug may be fixed for the next stunnel release. Additionally, I'd like verification that my fix makes sense given the structure of the stunnel code. From doing some searches in the stunnel source, I could not find any existing code that initialized the PBE table so adding the call to ssl_init() made sense _to me_. Michal: do you agree?