- stunnel-users - stunnel.org

segfault with v5.55
by Mark Brookes 30 Sep '19

30 Sep '19

We are currently experiencing a segfault with version 5.55 openssl version 1.1.0k and are trying to narrow down what the issue may be. The first 2 segfaults are about 10 mins apart. The below is from the stunnel log just before and then after the event. Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Deallocating application specific data for session connect address Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Deallocating application specific data for session connect address Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Deallocating application specific data for session connect address Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Deallocating application specific data for session connect address Sep 23 11:47:31 xxxx stunnel: LOG7[77582]: TLS state (accept): SSLv3/TLS write finished Sep 23 11:47:31 xxxx stunnel: LOG7[77582]: TLS state (accept): SSLv3/TLS read change cipher spec Sep 23 11:47:31 xxxx stunnel: LOG7[77582]: TLS state (accept): SSLv3/TLS read finished Sep 23 11:47:31 xxxx stunnel: LOG7[77580]: TLS state (accept): SSLv3/TLS write finished Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Deallocating application specific data for session connect address Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Deallocating application specific data for session connect address Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Deallocating application specific data for session connect address Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Deallocating application specific data for session connect address Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Deallocating application specific data for session connect address Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Deallocating application specific data for session connect address Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Deallocating application specific data for session connect address Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Deallocating application specific data for session connect address Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Deallocating application specific data for session connect address Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Deallocating application specific data for session connect address Sep 23 11:47:31 xxxx stunnel: LOG7[77581]: Remove session callback Sep 23 11:52:58 xxxx stunnel: LOG7[ui]: Clients allowed=62937 Sep 23 11:52:58 xxxx stunnel: LOG5[ui]: stunnel 5.55 on x86_64-pc-linux-gnu platform Sep 23 11:52:58 xxxx stunnel: LOG5[ui]: Compiled/running with OpenSSL 1.1.0k 28 May 2019 Sep 23 11:52:58 xxxx stunnel: LOG5[ui]: Threading:PTHREAD Sockets:POLL,IPv6 TLS:ENGINE,OCSP,PSK,SNI Sep 23 11:52:58 xxxx stunnel: LOG7[ui]: errno: (*__errno_location ()) The following is from the kernel log. stunnel[31253]: segfault at 0 ip 00007f91214cacea sp 00007f91205ea9f8 error 6 in libpthread-2.12.so[7f91214c0000+17000] The kernel log after the second segfault- stunnel[1868]: segfault at 0 ip 00007f5b35398cea sp 00007f5b344c59f8 error 6 in libpthread-2.12.so[7f5b3538e000+17000] This is all the information we have at the moment. -- Mark Brookes Loadbalancer.org Ltd. www.loadbalancer.org +44 (0)330 380 1064 mark(a)loadbalancer.org

1 0

Re: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections
by Lynch, Andrew 27 Sep '19

27 Sep '19

I’ve been digging around the code and I am no longer sure whether stunnel is at fault or it is actually an issue in OpenSSL! load_key_engine() in ctx.c shoves the key into the SSL context – SSL_CTX_use_PrivateKey(section->ctx, pkey). So the key should be freed when that context is disposed of. The SIGHUP ends up in case SIGNAL_RELOAD_CONFIG of signal_pipe_dispatch(), which calls options_free() [stunnel.c line 830]. Eventually there is an SSL_CTX_free() call, but it looks like this does not actually free the private key. Ephemeral keys are freed and loads of other stuff, but not the private key associated with the certificate. I may try to build a hacked OpenSSL on Monday to see what happens if SSL_CTX_free also frees the private key… Regards, Andrew. From: Brent Kimberley [mailto:[email protected]] Sent: Friday, September 27, 2019 4:29 PM To: stunnel-users(a)stunnel.org; Lynch, Andrew Subject: Re: stunnel-users Digest, Vol 182, Issue 14 >> Over time, after numerous reloads of stunnel (kill -HUP) the HSM reports that its connection table is full. >> Logging from the engine shows that stunnel is never freeing the keys and therefore the engine is not closing the associated sessions with the HSM. Each stunnel reload opens 70 new sessions until eventually the HSM's configured limit is exceeded. Where does the signal dispatcher free keys on SIG_HUP? https://github.com/mtrojnar/stunnel/blob/master/src/stunnel.c Date: Fri, 27 Sep 2019 07:23:29 +0000 From: "Lynch, Andrew" <andrew.lynch(a)atos.net<mailto:[email protected]>> To: Eric Eberhard <flash(a)vicsmba.com<mailto:[email protected]>> Cc: "stunnel-users(a)stunnel.org<mailto:[email protected]>" <stunnel-users(a)stunnel.org<mailto:[email protected]>> Subject: Re: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections Hi Eric, Thank you for your suggestion. It is certainly worth looking into, although I suspect it may be impractical in our environment. Now we have a single stunnel.conf with 70 services which would translate into 70 additional entries in inetd.conf. Each of those entries then needs to have its own reduced stunnel.conf so that it will only load the cert and key for its endpoint. The stunnel HOWTO recommends daemon mode over inetd mode due to the overhead for forking and SSL initialization. We would have additional overhead for engine init and HSM connection. Regards, Andrew. -----Original Message----- From: Eric Eberhard [mailto:[email protected]<mailto:[email protected]>] Sent: Thursday, September 26, 2019 7:20 PM To: Lynch, Andrew; stunnel-users(a)stunnel.org<mailto:[email protected]> Subject: RE: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections? Try running in inetd mode -- even if you don't like this you will learn something. Inetd will close connections as needed. E -----Original Message----- From: stunnel-users [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Lynch, Andrew Sent: Thursday, September 26, 2019 5:56 AM To: stunnel-users(a)stunnel.org<mailto:[email protected]> Subject: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections? Hi, We are using stunnel as a server to terminate incoming TLS connections. The config has around 70 services with certificates whose EC private keys are stored in an HSM and accessed using an OpenSSL engine. Over time, after numerous reloads of stunnel (kill -HUP) the HSM reports that its connection table is full. Logging from the engine shows that stunnel is never freeing the keys and therefore the engine is not closing the associated sessions with the HSM. Each stunnel reload opens 70 new sessions until eventually the HSM's configured limit is exceeded. This behaviour has been observed on Suse Enterprise Linux 12.3 with the system-provided stunnel-5.00-4.3.4, but I can reproduce it with my own build of the current version 5.55. Is this a known issue? It appears that other (ephemeral) keys are being freed, just not those associated with the service certificates. Currently our workaround is to perform a full restart instead of a reload. This closes all HSM sessions when the process terminates, but of course it also kills any open client connections so it can only be done during the scheduled maintenance windows. Regards, Andrew. _______________________________________________ stunnel-users mailing list stunnel-users(a)stunnel.org<mailto:[email protected]> https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users ------------------------------ Subject: Digest Footer _______________________________________________ stunnel-users mailing list stunnel-users(a)stunnel.org<mailto:[email protected]> https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users ------------------------------ End of stunnel-users Digest, Vol 182, Issue 14 **********************************************

2 1

Re: [stunnel-users] stunnel-users Digest, Vol 182, Issue 14
by Eric Eberhard 27 Sep '19

27 Sep '19

I don’t suggest it as a final solution – although for me it works great and we have customers exchanging 2-4 million XML documents per day with no performance issues – BUT no matter what it is a good debugging tool. Inetd gets all the initial connections and closes them when done. It might or might or might not be involved with the HSM at all. You might also look at the process table (some form of ps) and grep for stunnel and relatives and see if they are hanging around. In theory, if a process dies, it “should” close it’s connections. If it is floating in space it will not. Inetd will kill the process – at least on Centos and AIX and SCO – for sure and for certain. E From: stunnel-users [mailto:[email protected]] On Behalf Of Brent Kimberley Sent: Friday, September 27, 2019 7:29 AM To: stunnel-users(a)stunnel.org; andrew.lynch(a)atos.net Subject: Re: [stunnel-users] stunnel-users Digest, Vol 182, Issue 14 >> Over time, after numerous reloads of stunnel (kill -HUP) the HSM reports that its connection table is full. >> Logging from the engine shows that stunnel is never freeing the keys and therefore the engine is not closing the associated sessions with the HSM. Each stunnel reload opens 70 new sessions until eventually the HSM's configured limit is exceeded. Where does the signal dispatcher free keys on SIG_HUP? https://github.com/mtrojnar/stunnel/blob/master/src/stunnel.c Date: Fri, 27 Sep 2019 07:23:29 +0000 From: "Lynch, Andrew" <andrew.lynch(a)atos.net <mailto:[email protected]> > To: Eric Eberhard <flash(a)vicsmba.com <mailto:[email protected]> > Cc: "stunnel-users(a)stunnel.org <mailto:[email protected]> " <stunnel-users(a)stunnel.org <mailto:[email protected]> > Subject: Re: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections Hi Eric, Thank you for your suggestion. It is certainly worth looking into, although I suspect it may be impractical in our environment. Now we have a single stunnel.conf with 70 services which would translate into 70 additional entries in inetd.conf. Each of those entries then needs to have its own reduced stunnel.conf so that it will only load the cert and key for its endpoint. The stunnel HOWTO recommends daemon mode over inetd mode due to the overhead for forking and SSL initialization. We would have additional overhead for engine init and HSM connection. Regards, Andrew. -----Original Message----- From: Eric Eberhard [mailto:[email protected] <mailto:[email protected]> ] Sent: Thursday, September 26, 2019 7:20 PM To: Lynch, Andrew; stunnel-users(a)stunnel.org <mailto:[email protected]> Subject: RE: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections? Try running in inetd mode -- even if you don't like this you will learn something. Inetd will close connections as needed. E -----Original Message----- From: stunnel-users [mailto:[email protected] <mailto:[email protected]> ] On Behalf Of Lynch, Andrew Sent: Thursday, September 26, 2019 5:56 AM To: stunnel-users(a)stunnel.org <mailto:[email protected]> Subject: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections? Hi, We are using stunnel as a server to terminate incoming TLS connections. The config has around 70 services with certificates whose EC private keys are stored in an HSM and accessed using an OpenSSL engine. Over time, after numerous reloads of stunnel (kill -HUP) the HSM reports that its connection table is full. Logging from the engine shows that stunnel is never freeing the keys and therefore the engine is not closing the associated sessions with the HSM. Each stunnel reload opens 70 new sessions until eventually the HSM's configured limit is exceeded. This behaviour has been observed on Suse Enterprise Linux 12.3 with the system-provided stunnel-5.00-4.3.4, but I can reproduce it with my own build of the current version 5.55. Is this a known issue? It appears that other (ephemeral) keys are being freed, just not those associated with the service certificates. Currently our workaround is to perform a full restart instead of a reload. This closes all HSM sessions when the process terminates, but of course it also kills any open client connections so it can only be done during the scheduled maintenance windows. Regards, Andrew. _______________________________________________ stunnel-users mailing list stunnel-users(a)stunnel.org <mailto:[email protected]> https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users ------------------------------ Subject: Digest Footer _______________________________________________ stunnel-users mailing list stunnel-users(a)stunnel.org <mailto:[email protected]> https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users ------------------------------ End of stunnel-users Digest, Vol 182, Issue 14 **********************************************

1 0

Re: [stunnel-users] stunnel-users Digest, Vol 182, Issue 14
by Brent Kimberley 27 Sep '19

27 Sep '19

>> Over time, after numerous reloads of stunnel (kill -HUP) the HSM reports that its connection table is full. >> Logging from the engine shows that stunnel is never freeing the keys and therefore the engine is not closing the associated sessions with the HSM. Each stunnel reload opens 70 new sessions until eventually the HSM's configured limit is exceeded. Where does the signal dispatcher free keys on SIG_HUP?https://github.com/mtrojnar/stunnel/blob/master/src/stunnel.c Date: Fri, 27 Sep 2019 07:23:29 +0000 From: "Lynch, Andrew" <andrew.lynch(a)atos.net> To: Eric Eberhard <flash(a)vicsmba.com> Cc: "stunnel-users(a)stunnel.org" <stunnel-users(a)stunnel.org> Subject: Re: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections Hi Eric, Thank you for your suggestion. It is certainly worth looking into, although I suspect it may be impractical in our environment. Now we have a single stunnel.conf with 70 services which would translate into 70 additional entries in inetd.conf. Each of those entries then needs to have its own reduced stunnel.conf so that it will only load the cert and key for its endpoint. The stunnel HOWTO recommends daemon mode over inetd mode due to the overhead for forking and SSL initialization. We would have additional overhead for engine init and HSM connection. Regards, Andrew. -----Original Message----- From: Eric Eberhard [mailto:[email protected]] Sent: Thursday, September 26, 2019 7:20 PM To: Lynch, Andrew; stunnel-users(a)stunnel.org Subject: RE: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections? Try running in inetd mode -- even if you don't like this you will learn something. Inetd will close connections as needed. E -----Original Message----- From: stunnel-users [mailto:[email protected]] On Behalf Of Lynch, Andrew Sent: Thursday, September 26, 2019 5:56 AM To: stunnel-users(a)stunnel.org Subject: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections? Hi, We are using stunnel as a server to terminate incoming TLS connections. The config has around 70 services with certificates whose EC private keys are stored in an HSM and accessed using an OpenSSL engine. Over time, after numerous reloads of stunnel (kill -HUP) the HSM reports that its connection table is full. Logging from the engine shows that stunnel is never freeing the keys and therefore the engine is not closing the associated sessions with the HSM. Each stunnel reload opens 70 new sessions until eventually the HSM's configured limit is exceeded. This behaviour has been observed on Suse Enterprise Linux 12.3 with the system-provided stunnel-5.00-4.3.4, but I can reproduce it with my own build of the current version 5.55. Is this a known issue? It appears that other (ephemeral) keys are being freed, just not those associated with the service certificates. Currently our workaround is to perform a full restart instead of a reload. This closes all HSM sessions when the process terminates, but of course it also kills any open client connections so it can only be done during the scheduled maintenance windows. Regards, Andrew. _______________________________________________ stunnel-users mailing list stunnel-users(a)stunnel.org https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users ------------------------------ Subject: Digest Footer _______________________________________________ stunnel-users mailing list stunnel-users(a)stunnel.org https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users ------------------------------ End of stunnel-users Digest, Vol 182, Issue 14 **********************************************

1 0

Re: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections?
by Eric Eberhard 27 Sep '19

27 Sep '19

Try running in inetd mode -- even if you don't like this you will learn something. Inetd will close connections as needed. E -----Original Message----- From: stunnel-users [mailto:[email protected]] On Behalf Of Lynch, Andrew Sent: Thursday, September 26, 2019 5:56 AM To: stunnel-users(a)stunnel.org Subject: [stunnel-users] stunnel with OpenSSL engine: reload leaks HSM connections? Hi, We are using stunnel as a server to terminate incoming TLS connections. The config has around 70 services with certificates whose EC private keys are stored in an HSM and accessed using an OpenSSL engine. Over time, after numerous reloads of stunnel (kill -HUP) the HSM reports that its connection table is full. Logging from the engine shows that stunnel is never freeing the keys and therefore the engine is not closing the associated sessions with the HSM. Each stunnel reload opens 70 new sessions until eventually the HSM's configured limit is exceeded. This behaviour has been observed on Suse Enterprise Linux 12.3 with the system-provided stunnel-5.00-4.3.4, but I can reproduce it with my own build of the current version 5.55. Is this a known issue? It appears that other (ephemeral) keys are being freed, just not those associated with the service certificates. Currently our workaround is to perform a full restart instead of a reload. This closes all HSM sessions when the process terminates, but of course it also kills any open client connections so it can only be done during the scheduled maintenance windows. Regards, Andrew. _______________________________________________ stunnel-users mailing list stunnel-users(a)stunnel.org https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users

2 1

stunnel with OpenSSL engine: reload leaks HSM connections?
by Lynch, Andrew 26 Sep '19

26 Sep '19

Hi, We are using stunnel as a server to terminate incoming TLS connections. The config has around 70 services with certificates whose EC private keys are stored in an HSM and accessed using an OpenSSL engine. Over time, after numerous reloads of stunnel (kill -HUP) the HSM reports that its connection table is full. Logging from the engine shows that stunnel is never freeing the keys and therefore the engine is not closing the associated sessions with the HSM. Each stunnel reload opens 70 new sessions until eventually the HSM's configured limit is exceeded. This behaviour has been observed on Suse Enterprise Linux 12.3 with the system-provided stunnel-5.00-4.3.4, but I can reproduce it with my own build of the current version 5.55. Is this a known issue? It appears that other (ephemeral) keys are being freed, just not those associated with the service certificates. Currently our workaround is to perform a full restart instead of a reload. This closes all HSM sessions when the process terminates, but of course it also kills any open client connections so it can only be done during the scheduled maintenance windows. Regards, Andrew.

1 0

Re: [stunnel-users] Problems when using PSK between a version of stunnel usingopenssl 1.0 and a version of stunnel using openssl 1.1.1 in stunnel-users Digest, Vol 182, Issue 7
by Koperberg, J. (Hannah) 25 Sep '19

25 Sep '19

Hi Brent Thank you for your suggestions. I installed stunnel version 5.55 on the client test side (where the openssl version OpenSSL 1.1.1c is). That has solved the problem. Kind regards and thanks again for your help, Hannah Koperberg, de Volksbank N.V., The Netherlands From: Brent Kimberley <brent_kimberley(a)rogers.com> Sent: woensdag 18 september 2019 18:30 To: stunnel-users(a)stunnel.org Subject: Re: [stunnel-users] stunnel-users Digest, Vol 182, Issue 7 psk_find() and psk_server_callback() are marked NOEXPORT. You can try whitebox - decompile ctx.c:: psk_server_callback() and/or blackbox - breaking the problem into parts to see what works/fails. Decomposing the problem into parts : a) Two differing stunnel versions and a common openssl library. b) One common stunnel version and two differing openssl libraries. On Wednesday, September 18, 2019, 11:28:48 a.m. EDT, Brent Kimberley <brent_kimberley(a)rogers.com<mailto:[email protected]>> wrote: >> 2019.09.16 13:36:28 LOG6[3]: PSK identity not found (session resumption?) Try dumping the symbols for ctx.c::psk_find(), ssl_lib.c::SSL_get_ex_data(), and ctx.c:: psk_server_callback(). ________________________________ Dit e-mailbericht is alleen bestemd voor de geadresseerden. Indien dit bericht niet voor u is bedoeld, wordt u verzocht de afzender hiervan op de hoogte te stellen door het bericht te retourneren en de inhoud niet te gebruiken. This e-mail message is intended to be exclusively for the addressee. If you are not the intended recipient you are kindly requested not to make any use whatsoever of the contents and to notify the sender immediately by returning this e-mail message.

2 1

Re: [stunnel-users] stunnel-users Digest, Vol 182, Issue 11
by Brent Kimberley 24 Sep '19

24 Sep '19

>>We are running stunnel-5.49-1.ppc on AIX 7100-05-03-1846. >>It will stay up awhile and our connection tests work, but it core dumps. Here’s the message in the log file. Thoughts? >>INTERNAL ERROR: Out of memory at s3_both.c, line 626 Hi Gerald. You may have found a memory leak. Try looking at: 1) line 626 of OpenSSL::s3_booth.c https://github.com/openssl/openssl/blob/OpenSSL_1_0_2-stable/ssl/s3_both.c 2) line 296 of stunnel::str.c ; and/or https://github.com/mtrojnar/stunnel/blob/stunnel-5.49/src/str.c 3) dbx, truss, or probevue - in an offline dev environment. http://archive.ibmsystemsmag.com/aix/administrator/performance/debugging-me…

1 0

AIX:stunnel core dumping
by Gerald Shaw 24 Sep '19

24 Sep '19

Hello, We are running stunnel-5.49-1.ppc on AIX 7100-05-03-1846. It will stay up awhile and our connection tests work, but it core dumps. Here’s the message in the log file. Thoughts? INTERNAL ERROR: Out of memory at s3_both.c, line 626 Best, Gerald Shaw

1 0

Cannot create .crt file
by John Khoo 24 Sep '19

24 Sep '19

Hi, I am new to STunnel. I come to know STunnel thru this link: http://www.ibphoenix.com/downloads/Secure_connections_to_Firebird_with_Stun… Based on the instruction, I tried create a certificate using this line below: openssl ca -config "C:\Program Files (x86)\stunnel\config\openssl.conf" -policy policy_anything -notext -in C:\CA\temp\vnc_server\server.req -out C:\CA\temp\vnc_server\server.crt However, no crt file is created and no error is prompted. I hope that someone can help me. Thanks.

1 0