Peter,
Thanks again for your quick reply.
On 2/13/19 15:09, Peter Pentchev wrote:
On Wed, Feb 13, 2019 at 01:45:10PM -0500, Christopher Schultz wrote:
Peter, On 2/13/19 13:09, Peter Pentchev wrote:
On Wed, Feb 13, 2019 at 12:17:24PM -0500, Christopher Schultz wrote:
Hello,
I'm new to the list so I'm sorry if this isn't the right place to report this.
I had a server stop responding to stunnel connections sometime yesterday and the resolution was ultimately to reboot the server and everything was okay. Restarting the stunnel service was not enough to get things working again.
[snip]
Looking through the log file, I could see that there were some odd messages coming from stunnel in the daemon.log file suggesting that there might be a memory leak. I won't post them here unless requested, as they may represent a potential security issue.
Hi,
Unfortunately this is a known problem with the Debian package of stunnel; I did not get around to backporting the fixes to the stunnel version that is in Debian 9.x (stretch); I'm really sorry about that.
Confirmed: I'm running Debian Stretch:
$ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 9.7 (stretch) Release: 9.7 Codename: stretch
I could make a stretch backport of the stunnel version that is in unstable/testing right now - the one that will be in the upcoming Debian 10.x (buster). Actually I'll build a package later tonight and let you know where you can download it from, then I'll upload it to the stretch-backports suite and wait for the backports admins to accept it. This package will not have the memory exhaustion problem that the version in stretch does have.
That would be great. Will I (eventually) end up getting it via the normal apt update mechanism? I only have these apt sources configured:
deb http://ftp.us.debian.org/debian/ stretch main deb http://security.debian.org/ stretch/updates main
Well, to be honest, I'm not sure how easy it will be to backport the change that fixes the memory exhaustion without bringing in a lot of other changes in later versions of stunnel.
I believe your best bet would be to add the stretch-backports suite as described on https://backports.debian.org/Instructions/ and then install only the stunnel4 package from that suite:
apt-get install -t stretch-backports stunnel4
This will bring in this package, and only this package, from the backports suite. The package there is exactly the version that has been in the testing distribution for two months now (it was accepted almost immediately after I uploaded it, it didn't really need to go through human approval), and there have been no reports of trouble with it since then.
I can look into that. I don't exactly need an act of congress to add an apt source and update a package. I might want to do it on a "slow day", though :)
Is there a safe mitigation I can put into place before a patch? I believe Debian's stunnel service scripts completely shut-down one stunnel process and then start up a new one, so I might have a (very short) interruption. I can schedule that via cron if necessary, but I'd prefer to avoid even a small interruption if possible.
Could you just confirm that you are indeed running Debian 9.x (stretch)? (it seems this way from the kernel version that you are running and from the symptoms; stunnel 5.06 in jessie does not have this problem)
I have 5.39. Is this older than 5.06? Or is that a typo?
No, it was not a typo; I'm sorry if I was unclear. When I said "jessie", I meant Debian 8.x (jessie) - the previous release of Debian that had a much older version of stunnel that did not have the changes that, unfortunately, led to the memory leak.
Duh. I didn't even read "jessie", there. Thanks for the clarification.
To be totally honest, the leak itself is partly my fault, since it was introduced in a patch for an issue that was fixed in a more elaborate way in later versions of stunnel.>
Sorry again, and thanks for your understanding!
No, it's great to get this kind of feedback right away.
I *am* curious about why a service-restart did not seem to fix things. With the above memory-exhaustion, would you have expected a service-restart to fix everything? (I certainly did.) Is it likely that the "restart" didn't actually restart the service, and I was just repeatedly testing a borked server process?
I think that it is possible that this might have been the case; there was actually a bug in the stunnel4 init script that was fixed in a later Debian upload (the one that was in the stretch-backports suite until now). The bug was that the script did not wait for the old process to really end before attempting to start a new one - and the new one would not really start, since the kernel would not allow it to listen on the same ports that the old one was still listening on. I didn't think of this bug at once when I read your first message, but it makes sense now.
I almost want to wait until it fails again and then inspect it. If a failing-restart does not explain it, then it suggests some kind of corruption of the kernel's TCP/IP stack which doesn't sound like the kind of thing that should be possible.
So, yeah, it seems that I'm guilty of not fixing not one, but two serious problems with the version of stunnel in the stable Debian release... I could try to figure out a way to fix the memory exhaustion one, but it may take me a couple of days. Right now, your best bet (and the course of action that I recommend) is to install stunnel 5.50 from the stretch-backports suite.
I'll give that a shot and see how things go. If all else fails, I do know that a server reboot will fix the issue. Of course, if I wanted to reboot servers to get them to work, I'd have built everything on Windows :P
Thanks, -chris