Peter, On 2/13/19 15:09, Peter Pentchev wrote: > On Wed, Feb 13, 2019 at 01:45:10PM -0500, Christopher Schultz wrote: >> Peter, >> >> Thanks for the quick reply. >> >> On 2/13/19 13:09, Peter Pentchev wrote: >>> On Wed, Feb 13, 2019 at 12:17:24PM -0500, Christopher Schultz wrote: >>>> Hello, >>>> >>>> I'm new to the list so I'm sorry if this isn't the right place to report >>>> this. >>>> >>>> I had a server stop responding to stunnel connections sometime yesterday >>>> and the resolution was ultimately to reboot the server and everything >>>> was okay. Restarting the stunnel service was not enough to get things >>>> working again. >>> [snip] >>>> >>>> Looking through the log file, I could see that there were some odd >>>> messages coming from stunnel in the daemon.log file suggesting that >>>> there might be a memory leak. I won't post them here unless requested, >>>> as they may represent a potential security issue. >>> >>> Hi, >>> >>> Unfortunately this is a known problem with the Debian package of >>> stunnel; I did not get around to backporting the fixes to the stunnel >>> version that is in Debian 9.x (stretch); I'm really sorry about that. >> >> Confirmed: I'm running Debian Stretch: >> >> $ lsb_release -a >> No LSB modules are available. >> Distributor ID: Debian >> Description: Debian GNU/Linux 9.7 (stretch) >> Release: 9.7 >> Codename: stretch >> >>> I could make a stretch backport of the stunnel version that is in >>> unstable/testing right now - the one that will be in the upcoming >>> Debian 10.x (buster). Actually I'll build a package later tonight and >>> let you know where you can download it from, then I'll upload it >>> to the stretch-backports suite and wait for the backports admins to >>> accept it. This package will not have the memory exhaustion problem >>> that the version in stretch does have. >> >> That would be great. Will I (eventually) end up getting it via the >> normal apt update mechanism? I only have these apt sources configured: >> >> deb http://ftp.us.debian.org/debian/ stretch main >> deb http://security.debian.org/ stretch/updates main > > Well, to be honest, I'm not sure how easy it will be to backport > the change that fixes the memory exhaustion without bringing in > a lot of other changes in later versions of stunnel. > > I believe your best bet would be to add the stretch-backports suite > as described on https://backports.debian.org/Instructions/ and > then install only the stunnel4 package from that suite: > > apt-get install -t stretch-backports stunnel4 > > This will bring in this package, and only this package, from > the backports suite. The package there is exactly the version that > has been in the testing distribution for two months now (it was > accepted almost immediately after I uploaded it, it didn't really > need to go through human approval), and there have been no reports of > trouble with it since then. > >> Is there a safe mitigation I can put into place before a patch? I >> believe Debian's stunnel service scripts completely shut-down one >> stunnel process and then start up a new one, so I might have a (very >> short) interruption. I can schedule that via cron if necessary, but I'd >> prefer to avoid even a small interruption if possible. >> >>> Could you just confirm that you are indeed running Debian 9.x (stretch)? >>> (it seems this way from the kernel version that you are running and >>> from the symptoms; stunnel 5.06 in jessie does not have this problem) >> >> I have 5.39. Is this older than 5.06? Or is that a typo? > > No, it was not a typo; I'm sorry if I was unclear. When I said "jessie", > I meant Debian 8.x (jessie) - the previous release of Debian that had > a much older version of stunnel that did not have the changes that, > unfortunately, led to the memory leak. To be totally honest, the leak > itself is partly my fault, since it was introduced in a patch for > an issue that was fixed in a more elaborate way in later versions of > stunnel. > >>> Sorry again, and thanks for your understanding! >> >> No, it's great to get this kind of feedback right away. >> >> I *am* curious about why a service-restart did not seem to fix things. >> With the above memory-exhaustion, would you have expected a >> service-restart to fix everything? (I certainly did.) Is it likely that >> the "restart" didn't actually restart the service, and I was just >> repeatedly testing a borked server process? > > I think that it is possible that this might have been the case; > there was actually a bug in the stunnel4 init script that was fixed > in a later Debian upload (the one that was in the stretch-backports > suite until now). The bug was that the script did not wait for > the old process to really end before attempting to start a new one - > and the new one would not really start, since the kernel would not > allow it to listen on the same ports that the old one was still > listening on. I didn't think of this bug at once when I read your > first message, but it makes sense now. > > So, yeah, it seems that I'm guilty of not fixing not one, but two > serious problems with the version of stunnel in the stable Debian > release... I could try to figure out a way to fix the memory > exhaustion one, but it may take me a couple of days. Right now, > your best bet (and the course of action that I recommend) is to > install stunnel 5.50 from the stretch-backports suite.
So, my application servers are evidently quite regular and predictable. And they stay up for a long time. Therefore, last night just before bedtime, another application server's stunnel service stopped responding, having suffered the same fate as the one earlier in the day. This time I was prepared for it and ready for some forensics. Shutting down the service via `/etc/init.d/stunnel4 stop` appeared to succeed (systemd reported "service stopped") but then I did a `ps` and I could plainly see that the process was still running. I have two different stunnel configuration files, so I usually get more than one stunnel process listening, and only one of them had stopped (the one that did NOT run out of memory). Performing a `kill -9` on the process did indeed kill it, and then a standard service-start for stunnel got it working again. One of my application servers has been upgraded to the backports version (5.44) and I'll let that one run for a bit before upgrading the others, just to Make Sure :) Thanks for your help with this. I really do think that the stable package needs to be fixed. I use Debian primarily because it's known to be rock-solid and I'd like it to stay that way. :) Not everyone will be upgrading to Buster as soon as it's released, so it would be good to get that memory fix back-ported to Stretch. Thanks again for your work on this. -chris
signature.asc
Description: OpenPGP digital signature
_______________________________________________ stunnel-users mailing list [email protected] https://www.stunnel.org/cgi-bin/mailman/listinfo/stunnel-users
