Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On 31.03.19 00:06, Sven Hartge wrote: > On 25.03.19 20:25, Sven Hartge wrote: >> On 24.03.19 02:02, Sven Hartge wrote: >> >>> So far, so good. I have your packages running on the main webmail server >>> and the main web server for my university and so far everything is fine, >>> while default packages and the test1 packages with mpm_event would >>> normally start showing the symptoms after ~12 hours. >> >> Still no problems here and both systems have seen some serious traffic >> the last days, with the new semester starting next week and all. > > One week later: still no problems here. > > I also did a cross-check with 2.4.25-3+deb9u6 and it died after ~12 hours. > > So from my perspective 2.4.25-3+deb9u7~test2 fixes the problems with > mpm_event for me. Last report, because I replaced the test-packages with the ones from the security repository and switched back to mpm_worker: I've not seen and problems or even an indication of a problem when I was runnning the apache2 packages with the fully backported mpm_event. All servers I tested this on ran flawless, even with active mod_http2. All those servers would die at least once a week, often more often than that when running mpm_event with the old packages. I say: go for it. Please provide this change for the next point release. Grüße, Sven. signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On 25.03.19 20:25, Sven Hartge wrote: > On 24.03.19 02:02, Sven Hartge wrote: > >> So far, so good. I have your packages running on the main webmail server >> and the main web server for my university and so far everything is fine, >> while default packages and the test1 packages with mpm_event would >> normally start showing the symptoms after ~12 hours. > > Still no problems here and both systems have seen some serious traffic > the last days, with the new semester starting next week and all. One week later: still no problems here. I also did a cross-check with 2.4.25-3+deb9u6 and it died after ~12 hours. So from my perspective 2.4.25-3+deb9u7~test2 fixes the problems with mpm_event for me. S! signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On 3/25/19 8:25 PM, Sven Hartge wrote: > On 24.03.19 02:02, Sven Hartge wrote: > >> So far, so good. I have your packages running on the main webmail server >> and the main web server for my university and so far everything is fine, >> while default packages and the test1 packages with mpm_event would >> normally start showing the symptoms after ~12 hours. > > Still no problems here and both systems have seen some serious traffic > the last days, with the new semester starting next week and all. This is good news! I'll try and see if I can install the packages on a few servers myself this week. Thanks!! Kind regards, Martijn. signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On 24.03.19 02:02, Sven Hartge wrote: > So far, so good. I have your packages running on the main webmail server > and the main web server for my university and so far everything is fine, > while default packages and the test1 packages with mpm_event would > normally start showing the symptoms after ~12 hours. Still no problems here and both systems have seen some serious traffic the last days, with the new semester starting next week and all. Grüße, Sven. signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On 22.03.19 21:19, Sven Hartge wrote: > On 22.03.19 21:14, Stefan Fritsch wrote: > >> Second try with different approach is at >> >> https://people.debian.org/~sf/apache2-mpm-event-902493/2.4.25-3+deb9u7~test2/ >> >> I have backported mpm_event from 2.4.28 and reverted one commit that was >> incompatible with 2.4.25. This was quite painless and I am more confident >> about >> this diff than the first one (even if it's three times the size). > > Thank you. I've thrown the packages into the snake pit and will report > back should anything break. So far, so good. I have your packages running on the main webmail server and the main web server for my university and so far everything is fine, while default packages and the test1 packages with mpm_event would normally start showing the symptoms after ~12 hours. Grüße, Sven. signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On 22.03.19 21:14, Stefan Fritsch wrote: > Second try with different approach is at > > https://people.debian.org/~sf/apache2-mpm-event-902493/2.4.25-3+deb9u7~test2/ > > I have backported mpm_event from 2.4.28 and reverted one commit that was > incompatible with 2.4.25. This was quite painless and I am more confident > about > this diff than the first one (even if it's three times the size). Thank you. I've thrown the packages into the snake pit and will report back should anything break. Grüße, Sven. signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On Monday, 11 March 2019 09:35:45 CET Sven Hartge wrote: > This breaks quite fast, resulting in apache2 processes at 100% CPU, doing > nothing but: Thanks for the quick feed-back. Second try with different approach is at https://people.debian.org/~sf/apache2-mpm-event-902493/2.4.25-3+deb9u7~test2/ I have backported mpm_event from 2.4.28 and reverted one commit that was incompatible with 2.4.25. This was quite painless and I am more confident about this diff than the first one (even if it's three times the size). Cheers, Stefan sha256 sums: 84b8b11ca6973144a3212fe7b65b12010a3118f0e4549afd70c30da41bffc56d apache2_2.4.25-3+deb9u7~test2_amd64.build 1ab44b8d84c5fd30aed5d6453dd6971261ab71872c01b6d3613cd40738652662 apache2_2.4.25-3+deb9u7~test2_amd64.buildinfo 0a7a1b5b90523761721dfb61acce9fb7deb27449b9f16be69d10b5e2cb2822c3 apache2_2.4.25-3+deb9u7~test2_amd64.changes 75829bb4280312ce340f49dad1c426b054ca867c611da27493da87b45a24892d apache2_2.4.25-3+deb9u7~test2_amd64.deb 9646a9cbe97942ba52077947e97ab573c2c79222c51cc5ca14980b1e9cf5651d apache2_2.4.25-3+deb9u7~test2.debian.tar.xz d8b07337d522b806958865fff5c1f0f3274c0b5e5ebd12d41d4041ea49ab64c7 apache2_2.4.25-3+deb9u7~test2.dsc 74d43eb1a85304882854d013e4b3270fbb13654e11e94b6ab12feea6574dada5 apache2- bin_2.4.25-3+deb9u7~test2_amd64.deb 6b61dae59e72adc8282498d344a1c9714bb309fe33be76bb6e4e93e35473faeb apache2- data_2.4.25-3+deb9u7~test2_all.deb 778eb1ae5600af495fddbd598402f357560228c3b96eb2ae560e83e037abe244 apache2- dbg_2.4.25-3+deb9u7~test2_amd64.deb 48073fe04b36539cc7517ae2a9e2d28fd5a74da8d7ce4f03d2ebd85b8787dfa5 apache2- dev_2.4.25-3+deb9u7~test2_amd64.deb 316f391d37266fffc09ca0d628a208dd46e695bdaafd63018259ad895e4e62bb apache2- doc_2.4.25-3+deb9u7~test2_all.deb ccd85a18e88827a78b60c47c711b1d1a0e04067b5bc135c092aa2374a010202c apache2-ssl- dev_2.4.25-3+deb9u7~test2_amd64.deb 839b59f6d3302132c21b5a5342d13d9dc31af4a5f0dee450c79e76bd078ef9fa apache2- suexec-custom_2.4.25-3+deb9u7~test2_amd64.deb cb2da63a2e62a70d4f1e0d463c7e5b47ab37f79e2ff2b2e1fa7bf92f5d3971c4 apache2- suexec-pristine_2.4.25-3+deb9u7~test2_amd64.deb 200514bc47564931121a8a3c654857ee62d865247c8fcf719fa6fd27508ef19e apache2- utils_2.4.25-3+deb9u7~test2_amd64.deb signature.asc Description: This is a digitally signed message part.
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
Um 08:56 Uhr am 11.03.19 schrieb Sven Hartge: > I am going to test these package on the systems which have shown to be > hit by regularly this problem here, so I am confident I will be able to > report within two weeks if there have been any problems and if your > change did indeed fix the problem. This breaks quite fast, resulting in apache2 processes at 100% CPU, doing nothing but: [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) [pid 116644] epoll_ctl(26, EPOLL_CTL_DEL, -1, 0x7fbf20fe8d50) = -1 EBADF (Bad file descriptor) Grüße, Sven.
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On 10.03.19 12:51, Stefan Fritsch wrote: > I am not comfortable with switching to mpm_worker, either, since this would > be > a significant behavior change. > > I have however tried a backport of the patch referenced in the upstream bug > report and put a build here: > > https://people.debian.org/~sf/apache2-mpm-event-902493/2.4.25-3+deb9u7~test1/ > (sha256 sums below) > > I don't think that I will put this patch into the next point release (9.9), > but if there are a fair number of people who test this on their systems and > report back, I may consider it for the 9.10 point release. So, please test > this and report back after maybe 1-2 weeks. > > * if it fixes the bug > * if you have seen any new issues > * on how many systems and how long you have tested it and how much load > (requests/day) those systems see Thank you. I am going to test these package on the systems which have shown to be hit by regularly this problem here, so I am confident I will be able to report within two weeks if there have been any problems and if your change did indeed fix the problem. Grüße, Sven. signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
Hi, I am not comfortable with switching to mpm_worker, either, since this would be a significant behavior change. I have however tried a backport of the patch referenced in the upstream bug report and put a build here: https://people.debian.org/~sf/apache2-mpm-event-902493/2.4.25-3+deb9u7~test1/ (sha256 sums below) I don't think that I will put this patch into the next point release (9.9), but if there are a fair number of people who test this on their systems and report back, I may consider it for the 9.10 point release. So, please test this and report back after maybe 1-2 weeks. * if it fixes the bug * if you have seen any new issues * on how many systems and how long you have tested it and how much load (requests/day) those systems see Cheers, Stefan e1a038b77c952006d1a00ea80b83138e0a949f469798ce9eb14c9b403e2517be apache2_2.4.25-3+deb9u7~test1_amd64.build c353243745bd936392d29b886fe89aeae76ad8afec153a93302bdd6e0e0d48cf apache2_2.4.25-3+deb9u7~test1_amd64.buildinfo 68b58ba59aa9cc24ac1ad1e3b515446dc1ac22a99e579d8747931363b1cce04c apache2_2.4.25-3+deb9u7~test1_amd64.changes e7b8d0151f0dc960b21b78b778be7d15d7262344cbcc9b48c4bea2391f7cedcb apache2_2.4.25-3+deb9u7~test1_amd64.deb e59734e7397ab42128f7baceec14386265f0dfa7312ba7d2bf6d0a8d3ae963cf apache2_2.4.25-3+deb9u7~test1.debian.tar.xz bee5f76574ee771778ce592f11bf708ebeb2a34a4cc99c2770647684f026fd1b apache2_2.4.25-3+deb9u7~test1.dsc f87ec2df1c9fee3e6bfde3c8b855a3ddb7ca1ab20ca877bd0e2b6bf3f05c80b2 apache2_2.4.25.orig.tar.bz2 4a51a890d6056042928d6ec026095b8bdf01f207012fbdfb12eb72228a07bb98 apache2- bin_2.4.25-3+deb9u7~test1_amd64.deb cb9116f2c64521530f013676a0617c0d96625227e60948b2e294307c154846b2 apache2- data_2.4.25-3+deb9u7~test1_all.deb 1c320faf6a36e39a12053d78bad3a0289bcdb70b6befb9171f5d0f04b1096a57 apache2- dbg_2.4.25-3+deb9u7~test1_amd64.deb 7bfa7628d19b35606bf3c843268c23021685b6f5f138b8b79b581ba489619579 apache2- dev_2.4.25-3+deb9u7~test1_amd64.deb d9bb79db08c943b9540e7ce3e0e22331a163a7fcd3df7b761f51d91894c5459e apache2- doc_2.4.25-3+deb9u7~test1_all.deb bebf38cee176889918824b9f34366f1762b940b936e9c6a5c67ffaf32b4f2807 apache2-ssl- dev_2.4.25-3+deb9u7~test1_amd64.deb 445337a0b77a9ceaab13210831ac8159f60b2b085ed7b262bd718a4eb8e4cb4c apache2- suexec-custom_2.4.25-3+deb9u7~test1_amd64.deb 9680bcdda490feccdf1079d7f3406d13e2f8ddfa64eec9d40a921db2e59cc76f apache2- suexec-pristine_2.4.25-3+deb9u7~test1_amd64.deb 49a29207e44cdd16383c0d4d7f51ac8357a0fb3d80f284f30afaaf785d2fcd4c apache2- utils_2.4.25-3+deb9u7~test1_amd64.deb signature.asc Description: This is a digitally signed message part.
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On Tuesday, 12 February 2019 17:44:39 CET Gedalya wrote: > On 2/13/19 12:38 AM, Jan Wagner wrote: > > backports is not meant for fixing things. beside that it would require > > all rebuilding most of the additional apache modules not shiped by the > > apache2 source package. > > So we're back to doing nothing at all? > > I'm not at all advocating backports as a solution, rather suggesting that it > would be _something_ that could be done while the issue is not fixed, for > whatever reason. > > As for rebuilding other sources packages, yes, but like you said, not only > is backports not for fixing things, it has been traditionally known to > break things as well. Packages from backports are indeed sometimes not > installable to some users. > > In my case I wasn't using any other modules (PHP via fcgi), so apache2, apr > and apr-util were enough. Again, we're not forcing anyone to use this, nor > are we saying that this addresses the issue. I will think about it. It I definitely won't have time in the next 2-3 weeks. Cheers, Stefan
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On 2/13/19 12:54 AM, Sven Hartge wrote: > Maybe adding the Release to the first sentence to convey that this is > only a problem for Stretch and not Buster Yea. Well, since mpm_event is still the default in buster :-) --->8--- By default, Apache 2.4 as provided by Debian uses mpm_event to process requests. The version of Apache included in Debian stretch, when used with TLS and under very specific circumstances, may stop accepting new connections for a certain period of time. To work around this issue, you may switch to mpm_worker or mpm_prefork. Apache in buster and later releases is not affected by this issue. For further discussion, see https://bugs.debian.org/902493 and https://bz.apache.org/bugzilla/show_bug.cgi?id=60956
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On 12.02.19 17:51, Gedalya wrote: > On Mon, 11 Feb 2019 14:28:00 +0100 Sven Hartge wrote: >> >> Language of course needs a bit of refining from my German-English. >> > > > Nothing atrocious, really, but I was able to come up with something a bit > more concise. > > > --->8--- > By default, Apache 2.4 as provided by Debian uses mpm_event to process > requests. > > When used with TLS and under very specific circumstances, Apache will stop > accepting new connections for a certain period of time. > > To work around this issue, you may switch to mpm_worker or mpm_prefork. > > For further discussion, see https://bugs.debian.org/902493 and > https://bz.apache.org/bugzilla/show_bug.cgi?id=60956 Maybe adding the Release to the first sentence to convey that this is only a problem for Stretch and not Buster: --->8--- By default, Apache 2.4 as provided by Debian Stretch uses mpm_event to process requests. When used with TLS and under very specific circumstances, Apache will stop accepting new connections for a certain period of time. To work around this issue, you may switch to mpm_worker or mpm_prefork. For further discussion, see https://bugs.debian.org/902493 and https://bz.apache.org/bugzilla/show_bug.cgi?id=60956 signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On Mon, 11 Feb 2019 14:28:00 +0100 Sven Hartge wrote: > > Language of course needs a bit of refining from my German-English. > Nothing atrocious, really, but I was able to come up with something a bit more concise. --->8--- By default, Apache 2.4 as provided by Debian uses mpm_event to process requests. When used with TLS and under very specific circumstances, Apache will stop accepting new connections for a certain period of time. To work around this issue, you may switch to mpm_worker or mpm_prefork. For further discussion, see https://bugs.debian.org/902493 and https://bz.apache.org/bugzilla/show_bug.cgi?id=60956
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
Am 12.02.19 um 17:44 schrieb Gedalya: > On 2/13/19 12:38 AM, Jan Wagner wrote: >> backports is not meant for fixing things. beside that it would require >> all rebuilding most of the additional apache modules not shiped by the >> apache2 source package. > So we're back to doing nothing at all? Just to point out: I'm not (one of) the package maintainer(s), I'm just speaking here as user. signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On 2/13/19 12:38 AM, Jan Wagner wrote: > backports is not meant for fixing things. beside that it would require > all rebuilding most of the additional apache modules not shiped by the > apache2 source package. So we're back to doing nothing at all? I'm not at all advocating backports as a solution, rather suggesting that it would be _something_ that could be done while the issue is not fixed, for whatever reason. As for rebuilding other sources packages, yes, but like you said, not only is backports not for fixing things, it has been traditionally known to break things as well. Packages from backports are indeed sometimes not installable to some users. In my case I wasn't using any other modules (PHP via fcgi), so apache2, apr and apr-util were enough. Again, we're not forcing anyone to use this, nor are we saying that this addresses the issue.
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
Am 12.02.19 um 17:28 schrieb Gedalya: > So how about backporting it to stretch-backports? Isn't that what the > backports section is for? backports is not meant for fixing things. beside that it would require all rebuilding most of the additional apache modules not shiped by the apache2 source package. with regards, jan. -- Never write mail to , you have been warned! -BEGIN GEEK CODE BLOCK- Version: 3.12 GIT d-- s+: a C+++ UL P+ L+++ E--- W+++ N+++ o++ K++ w--- O M+ V- PS PE Y++ PGP++ t-- 5 X R tv- b+ DI D+ G++ e++ h r+++ y --END GEEK CODE BLOCK-- signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On Sun, 10 Feb 2019 14:36:11 +0100 (CET) Stefan Fritsch wrote: > If we did a backport, rhe risk of introducing regressions would be quite high. So how about backporting it to stretch-backports? Isn't that what the backports section is for? It would be then available to those who have interest in it and are willing to accept potential side effects. If we are _not_ going to fix it in stable, I think this could still provide some convenience to some users.
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On 11.02.19 13:36, Jan Wagner wrote: > Am 11.02.19 um 13:12 schrieb Sven Hartge: >>> Okay ... here is an area where you can push forward. What about >>> providing documentation patches? >> Sure. What kind of documentation. NEWS.Debian? > where would you expect such a documentation, as you suggested it? I'd expect this information in NEWS.Debian. Maybe something like this: ---8<-- Apache2.4 in Debian uses mpm_event to process requests. When used with SSL and under very specific circumstances, Apache will stop accepting new connections until a certain timeout has occurred. This causes a service disruption as the webserver not longer processes new request. To work around this issue, switch to mpm_worker or mpm_prefork, which both don't show this problem. This issue has been documented as https://bugs.debian.org/902493 and https://bz.apache.org/bugzilla/show_bug.cgi?id=60956 ---8<-- Language of course needs a bit of refining from my German-English. Again: I still propose to switch to mpm_worker as default MPM for Stretch so not every admin has to hit the same wall at some time. Grüße, Sven. signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
Am 11.02.19 um 13:12 schrieb Sven Hartge: >> Okay ... here is an area where you can push forward. What about >> providing documentation patches? > Sure. What kind of documentation. NEWS.Debian? where would you expect such a documentation, as you suggested it? Cheers, Jan. -- Never write mail to , you have been warned! -BEGIN GEEK CODE BLOCK- Version: 3.12 GIT d-- s+: a C+++ UL P+ L+++ E--- W+++ N+++ o++ K++ w--- O M+ V- PS PE Y++ PGP++ t-- 5 X R tv- b+ DI D+ G++ e++ h r+++ y --END GEEK CODE BLOCK-- signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On 11.02.19 11:29, Jan Wagner wrote: > Okay ... here is an area where you can push forward. What about > providing documentation patches? Sure. What kind of documentation. NEWS.Debian? Grüße, Sven. signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
Am 11.02.19 um 10:46 schrieb Sven Hartge: > But this bug has been encountered frequently enough (and is difficult to > spot, if you don't exactly know what to search for) and with increasing Beeing there. Searched >3 weeks before I thought it would be a idea to switch the mpm. > adoption of SSL more and more people will hit it, that I think at least > *some* action is warranted. > > Maybe better documentation to help people encountering this Okay ... here is an area where you can push forward. What about providing documentation patches? With kind regards, Jan. -- Never write mail to , you have been warned! -BEGIN GEEK CODE BLOCK- Version: 3.12 GIT d-- s+: a C+++ UL P+ L+++ E--- W+++ N+++ o++ K++ w--- O M+ V- PS PE Y++ PGP++ t-- 5 X R tv- b+ DI D+ G++ e++ h r+++ y --END GEEK CODE BLOCK-- signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On 11.02.19 10:29, Jan Wagner wrote: > Am 11.02.19 um 09:51 schrieb Sven Hartge: >> Also I am a bit disappointed by you invoking the "the next release is >> near" argument. Most of my servers for example won't get Buster until >> early to mid 2020 and I think many of others are in the same boat. > just to point this out. You prefer an invasive backport and risk to > stability in other areas? The update policy of Debian in the past was, > that this should be avoided. No, I am disappointed in the "let's do nothing" stance. I can see why backporting the newer mpm_event is risky and that it should be avoided. I can also know that just throwing in a completely new Apache is something Debian does not do, I've been using Debian for the last 20 years because of exactly that guarantee, to not get surprised by mid-release major changes. But this bug has been encountered frequently enough (and is difficult to spot, if you don't exactly know what to search for) and with increasing adoption of SSL more and more people will hit it, that I think at least *some* action is warranted. Maybe better documentation to help people encountering this or maybe changing the default MPM for Stretch on new installs, since mpm_event in Stretch clearly is flawed and buggy with SSL. But just saying "Buster is release soon" can't be the right solution here. Stretch will likely be used for at least 3 more years before it is phased out, keeping a *known* bug with an easy workaround active for that long because of "we don't change Debian Stable *ever*" seems wrong to me. Grüße, Sven. signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
Hi Sven, Am 11.02.19 um 09:51 schrieb Sven Hartge: > Also I am a bit disappointed by you invoking the "the next release is > near" argument. Most of my servers for example won't get Buster until > early to mid 2020 and I think many of others are in the same boat. just to point this out. You prefer an invasive backport and risk to stability in other areas? The update policy of Debian in the past was, that this should be avoided. Cheers, Jan. -- Never write mail to , you have been warned! -BEGIN GEEK CODE BLOCK- Version: 3.12 GIT d-- s+: a C+++ UL P+ L+++ E--- W+++ N+++ o++ K++ w--- O M+ V- PS PE Y++ PGP++ t-- 5 X R tv- b+ DI D+ G++ e++ h r+++ y --END GEEK CODE BLOCK-- signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On 10.02.19 14:36, Stefan Fritsch wrote: > Between 2.4.25 and the fix for this issue, there were some intrusive > changes in mpm_evnt. If we did a backport, rhe risk of introducing > regressions would be quite high. Therefore, and because the next Debian > stable release is quite near, I don't think it makes sense to backport the > fix. Maybe, as a compromise, switch to using mpm_worker as the default MPM instead of mpm_event? I've seen this problem here in more and more servers of mine, I had to switch all of them to mpm_worker to avoid this nasty bug. Also I am a bit disappointed by you invoking the "the next release is near" argument. Most of my servers for example won't get Buster until early to mid 2020 and I think many of others are in the same boat. Grüße, Sven. signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
Between 2.4.25 and the fix for this issue, there were some intrusive changes in mpm_evnt. If we did a backport, rhe risk of introducing regressions would be quite high. Therefore, and because the next Debian stable release is quite near, I don't think it makes sense to backport the fix.
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
I've had seemingly the same issue. After several weeks of running a backported apache2 2.4.37-1 the issues are gone. Previously it was happening several times every day, with the outage lasting sometimes 10 minutes or so. This was very difficult to troubleshoot, as nothing is logged, and it was relatively hard to find this bug report and find my way towards a solution. Applying the fix to stretch might help others who are struggling to understand the issue they are facing and are therefore not being heard.
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
Hi all, On Fri, 5 Oct 2018 14:02:41 +0200 Sven Hartge wrote: On Wed, 27 Jun 2018 10:39:51 +0200 Martijn Grendelman wrote: > Some of our Debian Stretch based Apache webservers suffer from > intermittent connection timeouts. > > We have been trying to pin down the problem for a while, and eventually, > we found this bug report in Apache's Bugzilla, that seems to fit our > problem perfectly: > > https://bz.apache.org/bugzilla/show_bug.cgi?id=60956 I can verifiy and this bug and also had to change to mpm_worker to work around this bug. A backport of the changes in mpm_event made for 2.4.28 would be very nice, just like mod_http2 was backported from a newer version of apache2. After suffering (probably) from this issue on multiple systems, I'd appreciate backporting the fix from Apache 2.4.28 to Stretch as well. It could be just my personal impression, but it seems like this is affecting more systems over time. We spotted the issue (Apache hangs without warning and without logs until restarted or the timeout clears) on just one system, now multiple systems are affected, even with low or even just internal (browser clients behind VPN) traffic. Switching to MPM_Worker helped to solve this in the meantime. Best, Anton Dollmaier
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
On Wed, 27 Jun 2018 10:39:51 +0200 Martijn Grendelman wrote: > Some of our Debian Stretch based Apache webservers suffer from > intermittent connection timeouts. > > We have been trying to pin down the problem for a while, and eventually, > we found this bug report in Apache's Bugzilla, that seems to fit our > problem perfectly: > > https://bz.apache.org/bugzilla/show_bug.cgi?id=60956 I can verifiy and this bug and also had to change to mpm_worker to work around this bug. A backport of the changes in mpm_event made for 2.4.28 would be very nice, just like mod_http2 was backported from a newer version of apache2. Grüße, Sven Hartge. signature.asc Description: OpenPGP digital signature
Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns
Package: apache2-bin Version: 2.4.25-3+deb9u4 Severity: important Tags: patch upstream Dear Maintainer, Some of our Debian Stretch based Apache webservers suffer from intermittent connection timeouts. We have been trying to pin down the problem for a while, and eventually, we found this bug report in Apache's Bugzilla, that seems to fit our problem perfectly: https://bz.apache.org/bugzilla/show_bug.cgi?id=60956 The short version of the story is, that under very specific circumstances, Apache will stop accepting new connections until a certain timeout has occurred. The source of this behaviour is in the event MPM's code for cleaning up stale connections, which may block in an unexpected way. It seems that the bug has been present in Apache since v2.4.12, and has been fixed in v2.4.28. The bug report above contains a patch that fixes the problem. I suspect that this isn't a real problem for many users, because it took the upstream community a long time to find it, and it doesn't seem to be a common issue, if you start looking around. However, I have been able to identify this problem on almost all of our Stretch webservers, even if its occurrences are quite rare. Some of our less-loaded servers only show it once every few weeks. One of them, however, has been suffering from it multiple times daily for the past couple of weeks, up to a point that Apache was considered unusable. Also, we are not the only ones having this problem, for example see: https://serverfault.com/questions/819717/apache-event-mpm-hangs-sporadicly On top of that, if the circumstances are right, the bug can be triggered from a malicious client, leading to denial of service. As such, I would think this can be considered a security vulnerability. Given that this is a real bug, having the scent of a security problem, that causes a real problem for us and at least a few other people, I kindly request to see if the patch from the mentioned Bugzilla report can be applied to Apache 2.4.25 in Stretch. I already know it doesn't apply cleanly, and I don't have the necessary C-skills to reliably backport the changes, I'm afraid. We 'solved' the problem in our shop by backporting Apache 2.4.33 from Buster to Stretch, but you'll understand this this is not a great solution from a security perspective. -- Package-specific info: -- System Information: Debian Release: 9.4 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 4.9.0-6-amd64 (SMP w/1 CPU core) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages apache2-bin depends on: ii libapr1 1.5.2-5 ii libaprutil1 1.5.4-3 ii libaprutil1-dbd-sqlite3 1.5.4-3 ii libaprutil1-ldap 1.5.4-3 ii libc62.24-11+deb9u3 ii libldap-2.4-22.4.44+dfsg-5+deb9u1 ii liblua5.2-0 5.2.4-1.1+b2 ii libnghttp2-141.18.1-1 ii libpcre3 2:8.39-3 ii libssl1.0.2 1.0.2l-2+deb9u3 ii libxml2 2.9.4+dfsg1-2.2+deb9u2 ii perl 5.24.1-3+deb9u4 ii zlib1g 1:1.2.8.dfsg-5 apache2-bin recommends no packages. Versions of packages apache2-bin suggests: pn apache2-doc pn apache2-suexec-pristine | apache2-suexec-custom ii lynx [www-browser] 2.8.9dev11-1 Versions of packages apache2 depends on: ii apache2-data 2.4.25-3+deb9u4 ii apache2-utils2.4.25-3+deb9u4 ii dpkg 1.18.24 ii init-system-helpers 1.48 ii lsb-base 9.20161125 ii mime-support 3.60 ii perl 5.24.1-3+deb9u4 ii procps 2:3.3.12-3+deb9u1 Versions of packages apache2 recommends: ii ssl-cert 1.0.39 Versions of packages apache2 suggests: pn apache2-doc pn apache2-suexec-pristine | apache2-suexec-custom ii lynx [www-browser] 2.8.9dev11-1 Versions of packages apache2-bin is related to: ii apache2 2.4.25-3+deb9u4 ii apache2-bin 2.4.25-3+deb9u4 -- no debconf information
