Re: [users@httpd] MPM event unstable behind AWS ALB
I'm a beginner but I did install fpm to see how it works. Did you set the fpm acces logs? ; The access log file ; Default: not set access.log = "/usr/local/etc/php/8.0/log/www.access.log" And the status page? ; The URI to view the FPM status page. If this value is not set, no URI will be ; recognized as a status page. It shows the following information: pm.status_path = /fpm-status I don't know how ALB works, but does it make different requests/hit? If not, that's maybe why FPM coughs Cordially Luis On 20/12/2021 01:08, Dan Washusen wrote: Also worth noting that Nginx OS does not experience the issues (otherwise same setup)... Dan On Mon, Dec 20, 2021 at 10:58 AM Dan Washusen wrote: Yeah, all those parameters are configured and everything works fine when I switch to the MPM worker module. The 408 errors are returned by Jetty and ProxyPass is configured to not reuse connections (which seems to suggest not a KeepAlive issue), Jetty is erroring because it receives no request data. Kind of seems like the MPM event module is missing events/data (when behind the ALB)... Given that the TCP network load balancer doesn't show the issue, maybe it's something to do with how the ALB sends the data to the instance (e.g. Jumbo frames vs the standard MTU)...? On Mon, Dec 20, 2021 at 10:37 AM Igor Cicimov wrote: I will still say this is a timeout issue. Set the ALB timeout to 3600 which is the max possible in case you do not know where to start from. I guess you already have ALB logs enabled. In apache check the KeepAliveTimeout, RequestReadTimeout and ProxyTimeout and make sure they make sense for your user case. Enable debug logs too for more details. It is tough to guess without knowing your relevant configuration but 408 is usually caused by client connection being closed while keep-alive in use. On Mon, Dec 20, 2021 at 10:12 AM Dan Washusen wrote: Thanks for the response. Timeouts are configured appropriately... To clarify; everything works fine through a TCP Network Load Balancer pointing at the same infrastructure. There is something about having a HTTP based Application Load Balancer in front of an MPM event configuration that's causing issues... Dan On Mon, Dec 20, 2021 at 10:01 AM Igor Cicimov wrote: In a proxy chains like this getting the timeouts in sync is the most important thing. Make sure that you have done that. On Mon, 20 Dec 2021, 08:37 Dan Washusen, wrote: Hi All, I've been experimenting with the MPM event module with Apache instances sitting behind an AWS Application Load Balancer (ALB) and it really doesn't seem to be working well. Response times shoot up (compared to MPM event worker) and we see a fair few 502 errors returned (by the AWS ALB. The basic layout is: AWS Application Load Balancer -> Apache 2.4.x -> AWS Internal TCP Load Balancer (NLB) -> Jetty App Servers Debugging the issue I think I traced it down to Jetty returning a 408 error because it can't read the request body in a timely manner. So it seems like for some reason MPM is sending the request body...? We're running Ubuntu 20.04 with Apache 2.4.41-4ubuntu3.8 with the following worker configuration: |ServerLimit 250 StartServers 100 MinSpareThreads 75 MaxSpareThreads 250 ThreadLimit 64 ThreadsPerChild 64 MaxRequestWorkers 8000| I've come across several random posts mentioning that the MPM event module doesn't work behind an ALB but no-one seems to be into any detail. Anyone have some debugging/configuration suggestions? Thanks, Dan p.s. I've created a serverfault post showing graphs etc: https://serverfault.com/questions/1087747/apache-2-4-mpm-event-module-causing-intermittent-502-errors-and-slow-response-ti
Re: [users@httpd] MPM event unstable behind AWS ALB
Yeah the 408 is not easy to troubleshoot it might be anything from malformed HTTP header and slow client (i.e. legit DoS attack where a malicious client sends the headers and then delais sending the request body in order to keep the connection open and exhaust server side resources) to a legit network issue. Apart from trying to find the clew in the apache debug logs I don't know what else to suggest. On Mon, Dec 20, 2021 at 11:09 AM Dan Washusen wrote: > Also worth noting that Nginx OS does not experience the issues (otherwise > same setup)... > > Dan > > > On Mon, Dec 20, 2021 at 10:58 AM Dan Washusen wrote: > >> Yeah, all those parameters are configured and everything works fine when >> I switch to the MPM worker module. The 408 errors are returned by Jetty and >> ProxyPass is configured to not reuse connections (which seems to suggest >> not a KeepAlive issue), Jetty is erroring because it receives no request >> data. Kind of seems like the MPM event module is missing events/data (when >> behind the ALB)... >> >> Given that the TCP network load balancer doesn't show the issue, maybe >> it's something to do with how the ALB sends the data to the instance (e.g. >> Jumbo frames vs the standard MTU)...? >> >> On Mon, Dec 20, 2021 at 10:37 AM Igor Cicimov wrote: >> >>> I will still say this is a timeout issue. Set the ALB timeout to 3600 >>> which is the max possible in case you do not know where to start from. I >>> guess you already have ALB logs enabled. >>> >>> In apache check the KeepAliveTimeout, RequestReadTimeout and >>> ProxyTimeout and make sure they make sense for your user case. Enable debug >>> logs too for more details. >>> >>> It is tough to guess without knowing your relevant configuration but 408 >>> is usually caused by client connection being closed while keep-alive in use. >>> >>> On Mon, Dec 20, 2021 at 10:12 AM Dan Washusen wrote: >>> Thanks for the response. Timeouts are configured appropriately... To clarify; everything works fine through a TCP Network Load Balancer pointing at the same infrastructure. There is something about having a HTTP based Application Load Balancer in front of an MPM event configuration that's causing issues... Dan On Mon, Dec 20, 2021 at 10:01 AM Igor Cicimov wrote: > In a proxy chains like this getting the timeouts in sync is the most > important thing. Make sure that you have done that. > > On Mon, 20 Dec 2021, 08:37 Dan Washusen, wrote: > >> Hi All, >> I've been experimenting with the MPM event module with Apache >> instances sitting behind an AWS Application Load Balancer (ALB) and it >> really doesn't seem to be working well. Response times shoot up (compared >> to MPM event worker) and we see a fair few 502 errors returned (by the >> AWS >> ALB. >> >> The basic layout is: AWS Application Load Balancer -> Apache 2.4.x -> >> AWS Internal TCP Load Balancer (NLB) -> Jetty App Servers >> >> Debugging the issue I think I traced it down to Jetty returning a 408 >> error because it can't read the request body in a timely manner. So it >> seems like for some reason MPM is sending the request body...? >> >> We're running Ubuntu 20.04 with Apache 2.4.41-4ubuntu3.8 with the >> following worker configuration: >> >> ServerLimit 250 >> StartServers 100 >> MinSpareThreads 75 >> MaxSpareThreads 250 >> ThreadLimit 64 >> ThreadsPerChild 64 >> MaxRequestWorkers 8000 >> >> I've come across several random posts mentioning that the MPM event >> module doesn't work behind an ALB but no-one seems to be into any detail. >> Anyone have some debugging/configuration suggestions? >> >> Thanks, >> Dan >> >> p.s. I've created a serverfault post showing graphs etc: >> https://serverfault.com/questions/1087747/apache-2-4-mpm-event-module-causing-intermittent-502-errors-and-slow-response-ti >> >
Re: [users@httpd] MPM event unstable behind AWS ALB
Also worth noting that Nginx OS does not experience the issues (otherwise same setup)... Dan On Mon, Dec 20, 2021 at 10:58 AM Dan Washusen wrote: > Yeah, all those parameters are configured and everything works fine when I > switch to the MPM worker module. The 408 errors are returned by Jetty and > ProxyPass is configured to not reuse connections (which seems to suggest > not a KeepAlive issue), Jetty is erroring because it receives no request > data. Kind of seems like the MPM event module is missing events/data (when > behind the ALB)... > > Given that the TCP network load balancer doesn't show the issue, maybe > it's something to do with how the ALB sends the data to the instance (e.g. > Jumbo frames vs the standard MTU)...? > > On Mon, Dec 20, 2021 at 10:37 AM Igor Cicimov wrote: > >> I will still say this is a timeout issue. Set the ALB timeout to 3600 >> which is the max possible in case you do not know where to start from. I >> guess you already have ALB logs enabled. >> >> In apache check the KeepAliveTimeout, RequestReadTimeout and ProxyTimeout >> and make sure they make sense for your user case. Enable debug logs too for >> more details. >> >> It is tough to guess without knowing your relevant configuration but 408 >> is usually caused by client connection being closed while keep-alive in use. >> >> On Mon, Dec 20, 2021 at 10:12 AM Dan Washusen wrote: >> >>> Thanks for the response. Timeouts are configured appropriately... >>> >>> To clarify; everything works fine through a TCP Network Load Balancer >>> pointing at the same infrastructure. There is something about having a HTTP >>> based Application Load Balancer in front of an MPM event configuration >>> that's causing issues... >>> >>> Dan >>> >>> >>> >>> On Mon, Dec 20, 2021 at 10:01 AM Igor Cicimov >>> wrote: >>> In a proxy chains like this getting the timeouts in sync is the most important thing. Make sure that you have done that. On Mon, 20 Dec 2021, 08:37 Dan Washusen, wrote: > Hi All, > I've been experimenting with the MPM event module with Apache > instances sitting behind an AWS Application Load Balancer (ALB) and it > really doesn't seem to be working well. Response times shoot up (compared > to MPM event worker) and we see a fair few 502 errors returned (by the AWS > ALB. > > The basic layout is: AWS Application Load Balancer -> Apache 2.4.x -> > AWS Internal TCP Load Balancer (NLB) -> Jetty App Servers > > Debugging the issue I think I traced it down to Jetty returning a 408 > error because it can't read the request body in a timely manner. So it > seems like for some reason MPM is sending the request body...? > > We're running Ubuntu 20.04 with Apache 2.4.41-4ubuntu3.8 with the > following worker configuration: > > ServerLimit 250 > StartServers 100 > MinSpareThreads 75 > MaxSpareThreads 250 > ThreadLimit 64 > ThreadsPerChild 64 > MaxRequestWorkers 8000 > > I've come across several random posts mentioning that the MPM event > module doesn't work behind an ALB but no-one seems to be into any detail. > Anyone have some debugging/configuration suggestions? > > Thanks, > Dan > > p.s. I've created a serverfault post showing graphs etc: > https://serverfault.com/questions/1087747/apache-2-4-mpm-event-module-causing-intermittent-502-errors-and-slow-response-ti >
Re: [users@httpd] MPM event unstable behind AWS ALB
Yeah, all those parameters are configured and everything works fine when I switch to the MPM worker module. The 408 errors are returned by Jetty and ProxyPass is configured to not reuse connections (which seems to suggest not a KeepAlive issue), Jetty is erroring because it receives no request data. Kind of seems like the MPM event module is missing events/data (when behind the ALB)... Given that the TCP network load balancer doesn't show the issue, maybe it's something to do with how the ALB sends the data to the instance (e.g. Jumbo frames vs the standard MTU)...? On Mon, Dec 20, 2021 at 10:37 AM Igor Cicimov wrote: > I will still say this is a timeout issue. Set the ALB timeout to 3600 > which is the max possible in case you do not know where to start from. I > guess you already have ALB logs enabled. > > In apache check the KeepAliveTimeout, RequestReadTimeout and ProxyTimeout > and make sure they make sense for your user case. Enable debug logs too for > more details. > > It is tough to guess without knowing your relevant configuration but 408 > is usually caused by client connection being closed while keep-alive in use. > > On Mon, Dec 20, 2021 at 10:12 AM Dan Washusen wrote: > >> Thanks for the response. Timeouts are configured appropriately... >> >> To clarify; everything works fine through a TCP Network Load Balancer >> pointing at the same infrastructure. There is something about having a HTTP >> based Application Load Balancer in front of an MPM event configuration >> that's causing issues... >> >> Dan >> >> >> >> On Mon, Dec 20, 2021 at 10:01 AM Igor Cicimov wrote: >> >>> In a proxy chains like this getting the timeouts in sync is the most >>> important thing. Make sure that you have done that. >>> >>> On Mon, 20 Dec 2021, 08:37 Dan Washusen, wrote: >>> Hi All, I've been experimenting with the MPM event module with Apache instances sitting behind an AWS Application Load Balancer (ALB) and it really doesn't seem to be working well. Response times shoot up (compared to MPM event worker) and we see a fair few 502 errors returned (by the AWS ALB. The basic layout is: AWS Application Load Balancer -> Apache 2.4.x -> AWS Internal TCP Load Balancer (NLB) -> Jetty App Servers Debugging the issue I think I traced it down to Jetty returning a 408 error because it can't read the request body in a timely manner. So it seems like for some reason MPM is sending the request body...? We're running Ubuntu 20.04 with Apache 2.4.41-4ubuntu3.8 with the following worker configuration: ServerLimit 250 StartServers 100 MinSpareThreads 75 MaxSpareThreads 250 ThreadLimit 64 ThreadsPerChild 64 MaxRequestWorkers 8000 I've come across several random posts mentioning that the MPM event module doesn't work behind an ALB but no-one seems to be into any detail. Anyone have some debugging/configuration suggestions? Thanks, Dan p.s. I've created a serverfault post showing graphs etc: https://serverfault.com/questions/1087747/apache-2-4-mpm-event-module-causing-intermittent-502-errors-and-slow-response-ti >>>
Re: [users@httpd] MPM event unstable behind AWS ALB
I will still say this is a timeout issue. Set the ALB timeout to 3600 which is the max possible in case you do not know where to start from. I guess you already have ALB logs enabled. In apache check the KeepAliveTimeout, RequestReadTimeout and ProxyTimeout and make sure they make sense for your user case. Enable debug logs too for more details. It is tough to guess without knowing your relevant configuration but 408 is usually caused by client connection being closed while keep-alive in use. On Mon, Dec 20, 2021 at 10:12 AM Dan Washusen wrote: > Thanks for the response. Timeouts are configured appropriately... > > To clarify; everything works fine through a TCP Network Load Balancer > pointing at the same infrastructure. There is something about having a HTTP > based Application Load Balancer in front of an MPM event configuration > that's causing issues... > > Dan > > > > On Mon, Dec 20, 2021 at 10:01 AM Igor Cicimov wrote: > >> In a proxy chains like this getting the timeouts in sync is the most >> important thing. Make sure that you have done that. >> >> On Mon, 20 Dec 2021, 08:37 Dan Washusen, wrote: >> >>> Hi All, >>> I've been experimenting with the MPM event module with Apache instances >>> sitting behind an AWS Application Load Balancer (ALB) and it really doesn't >>> seem to be working well. Response times shoot up (compared to MPM event >>> worker) and we see a fair few 502 errors returned (by the AWS ALB. >>> >>> The basic layout is: AWS Application Load Balancer -> Apache 2.4.x -> >>> AWS Internal TCP Load Balancer (NLB) -> Jetty App Servers >>> >>> Debugging the issue I think I traced it down to Jetty returning a 408 >>> error because it can't read the request body in a timely manner. So it >>> seems like for some reason MPM is sending the request body...? >>> >>> We're running Ubuntu 20.04 with Apache 2.4.41-4ubuntu3.8 with the >>> following worker configuration: >>> >>> ServerLimit 250 >>> StartServers 100 >>> MinSpareThreads 75 >>> MaxSpareThreads 250 >>> ThreadLimit 64 >>> ThreadsPerChild 64 >>> MaxRequestWorkers 8000 >>> >>> I've come across several random posts mentioning that the MPM event >>> module doesn't work behind an ALB but no-one seems to be into any detail. >>> Anyone have some debugging/configuration suggestions? >>> >>> Thanks, >>> Dan >>> >>> p.s. I've created a serverfault post showing graphs etc: >>> https://serverfault.com/questions/1087747/apache-2-4-mpm-event-module-causing-intermittent-502-errors-and-slow-response-ti >>> >>
Re: [users@httpd] MPM event unstable behind AWS ALB
Thanks for the response. Timeouts are configured appropriately... To clarify; everything works fine through a TCP Network Load Balancer pointing at the same infrastructure. There is something about having a HTTP based Application Load Balancer in front of an MPM event configuration that's causing issues... Dan On Mon, Dec 20, 2021 at 10:01 AM Igor Cicimov wrote: > In a proxy chains like this getting the timeouts in sync is the most > important thing. Make sure that you have done that. > > On Mon, 20 Dec 2021, 08:37 Dan Washusen, wrote: > >> Hi All, >> I've been experimenting with the MPM event module with Apache instances >> sitting behind an AWS Application Load Balancer (ALB) and it really doesn't >> seem to be working well. Response times shoot up (compared to MPM event >> worker) and we see a fair few 502 errors returned (by the AWS ALB. >> >> The basic layout is: AWS Application Load Balancer -> Apache 2.4.x -> AWS >> Internal TCP Load Balancer (NLB) -> Jetty App Servers >> >> Debugging the issue I think I traced it down to Jetty returning a 408 >> error because it can't read the request body in a timely manner. So it >> seems like for some reason MPM is sending the request body...? >> >> We're running Ubuntu 20.04 with Apache 2.4.41-4ubuntu3.8 with the >> following worker configuration: >> >> ServerLimit 250 >> StartServers 100 >> MinSpareThreads 75 >> MaxSpareThreads 250 >> ThreadLimit 64 >> ThreadsPerChild 64 >> MaxRequestWorkers 8000 >> >> I've come across several random posts mentioning that the MPM event >> module doesn't work behind an ALB but no-one seems to be into any detail. >> Anyone have some debugging/configuration suggestions? >> >> Thanks, >> Dan >> >> p.s. I've created a serverfault post showing graphs etc: >> https://serverfault.com/questions/1087747/apache-2-4-mpm-event-module-causing-intermittent-502-errors-and-slow-response-ti >> >
Re: [users@httpd] MPM event unstable behind AWS ALB
In a proxy chains like this getting the timeouts in sync is the most important thing. Make sure that you have done that. On Mon, 20 Dec 2021, 08:37 Dan Washusen, wrote: > Hi All, > I've been experimenting with the MPM event module with Apache instances > sitting behind an AWS Application Load Balancer (ALB) and it really doesn't > seem to be working well. Response times shoot up (compared to MPM event > worker) and we see a fair few 502 errors returned (by the AWS ALB. > > The basic layout is: AWS Application Load Balancer -> Apache 2.4.x -> AWS > Internal TCP Load Balancer (NLB) -> Jetty App Servers > > Debugging the issue I think I traced it down to Jetty returning a 408 > error because it can't read the request body in a timely manner. So it > seems like for some reason MPM is sending the request body...? > > We're running Ubuntu 20.04 with Apache 2.4.41-4ubuntu3.8 with the > following worker configuration: > > ServerLimit 250 > StartServers 100 > MinSpareThreads 75 > MaxSpareThreads 250 > ThreadLimit 64 > ThreadsPerChild 64 > MaxRequestWorkers 8000 > > I've come across several random posts mentioning that the MPM event module > doesn't work behind an ALB but no-one seems to be into any detail. Anyone > have some debugging/configuration suggestions? > > Thanks, > Dan > > p.s. I've created a serverfault post showing graphs etc: > https://serverfault.com/questions/1087747/apache-2-4-mpm-event-module-causing-intermittent-502-errors-and-slow-response-ti >
[users@httpd] MPM event unstable behind AWS ALB
Hi All, I've been experimenting with the MPM event module with Apache instances sitting behind an AWS Application Load Balancer (ALB) and it really doesn't seem to be working well. Response times shoot up (compared to MPM event worker) and we see a fair few 502 errors returned (by the AWS ALB. The basic layout is: AWS Application Load Balancer -> Apache 2.4.x -> AWS Internal TCP Load Balancer (NLB) -> Jetty App Servers Debugging the issue I think I traced it down to Jetty returning a 408 error because it can't read the request body in a timely manner. So it seems like for some reason MPM is sending the request body...? We're running Ubuntu 20.04 with Apache 2.4.41-4ubuntu3.8 with the following worker configuration: ServerLimit 250 StartServers 100 MinSpareThreads 75 MaxSpareThreads 250 ThreadLimit 64 ThreadsPerChild 64 MaxRequestWorkers 8000 I've come across several random posts mentioning that the MPM event module doesn't work behind an ALB but no-one seems to be into any detail. Anyone have some debugging/configuration suggestions? Thanks, Dan p.s. I've created a serverfault post showing graphs etc: https://serverfault.com/questions/1087747/apache-2-4-mpm-event-module-causing-intermittent-502-errors-and-slow-response-ti
