Re: [users@httpd] timeout and keepalive parameter in ProxyPass

[Yann Ylavic]

On Wed, Oct 13, 2021 at 4:22 PM Usha Nayak  wrote:
>
> Need help in understanding the highlighted parameters:
>
> ProxyPass "/example" "http://backend.example.com;  timeout=3600 keepalive=On
>
> keepalive parameter - As I understand relates to TCP or socket keep alive 
> probes that are sent to prevent idle connection.
>
> My confusion is :
>
> Assuming the backend server takes 4 hrs to process a request and I have 
> 'keepalive on' in Apache httpd with the OS setting to send tcp probes every 2 
> minutes. Backend server and Apache httpd are sending TCP keep alive packets 
> to indicate the socket connection is not idle.
>
>  In this case, would the 'apache httpd' close the connection in 1 hr ( 
> timeout = 3600) because the 'apache httpd' didn't get the http data?

Yes, the timeout= parameter sets the inactivity timeout for the
backend connection. If reading from or writing to this connection
takes more time than the timeout= va
lue then the connection is forcibly closed and an error (504) is
returned to the client.

The keepalive= parameter is to enable the TCP keepalive mechanism on
the connection at the operating system level (SO_KEEPALIVE socket
option).
As you noted it consists of periodic TCP probes sent by the OS on the
connection to prevent intermediaries/routers/firewalls from closing it
due to their own TCP inactivity timeout, and also to detect whether
the peer is still connected (by acking the probes). The settings for
the probes period, ack timeout and number of retries happen at the OS
level for all the connections, that's why keepalive= is a simple
on/off.

Note that the ProxyPass keepalive= parameter must not be confused with
the "KeepAlive on/off" directive pertaining to the client connection.
The KeepAlive and KeepAliveTimeout directives are about the lifetime
of the client connection in between HTTP transactions. Once a request
has been handled and a response was sent to the client the
KeepAliveTimeout is the time to wait for the next request on the same
connection (if "KeepAlive off" the connection is closed immediately
and thus each connection handles a single request/response only).

FWIW, here is the correspondence between the client side and backend
side settings regarding timeout and keepalive:

Client side:  Proxy/Backend side:
  Timeout N   <=>   timeout=N
  KeepAlive On/Off<=>   enablereuse=On/Off
  KeepAliveTimeout N  <=>   ttl=N
  [*Always On*]   <=>   keepalive=On/Off

Hope that helps..


Regards;
Yann.

-
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org

Re: [users@httpd] Regarding CVE-2021-40438

[alchemist vk]

Thanks Nick for your response.
I recently started looking to webserver and getting hands-on with these
things.
So I thought I can get higher exposure if I learn more about these CVEs and
implications. So I posted the question in this forum.

With Regards,
Venkatesh

On Tue, Oct 19, 2021 at 4:05 PM Nick Folino  wrote:

> Nobody here is going to tell you how to exploit vulnerabilities.
> If you can't figure it out by reading the code then upgrade to the fixed
> version.
>
> Nick
>
> On Tue, Oct 19, 2021 at 2:49 AM alchemist vk 
> wrote:
>
>> Hi All,
>>  I understand that, CVE-2021-40438 is fixed in httpd release 2.4.50
>> onwards.
>> But I would like to know more about, how this issue can be exploitable in
>> prior versions and can I know the commit id/patch details for this issue.
>>
>> Tried looking into commit details in github apache repo, but couldnt find
>> anything specific to CVE-2021-40438.
>>
>> Please help me in this regard,
>>
>> With Regards
>> Venkatesh
>>
>

[users@httpd] RE: [EXTERNAL] Re: [users@httpd] timeout and keepalive parameter in ProxyPass

[Orendt, John]

Hi

I use a different approach to these types of requests.

The back end request immediately returns a status message ”Request Received”

On Subsequent Requests returns a status message ”Working”

Finally, Subsequent Request returns a status message ”Ready” with results.

John Orendt
john.p.ore...@medtronic.com

From: Usha Nayak 
Sent: Tuesday, October 19, 2021 10:37 AM
To: users@httpd.apache.org
Subject: [EXTERNAL] Re: [users@httpd] timeout and keepalive parameter in 
ProxyPass

Hello

Thank you for your response. However, I'm still a bit fuzzy when it comes to 
the keepalive and timeout attribute.

I understand keepalive=On attribute by itself but in combination with timeout 
is where I'm unclear.

In the scenario that I presented with following:

  *timeout=3600 keepalive=On attributes
  *backend server taking a long time processing the request sent by client 
( note: no streaming any data from server yet since it's busy processing the 
request, let's say processing time on server is 4 hrs )
  *OS settings are set to send TCP Keep alive packet in every 2 mins

My guess:

  *   With TCP keep alive probes sent and ACK between 'Apache httpd' machine 
and 'Backend server' machine, we can be assured that there's no inactive or 
idle TCP connection.
  *   But, from the 'Apache Httpd' application point of view, it hasn't 
received the data from the server in 1 hr(3600), and therefore 'Apache Httpd' 
will initiate a timeout. When the 'Backend Server' is ready in 4 hrs to stream 
the response back, there's no ' Apache Httpd' client to receive the response.
Please do correct me if I'm thinking this incorrectly,

Appreciate all the help

Thanks.


On Wed, Oct 13, 2021 at 10:03 AM Dino Ciuffetti 
mailto:d...@tuxweb.it>> wrote:
The global "Timeout" parameter is needed to set the length of time Apache httpd 
will wait for I/O in various circumstances.
https://httpd.apache.org/docs/2.4/mod/core.html#timeout


The "keepalive" attribute of ProxyPass can be used to fool your network 
equipments (like intermediate firewalls) to mantain the tcp connection between 
apache and its backends open even when no data is passing from and to the 
wires. Some network equipment like firewalls tend to drop inactive (silent) 
connections too early (from my experience ~30 min but your mileage may vary).

The "timeout" attribute of ProxyPass is the number of seconds Apache httpd 
waits for data sent by / to the backend. So in your case apache will close a 
inactive connection with the backend in 1hr, but while your backend continue to 
stream data the connection will be kept open. Yes, it's a sort of application 
timeout.

There is also the "connectiontimeout" attribute of ProxyPass that will consider 
a new TCP connection with the backend not valid if it takes the specified time 
to complete (TCP SYN + SYN/ACK + ACK) so it can be retried earlier.




13 ottobre 2021 16:22, "Usha Nayak" 
mailto:usha...@gmail.com?to=%22usha%20nayak%22%20%3cusha...@gmail.com%3E>>
 wrote:
Hello
Need help in understanding the highlighted parameters:
ProxyPass "/example" 
"http://backend.example.com"
 timeout=3600 keepalive=On
keepalive parameter - As I understand relates to TCP or socket keep alive 
probes that are sent to prevent idle connection.
[cid:image001.png@01D7C4D6.CF3A0310]
My confusion is :
Assuming the backend server takes 4 hrs to process a request and I have 
'keepalive on' in Apache httpd with the OS setting to send tcp probes every 2 
minutes. Backend server and Apache httpd are sending TCP keep alive packets to 
indicate the socket connection is not idle.

  *   In this case, would the 'apache httpd' close the connection in 1 hr ( 
timeout = 3600) because the 'apache httpd' didn't get the http data? Is this 
timeout 'apache httpd' application timeout ?
Appreciate help.
Thanks.

[CONFIDENTIALITY AND PRIVACY NOTICE] Information transmitted by this email is 
proprietary to Medtronic and is intended for use only by the individual or 
entity to which it is addressed, and may contain information that is private, 
privileged, confidential or exempt from disclosure under applicable law. If you 
are not the intended recipient or it appears that this mail has been forwarded 
to you without proper authority, you are notified that any use or dissemination 
of this information in any manner is strictly prohibited. In such cases, please 
delete this mail from your records. To view this notice in other languages you 
can either select the following link or manually copy and paste the link into 
the address bar of a web browser: http://emaildisclaimer.medtronic.com

Re: [users@httpd] timeout and keepalive parameter in ProxyPass

[Usha Nayak]

Hello

Thank you for your response. However, I'm still a bit fuzzy when it comes
to the keepalive and timeout attribute.

I understand keepalive=On attribute by itself but in combination with
timeout is where I'm unclear.

In the scenario that I presented with following:

   -  timeout=3600 keepalive=On attributes
   -  backend server taking a long time processing the request sent by
   client ( note: no streaming any data from server yet since it's busy
   processing the request, let's say processing time on server is 4 hrs )
   -  OS settings are set to send TCP Keep alive packet in every 2 mins


My guess:

   - With TCP keep alive probes sent and ACK between 'Apache httpd' machine
   and 'Backend server' machine, we can be assured that there's no inactive or
   idle TCP connection.
   - But, from the 'Apache Httpd' application point of view, it hasn't
   received the data from the server in 1 hr(3600), and therefore 'Apache
   Httpd' will initiate a timeout. When the 'Backend Server' is ready in 4 hrs
   to stream the response back, there's no ' Apache Httpd' client to receive
   the response.

Please do correct me if I'm thinking this incorrectly,

Appreciate all the help

Thanks.


On Wed, Oct 13, 2021 at 10:03 AM Dino Ciuffetti  wrote:

> The global "Timeout" parameter is needed to set the length of time Apache
> httpd will wait for I/O in various circumstances.
> https://httpd.apache.org/docs/2.4/mod/core.html#timeout
>
>
> The "keepalive" attribute of ProxyPass can be used to fool your network
> equipments (like intermediate firewalls) to mantain the tcp connection
> between apache and its backends open even when no data is passing from and
> to the wires. Some network equipment like firewalls tend to drop inactive
> (silent) connections too early (from my experience ~30 min but your mileage
> may vary).
>
> The "timeout" attribute of ProxyPass is the number of seconds Apache httpd
> waits for data sent by / to the backend. So in your case apache will close
> a inactive connection with the backend in 1hr, but while your backend
> continue to stream data the connection will be kept open. Yes, it's a sort
> of application timeout.
>
> There is also the "connectiontimeout" attribute of ProxyPass that will
> consider a new TCP connection with the backend not valid if it takes the
> specified time to complete (TCP SYN + SYN/ACK + ACK) so it can be retried
> earlier.
>
>
>
>
> 13 ottobre 2021 16:22, "Usha Nayak"  >
> wrote:
>
> Hello
> Need help in understanding the highlighted parameters:
> ProxyPass "/example" "http://backend.example.com; timeout=3600
> keepalive=On
> keepalive parameter - As I understand relates to TCP or socket keep alive
> probes that are sent to prevent idle connection.
> [image: image.png]
> My confusion is :
>
> Assuming the backend server takes 4 hrs to process a request and I have
> 'keepalive on' in Apache httpd with the OS setting to send tcp probes every
> 2 minutes. Backend server and Apache httpd are sending TCP keep alive
> packets to indicate the socket connection is not idle.
>
>
>- In this case, would the 'apache httpd' close the connection in 1 hr
>( timeout = 3600) because the 'apache httpd' didn't get the http data? Is
>this timeout 'apache httpd' application timeout ?
>
> Appreciate help.
> Thanks.
>
>
>
>

Re: [users@httpd] How to display the True-Client-IP header in the access log

[Rainer Canavan]

On Tue, Oct 19, 2021 at 1:44 PM Mason Hayes  wrote:
>
> Hi, All
>
> When Apache is accessed via a CDN (Akamai), I would like to record the IP of 
> the accessing client in the Apache logs.
> In order to display the True-Client-IP header sent by Akamai in the access 
> log like X-Forward-For, do I have to change the Logformat setting in 
> httpd.conf as follows?
>
> Logformat
> "%{True-Client-IP}i %h %l %u %t˶~˵"%r\" %>s %b˶~˵"%{Referer}i\" 
> \%{User-Agent}i\" combined

That looks OK, but you may want to look into using
https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html
You would have to set RemoteIPHeader to True-Client-IP and, since
Akamai to my knowledge doesn't publish a list of its source IPs,
consider some kind of authentication, e.g. basic auth
https://httpd.apache.org/docs/2.4/mod/mod_auth_basic.html to protect
the vhost from access without Akamai. Otherwise anyone would be able
to fake an arbitrary source IP in your logs.

rainer

-
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org

Re: [users@httpd] How to display the True-Client-IP header in the access log

[Jim Albert]

On 10/19/2021 7:43 AM, Mason Hayes wrote:

Hi, All

When Apache is accessed via a CDN (Akamai), I would like to record the 
IP of the accessing client in the Apache logs.
In order to display the True-Client-IP header sent by Akamai in the 
access log like X-Forward-For, do I have to change the Logformat 
setting in httpd.conf as follows?


Logformat
"%{True-Client-IP}i %h %l %u %t˶~˵"%r\" %>s %b˶~˵"%{Referer}i\" 
\%{User-Agent}i\" combined


If anyone has had any success with True-Client-IP showing up in the 
logs, please let me know.


Regards,



If the real client IP is not in X-Forwarded-For you'll need to know what 
environment variable it is supplied in. You should be able to write some 
server side code to list all the environment variables and their values 
presented to you by the web server. That variable in Apache for me  is 
X-Forwarded-For.


Jim


-
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org

RE: [users@httpd] How to display the True-Client-IP header in the access log

[Marc]

With haproxy you have an option to enable a proxy protocol, this transmits the 
client ip. I guess something similar must exist in your case.

> 
> When Apache is accessed via a CDN (Akamai), I would like to record the
> IP of the accessing client in the Apache logs.
> In order to display the True-Client-IP header sent by Akamai in the
> access log like X-Forward-For, do I have to change the Logformat setting
> in httpd.conf as follows?
> 
> Logformat
> "%{True-Client-IP}i %h %l %u %t˶~˵"%r\" %>s %b˶~˵"%{Referer}i\" \%{User-
> Agent}i\" combined
> 
> If anyone has had any success with True-Client-IP showing up in the
> logs, please let me know.
> 
> Regards,
> 


-
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org

[users@httpd] How to display the True-Client-IP header in the access log

[Mason Hayes]

Hi, All

When Apache is accessed via a CDN (Akamai), I would like to record the IP
of the accessing client in the Apache logs.
In order to display the True-Client-IP header sent by Akamai in the access
log like X-Forward-For, do I have to change the Logformat setting in
httpd.conf as follows?

Logformat
"%{True-Client-IP}i %h %l %u %t˶~˵"%r\" %>s %b˶~˵"%{Referer}i\"
\%{User-Agent}i\" combined

If anyone has had any success with True-Client-IP showing up in the logs,
please let me know.

Regards,

Re: [users@httpd] Regarding CVE-2021-40438

[Nick Folino]

Nobody here is going to tell you how to exploit vulnerabilities.
If you can't figure it out by reading the code then upgrade to the fixed
version.

Nick

On Tue, Oct 19, 2021 at 2:49 AM alchemist vk  wrote:

> Hi All,
>  I understand that, CVE-2021-40438 is fixed in httpd release 2.4.50
> onwards.
> But I would like to know more about, how this issue can be exploitable in
> prior versions and can I know the commit id/patch details for this issue.
>
> Tried looking into commit details in github apache repo, but couldnt find
> anything specific to CVE-2021-40438.
>
> Please help me in this regard,
>
> With Regards
> Venkatesh
>

Re: [users@httpd] Issue with Apache 2.4.51 hanging

[Deepak Goel]

Hi

Looks like the step 2 in your process is not working in the upgraded
version of apache.

Therefore it is vomiting out the following:
 server reached MaxRequestWorkers setting, consider raising the
MaxRequestWorkers setting

Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"

+91 73500 12833
deic...@gmail.com

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

Make In India : http://www.makeinindia.com/home


On Mon, Oct 18, 2021 at 2:57 PM Patrick Verdon 
wrote:

> Hi All,
>
> I'd appreciate some feedback on an issue I'm experiencing. I've spent
> quite some time researching the problem as it causes a serious outage in
> our application. I've searched the Web, Stack Overflow, this list's mail
> archives, the latest Apache bugs, and more, but have not been able to find
> any reports of a similar issue.
>
> Background. I'm running the latest Apache 2.4.51 on Amazon Linux with
> mod_proxy, mod_php and mod_ssl with varnish in front. Some requests to our
> application take about 45 seconds to complete so there is a warm-up cache
> procedure at regular intervals during the day which primes the varnish
> cache. The following steps reliably cause Apache to hang, requiring a
> manual restart:
>
>1. Varnish cache is cleared, causing spike in load on httpd
>2. Warm-up cache process kicks off with 2 long running requests (45
>seconds each). This is a PHP application running under mod_php - each
>process grows up to 700 MB, so the application kills the httpd child
>process at the end to release the memory, using posix_kill(PID, 28).
>3. Apache hangs and does not recover. Varnish serves 503s.
>4. Manual restart required: service httpd restart
>5. Errors in the log show that 2 children had segmentation faults,
>presumably the 2 with long running processes.
>
>
> Albeit ugly, this process has been running for a year and a half without
> any issues. We traced the date that crashes started to the date Apache was
> upgraded from version 2.4.46 to 2.4.48 and as you can see it's still an
> issue in 2.4.51.
>
> See the error_log below and details about the installation.
>
> Any feedback on where to report this issue would be much appreciated.
>
> Thanks.
>
> Patrick
>
> --
>
> # cat /var/log/httpd/error_log
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> *** Error in `/usr/sbin/httpd': corrupted size vs. prev_size:
> 0x557f94567e4f ***
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> httpd: misc/apr_reslist.c:161: reslist_cleanup: Assertion `rl->ntotal ==
> 0' failed.
> [Sun Oct 17 15:53:47.990497 2021] [core:notice] [pid 2620] AH00052: child
> pid 3166 exit signal Aborted (6)
> [Sun Oct 17 15:53:47.990531 2021] [core:notice] [pid 2620] AH00052: child
> pid 3483 exit signal Aborted (6)
> [Sun Oct 17 15:53:47.990545 2021] [core:notice] [pid 2620] AH00052: child
> pid 2657 exit signal Aborted (6)
> [Sun Oct 17 15:53:47.990557 2021] [core:notice] [pid 2620] AH00052: child
> pid 2660 exit signal Aborted (6)
> [Sun Oct 17 15:53:47.990568 2021] [core:notice] [pid 2620] AH00052: 

[users@httpd] Regarding CVE-2021-40438

[alchemist vk]

Hi All,
 I understand that, CVE-2021-40438 is fixed in httpd release 2.4.50 onwards.
But I would like to know more about, how this issue can be exploitable in
prior versions and can I know the commit id/patch details for this issue.

Tried looking into commit details in github apache repo, but couldnt find
anything specific to CVE-2021-40438.

Please help me in this regard,

With Regards
Venkatesh