Re: [users@httpd] missing image
On 2/1/24 8:25 AM, Adam Weremczuk wrote: Hi Sherrard, My index.html is super simple: sorry if i wasn't clear. i was asking if you had the version with the absolute path posted on your site. no matter, see the below. After I replace: img src="ms-logo.png" with img src="/var/www/html/holding/ms-logo.png" (valid path) the image doesn't even show on the home page. that is not the correct form of the absolute URL. absolute URLs are based at your DocumentRoot. so since your DocumentRoot is DocumentRoot /var/www/html/holding the correct absolute URL would be "/ms-logo.png"", and the resulting img tag would be HTH best, Sherrard - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] graceful-stop closes established connections without response
On 1/31/24 3:11 PM, Sherrard Burton wrote: two is about par for the course _when there are resets_. but it doesn't necessarily happen on every test run. for example, after initially applying the v1 patch (and having fully composed a response to say that the patch seemed to have worked :-)), i discovered that i had forgotten to switch back from prefork to event. ie, i ran the test a few times in a row with no resets, even though the patch was not in play. i have not previously, but i will try to gather some statistics about number of resets per failed run and failed vs successful runs, since the combination of the two is needed to make any quantitative assessment. correction: two is par for the course when the client connection is remote. in order to have an iterable test setup, i created a bash loop that started the tcpdump, strace and curl instances all on the server while simultaneously gracefully stopping apache. with client connection coming from the localhost, we end up with many more reset connections per run, and failures present in each test. but ultimately, the results are such that i don't think any changes are warranted. stock debian apache 2.4.57-2, 100 iterations: average number of established connections/submitted requests: 67284 / 100 = 672.84 average number of responses received: 65162 / 100 = 651.62 average number of reset connections: 2122 / 100 = 21.22 patched with v3, 100 iterations: average number of established connections/submitted requests: 68319 / 100 = 683.19 average number of responses received: 66082 / 100 = 660.82 average number of reset connections: 2237 / 100 = 22.37 not sure if the slight increase in connections established and handled under the patched version indicates that there might be some benefit to the patch, or if the difference is small enough that it can be chalked up slight differences in resource demands in the VM host at the time that the tests were running. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] graceful-stop closes established connections without response
On 1/31/24 06:24 AM, Yann Ylavic wrote: On Tue, Jan 30, 2024 at 8:24 PM Sherrard Burton wrote: i have confirmed that the patch has been applied, and the behavior still persists, as confirmed by comparing the counts of [SYN,ACK] and accept() ~$ tcpdump -n -r /tmp/tcpdump.pcap | grep -Fc '[S.]'; grep -Fh 'accept4' /tmp/strace-apache2.out.* | grep -Fc .240.209 reading from file /tmp/tcpdump.pcap, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 Warning: interface names might be incorrect 3485 3483 This means those two connections came in (or were made available by the system) after the last accept() call, which is the race condition that httpd can do nothing about unfortunately. How much does it improve compared to non-patched httpd, how many reset connections without the patch? If not significant I don't think it's worth attempting to do something about it.. two is about par for the course _when there are resets_. but it doesn't necessarily happen on every test run. for example, after initially applying the v1 patch (and having fully composed a response to say that the patch seemed to have worked :-)), i discovered that i had forgotten to switch back from prefork to event. ie, i ran the test a few times in a row with no resets, even though the patch was not in play. i have not previously, but i will try to gather some statistics about number of resets per failed run and failed vs successful runs, since the combination of the two is needed to make any quantitative assessment. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] missing image
On 1/31/24 02:26 PM, Adam Weremczuk wrote: I've already tried replacing relative path to the image with absolute but it made no difference. Any ideas? do you have a live example with the absolute path? the broken ones that i looked at all had the relative paths which (understandably) doesn't work. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] graceful-stop closes established connections without response
i have confirmed that the patch has been applied, and the behavior still persists, as confirmed by comparing the counts of [SYN,ACK] and accept() ~$ tcpdump -n -r /tmp/tcpdump.pcap | grep -Fc '[S.]'; grep -Fh 'accept4' /tmp/strace-apache2.out.* | grep -Fc .240.209 reading from file /tmp/tcpdump.pcap, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 Warning: interface names might be incorrect 3485 3483 as well as ~$ for i in '> GET /index.html' '< HTTP/1.1 200 OK'; do echo -e "'$i':\t$( cat /tmp/curl-[0-9]*.out | grep -Fc "$i" )"; done '> GET /index.html': 3485 '< HTTP/1.1 200 OK': 3483 ~$ grep -F reset /tmp/curl-[0-9]*.out /tmp/curl-13.out:* Recv failure: Connection reset by peer /tmp/curl-13.out:curl: (56) Recv failure: Connection reset by peer /tmp/curl-2.out:* Recv failure: Connection reset by peer /tmp/curl-2.out:curl: (56) Recv failure: Connection reset by peer which all seem to agree that there were two failed connections and, in case this shines some light on things: ~$ grep -F '19:10:50' /var/log/apache2/error.log | grep -F XXX [Tue Jan 30 19:10:50.357074 2024] [mpm_event:notice] [pid 151381:tid 140267709454016] XXX: may exit (0, 1) [Tue Jan 30 19:10:50.357498 2024] [mpm_event:notice] [pid 151382:tid 140267709454016] XXX: may exit (0, 2) [Tue Jan 30 19:10:50.357610 2024] [mpm_event:notice] [pid 151381:tid 140267709454016] XXX: draining [Tue Jan 30 19:10:50.358011 2024] [mpm_event:notice] [pid 151381:tid 140267709454016] XXX: exiting [Tue Jan 30 19:10:50.358091 2024] [mpm_event:notice] [pid 151382:tid 140267709454016] XXX: draining [Tue Jan 30 19:10:50.358201 2024] [mpm_event:notice] [pid 151381:tid 140267709454016] XXX: closing [Tue Jan 30 19:10:50.358492 2024] [mpm_event:notice] [pid 151382:tid 140267709454016] XXX: exiting [Tue Jan 30 19:10:50.358906 2024] [mpm_event:notice] [pid 151382:tid 140267709454016] XXX: closing [Tue Jan 30 19:10:50.366948 2024] [mpm_event:notice] [pid 151381:tid 140267709454016] XXX: exited [Tue Jan 30 19:10:50.367124 2024] [mpm_event:notice] [pid 151382:tid 140267709454016] XXX: exited On 1/30/24 06:09 AM, Yann Ylavic wrote: On Tue, Jan 30, 2024 at 11:54 AM Yann Ylavic wrote: On Tue, Jan 30, 2024 at 4:37 AM Sherrard Burton wrote: i was going to add some debugging lines, but when i took a quick look at the patch, i wasn't clear on which sections of the code i should be guaranteed to hit. can you be so kind as to send an updated patch with some gratuitous logging in the appropriate sections so that there will be positive affirmation that the patch has (or hasn't) been applied and is falling into the expected sections? Sure, here is a v2 (which also includes a fix w.r.t. v1). Argh, please use this v3 instead, I missed that EINTR could interfere and should be ignored while draining. Regards; Yann. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] graceful-stop closes established connections without response
On 1/29/24 12:25 PM, Yann Ylavic wrote: That's where we are, I think, if this first/light patch eventually helps significantly with the "local" graceful-stop which you care about still, it's possibly worth it since it requires no opt-in (but needs testing..), but going further looks overkill/risky for httpd. i (theoretically) applied your patch to the debian source and rebuilt and installed the packages, but the behavior still persists. it has been a long time since i have had to patch debian source (it was apache back then as well :-) thundering herd patch IIRC) so i'm not 100% sure that i've gotten it right. mod_mpm_event.so does indeed have an updated modtime -rw-r--r-- 1 root root 75984 Jan 30 00:13 /usr/lib/apache2/modules/mod_mpm_event.so vs -rw-r--r-- 1 root root 75984 Apr 13 2023 /usr/lib/apache2/modules/mod_mpm_event.so but the identical file size makes me wonder whether i was successful in applying the patch. i was going to add some debugging lines, but when i took a quick look at the patch, i wasn't clear on which sections of the code i should be guaranteed to hit. can you be so kind as to send an updated patch with some gratuitous logging in the appropriate sections so that there will be positive affirmation that the patch has (or hasn't) been applied and is falling into the expected sections? - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] graceful-stop closes established connections without response
On 1/29/24 10:17 AM, Yann Ylavic wrote: On Mon, Jan 29, 2024 at 3:06 PM Eric Covener wrote: The patch helps in this case because we no longer close the listening sockets unconditionally, I mean without first checking if there are new connections in the backlog. So I thought the option was needed because if nothing stops new connections from arriving it could prevent the child from stopping indefinitely? How could we know if a LB/VIP is in place? it sounds like this issue is all but resolved, but i would like to understand whether the above (preventing the child from stopping indefinitely) is an actual possibility. my (naive) expectation is that if a given child has been signaled while handling an existing request then it "knows" not to accept() a new request after completing the existing request. so it seems that the child is not under any danger of continuing indefinitely, regardless of the contents of the backlog. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] graceful-stop closes established connections without response
On 1/29/24 9:05 AM, Eric Covener wrote: Maybe I wasn't clear enough but this patch makes sense only if there is something in place that prevents new connections from arriving at the stopping httpd children processes (like a frontend/load-balancer or a tcp/bpf filter), otherwise they may never really stop which does not help for a graceful stop/restart obviously. So this change (if useful) should be guarded by a GracefulDrain on/off or something config option to not hurt the other use cases. Thanks Yann! It seems to me If there is no such LB/VIP that stops new connections from landing on this server, the new option should be avoided. But if there is such a LB/VIP, the option is not really needed. Is it fair? indeed. i tried to convey this in my most-recent response, but i think you put it much more clearly. we have essentially starting removing servers from the backend pool before issuing the graceful-stop. _when_ this is feasible, it does obviate the need for any changes at the httpd level. but it still leaves us with the possibility of resetting established connections during unplanned shutdown, such as an unexpected host reboot. but, as indicated previously, i think i am satisfied with this approach after gaining understanding the underlying mechanisms. - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] graceful-stop closes established connections without response
On 1/29/24 8:59 AM, Yann Ylavic wrote: Maybe I wasn't clear enough but this patch makes sense only if there is something in place that prevents new connections from arriving at the stopping httpd children processes (like a frontend/load-balancer or a tcp/bpf filter), otherwise they may never really stop which does not help for a graceful stop/restart obviously. So this change (if useful) should be guarded by a GracefulDrain on/off or something config option to not hurt the other use cases. Yann, thank you for the feedback and the thorough explanation (i hadn't thought about the OS-level backlog). after discovery of this problem i began removing the target server from the backend pool in the load-balancer before issuing the graceful-stop. this seems to work as expected for planned maintenance, but the problem can still occur if the backend server ends up needing to be stopped outside of my direct command. we know that we cannot guard against all situations, but the systemd unit files on debian seem to trigger graceful-stop for its 'stop' action ExecStop=/usr/sbin/apachectl graceful-stop so if indeed this were addressable within httpd (which it sounds like it is not) then we would theoretically be protected from all but unexpected hard crashes in httpd or at the OS level. my takeaway at this point, not having yet finished reading Eric's other responses on this thread, is that this mostly "is what it is", and i think that i am (provisionally) satisfied with knowing that the behavior that i am observing is not a bug, per-se. ps--is there any possibility of "shrinking" this window by tuning either ListenBacklog or any analogous settings at the OS level? or do i risk creating a different problem? i ask because theoretically, the backlog/queuing (from the client perspective) should mostly be handled by the load-balancer, which is currently L4, not L7. so if reducing/eliminating the backlog on the server might positively impact this situation then i may spend some time on it. thanks again - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
Re: [users@httpd] graceful-stop closes established connections without response
Eric, thanks for the quick reply. follow-up inline below: On 1/27/24 09:46 PM, Eric Covener wrote: apache2: 2.4.56-1~deb11u2, prefork MPM, mod_perl I think it's a large window on prefork where this can happen. If any process is busy processing a request, it cannot close its copy of the listening socket. The OS will continue to complete TCP connections and acknowledge (some) data with nobody calling accept(). When the last of the listening sockets is finally closed, the TCP connections that arrived in this timeframe will be abruptly closed. is this window essentially the time between when the parent process calls close() on its copy of the listening socket and when the last of the child processes is handling its request and can call close() on the last remaining copy of the listening socket? if so, then it sounds like the window becomes wider and this situation becomes more likely if the children are handling longer-running requests. Both worker and event MPMs have a dedicated listener thread per child process, so it will close those copies of the listening sockets much more quickly. so that i am clear, are you saying that this behavior is still possible, although less likely under the worker and event MPMs? NVM, actually, it appears that i may have already answered this question by not paying enough attention to detail when setting up my test environment. i unintentionally replicated the behavior under the event MPM when my intention was to use prefork for the sake of consistency. but i would still like to better understand "how much" the window is reduced under worker and event. for example, would they be similarly-effected by a child that is still handling a long-running request, or would the listener thread be able to close() its copy of the socket while the handling thread is still serving content to the client? thanks again - To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
[users@httpd] graceful-stop closes established connections without response
good day all,
i have an issue where for a small subset of requests issued during a
graceful-stop the client is not receiving connection refused (immediate
[RST,ACK] after [SYN]), nor receiving a full response, but the client is
rather completing the handshake, receiving an [ACK] after being allowed
to send the full request, but then receiving a [RST,ACK] immediately
after that [ACK].
as a result, a client with direct access reports 'connection reset by
peer', a client with access through a L4 load balancer reports 'empty
reply from server' and a client accessing via a reverse proxy receives a
502. all of these seem to be consistent with the observed behavior of
abruptly closing the connection.
i have the tcpdumps and the strace output to accompany the above
behavior, but before deluging the list with attachments, i wanted to
first see
a) if anyone has experienced similar behavior during graceful-stop
b) if anyone can point me to an explanation of this behavior (and let me
know where my search skills are lacking :-))
c) if it is my understanding of the expected behavior of graceful-stop
is at fault here
the particulars of my setup:
Debian GNU/Linux 11 (bullseye)
apache2: 2.4.56-1~deb11u2, prefork MPM, mod_perl
my testing involves starting a series of curl loops in parallel, then
calling graceful-stop on the running server.
client:
~$ for i in {1..6}; do for j in {1..1000}; do curl ...; done; done
server:
~$ apache2ctl graceful-stop && sleep 5 && systemctl start apache2
and the above behavior can usually be observed for a handful of requests
during each test run. and from comparing the strace and tcpdump output,
in _appears_ that the connection is being dropped between the time that
it is received by the parent apache process, but before one of the child
processes can accept() it. ie, for the ephemeral port used for the
errant connections, there is no matching accept() in any of the strace
output.
i hope that i have provided enough initial information and a sufficient
description. thanks in advance for any help, pointers or correction.
-
To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org
For additional commands, e-mail: users-h...@httpd.apache.org
[us...@httpd] cache key generation for reverse proxy
please forgive me if this is the wrong place for this question, or if this has been discussed elsewhere. i searched most of the night and morning, and then started pouring through the source code, and i'm pretty sure i've isolated the "issue" but need some advice as to where to go. we have been using apache 1.3 as a caching reverse proxy in front of mod_perl backend servers for several years, and it has worked flawlessly. we recently took the leap to upgrade to 2.2, and everything is working generally well, but we are having some issues with on-disk caching using mod_disk_cache. for each request, we determine which backend server to proxy off to based on various complex criteria, using mod_rewite. but our setup can be generalized as follows: we send "human" traffic to one set of machines and "bot" traffic to another. the content returned for each is a little different, mostly based on things like not including image-heavy widgets and links to pages or section of the site that require user login. with apache 1.3, this worked swimmingly, as the key for the on-disk cache seemed to be generated based on the proxy target url, as opposed to the canonical request url. $ sudo grep -rF -m1 '/stories/201007010002.html' . ./R/U/dsvSj2WzpWj5Do8S0Lcw:X-URL: http://crawler:8082/stories/201007010002.html ./x/h/e...@6q5zmjcpunuqnag:X-URL: http://backend:8081/stories/201007010002.html with the move to apache 2.2, it appears, based on the behavior we are seeing, as well as the comments in cache_storage.c, that the default key generation method is based on the canonical request url, taken before the translate name hook runs. the end result is that, based on our setup, we end up with "practical key collisions", although there is not technically a collision. so in the above example, humans might see the cached version of the page as generated for bot consumption, or vice-versa. looking at the cache key generation methods, and where they are called from, i could not see any way that i could, through the apache configuration, change this behavior. so is there a directive or module that i'm overlooking, or is there even a "simple" patch that i could apply that would get us back to the key generation behavior of 1.3, or something similar? thanks in advance. - The official User-To-User support forum of the Apache HTTP Server Project. See http://httpd.apache.org/userslist.html> for more info. To unsubscribe, e-mail: users-unsubscr...@httpd.apache.org " from the digest: users-digest-unsubscr...@httpd.apache.org For additional commands, e-mail: users-h...@httpd.apache.org
