hung apache during shutdown, was: one word syncronize once more
From another thread Re: one word syncronize once more, new thread created as I also don't believe the issues are related. Greg Ames wrote: if you have a hung server, I would do some more diagnosis and not get sidetracked by MRPC. have you looked for unusual messages in the error log, especially messages that mention MaxClients? have you gotten samples of the server-status page with ExtendedStatus on? do you see a pattern with certain URLs that hang and others that do not? are small local static files always served quickly? The hung apache runtime always occur after the leader apache process has been given its signal to terminate. So talking in terms of what web pages it serves makes no sense, since it has already shutdown all further webpage processing. The fatal bug is that the apache leader process does not terminate in good time and remains running. I believe it also hang onto the bound listerning tcp sockets as well, obvious a kill -9 gets rid of it. I'm not sure without any extra information if this list will be able to assist in the matter, maybe there is someway to have apache emit debug logging information to stdout/stderr which I can capture to see where in apache it is hung up and what it is waiting for, maybe a sledgehammer approach of using a SIGALRM and new handler to be put in place as soon as apache begins its shutdown. This new handler will raise a SIGKILL on itself should it go off. Its unlikely I'd have chance to investigate this matter in the short term anyway. I was just using the previous thread as an example to get insight into what actual real life problems may result from lost increment/decrements due to SMP concurrent issues. Darryl
Re: one word syncronize once more
the disease: on an SMP system, multiple CPUs could try to update requests_this_child simultaneously, and only one of the updates will take effect. this is a rare timing situation. the SMP issue is not important because the variable will still be updated and continue to do its job. requests_this_child is the variable used to implement MaxRequestsPerChild. this is used to periodically quiesce worker processes which are replaced by new workers and keep the server going if there are slow memory or resource leaks due to bugs. the hope is a worker process that is using more and more memory or file descriptors over time will be shut down before the situation gets serious. it counts the number of connections served by a worker process, contrary to its name. the admin picks a big number out of the air, let's say 1, and sets MaxRequestsPerChild to that value. if he observes that that workers are now being shut down before the leak gets too bad and there isn't a lot of unneccessary process forking/exiting, great! he's done until he has time to address the software bug which is the root cause of the problem. if the workers are still getting too big before quiescing, he lowers MRPC. if there is no more worker growth problem but the CPU is high and he can see workers getting forked very rapidly, he raises MRPC. so if internally the httpd actually processes 10002 connections instead of 1 before quescing some worker process, it isn't important enough to justify slowing down the mainline path. if you have a hung server, I would do some more diagnosis and not get sidetracked by MRPC. have you looked for unusual messages in the error log, especially messages that mention MaxClients? have you gotten samples of the server-status page with ExtendedStatus on? do you see a pattern with certain URLs that hang and others that do not? are small local static files always served quickly? Greg - Original Message From: Darryl Miles [EMAIL PROTECTED] To: dev@httpd.apache.org Sent: Friday, July 20, 2007 1:46:53 PM Subject: Re: one word syncronize once more Greg Ames wrote: please see rev. 558039. requests_this_child does not need to be 100% accurate. the cure below is worse than the disease. Greg -requests_this_child--; /* FIXME: should be synchronized - aaron */ +apr_atomic_dec32(requests_this_child); /* much slower than important */ Maybe someone could reconfirm to the list what exactly the disease is ? Got a little couch potato? Check out fun summer activities for kids. http://search.yahoo.com/search?fr=oni_on_mailp=summer+activities+for+kidscs=bz
Re: one word syncronize once more
please see rev. 558039. requests_this_child does not need to be 100% accurate. the cure below is worse than the disease. Greg - Original Message From: Dmytro Fedonin [EMAIL PROTECTED] To: dev@httpd.apache.org Sent: Thursday, June 14, 2007 11:49:42 AM Subject: one word syncronize once more Hi all, I've got some response which shows that I was not clear enough in my previous post. Fullproof solution would be: Index: server/mpm/worker/worker.c === --- server/mpm/worker/worker.c (revision 545597) +++ server/mpm/worker/worker.c (working copy) @@ -892,7 +887,7 @@ bucket_alloc = apr_bucket_alloc_create(ptrans); process_socket(ptrans, csd, process_slot, thread_slot, bucket_alloc); worker_sockets[thread_slot] = NULL; -requests_this_child--; /* FIXME: should be synchronized - aaron */ +apr_atomic_dec32(requests_this_child); /* much slower than important */ apr_pool_clear(ptrans); last_ptrans = ptrans; } Because we don't care about if (requests_this_child = 0) it would be enough. But it is too way slow and is not so important. Get the free Yahoo! toolbar and rest assured with the added security of spyware protection. http://new.toolbar.yahoo.com/toolbar/features/norton/index.php
Re: one word syncronize once more
Greg Ames wrote: please see rev. 558039. requests_this_child does not need to be 100% accurate. the cure below is worse than the disease. Greg -requests_this_child--; /* FIXME: should be synchronized - aaron */ +apr_atomic_dec32(requests_this_child); /* much slower than important */ Maybe someone could reconfirm to the list what exactly the disease is ? If this counter is not accurate (meaning it may loose increments/decrements) does the end the world cave in ? Maybe someone could explain how and where this particular variable requests_this_child is used. For example if its just to provide guideline information to allow termination after 1 requests then thats less of a problem than if someone configured it to terminate after 2 requests. Being 0.0001% out is less of a problem than 100% out. For example if the counter is used in some way to allow apache to restart/logrotate and if it does not come back to zero reliabily then apache will remain hung trying to shutdown (as it often the case in my real world experience) then the disease would be a hung webserver during shutdown and taking. This is much more fatal. I'm not suggesting this particular counter requests_this_child has a direct cause with hung apache instances but I'm asking those that understand the disease(s) if they could explain exactly what they are. I'm not familiar with the apr_atomic_dec32() API, is this correctly optimized to a single asm instruction lock decl 0xaddr32 on Intel IA32 ? How many clock cycles do you think that operations takes, what technical understanding makes you think such a cure should be utilized with caution ? My understanding of the cure is that its very very light weight so light weight I'd challenge you to prove the mythical performance issue in the situation its being used in apache. To give this some weight, I'm was recently involved in an multi-threaded ethernet packet processing application dealing with 100's of mbit and each packet that came updated multiple counters using the IA32 LOCK prefix but there was zero noticeable contention. Indeed this is the purpose of the assembly primitive to implement something in hardware that is most efficiently done there. Darryl
Re: one word syncronize
sebb wrote:
On 14/06/07, Dmytro Fedonin [EMAIL PROTECTED] wrote:
Looking through 'server/mpm/worker/worker.c' I have found such a
combination of TODO/FIXME comments:
1)
/* TODO: requests_this_child should be synchronized - aaron */
if (requests_this_child = 0) {
2)
requests_this_child--; /* FIXME: should be synchronized - aaron */
And I can not see any point here. These are one word CPU operations,
thus there is no way to preempt inside this kind of operation. So, one
CPU is safe by nature of basic operation. If we have several CPUs they
will synchronize caches any way, thus we will never get inconsistent
state here. We can only lose time trying to synchronize it in code. Am I
not right?
The decrement operation is a read-modify-write cycle, it is possible for
2 CPUs to overlap their operations, ending up with a observable lost
decrement. Since they both end up reading the same initial value.
On IA32/x86 the DEC assembly instruction operation can be prefixed by
the LOCK instruction, this makes the CPU continue to assert memory bus
locking for the duration of the instruction so there is no way for CPU2
to perform a read access until CPU1 releases control of the memory bus
when it completes the instruction, this is effectively what atomic_dec()
enforces.
The amount of performance lost by using atomic_xxx() really is minimal,
with any luck it might only be that cache-line that remains locked not
the entire memory bus.
The decrement operation may be handled as load, decrement, store on
some architectures, so can be pre-empted by a different CPU.
There is no other way to handle it :) Memory itself can't perform
arithmetic operations, so the decrement always happens inside the ALU
inside the CPU.
It is true that non-SMP aware CPUs might maintain memory bus acquisition
during the 'decrement' (aka modify) phase of the operation since there
is no reason not to give it up as they are the only user of memory.
This becomes a performance bottleneck for any SMP capable CPU which has
a cache that can operate at full CPU clock speeds. As the 'decrement'
(aka modify) phase is going to require at least 1 clock cycle to perform
so why not let another CPU make use of the memory bus.
Also some hardware architectures (e.g. HP Alpha) have an unusual
memory model. One CPU may see memory updates in a different order from
another CPU. Software that relies on the updates being seen across all
CPUs must use the appropriate memory synchronisation instructions.
I don't know if these considerations apply to this code.
Memory update ordering applies when considering how 2 or more distinct
machine words are updated with respect to themselves when those updates
are observed from another CPU.
The example here is with concerns over a single machine word being
updated on SMP systems.
Darryl
one word syncronize
Hi all,
Looking through 'server/mpm/worker/worker.c' I have found such a
combination of TODO/FIXME comments:
1)
/* TODO: requests_this_child should be synchronized - aaron */
if (requests_this_child = 0) {
2)
requests_this_child--; /* FIXME: should be synchronized - aaron */
And I can not see any point here. These are one word CPU operations,
thus there is no way to preempt inside this kind of operation. So, one
CPU is safe by nature of basic operation. If we have several CPUs they
will synchronize caches any way, thus we will never get inconsistent
state here. We can only lose time trying to synchronize it in code. Am I
not right?
PS My assumptions are several threads of the same process are dealing
with one word of common memory.
--
Best regards,
Dmytro
one word syncronize once more
Hi all, I've got some response which shows that I was not clear enough in my previous post. Fullproof solution would be: Index: server/mpm/worker/worker.c === --- server/mpm/worker/worker.c (revision 545597) +++ server/mpm/worker/worker.c (working copy) @@ -892,7 +887,7 @@ bucket_alloc = apr_bucket_alloc_create(ptrans); process_socket(ptrans, csd, process_slot, thread_slot, bucket_alloc); worker_sockets[thread_slot] = NULL; -requests_this_child--; /* FIXME: should be synchronized - aaron */ +apr_atomic_dec32(requests_this_child); /* much slower than important */ apr_pool_clear(ptrans); last_ptrans = ptrans; } Because we don't care about if (requests_this_child = 0) it would be enough. But it is too way slow and is not so important. So, the question is: If on x86 we have native atomic 'subl $0x1,mem' which is almost as fast as optimized mov dec mov, why not to use platform specific feature. As for now x86 represents majority of hosts. PS I have found 21 atomic dec/inc in trunk and more than half of them are about counters similar to the case above. -- Best regards, Dmytro
Re: one word syncronize
On 14/06/07, Dmytro Fedonin [EMAIL PROTECTED] wrote:
Hi all,
Looking through 'server/mpm/worker/worker.c' I have found such a
combination of TODO/FIXME comments:
1)
/* TODO: requests_this_child should be synchronized - aaron */
if (requests_this_child = 0) {
2)
requests_this_child--; /* FIXME: should be synchronized - aaron */
And I can not see any point here. These are one word CPU operations,
thus there is no way to preempt inside this kind of operation. So, one
CPU is safe by nature of basic operation. If we have several CPUs they
will synchronize caches any way, thus we will never get inconsistent
state here. We can only lose time trying to synchronize it in code. Am I
not right?
The decrement operation may be handled as load, decrement, store on
some architectures, so can be pre-empted by a different CPU.
Also some hardware architectures (e.g. HP Alpha) have an unusual
memory model. One CPU may see memory updates in a different order from
another CPU. Software that relies on the updates being seen across all
CPUs must use the appropriate memory synchronisation instructions.
I don't know if these considerations apply to this code.
PS My assumptions are several threads of the same process are dealing
with one word of common memory.
--
Best regards,
Dmytro
