Re: event mpm and mod_status
On Sat, 12 Jan 2013, Stefan Fritsch wrote:
On Thursday 10 January 2013, Niklas Edmundsson wrote:
To reiterate back to the event mpm / mod_status integration, are
there any work in progress on implementing a more verbose status
display for the event mpm? I'm thinking of something that can show
all requests currently being processed like we have today for
prefork/worker. The current "thread status" thingie is probably
interesting too, but more from a developer/server diag point of
view.
I am not aware of anyone working on this.
OK.
I agree that mod_socache may be a bit heavy-weight for this, but one
would need to test that.
My guess is that it's the one-big-lock requirement that bogs this
down for high request/s rates, but for us it might be good enough.
There is also mod_slotmem_shm, but AFAICS
that requires to know the size in advance. So one would have to
allocate AsyncRequestWorkerFactor times (max number of threads) slots
to have one slot per connection.
I'll investigate a bit when time allows. Part of the problem is to
find something useful to use as an identifier/index, process/thread is
obviously not enough.
This would still not give the full picture, because with pipelining,
there can be several (5 or 6, I think) requests active on one
connection: zero or one being processed by a worker and the rest doing
write completion. I am not sure how much effort one should put into
displaying that correctly in mod_status, though.
Given that all file delivery will be done by write completion I think
we'll want this to be rather accurate, at least for the connections
doing write completion (think a server being overloaded serving DVD
images, it gets easier to diagnose if you can easily figure out the
culprit).
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | ni...@acc.umu.se
---
If it's useless, it will have to be documented.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: event mpm and mod_status
On Thursday 10 January 2013, Niklas Edmundsson wrote: > To reiterate back to the event mpm / mod_status integration, are > there any work in progress on implementing a more verbose status > display for the event mpm? I'm thinking of something that can show > all requests currently being processed like we have today for > prefork/worker. The current "thread status" thingie is probably > interesting too, but more from a developer/server diag point of > view. > > How hitech does it have to be? Would using mod_socache for storage > be a reasonable way to do it? Personally I find it a bit rough > around the edges (no locking for example, so all users have to use > a single big lock for all access), but perhaps it's good enough > anyway? > > As it stands this seems to be the only blocker for us trying to use > the event mpm for file archive serving, so if there's any work > being done on the subject I'm very interested :) I am not aware of anyone working on this. I agree that mod_socache may be a bit heavy-weight for this, but one would need to test that. There is also mod_slotmem_shm, but AFAICS that requires to know the size in advance. So one would have to allocate AsyncRequestWorkerFactor times (max number of threads) slots to have one slot per connection. This would still not give the full picture, because with pipelining, there can be several (5 or 6, I think) requests active on one connection: zero or one being processed by a worker and the rest doing write completion. I am not sure how much effort one should put into displaying that correctly in mod_status, though.
Re: event mpm and mod_status
To reiterate back to the event mpm / mod_status integration, are there
any work in progress on implementing a more verbose status display for
the event mpm? I'm thinking of something that can show all requests
currently being processed like we have today for prefork/worker. The
current "thread status" thingie is probably interesting too, but more
from a developer/server diag point of view.
How hitech does it have to be? Would using mod_socache for storage be
a reasonable way to do it? Personally I find it a bit rough around the
edges (no locking for example, so all users have to use a single big
lock for all access), but perhaps it's good enough anyway?
As it stands this seems to be the only blocker for us trying to use
the event mpm for file archive serving, so if there's any work being
done on the subject I'm very interested :)
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | ni...@acc.umu.se
---
Old Vulcan Proverb "Only Nixon could go to China"
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: event mpm and mod_status
On Monday 07 January 2013, Daniel Lescohier wrote: > I see that event mpm uses a worker queue that uses a condition > variable, and it does a condition variable signal when something > is pushed onto it. If all of the cpu cores are doing useful work, > the signal is not going to force a context switch out of a thread > doing useful work, the thread will continue working until it's > used up it's timeslice, as long as it doesn't end early because it > becomes blocked on i/o or something like that. So, in that case, > the first thread to finish serving it's request and gets to > ap_queue_pop_something(worker_queue...) will get the item on the > queue, and it will have hot caches. On the other hand, if only > some of the cpu cores are active, but the threads that are active > are busy in the middle of a request, the condition variable signal > will wake up one of the idle threads, the thread will be scheduled > onto an idle core, and that thread can start doing useful > work. That thread may have colder caches, but the other cores > were busy doing useful work any way, so it's worth activating this > thread on an idle core. But on the idle core, it may make a difference which thread is used. A common pattern with event mpm is that a work thread does only a very small piece of work when writing some additional data to the network during write completion. It will then block again until ap_queue_pop_something(worker_queue...) returns with some more work to do. The likelyhood of warm caches is much higher if the same thread (or the same set of a small number of threads) always does these small bits of work, compared to the case that all (e.g. 30) idle threads each doing a small piece of work in a round robin fashion. Of course, one needs to do real benchmarks when trying to optimize this. > Also, if a worker thread is idle on waiting on > ap_queue_pop_something(worker_queue...), and the thread was > unscheduled by the OS process scheduler, when you wake it up > again, the OS process scheduler won't necessarily schedule it on > the same cpu core it was on before. So, it won't necessarily have > a warm cache after you wake it up again. You're only guaranteed a > warm cache if the thread remains scheduled by the OS. The current > queue/condvar implementation favors sending new requests to > threads that remain scheduled by the OS.
Re: event mpm and mod_status
I see that event mpm uses a worker queue that uses a condition variable, and it does a condition variable signal when something is pushed onto it. If all of the cpu cores are doing useful work, the signal is not going to force a context switch out of a thread doing useful work, the thread will continue working until it's used up it's timeslice, as long as it doesn't end early because it becomes blocked on i/o or something like that. So, in that case, the first thread to finish serving it's request and gets to ap_queue_pop_something(worker_queue...) will get the item on the queue, and it will have hot caches. On the other hand, if only some of the cpu cores are active, but the threads that are active are busy in the middle of a request, the condition variable signal will wake up one of the idle threads, the thread will be scheduled onto an idle core, and that thread can start doing useful work. That thread may have colder caches, but the other cores were busy doing useful work any way, so it's worth activating this thread on an idle core. Also, if a worker thread is idle on waiting on ap_queue_pop_something(worker_queue...), and the thread was unscheduled by the OS process scheduler, when you wake it up again, the OS process scheduler won't necessarily schedule it on the same cpu core it was on before. So, it won't necessarily have a warm cache after you wake it up again. You're only guaranteed a warm cache if the thread remains scheduled by the OS. The current queue/condvar implementation favors sending new requests to threads that remain scheduled by the OS. On Sat, Jan 5, 2013 at 9:37 AM, Jim Jagielski wrote: > +1... a lot of little improvements can result in a BIG > improvement. > > On Jan 5, 2013, at 8:34 AM, Graham Leggett wrote: > > > On 05 Jan 2013, at 2:05 AM, Stefan Fritsch wrote: > > > >> For 1., a better thread selection would definitely be a win. For 2. > >> and 3., it is less obvious. > > > > +1. > > > > Just because in some cases a cache might not help, doesn't mean we > shouldn't take advantage of the cache when it will help. > > > > Regards, > > Graham > > -- > > > >
Re: event mpm and mod_status
+1... a lot of little improvements can result in a BIG improvement. On Jan 5, 2013, at 8:34 AM, Graham Leggett wrote: > On 05 Jan 2013, at 2:05 AM, Stefan Fritsch wrote: > >> For 1., a better thread selection would definitely be a win. For 2. >> and 3., it is less obvious. > > +1. > > Just because in some cases a cache might not help, doesn't mean we shouldn't > take advantage of the cache when it will help. > > Regards, > Graham > -- >
Re: event mpm and mod_status
On 05 Jan 2013, at 2:05 AM, Stefan Fritsch wrote: > For 1., a better thread selection would definitely be a win. For 2. > and 3., it is less obvious. +1. Just because in some cases a cache might not help, doesn't mean we shouldn't take advantage of the cache when it will help. Regards, Graham -- smime.p7s Description: S/MIME cryptographic signature
Re: event mpm and mod_status
On Friday 04 January 2013, Daniel Lescohier wrote: > I just saw this from last month from Stefan Fritsch and Niklas > Edmundsson: > > The fact that the client ip shows up on all threads points to some > > >> potential optimization: Recently active threads should be > >> preferred, because their memory is more likely to be in the cpu > >> caches. Right now, the thread that has been idle the longest > >> time will do the work. > >> > >> Ah, virtually guaranteeing that the thread with the coldest > >> cache gets to > > > > do the work... > > I definitely agree on the potential for improvement here, you would > most > > > likely want to select either the thread that processed this > > request last time, or the most recently active idle thread. > > These two conditions kinda collides though, so the challenge is > > probably to come up with some rather cheap selection algorithm > > that is good enough. > > Which CPU memory caches are you referring to? > >1. For stack, on a new request, you're writing a new call stack, > the prior request's stack was unwound. Even if the previous request's stack memory is freed, the new request will use the same memory which will likely be in the cpu cache. And since the stack is written to in portions smaller than a cache line, the cpu would have to read the stack from memory otherwise. >2. For heap, you're creating a new request pool, so you will be > popping an apr memnode from the head of the allocator's free list. > It may even be the same memnode that was pushed onto the free > list when the prior request's apr pool was freed. Mpm event uses per-connection allocators. The most recently freed allocator will be used first. If this is done by the most recently active thread, it may be more likely that the allocator memory is in the cache of the correct cpu. But this may be more complex in practice. Besides, it is quite possible that the most recently used memnode is not at the top of the allocator. >3. On a new request, the next code/instructions it will execute > will be the functions that reads the http request and populates > the request_rec. That's different code than happened at the end of > the prior request (serving the request, logging the request, > etc.). If a request is handled by a previously inactive thread, it may be more likely that that thread is scheduled on a cpu core that did not previously run httpd at all. This would then cause a cold instruction cache and require a context switch. However, this depends very much on the OS and the details of the workload. > So, I'm not convinced a thread selection algorithm is needed. For 1., a better thread selection would definitely be a win. For 2. and 3., it is less obvious.
Re: event mpm and mod_status
I just saw this from last month from Stefan Fritsch and Niklas Edmundsson: The fact that the client ip shows up on all threads points to some >> potential optimization: Recently active threads should be preferred, >> because their memory is more likely to be in the cpu caches. Right >> now, the thread that has been idle the longest time will do the work. >> >> Ah, virtually guaranteeing that the thread with the coldest cache gets to > do the work... I definitely agree on the potential for improvement here, you would most > likely want to select either the thread that processed this request last > time, or the most recently active idle thread. These two conditions kinda > collides though, so the challenge is probably to come up with some rather > cheap selection algorithm that is good enough. > Which CPU memory caches are you referring to? 1. For stack, on a new request, you're writing a new call stack, the prior request's stack was unwound. 2. For heap, you're creating a new request pool, so you will be popping an apr memnode from the head of the allocator's free list. It may even be the same memnode that was pushed onto the free list when the prior request's apr pool was freed. 3. On a new request, the next code/instructions it will execute will be the functions that reads the http request and populates the request_rec. That's different code than happened at the end of the prior request (serving the request, logging the request, etc.). So, I'm not convinced a thread selection algorithm is needed.
Re: event mpm and mod_status
On Mon, 17 Dec 2012, Stefan Fritsch wrote:
On Sunday 16 December 2012, Niklas Edmundsson wrote:
I'm playing around with the event mpm in httpd 2.4.3, and I've come
to the conclusion that the stats reported by mod_status is rather
strange. I'm not sure if it's a bug or simply not implemented...
My test case is just a simple file transfer of a DVD image, no
mod_cache etc. involved.
I then, naively perhaps, expect that I should se this file transfer
on /server-status, but I see little to no trace of it.
I would at least expect it to consistently show up as writing
somewhere, but "writing" under "Async connections" mostly shows 0.
It should alternate between "Async connections/writing" and
"threads/busy". If not, then that's a bug.
Ah, it should alternate. I'll check this for this when I get round to
doing some more testing.
I get the underlying reason for this behaviour, but from the viewpoint
of a generic server admin without grasp of the underlying code it gets
slightly confusing when stuff bounces around like that.
I'll double check if I have missed some docco, and cobble up a
suggestion for improvement if I find that it's unclear/missing.
Another interesting thing is that the client ip shows up all over
the place in the server/thread list, but without any info on what
the client is doing. I'm guessing that this is the async
processing going on, but the status shown on /server-status is
just plain confusing.
Am I missing something?
Maybe a scoreboard update call is missing at some point. I will have
to look at it.
The fact that the client ip shows up on all threads points to some
potential optimization: Recently active threads should be preferred,
because their memory is more likely to be in the cpu caches. Right
now, the thread that has been idle the longest time will do the work.
Ah, virtually guaranteeing that the thread with the coldest cache gets
to do the work...
I definitely agree on the potential for improvement here, you would
most likely want to select either the thread that processed this
request last time, or the most recently active idle thread. These two
conditions kinda collides though, so the challenge is probably to come
up with some rather cheap selection algorithm that is good enough.
Sure, I can look in the server logs to find out what files have
been sent, but that's not quite the same as the "what is happening
right now" I'm used to /server-status presenting to me.
Is it possible to patch mod_status to show something useful for the
event mpm too, or is the information needed not available? For
example, I would expect to find a list of all the current requests
being processed somehow. As it currently stands it seems that all
replies currently being sent in async mode is a "black hole" on
/server-status.
The problem is that there is no fixed limit for the number of
connections being processed, which makes it hard to put that
information into a shared memory table. And without that, there is no
way for the process handling the server-info request to get the
information from the other processes. But I agree that this should be
fixed somehow.
Some kind of mechanism to extract the info when needed might be
preferred, as it would mean zero impact except when you're accessing
/server-status, but I don't know enough of the innards of httpd/event
to say if it's even possible.
An alternative could be to use the socache stuff to track this for
those that can take the performance impact (in our application,
large-file serving, it would be negligable impact for example).
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | ni...@acc.umu.se
---
Something Evil has happened unexpectedly.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: event mpm and mod_status
On Sunday 16 December 2012, Niklas Edmundsson wrote: > I'm playing around with the event mpm in httpd 2.4.3, and I've come > to the conclusion that the stats reported by mod_status is rather > strange. I'm not sure if it's a bug or simply not implemented... > > My test case is just a simple file transfer of a DVD image, no > mod_cache etc. involved. > > I then, naively perhaps, expect that I should se this file transfer > on /server-status, but I see little to no trace of it. > > I would at least expect it to consistently show up as writing > somewhere, but "writing" under "Async connections" mostly shows 0. It should alternate between "Async connections/writing" and "threads/busy". If not, then that's a bug. > Another interesting thing is that the client ip shows up all over > the place in the server/thread list, but without any info on what > the client is doing. I'm guessing that this is the async > processing going on, but the status shown on /server-status is > just plain confusing. > > Am I missing something? Maybe a scoreboard update call is missing at some point. I will have to look at it. The fact that the client ip shows up on all threads points to some potential optimization: Recently active threads should be preferred, because their memory is more likely to be in the cpu caches. Right now, the thread that has been idle the longest time will do the work. > Sure, I can look in the server logs to find out what files have > been sent, but that's not quite the same as the "what is happening > right now" I'm used to /server-status presenting to me. > > Is it possible to patch mod_status to show something useful for the > event mpm too, or is the information needed not available? For > example, I would expect to find a list of all the current requests > being processed somehow. As it currently stands it seems that all > replies currently being sent in async mode is a "black hole" on > /server-status. The problem is that there is no fixed limit for the number of connections being processed, which makes it hard to put that information into a shared memory table. And without that, there is no way for the process handling the server-info request to get the information from the other processes. But I agree that this should be fixed somehow.
event mpm and mod_status
Hi all.
I'm playing around with the event mpm in httpd 2.4.3, and I've come to
the conclusion that the stats reported by mod_status is rather
strange. I'm not sure if it's a bug or simply not implemented...
My test case is just a simple file transfer of a DVD image, no
mod_cache etc. involved.
I then, naively perhaps, expect that I should se this file transfer on
/server-status, but I see little to no trace of it.
I would at least expect it to consistently show up as writing
somewhere, but "writing" under "Async connections" mostly shows 0.
Another interesting thing is that the client ip shows up all over the
place in the server/thread list, but without any info on what the
client is doing. I'm guessing that this is the async processing going
on, but the status shown on /server-status is just plain confusing.
Am I missing something?
Sure, I can look in the server logs to find out what files have been
sent, but that's not quite the same as the "what is happening right
now" I'm used to /server-status presenting to me.
Is it possible to patch mod_status to show something useful for the
event mpm too, or is the information needed not available? For
example, I would expect to find a list of all the current requests
being processed somehow. As it currently stands it seems that all
replies currently being sent in async mode is a "black hole" on
/server-status.
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | ni...@acc.umu.se
---
OH NO, my wife burned the rice crispies--AGAIN!!
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
