Re: Questions about mod_event's documentation

[Luca Toscano]

Hi Rainer,

thank you 100 times for this email, it was really helpful! Comments inline:

2016-02-02 17:12 GMT+01:00 Rainer Jung :
>
>
> The number of worker threads per process is constant during the lifetime
> from the process creation to its end. It is equals to ThreadsPerChild and
> the sole purpose of this configuration item.
>

This information is not completely trivial to establish simply reading the
documentation, even if all the information are there. Reading mod_worker's
doc can lead to think about this option, but there is always the space for
some doubts..


>
> AsyncRequestWorkerFactor  comes into the image, because in contrast to the
> traditional MPMs prefork, worker and winnt, event does not have a 1:1
> relation between worker threads and client connections. Event is designed
> to scale better than those in terms of connections. It should handle a lot
> more connections with the same number of threads. How can this work? It
> works by freeing the worker threads from the connection handling in certain
> situations where the thread would actually simply wait until it can write
> back the next part of the response or until the next request on the same
> connection arrives. But this design results in some over commitment. What
> happens if by bad luck over time we accepted lots of connections, which
> were mostly idle and then suddenly many of them start activity which needs
> a worker thread for each of them. Then it might happen, that we do not have
> enough worker threads in the process that accepted the connections. We
> don't have a way to move such connections to another child process. Once
> the connection is accepted it stays with the same process.
>

The only thing that I would like to expand a bit would be: "in certain
situations where the thread would actually simply wait until it can write
back the next part of the response or until the next request on the same
connection arrives.", but as Eric told me on #httpd-dev it is a bit
complicate to summarise in few sentences (an easy one is keep alive). It
might not be needed in the documentation, but it would be good for the
people interested in an in depth look without reading the code.

The rest of the explanation is simply great, very easy and straightforward.
I would really like to add some of it to the how it works section because I
think it would help a lot of people.


> ...
>
> HTH a bit w.r.t. AsyncRequestWorkerFactor (and also hoping that I didn't
> make mistakes in explaining).
>
>
Same thing for the second part of the email, I believe that the directive
AsyncRequestWorkerFactor should get more information, maybe with a little
section explaining some directions to take for tuning it.

I want to re-state my point: httpd's documentation contains often all the
information needed but sometimes it misses the necessary background for the
people not heavily involved in the httpd community to get the whole picture
straight away.

I'll work on a small patch for the mod_event's documentation, and I'll send
it to the mailing list to get some feedback to get your opinion.

Thanks again!

Luca

Re: Questions about mod_event's documentation

[Rainer Jung]

Hi Luca,

some fragmentary answer:

Am 01.02.2016 um 10:17 schrieb Luca Toscano:

...


- AsyncRequestWorkerFactor is used to regulate the amount of requests
that a single process/threads block can handle, calculating the value
periodically using the idle threads/workers available. In case of
workers maxed out, the keep alive sockets/connections are closed to free
some space.


...


If my understanding is correct (I doubt it but let's assume this) then I
have the following questions:

- Would it be worth to add more info in the "how it works" section? A
first read may bring the user to think that the listening thread is the
one doing the actual work, rather than the workers, being a bit puzzled
when reading the AsyncRequestWorkerFactor section.


...


- The summary talks about "supporting threads" and given the fact that
AsyncRequestWorkerFactor is added to ThreadsPerChild, it raises the
question about how many of them are created at startup. Conversely, is
it a way to say: the number of threads for each process are
ThreadsPerChild but since they now perform also small bursts of work
(like keep alive house keeping and flushing data to clients) the total
amount of connections allowed should be more to make room for all these
connection/socket states?


The number of worker threads per process is constant during the lifetime 
from the process creation to its end. It is equals to ThreadsPerChild 
and the sole purpose of this configuration item.


AsyncRequestWorkerFactor  comes into the image, because in contrast to 
the traditional MPMs prefork, worker and winnt, event does not have a 
1:1 relation between worker threads and client connections. Event is 
designed to scale better than those in terms of connections. It should 
handle a lot more connections with the same number of threads. How can 
this work? It works by freeing the worker threads from the connection 
handling in certain situations where the thread would actually simply 
wait until it can write back the next part of the response or until the 
next request on the same connection arrives. But this design results in 
some over commitment. What happens if by bad luck over time we accepted 
lots of connections, which were mostly idle and then suddenly many of 
them start activity which needs a worker thread for each of them. Then 
it might happen, that we do not have enough worker threads in the 
process that accepted the connections. We don't have a way to move such 
connections to another child process. Once the connection is accepted it 
stays with the same process.


To reduce the likeliness of such shortage in free worker threads, the 
number of connections we accept per process is limited. The limit is not 
a fixed number or some fixed multiple of ThreadsPerChild, but instead it 
is calculated based on the number of idle worker threads. If that number 
gets small during run time, the number of additional connections we 
accept decreases until at some point that process won't accept any new 
connections. This situation can be monitored via mod_status.


Let's have a look at the formula: The docs say:

ThreadsPerChild + (AsyncRequestWorkerFactor * number of idle workers)

is the number of new connections a process will still accept (not 
counting connections in closing state). Ignoring for a moment the 
"closing state" stuff, we can write:


max_connections = ThreadsPerChild + (AsyncRequestWorkerFactor * 
idle_workers)


Let's replace ThreadsPerChild by (idle_workers + busy_workers):

max_connections = (idle_workers + busy_workers) + 
(AsyncRequestWorkerFactor * idle_workers)

   = busy_workers + (AsyncRequestWorkerFactor + 1) * idle_workers

Now splitting connections in busy and idle ones and defining a busy 
connection as one needing a worker thread so that the number of busy 
connections equals the number of busy workers, we get:


max_idle_connections + busy_workers = busy_workers + 
(AsyncRequestWorkerFactor + 1) * idle_workers


and thus

max_idle_connections = (AsyncRequestWorkerFactor + 1) * idle_workers

So although we only have idle_workers left, we accept up to 
(AsyncRequestWorkerFactor + 1) * idle_workers as the number of idle 
connections. Since AsyncRequestWorkerFactor by default has the value 2, 
it means by default we accept 3 times as many connections, as we have 
idle workers. That's a handy formulation for our type of over commitment.


Of course the formula is not always true. If many connections turn over 
into busy ones, than we might suddenly have more (old) idle connections 
than allowed by this formula. E.g. if for a moment all workers are idle, 
we would allow 3*ThreadsPerChild (idle) connections. Then assume 
ThreadsPerChild of them start to be busy, then we would allow no idle 
connections because no idle workers are left, but we would still have 
2*ThreadsPerChild old idle connections. The formula only tells us how 
many we are willing to handle unless we are already over the limit 
because to many turned into 

Re: Questions about mod_event's documentation

[Rainer Jung]

Am 02.02.2016 um 18:01 schrieb Luca Toscano:

Hi Rainer,

thank you 100 times for this email, it was really helpful! Comments inline:

2016-02-02 17:12 GMT+01:00 Rainer Jung >:


The number of worker threads per process is constant during the
lifetime from the process creation to its end. It is equals to
ThreadsPerChild and the sole purpose of this configuration item.



This information is not completely trivial to establish simply reading
the documentation, even if all the information are there. Reading
mod_worker's doc can lead to think about this option, but there is
always the space for some doubts..


You are right and it should be made clear in the docs, that all those 
MPMs work on a fixed number of threads per process basis. If they need 
to grow, they can only grow by producing new processes (with each again 
having the same fixed number of threads).



AsyncRequestWorkerFactor  comes into the image, because in contrast
to the traditional MPMs prefork, worker and winnt, event does not
have a 1:1 relation between worker threads and client connections.
Event is designed to scale better than those in terms of
connections. It should handle a lot more connections with the same
number of threads. How can this work? It works by freeing the worker
threads from the connection handling in certain situations where the
thread would actually simply wait until it can write back the next
part of the response or until the next request on the same
connection arrives. But this design results in some over commitment.
What happens if by bad luck over time we accepted lots of
connections, which were mostly idle and then suddenly many of them
start activity which needs a worker thread for each of them. Then it
might happen, that we do not have enough worker threads in the
process that accepted the connections. We don't have a way to move
such connections to another child process. Once the connection is
accepted it stays with the same process.


The only thing that I would like to expand a bit would be: "in certain
situations where the thread would actually simply wait until it can
write back the next part of the response or until the next request on
the same connection arrives.", but as Eric told me on #httpd-dev it is a
bit complicate to summarise in few sentences (an easy one is keep
alive). It might not be needed in the documentation, but it would be
good for the people interested in an in depth look without reading the code.


Maybe the mod_status output can be a guidline about interesting states. 
Have a look at the table part visible under


http://www.apache.org/server-status

The last three columns are all under "Async connections". Thos are 
connection states which would need a fixed worker thread in the old 
style MPMs (not event), but need no worker thread for event MPM:


- writing
  Writing back the answer to the client, but the TCP write buffer is 
full, because the connection to the client is too slow so packets pile 
up. When the next write would block, we instead free the worker thread 
and wait for that connection to get writable again. Not 100% sure, but I 
think this only happens when we serve static content. Not in the case of 
a proxy.


- keep-alive
  The response is finished and we wait for the next request to arrive, 
more precisely for the first packet of the next request.


- closing
  Sometimes we need to do a lingering close. That means we want to send 
back an error or similar, but the error condition happened, before we 
have read the whole request from the client. If we would simply write 
back the response and then close the connection, the client - still 
trying to send the rest of the request -would get a connection reset and 
could not read our answer. So in such cases, we try to read the rest of 
the request although we are not interested any more in it, simply to 
allow the client to consume our response after completing sending the 
request. There's a time limit for this lingering close but it can take 
relatively long time. The waits for the next packets from the clients 
here again happen without a dedicated worker thread.


These three cases are connections, that are not really idle, but need to 
wait for something to happen. During the waits we free the worker thread 
(decouple from the connection and allow it to work on another one).


Looking further at the status output, roughly the number "Connections 
total" would be equals to the sum of the three "Async connections" count 
plus "Threads busy". In theory this is not rue, because the numbers are 
retrieved while the output is generated, so some of them change during 
counting others (the numbers are not retrieved in an atomic way).


If you see in the "accepting" column a "no", it either means, that our 
over commitment limit is reached, or - more often - that this process is 

Re: Questions about mod_event's documentation

[Stefan Eissing]

Luca,

thanks for the long mail. I am looking forward to read the answers to your 
questions, as I have *assumptions* about them, but am myself not certain.

> Am 01.02.2016 um 10:17 schrieb Luca Toscano :
> 
> Hi Apache Devs!
> 
> I am trying to understand if https://httpd.apache.org/docs/2.4/mod/event.html 
> could use some documentation improvements or if I am the only one not getting 
> the whole picture correctly. 
> 
> I'd like to write a summary of my understanding of the module to get some 
> feedback:
> 
> - mod_event reuses mod_worker's architecture using forking processes that in 
> turn create several threads (controlled using the ThreadPerChild directive). 
> A special thread for each process is called the listener (like in worker) and 
> it keeps track of the connections/sockets "assigned" to its parent.
> 
> - mod_event's listener thread is smarter than its brother in mod_worker since 
> it keeps a list of sockets in: keep alive, flushing data to client only 
> (after the output chain has finished to process the response) and complete 
> new request/response to handle. 

My read: Not only in keepalive, but also in timeout, depending on the 
connection state.

> - when a socket changes its state, the listener checks free workers among its 
> thread pool and assign either some "small" work like handling keep alives or 
> flushing data, or a whole new request to handle. 

"changes its state" -> raises an event. If the event needs processing by a 
worker...

> - mod_ssl and mod_deflate are examples of filters that needs to act on the 
> whole response so a worker gets stuck flushing data to slow clients rather 
> than giving up the socket earlier to the listener and doing a different work.

Hmm, not sure I understand your point. Every part in the process of generating 
the bytes sent out on the socket is involved here. The crucial difference 
between the worker and the event mpm is:
- worker keeps the response state on the stack
- event keeps the response state in the heap
which means that calls to response processing 
- on worker, need to return when all is done
- on event, may return whenever a READ/WRITE event or TIMEOUT/KEEPALIVE is 
needed. 

In that way, writing a response with event is "stutter stepping" it. Write 
until EWOULDBLOCK, return, queued, event, write until EWOULDBLOCK,...

(my understanding)

> - AsyncRequestWorkerFactor is used to regulate the amount of requests that a 
> single process/threads block can handle, calculating the value periodically 
> using the idle threads/workers available. In case of workers maxed out, the 
> keep alive sockets/connections are closed to free some space.
> 
> 
> If my understanding is correct (I doubt it but let's assume this) then I have 
> the following questions:
> 
> - Would it be worth to add more info in the "how it works" section? A first 
> read may bring the user to think that the listening thread is the one doing 
> the actual work, rather than the workers, being a bit puzzled when reading 
> the AsyncRequestWorkerFactor section.
+1

> - Would it make sense to expand the summary to add more info about the fact 
> that most of worker's directives needs to be used? The "how it works" section 
> dominates and a reader is more keen to read it first and skip the summary in 
> my opinion. 
+1

> - An interesting question has been filed a while ago in the comments: "This 
> documentation does not make it clear whether the event MPM at least allows 
> for keepalives on SSL connections to conserve a thread.  Does it require the 
> use of a thread while transmitting only, or also while the kept-alive SSL 
> connection is idle?" - If my understanding is correct, the answer should be 
> that a slow client can still block a worker for a keep alive response due to 
> mod_ssl requirements, but that idle times are managed by the listener. 
There is some ongoing work done in trunk in regard to this...

> - The summary talks about "supporting threads" and given the fact that 
> AsyncRequestWorkerFactor is added to ThreadsPerChild, it raises the question 
> about how many of them are created at startup. Conversely, is it a way to 
> say: the number of threads for each process are ThreadsPerChild but since 
> they now perform also small bursts of work (like keep alive house keeping and 
> flushing data to clients) the total amount of connections allowed should be 
> more to make room for all these connection/socket states?
> 
> Apologies for the lng email, hope that what I've written makes sense! If 
> not, I'll start reading again! :)
> 
> Luca
> 
> 
> 

Re: Questions about mod_event's documentation

[Luca Toscano]

Hi Stefan,

thanks for the answers! Commenting inline:

2016-02-01 10:54 GMT+01:00 Stefan Eissing :
>
>
> > - mod_ssl and mod_deflate are examples of filters that needs to act on
> the whole response so a worker gets stuck flushing data to slow clients
> rather than giving up the socket earlier to the listener and doing a
> different work.
>
> Hmm, not sure I understand your point. Every part in the process of
> generating the bytes sent out on the socket is involved here. The crucial
> difference between the worker and the event mpm is:
> - worker keeps the response state on the stack
> - event keeps the response state in the heap
> which means that calls to response processing
> - on worker, need to return when all is done
> - on event, may return whenever a READ/WRITE event or TIMEOUT/KEEPALIVE is
> needed.
>
> In that way, writing a response with event is "stutter stepping" it. Write
> until EWOULDBLOCK, return, queued, event, write until EWOULDBLOCK,...
>
> (my understanding)
>

Really sorry for my poor explanation, but I am still in early phases when
you bang your head against httpd's code until you get something out of it :)

More context from the doc:

"A similar restriction is currently present for requests involving an
output filter that needs to read and/or modify the whole response body,
like for example mod_ssl, mod_deflate, or mod_include. If the connection to
the client blocks while the filter is processing the data, and the amount
of data produced by the filter is too big to be buffered in memory, the
thread used for the request is not freed while httpd waits until the
pending data is sent to the client."

After a chat in #httpd-dev I realised that this use case is not the
standard one for processing a https connection with event, but only a
corner case. For some reason this lead me to think that event was handling
HTTPS connections like worker in all the use cases (some sysadmin friends
thought the same too). It might be necessary only to add this note and
everything would be clearer in my opinion.

My goal is to reduce the time to understand mod-event (and to appreciate
all its greatness) for the "standard guy" like me. I now realise that the
docs are containing all the info that I needed but for some reason I wasn't
getting the correct picture out of it (some processing errors in my brain
might also be part of the problem but I can't easily fix it :)

Thanks! Looking forward to have other feedbacks!

Luca