2016-04-09 12:26 GMT+02:00 Luca Toscano <toscano.l...@gmail.com>: > Hi Apache devs, > > as part of my documentation duties I would like to add some details about > how connections are accepted in Event (for > http://httpd.apache.org/docs/current/misc/perf-tuning.html and > https://httpd.apache.org/docs/2.4/mod/event.html). I am not super expert > with the MPMs code so what I am going to say could be terribly wrong, > please be patient :) > > Everything started reading the documentation for SO_REUSEPORT, in which > event seems the only MPM non getting the same performance improvements as > worker/prefork. I tried to read the code to get a better idea about the > why, and I tried to compare the various MPMs. > > My understanding is: > > 1) only one listening socket configured will leverage modern kernel > features (when available) avoiding the use of a process mutex before accept > (essentially serializing the accept calls). > 2) multiple listening sockets configured needs some sort of control since > processes/threads needs to know what socket is ready for accept (or other > events of course). APR offers the apr_pollset_* functions to solve this > problem, for example using select/[e]poll over multiple listening sockets. > The thundering herd problem arises when select/[e]poll is used by all the > processes/threads over the same listening sockets, because each event wakes > all up at the same time causing extra kernel work (and cpu utilization). > > SAFE_ACCEPT() is usually implemented in the various MPMs to solve 2) > essentially serializing select/[e]poll/accept calls with a mutex (like SysV > semaphores). Prefork has to do this in all its processes because there is a > 1:1 correspondence between process and connection served, meanwhile Worker > is a bit smarter and delegates only one thread per process to the role of > "listener", assigning the accepted connection/fd to the first worker thread > available. > > I was a bit puzzled not finding any SAFE_ACCEPT in event, but eventually > (and thanks to the #dev IRC channel) I think I know why: Event uses more > recent epolls/kqueue/etc.. when available and sets the pollset to > APR_SO_NONBLOCK; apr_pollset_poll is configured to just wait a little > timeout to catch events, returning (and not blocking) otherwise. The only > lock used is thread based (one per process) since the pollset is updated by > the worker threads periodically (Keep alive, lingering close, etc. forces > the worker to give back control of the fd to the listener to be free to do > something else). The thundering herd issue in event is not really a major > problem since it has been mitigated using a combination of timeouts and non > blocking I/O in the listeners (plus the listeners are usually few compared > to the total number of worker threads). > > Last but not the least, > https://httpd.apache.org/docs/current/mod/core.html#mutex is therefore > not needed when using event. > > Does this vaguely resemble reality? If not, is there anybody kind enough > to give me some direction about what to look/read? Documentation will be > updated in return, I promise :) >
Sorry again me, still need to figure out if my thoughts are correct. The only doubt that I have (if the precedent email is somehow correct is how multiple listening sockets are managed by the listerner threads in event. The sockets are non blocking and without any guard before the apr_pollset_poll (between processes I mean) there might be the risk of having two or more listener threads trying to accept the same new connection, ending up in only one proceeding and the rest getting EAGAIN. I tried to read the following piece of code multiple times but I still don't have the full picture: https://github.com/apache/httpd/blob/fcf259f90c120d6ad63b639be5f9f720c300b327/server/mpm/event/event.c#L1948 I am pretty sure that I am missing a lot of things, so any hint about where to look would be really awesome. I'll try to add as much documentation as possible in return :) Thanks for the patience! Luca
