Re: Thundering herd and MPMs (for dummies)

[Yann Ylavic]

On Sat, Apr 16, 2016 at 2:17 PM, Yann Ylavic  wrote:
> Hi Luca,
>
> On Sat, Apr 16, 2016 at 12:07 PM, Luca Toscano  wrote:
>> The sockets are non blocking and without any guard before the
>> apr_pollset_poll (between processes I mean) there might be the risk of
>> having two or more listener threads trying to accept the same new
>> connection, ending up in only one proceeding and the rest getting EAGAIN.
>
> On modern systems, the thundering hurd is not an issue anymore (does
> not happen).
> There won't be multiple listeners (threads or processes) woken up at
> the same time for the same incoming connection when
> epoll_wait()/kevent()/... are used (see the corresponding man pages,
> EAGAIN is not a possible error, while it is for eg. poll()).
> So when accept() is called, we can be sure , so a fortiori for 
> epoll()+accept().

[Sorry, unexpected send...]
So when accept() is called, we can be sure that a connection is available.

>
> Since, as you noticed, mpm_event is meant for modern systems, not
> ACCEPT_MUTEX is implemented.

Re: Thundering herd and MPMs (for dummies)

[Yann Ylavic]

Hi Luca,

On Sat, Apr 16, 2016 at 12:07 PM, Luca Toscano  wrote:
> The sockets are non blocking and without any guard before the
> apr_pollset_poll (between processes I mean) there might be the risk of
> having two or more listener threads trying to accept the same new
> connection, ending up in only one proceeding and the rest getting EAGAIN.

On modern systems, the thundering hurd is not an issue anymore (does
not happen).
There won't be multiple listeners (threads or processes) woken up at
the same time for the same incoming connection when
epoll_wait()/kevent()/... are used (see the corresponding man pages,
EAGAIN is not a possible error, while it is for eg. poll()).
So when accept() is called, we can be sure , so a fortiori for epoll()+accept().

Since, as you noticed, mpm_event is meant for modern systems, not
ACCEPT_MUTEX is implemented.

Re: Thundering herd and MPMs (for dummies)

[Luca Toscano]

2016-04-09 12:26 GMT+02:00 Luca Toscano :

> Hi Apache devs,
>
> as part of my documentation duties I would like to add some details about
> how connections are accepted in Event (for
> http://httpd.apache.org/docs/current/misc/perf-tuning.html and
> https://httpd.apache.org/docs/2.4/mod/event.html). I am not super expert
> with the MPMs code so what I am going to say could be terribly wrong,
> please be patient :)
>
> Everything started reading the documentation for SO_REUSEPORT, in which
> event seems the only MPM non getting the same performance improvements as
> worker/prefork. I tried to read the code to get a better idea about the
> why, and I tried to compare the various MPMs.
>
> My understanding is:
>
> 1) only one listening socket configured will leverage modern kernel
> features (when available) avoiding the use of a process mutex before accept
> (essentially serializing the accept calls).
> 2) multiple listening sockets configured needs some sort of control since
> processes/threads needs to know what socket is ready for accept (or other
> events of course). APR offers the apr_pollset_* functions to solve this
> problem, for example using select/[e]poll over multiple listening sockets.
> The thundering herd problem arises when select/[e]poll is used by all the
> processes/threads over the same listening sockets, because each event wakes
> all up at the same time causing extra kernel work (and cpu utilization).
>
> SAFE_ACCEPT() is usually implemented in the various MPMs to solve 2)
> essentially serializing select/[e]poll/accept calls with a mutex (like SysV
> semaphores). Prefork has to do this in all its processes because there is a
> 1:1 correspondence between process and connection served, meanwhile Worker
> is a bit smarter and delegates only one thread per process to the role of
> "listener", assigning the accepted connection/fd to the first worker thread
> available.
>
> I was a bit puzzled not finding any SAFE_ACCEPT in event, but eventually
> (and thanks to the #dev IRC channel) I think I know why: Event uses more
> recent epolls/kqueue/etc.. when available and sets the pollset to
> APR_SO_NONBLOCK; apr_pollset_poll is configured to just wait a little
> timeout to catch events, returning (and not blocking) otherwise. The only
> lock used is thread based (one per process) since the pollset is updated by
> the worker threads periodically (Keep alive, lingering close, etc. forces
> the worker to give back control of the fd to the listener to be free to do
> something else). The thundering herd issue in event is not really a major
> problem since it has been mitigated using a combination of timeouts and non
> blocking I/O in the listeners (plus the listeners are usually few compared
> to the total number of worker threads).
>
> Last but not the least,
> https://httpd.apache.org/docs/current/mod/core.html#mutex is therefore
> not needed when using event.
>
> Does this vaguely resemble reality? If not, is there anybody kind enough
> to give me some direction about what to look/read? Documentation will be
> updated in return, I promise :)
>

Sorry again me, still need to figure out if my thoughts are correct. The
only doubt that I have (if the precedent email is somehow correct is how
multiple listening sockets are managed by the listerner threads in event.
The sockets are non blocking and without any guard before the
apr_pollset_poll (between processes I mean) there might be the risk of
having two or more listener threads trying to accept the same new
connection, ending up in only one proceeding and the rest getting EAGAIN. I
tried to read the following piece of code multiple times but I still don't
have the full picture:

https://github.com/apache/httpd/blob/fcf259f90c120d6ad63b639be5f9f720c300b327/server/mpm/event/event.c#L1948

I am pretty sure that I am missing a lot of things, so any hint about where
to look would be really awesome. I'll try to add as much documentation as
possible in return :)

Thanks for the patience!

Luca