Graham, thanks for the explanations and the hint to mod_proxy. I am now able to use pools/subpools and bucket_alloctors in a better way which improves overall performance.
Just for the record: the important thing to learn for me was that there is another layer underneath apr_pool_t called apr_allocator_t which is not thread-safe by default - but can be made so. With my own allocator per real connection and a mutex protecting it, I can pool and subpool to my heart's content across threads. In my measurements, the disadvantage of a mutex'ed allocator is much overcompensated by proper pool/memory reuse, e.g. it is faster. Cheers, Stefan > Am 17.03.2015 um 23:52 schrieb Graham Leggett <minf...@sharp.fm>: > > On 17 Mar 2015, at 15:59, Stefan Eissing <stefan.eiss...@greenbytes.de> wrote: > >> >> Hi, >> >> I have a question regarding my http2 module implementation, where I seem to >> fail to wrap my head around the limitations of pools and bucket brigades in >> a multi-threaded environment. As I understand it, httpd has - so far - the >> underlying assumption: >> >> 1 connection -> 1 request -> 1 thread >> >> at a single point in time. This used to be quite static, as in mpm_worker, >> and was made more dynamic with mpm_event, which may switch the thread >> around. However even though entities might be exchanged, it is always 1-1-1 >> at a single point in time. >> >> Now, as most of you are aware, with the HTTP/2 processing model, this holds >> no longer true. The relation there is 1-m-m* and, worse, the connection is >> multiplexing requests, so there are many active at the same time and their >> data will be interleaved. The request/thread coupling can stay as it is, but >> the connection relation is different. >> >> I today have a working implementation and it has major drawbacks (among all >> the bugs I need to find): >> >> 1. it mimicks httpd objects that are not really there, e.g. >> pseudo-connections, so that there seems to be a 1-1 with connections and >> requests again (just as mod_spdy does). Therefore it needs to fake HTTP/1.1 >> requests, so that core filter can parse them, etc. >> >> 2. beautiful apr infrastructure like bucket brigades are not used when data >> needs to be passed between threads. >> > >> I have made an attempt today to make bucket brigades work for passing data >> between threads, but things keep crashing on me when load grows. Which is >> all my fault for not understanding these things fully. So, am I on a fool's >> errand? Can I have brigade A in thread 1 and brigade B in thread 2 and pass >> buckets between them without excessive copying? I think I am doing it wrong >> and the cleanups of the allocators run in the wrong thread in the end which >> is not healthy. > > One place where you can get a concrete example of data in one pool lifetime > being transferred across to another is in mod_proxy - the backend connection > comes out of a pool, and may live longer or shorter than the request going > out the front. > > Getting this right is tricky, you need to be particular about the pool > lifetimes. > > Pools create a hierarchy, that is the key. If you have requests subordinate > to a connection, ie when the connection dies the requests die, let the > request be a subpool of the connection. > > You may have many independent requests linked to the same connection, just > make sure the requests don't leak into each other. > > Regards, > Graham > -- <green/>bytes GmbH Hafenweg 16, 48155 Münster, Germany Phone: +49 251 2807760. Amtsgericht Münster: HRB5782
