Re: mod_proxy_http2 questions (was: [VOTE] backport mod_proxy_http2 to 2.4.x as experimental)

[Yann Ylavic]

On Fri, Mar 11, 2016 at 11:33 AM, Stefan Eissing
 wrote:
>
>> Am 10.03.2016 um 22:24 schrieb Yann Ylavic :
>>
>> [...]
>>>
>>> Concurrent Handling
>>> ---
>>> I was then thinking: how do I get all the concurrent proxy_http2 requests to
>>> use the same connection? I was originally thinking about some dort of 
>>> de-mux,
>>> a sort of concentrator that sits on a single connection and communicates 
>>> with
>>> all the worker threads that have proxy requests...
>>>
>>> But that would be insane! There would be one worker sitting on the master
>>> connection, several workers for each single proxy request and one more
>>> worker that sits on the backend connection. And all the proxy requests
>>> would do is move data from the master connection thread to the backend
>>> one and vice versa. But what we need is just 2 workers: one for the master
>>> connection and one for the backend connection. They should handle as many
>>> concurrent requests as configured!
>>
>> Hmm, I'm not sure to follow here.
>> I read this as no need for workers because after all we don't need h1
>> requests to forward h2.
>
> That is not what I meant. I'd like to draw some pictures, but ASCII art
> in emails... I give this a try ;-)

Sorry, I misunderstood what you meant in the first part, and somehow
realized that by reading further (just leaved my first comments to be
at ease with nice explanations and ascii art you do so brilliantly :)
That's even clearer now, thanks!

>
>> But proxying is not only about routing, it's also rewriting, caching,
>> filtering, securing..., so who would do that actual work per request
>> if not workers?
>
> If I understand you correctly, all this is still happening. The things
> I describe above are only happening when the handler of mod_proxy_http2
> is invoked to open the backend connection.
>
> I assume caching, rewriting etc. happens before that?

Yes, all h1 stuff should work out of the box ;)

>> That's pretty much what the MPM (event) does, let's make it available
>> for backend connections (there is some code like this alredy in
>> mod_proxy_wstunnel which makes use of SUSPENDED and MPM callbacks,
>> that may be a way).
>
> Yes, that makes sense. Maybe we can go even further and make the
> whole request processing not rely on the stack so much. Then we could
> suspend/resume more often and, as I understand it, that is what
> Graham is working for also.

Yes, I'm looking forward Graham's work too.
Re. suspend/resume, we'll need to take care of thread/context switches
as much as possible (i.e. not waking threads for small chunks),
possibly by setting buckets aside until significant/whole data is
available, plus batching.

>
> (Comments on the rest in another mail)

Going there now...

Re: mod_proxy_http2 questions (was: [VOTE] backport mod_proxy_http2 to 2.4.x as experimental)

[Stefan Eissing]

> Am 10.03.2016 um 22:24 schrieb Yann Ylavic :
> 
> [...]
>> 
>> Concurrent Handling
>> ---
>> I was then thinking: how do I get all the concurrent proxy_http2 requests to
>> use the same connection? I was originally thinking about some dort of de-mux,
>> a sort of concentrator that sits on a single connection and communicates with
>> all the worker threads that have proxy requests...
>> 
>> But that would be insane! There would be one worker sitting on the master
>> connection, several workers for each single proxy request and one more
>> worker that sits on the backend connection. And all the proxy requests
>> would do is move data from the master connection thread to the backend
>> one and vice versa. But what we need is just 2 workers: one for the master
>> connection and one for the backend connection. They should handle as many
>> concurrent requests as configured!
> 
> Hmm, I'm not sure to follow here.
> I read this as no need for workers because after all we don't need h1
> requests to forward h2.

That is not what I meant. I'd like to draw some pictures, but ASCII art
in emails... I give this a try ;-)

Let's say we have a http/2 connection running on a regular mpm worker. Name
that "m1". This spawns off requests to h2_workers on slave connections
s1, s2, etc, which create and process the requests r1.3.
These do process the requests in the usual http/1.1 infrastructure
of httpd. This would look like this:

mpm-worker: m1
 h2-worker: |--- s1[r1]
 h2-worker: |--- s2[r2]
 h2-worker: |--- s3[r3]

This happens for *all* requests incoming on m1 - always.

If s1-3 encounter a typical mod_proxy_http configuration, the requests
would open backend connections b1-3 (let's keep caching etc. out of 
this for now):

mpm-worker: m1
 h2-worker: |--- s1[r1] (proxy_http) ---> b1
 h2-worker: |--- s2[r2] (proxy_http) ---> b2
 h2-worker: |--- s3[r3] (proxy_http) ---> b3

Now, with a h1 backend connection, we need three of them to process
three requests in parallel. Not so with h2. With proxy_http2, we'd like
b1-3 to be only a single connection.

So my original idea was to build something like this:

mpm-worker: m1
 h2-worker: |--- s1[r1] (proxy_http2) > h2-multiplexer <--> b1
 h2-worker: |--- s2[r2] (proxy_http2) --|
 h2-worker: |--- s3[r3] (proxy_http2) --|

But having 3 h2-workers occupied the whole time only to shuffle
request/response bodies between m1 and b2 is a waste. The current
design therefore establishes this:

mpm-worker: m1
 h2-worker: |--- proxy_http2_engine (s1[r1],s2[r2],s3[r3]) <--> b1

When a new request comes in, slave s4 is created and started
on a worker:

mpm-worker: m1
 h2-worker: |--- proxy_http2_engine (s1[r1],s2[r2],s3[r3]) <--> b1
 h2-worker: |--- s4[r4]

When that runs into the handler of mod_proxy_http2 and is about
to allocate a new backend connection, it first checks with m1 if
there is already some h2-worker processing requests for the same
backend. If there is, we make (assuming for the fun that r1 and
r2 have finished in the meantime):

A. same backend
mpm-worker: m1
 h2-worker: |--- proxy_http2_engine (s3[r3],s4[r4]) <--> b1

if b1 is to another backend however, we will establish this:

B. different backend
mpm-worker: m1
 h2-worker: |--- proxy_http2_engine (s3[r3]) <--> b1
 h2-worker: |--- proxy_http2_engine (s4[r4]) <--> b2

> But proxying is not only about routing, it's also rewriting, caching,
> filtering, securing..., so who would do that actual work per request
> if not workers?

If I understand you correctly, all this is still happening. The things
I describe above are only happening when the handler of mod_proxy_http2
is invoked to open the backend connection.

I assume caching, rewriting etc. happens before that? 

> I understand the need for an efficient pure h2 proxy which would only
> route or balance requests, but that's not the only use case, we need
> to handle the other cases too...

No, I did not build that. All request and response data are passed
through the normal request_rec->in/out filters. 

> Sure there could be a single pollset and queue(s) on each side
> (actually listeners, even if that may sound weird on the backend
> side), but I don't see how we would fill in those without (a pool of)
> workers that do the h1 work (if any).

I think a "proxy_http_engine", similar to http2 but with a separate
backend connection for each slave could be built and that could use
a pollset.

> That's pretty much what the MPM (event) does, let's make it available
> for backend connections (there is some code like this alredy in
> mod_proxy_wstunnel which makes use of SUSPENDED and MPM callbacks,
> that may be a way).

Yes, that makes sense. Maybe we can go even further and make the
whole request processing not rely on the stack so much. Then we could
suspend/resume more often and, as I understand it, that is what
Graham is working for also.

(Comments on the rest in another mail)

-Stefan

Re: mod_proxy_http2 questions (was: [VOTE] backport mod_proxy_http2 to 2.4.x as experimental)

[Yann Ylavic]

On Thu, Mar 10, 2016 at 5:38 PM, Stefan Eissing
 wrote:
>
> Iterative, the Common Case
> --
[]
>
> As to the input/output handling for that request_rec, that is basic mod_http2
> stuff. The core filters have been replaced with ones that shuffle the data
> to h2 internal things (mainly through the h2_mplx) and the response headers
> are taken directly from the request_rec->headers_out. The http2 usual way.

OK, got that, h2_filter_stream_output() is the network filter of the
slave (http/1) connections.

>
> Concurrent Handling
> ---
> I was then thinking: how do I get all the concurrent proxy_http2 requests to
> use the same connection? I was originally thinking about some dort of de-mux,
> a sort of concentrator that sits on a single connection and communicates with
> all the worker threads that have proxy requests...
>
> But that would be insane! There would be one worker sitting on the master
> connection, several workers for each single proxy request and one more
> worker that sits on the backend connection. And all the proxy requests
> would do is move data from the master connection thread to the backend
> one and vice versa. But what we need is just 2 workers: one for the master
> connection and one for the backend connection. They should handle as many
> concurrent requests as configured!

Hmm, I'm not sure to follow here.
I read this as no need for workers because after all we don't need h1
requests to forward h2.

But proxying is not only about routing, it's also rewriting, caching,
filtering, securing..., so who would do that actual work per request
if not workers?

I understand the need for an efficient pure h2 proxy which would only
route or balance requests, but that's not the only use case, we need
to handle the other cases too...

Sure there could be a single pollset and queue(s) on each side
(actually listeners, even if that may sound weird on the backend
side), but I don't see how we would fill in those without (a pool of)
workers that do the h1 work (if any).
That's pretty much what the MPM (event) does, let's make it available
for backend connections (there is some code like this alredy in
mod_proxy_wstunnel which makes use of SUSPENDED and MPM callbacks,
that may be a way).

>
> Backend Engines
> ---
> How can we do that? Let's discuss one master http2 connection with
> no proxy request ongoing. The next request which is handled by proxy_http2
> is the "first" - it sees no other ongoing request against the same
> backend. It registers itself at the master http2 session for the backend.
> It performs the request in the usual way, de-registers itself again
> and returns. Same as with HTTP/1, plus some registration dance of what
> I named a "request engine".
>
> If another worker handles a proxy_http2 request, while the "first"
> engine is ongoing, it sees that someone has registered for the backend
> that it also needs. What it then does is "hand over" the request_rec
> to the "first" engine and immediately returns as if done.

Which sends out the EOR since the core thinks the request is done, right?
If so, wouldn't returning SUSPENDED be more appropriate (that would
free the worker thread for some other work)?

>
> So, the "first" proxy_http2 worker collects all request_recs against
> the same backend and continues running until it is out of requests. Then
> it also shuts down.
>
> So, while several requests against the same backend are ongoing, there
> are two main workers handling it, plus, very shortly, some workers that
> create request_recs, find out that they are proxy_http2, hand them over
> and return.
>
> So, how do we hand request_recs over to another thread?
>
> Request Transfer
> 
> First of all, the request_recs inside a HTTP/2 connection are all created
> on top of slave connections and mod_http2 has a tight grip of the lifetime
> of those connections. So, all that has been allocated from that slave
> connection directly will still be there when the mod_proxy handler
> returns.
>
> But the request_rec itself and all filters that have been installed
> use a child pool and the usual runtime gets rid of that pool at the
> end of the request. How does it know that the request has ended? When
> the EOR bucket gets destroyed.
>
> So, mod_http2, when handing over request_recs, first *freezes* it
> by adding a top output filter that sets all buckets it gets aside. Which
> basically is the EOR bucket. It passes nothing on while frozen, so
> output filters further down never get called at this time.

That possibly shouldn't be needed, nothing would go out until the
backend provides the h1 response (or chunks, via h2 of course)
corresponding to a h1 request (both kept in relation with a
context/baton) and then the response bytes could go out through normal
filtering (finally h1_to_h2 on the client side if that's h2 there
too).

>
> When the proxy_http2 worker for that request returns, 

Re: mod_proxy_http2 questions (was: [VOTE] backport mod_proxy_http2 to 2.4.x as experimental)

[Stefan Eissing]

Hi Yann,

thanks for the questions. I'd really love to bounce the idea of my 
implementation
around here, because I hope that I got things right and, since I do not know all
corners and hooks in our little product, to get feedback what I have overlooked.

Iterative, the Common Case
--
First, if mod_proxy_http2 is called in a HTTP/1 connection, all should be pretty
much the same as for mod_proxy_http. I basically started with that code and 
added
the HTTP/2 session handling in. 

The HTTP/2 session is allocated with proxy_conn->connection->pool and registers
there for shutdown. So when mod_proxy_http2 get called serial for one request
after the other, it will find an existing HTTP/2 session on an already used
connection and use that again. That is nothing different than the HTTP/1.1 case
with some additional state piggy backed on the backend connection.

This was the first iteration of the module.

As to the input/output handling for that request_rec, that is basic mod_http2
stuff. The core filters have been replaced with ones that shuffle the data
to h2 internal things (mainly through the h2_mplx) and the response headers
are taken directly from the request_rec->headers_out. The http2 usual way.

Concurrent Handling
---
I was then thinking: how do I get all the concurrent proxy_http2 requests to
use the same connection? I was originally thinking about some dort of de-mux,
a sort of concentrator that sits on a single connection and communicates with
all the worker threads that have proxy requests...

But that would be insane! There would be one worker sitting on the master
connection, several workers for each single proxy request and one more
worker that sits on the backend connection. And all the proxy requests 
would do is move data from the master connection thread to the backend
one and vice versa. But what we need is just 2 workers: one for the master
connection and one for the backend connection. They should handle as many
concurrent requests as configured!

Backend Engines
---
How can we do that? Let's discuss one master http2 connection with
no proxy request ongoing. The next request which is handled by proxy_http2
is the "first" - it sees no other ongoing request against the same 
backend. It registers itself at the master http2 session for the backend.
It performs the request in the usual way, de-registers itself again
and returns. Same as with HTTP/1, plus some registration dance of what
I named a "request engine".

If another worker handles a proxy_http2 request, while the "first"
engine is ongoing, it sees that someone has registered for the backend
that it also needs. What it then does is "hand over" the request_rec
to the "first" engine and immediately returns as if done.

So, the "first" proxy_http2 worker collects all request_recs against
the same backend and continues running until it is out of requests. Then 
it also shuts down.

So, while several requests against the same backend are ongoing, there
are two main workers handling it, plus, very shortly, some workers that
create request_recs, find out that they are proxy_http2, hand them over 
and return.

So, how do we hand request_recs over to another thread?

Request Transfer

First of all, the request_recs inside a HTTP/2 connection are all created
on top of slave connections and mod_http2 has a tight grip of the lifetime
of those connections. So, all that has been allocated from that slave
connection directly will still be there when the mod_proxy handler
returns.

But the request_rec itself and all filters that have been installed
use a child pool and the usual runtime gets rid of that pool at the
end of the request. How does it know that the request has ended? When
the EOR bucket gets destroyed.

So, mod_http2, when handing over request_recs, first *freezes* it
by adding a top output filter that sets all buckets it gets aside. Which
basically is the EOR bucket. It passes nothing on while frozen, so 
output filters further down never get called at this time.

When the proxy_http2 worker for that request returns, mod_http2
notices the frozen request, thaws it and signals the backend connection
worker that the request is ready for processing. The backend worker
picks it up, handles it and *closes the output* which will write
the buckets set aside before. This will write the EOR and the 
request deallocates itself. During this time, all the input/output
filter etc. will remain in place and work.

Conclusion, TODOs
-
As to you question re filters on the backend connection: those are
used unchanged. However they will be used for HTTP/2 data and while
the byte counters will do, any request counter will not. I do not
know what else is done there...

Also I have not had the time to look at the logging of such request.
There might be some more work done there.

I hope this give an explanation of how mod_proxy_http2 works
together with mod_http2. If not, please 

mod_proxy_http2 questions (was: [VOTE] backport mod_proxy_http2 to 2.4.x as experimental)

[Yann Ylavic]

Hi Stefan,

sorry I didn't look closely enough in mod_proxy_http2's code yet to
answer the following questions by myself, so I'm asking here...

On Wed, Mar 9, 2016 at 1:53 PM, Stefan Eissing
 wrote:
>
> When called inside a HTTP/1.1 connection, it will open/reuse
> an existing HTTP/2 backend connection for this one request.
>
> When called inside a HTTP/2 connection, new requests can be
> transferred to an already ongoing backend HTTP/2 connection
> for the same master.

I can follow the request path which traverses the "normal" http/1
hooks/filters (after the http/2 ones in the latter case), but the
response seems to be handled solely in the http2 code.

Am I right to think that any registered request or connection output
filter will get called as usual with an HTTP/1 response?

It may not make much sense for things like mod_deflate (already
integrated in h2) but others like mod_headers/security/third-party
working at the request level would still want to play here...

Also, what about filters registered on the backend connection?
With mod_proxy_http the proxy_create_req hook allows this and for
example I (will) use it in a patch (about to be proposed) to
mod_logio/mod_status for per-vhost counters (rated incoming and
outgoing connections, requests, bytes, TTFB, TTLB, ..., on both the
client and backend sides, implemented with some (new) mod_logio
connection filters).
Since there is no "fake" request_rec used/needed in mod_proxy_http2,
the is no such hook, but I guess there can be one (or even several
more specific to h2, dunno)...
Anyway, that is surely not a primary goal, I'm just thinking out loud here :)

Thanks,
Yann.