async mpms + http2
I just committed a change in mod_http2 that will return idle connections back to the mpm, expecting to be invoked again when new data arrives. I learned quite something about event (have not looked much at motorz yet) and am thinking about improvements. If knowledgeable people could give me a hand here, I'd highly appreciate it. Observation 1: mapping conn_states As I understand it, the connection state model, as defined in conn_state_e, enables asynchronous processing in two states: a. CONN_STATE_CHECK_REQUEST_LINE_READABLE, check read, invoke ap_run_process_connection() on data b. CONN_STATE_WRITE_COMPLETION, check write, if data is still in out filters An invocation of ap_run_process_connection() is expected to return, usually in WRITE_COMPLETION state. Then any data is attempted to be written. When this succeeds, the next state depends on secondary flags such as c->aborted, c->keepalive and c->data_in_input_filters. HTTP/2 processing has one state where it fits into this model: when no more streams are open. This is what is currently implemented in trunk. However, there is another state in HTTP/2 processing, where a BLOCK_READ is performed: flow control on streaming out responses. Basically, the client has a window of n bytes, the server sends n response bytes and then needs to wait for the client to increase the window again. This would be a good time to resume processing back to the mpm, and initially I did so. But that breaks under load. Under load, the event mpm feels free to close connections in certain states - I think CONN_STATE_CHECK_REQUEST_LINE_READABLE. That is fine for a HTTP/1 connection as it always is between requests here. But HTTP/2 would also like to use read events, but unloading a child by closing such connections is not the right thing to do. Is there a way mod_http2 can achieve this or do we need another flag in conn_rec/cs? Observation 2: controlled close HTTP/2 would like to send out a last frame when closing a connection in a controlled/timeout way. Do we need another hook for that? Something like "pre_close_connection"? Observation 3: timeouts a. If admins would like to have another Timeout for HTTP/2 connections, they are currently stuck with configuring one Timeout which goes to server_rec->timeout, right? For idle connections, clients already try to keep it open longer. It would be nice to give admins a choice here for different values based on the selected protocol. How to best do this? b. Handling own timeouts: if I want, during processing the connection, a different socket timeout, I can just set it using apr_socket_timeout_set()? Shall I restore it back to server->timeout before returning? c. What would be the best way to wait for a file description *and* conditional event simultaneously? mod_http2 has now a sort of pool loop with double backup timer for situations where it needs to watch both client and backend processes. Cheers, Stefan
Re: async mpms + http2
On 15 Dec 2015, at 5:44 PM, Stefan Eissing wrote: > However, there is another state in HTTP/2 processing, where a BLOCK_READ is > performed: flow control on streaming out responses. Basically, the client has > a window of n bytes, the server sends n response bytes and then needs to wait > for the client to increase the window again. This would be a good time to > resume processing back to the mpm, and initially I did so. But that breaks > under load. > > Under load, the event mpm feels free to close connections in certain states > - I think CONN_STATE_CHECK_REQUEST_LINE_READABLE. That is fine for a HTTP/1 > connection as it always is between requests here. But HTTP/2 would also like > to use read events, but unloading a child by closing such connections is not > the right thing to do. > > Is there a way mod_http2 can achieve this or do we need another flag in > conn_rec/cs? Does the CONN_STATE_WRITE_COMPLETION state fit this case? The WRITE in the above name is a bit of a misnomer, because when mod_ssl returns WANTS_READ we return to the CONN_STATE_WRITE_COMPLETION state with the sense set to READ. This tells the MPM to block waiting until we’re readable, as opposed to block waiting until we’re writable. > Observation 2: controlled close > HTTP/2 would like to send out a last frame when closing a connection in a > controlled/timeout way. Do we need another hook for that? Something like > "pre_close_connection”? I’m assuming we would like a module to have control over the sending of that frame, if so, it makes sense yes. > Observation 3: timeouts > a. If admins would like to have another Timeout for HTTP/2 connections, they > are currently stuck with configuring one Timeout which goes to > server_rec->timeout, right? For idle connections, clients already try to keep > it open longer. It would be nice to give admins a choice here for different > values based on the selected protocol. How to best do this? Perhaps just as we can override the sense of the connection before returning, we might be able to modify the timeout before returning. > b. Handling own timeouts: if I want, during processing the connection, a > different socket timeout, I can just set it using apr_socket_timeout_set()? > Shall I restore it back to server->timeout before returning? That feels wrong, ideally we want to be formally telling the MPM “give us this timeout” rather than doing something sneaky behind it’s back. > c. What would be the best way to wait for a file description *and* > conditional event simultaneously? mod_http2 has now a sort of pool loop with > double backup timer for situations where it needs to watch both client and > backend processes. I’m not sure how portable it is watching for file descriptors. I have used libev to do this in the past and it was quite involved. However - if we can support this it would be really useful. Regards, Graham —
Re: async mpms + http2
> Am 15.12.2015 um 17:58 schrieb Graham Leggett : > > On 15 Dec 2015, at 5:44 PM, Stefan Eissing > wrote: > >> However, there is another state in HTTP/2 processing, where a BLOCK_READ is >> performed: flow control on streaming out responses. Basically, the client >> has a window of n bytes, the server sends n response bytes and then needs to >> wait for the client to increase the window again. This would be a good time >> to resume processing back to the mpm, and initially I did so. But that >> breaks under load. >> >> Under load, the event mpm feels free to close connections in certain states >> - I think CONN_STATE_CHECK_REQUEST_LINE_READABLE. That is fine for a HTTP/1 >> connection as it always is between requests here. But HTTP/2 would also like >> to use read events, but unloading a child by closing such connections is not >> the right thing to do. >> >> Is there a way mod_http2 can achieve this or do we need another flag in >> conn_rec/cs? > > Does the CONN_STATE_WRITE_COMPLETION state fit this case? As far as I figured out, that will only work when http2 processing moves into an output filter. Then the filter would have to mimic pending data and sense == READ. That would prevent event from closing the connection early and POLL_IN. Next attempt to write the output_filters could then read again. Other attempts to keep the connection although output filters are empty fail under load. I'll give the http2 output filter a try, but it feels a bit like hacking the current design (the design is not wrong, it just could not predict h2 usage). > The WRITE in the above name is a bit of a misnomer, because when mod_ssl > returns WANTS_READ we return to the CONN_STATE_WRITE_COMPLETION state with > the sense set to READ. This tells the MPM to block waiting until we’re > readable, as opposed to block waiting until we’re writable. I think this only happens, as long as data remains in the output filter. >> Observation 2: controlled close >> HTTP/2 would like to send out a last frame when closing a connection in a >> controlled/timeout way. Do we need another hook for that? Something like >> "pre_close_connection”? > > I’m assuming we would like a module to have control over the sending of that > frame, if so, it makes sense yes. Ok. >> Observation 3: timeouts >> a. If admins would like to have another Timeout for HTTP/2 connections, they >> are currently stuck with configuring one Timeout which goes to >> server_rec->timeout, right? For idle connections, clients already try to >> keep it open longer. It would be nice to give admins a choice here for >> different values based on the selected protocol. How to best do this? > > Perhaps just as we can override the sense of the connection before returning, > we might be able to modify the timeout before returning. I'd like that. >> b. Handling own timeouts: if I want, during processing the connection, a >> different socket timeout, I can just set it using apr_socket_timeout_set()? >> Shall I restore it back to server->timeout before returning? > > That feels wrong, ideally we want to be formally telling the MPM “give us > this timeout” rather than doing something sneaky behind it’s back. Yes. Do not like it either. >> c. What would be the best way to wait for a file description *and* >> conditional event simultaneously? mod_http2 has now a sort of pool loop with >> double backup timer for situations where it needs to watch both client and >> backend processes. > > I’m not sure how portable it is watching for file descriptors. I have used > libev to do this in the past and it was quite involved. However - if we can > support this it would be really useful. > > Regards, > Graham > — Cheers, Stefan