RE: No H2 Window updates!
> -Original Message- > From: Bert Huijben [mailto:b...@qqmail.nl] > Sent: vrijdag 27 november 2015 13:56 > To: b...@qqmail.nl > Subject: > > Hi, > > I finally took the time to diagnose that segfault I had, and I think it > points to a serious bug in httpd. > > To summarize this: I dont receive window updates. > > In this specific test we set a very huge amount of small requests (bodies of > 95 and 113 bytes), until we get out of the 65535 (or 65536) bytes of window > space I get from httpd at the connection level. > (Each stream doesnt get near its limit. I can try if I can receive window > updates there but currently I cant reproduce ever receiving a window > update) > > > Originally this caused a segfault in my code, but I fixed that one. But now > Im just stuck waiting to receive a window update from httpd > > > My last testing was against 2.4.x (to get the 2.4.18 goodness) And I think the combination of: === h2_session.c around line 707 === /* We need to handle window updates ourself, otherwise we * get flooded by nghttp2. */ nghttp2_option_set_no_auto_window_update(options, 1); And not a single call to nghttp2_submit_window_update() to find, explains the situation. I haven't tried what happens when I disable this auto_window call... but sending window updates is really required by the H2 specs. And I totally understand that this wasn't high priority... I worked around not sending updates in my implementation until yesterday :-) Bert
Re: No H2 Window updates!
Hmmm... this seems to me enough to warrant a hold on my T&R until we dig into this deeper.
RE: No H2 Window updates!
Well… it is not a regression, so can it be a show stopper? ☺… But I would like to see this fixed. I have no idea how hard it would be to fix this though. It could be as simple as removing that config line (which was probably added somewhere early on if I look at that comment). But windowing correctly for optimal performance isn’t easy. Luckily most of the windowing works in the other direction. Bert From: Jim Jagielski Sent: vrijdag 27 november 2015 17:55 To: dev@httpd.apache.org Subject: Re: No H2 Window updates! Hmmm... this seems to me enough to warrant a hold on my T&R until we dig into this deeper.
Re: No H2 Window updates!
Bert Huijben in gmane.comp.apache.devel (Fri, 27 Nov 2015 20:04:14 +0100): >Well it is not a regression, so can it be a show stopper? ? >But I would like to see this fixed. Curious: are you still testing this on Windows? If so, I guess you compiled your own httpd. I tried to do the same a couple of days ago, but ran into problems with Drupal7: the admin menu sometimes showed and sometimes did not show at all. I could not lay my finger on what went wrong. Because I did not have the problems with Apachelounge's 2.4.18-dev at https://www.apachelounge.com/viewtopic.php?t=6842 I checked out an earlier revision of the alpha branch: | svn co -r 1715218 http://svn.apache.org/repos/asf/httpd/httpd/branches/2.4-http2-alpha That revision compiled into a httpd with no problems. I am waiting now for Stefan Eissing to finish his work on mod_http2. BTW: did you switch to nghttp2 1.5.0 already? -- Jan
Re: No H2 Window updates!
Jan Ehrhardt in gmane.comp.apache.devel (Fri, 27 Nov 2015 22:34:53 +0100): >Bert Huijben in gmane.comp.apache.devel (Fri, 27 Nov 2015 20:04:14 +0100): >>Well it is not a regression, so can it be a show stopper? ? >>But I would like to see this fixed. > >Curious: are you still testing this on Windows? If so, I guess you >compiled your own httpd. I tried to do the same a couple of days ago, but >ran into problems with Drupal7: the admin menu sometimes showed and >sometimes did not show at all. I could not lay my finger on what went >wrong. > >Because I did not have the problems with Apachelounge's 2.4.18-dev at >https://www.apachelounge.com/viewtopic.php?t=6842 I checked out an earlier >revision of the alpha branch: >| svn co -r 1715218 >http://svn.apache.org/repos/asf/httpd/httpd/branches/2.4-http2-alpha > >That revision compiled into a httpd with no problems. I am waiting now for >Stefan Eissing to finish his work on mod_http2. Probably Stefan Eissing already completed his work for 2.4.18. I checked out the current 2.4.x branch with mod_http2 1.8.0 from http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/http2/ I rebuilt the complete branch with VC14 x86 and encountered the same problems with Drupal7. On my dev-server with Win 2008 R2 I now have 3 mod_http2's: #LoadModule http2_module modules/mod_http2.so LoadModule http2_module modules/mod_http2-r1715218.so #LoadModule http2_module modules/mod_http2_al.so The first one is mod_http 1.8.0, the second and the third are 1.5.0 (my build and Apachelounge's). PHP is 5.6.16 loaded with mod_fcgid 2.3.9, nghttp2 is version 1.4.0 (but I also tested with 1.5.0). Only mod_http2 1.8.0 sometimes just does not show the admin menu in Drupal7, the other 2 work fine. The same difference exists for VC9 httpd with nghttp2 1.5.0 compiled statically into mod_http2.so. mod_http2 1.5.0 runs without problems, mod_http2 1.8.0 sometimes just goes not show the Drupal 7 admin menu. @Steffen Land of Apachelounge: could you build mod_http2 1.8.0 before 2.4.18 before it goes into T&R? -- Jan
RE: No H2 Window updates!
> -Original Message-
> From: Jan Ehrhardt [mailto:php...@ehrhardt.nl]
> Sent: vrijdag 27 november 2015 22:35
> To: dev@httpd.apache.org
> Subject: Re: No H2 Window updates!
>
> Bert Huijben in gmane.comp.apache.devel (Fri, 27 Nov 2015 20:04:14 +0100):
> >Well. it is not a regression, so can it be a show stopper? ?.
> >But I would like to see this fixed.
>
> Curious: are you still testing this on Windows? If so, I guess you
> compiled your own httpd. I tried to do the same a couple of days ago, but
> ran into problems with Drupal7: the admin menu sometimes showed and
> sometimes did not show at all. I could not lay my finger on what went
> wrong.
>
> Because I did not have the problems with Apachelounge's 2.4.18-dev at
> https://www.apachelounge.com/viewtopic.php?t=6842 I checked out an
> earlier
> revision of the alpha branch:
> | svn co -r 1715218
> http://svn.apache.org/repos/asf/httpd/httpd/branches/2.4-http2-alpha
>
> That revision compiled into a httpd with no problems. I am waiting now for
> Stefan Eissing to finish his work on mod_http2.
>
> BTW: did you switch to nghttp2 1.5.0 already?
Hi Jan,
No I didn't switch to nghttp2 1.5.0 yet. Thanks for reminding me to check if
there is a new version :)
The code I wrote for Serf code doesn't use nghttp2... After reading the
specs I didn't think building on nghttp2 would win me a lot of time, and
having multiple separate implementations of the same specification/RFC in
the open source world would be a good thing. (Almost every recent H2 project
I see builds on top of nghttp2)
It is entirely possible that you hit the same problem as I did. (I'm
actually very surprised that I didn't hit this problem much earlier on.
There is just one test in the Subversion testsuite that sends more than 64
Kbyte of request bodies over a single connection... I'll fix that)
But back to that problem... There is an easy workaround, which I used on the
other side until two days ago: just making the H2 default window big enough
that you never get near it.
Just configure 'H2WindowSize' to be something like 1 GB and you probably
never have to think about window updates. (The max allowed value is 2 GB -1)
This windowing allows the server to throttle incoming DATA from the other
side, so servers like httpd really want to tune this dynamically. Note that
headers and new requests are not counted to this limit, so on the server
side it is really just request bodies... (Technically data on already closed
streams should be counted too... but that is an implementation detail).
I remember reading that Stefan Eissing is away for some time ('Re: NOTICE:
Intent to T&R 2.4.18' thread), so perhaps I should spend some time looking
at this myself.
Disabling that line that explicitly disables Window updates from the nghttp
library, could be an easy fix... but it might require some compensating
actions, like lowering the number of supported concurrent streams if that
comment is still up to date. Allowing up to 100 concurrent streams per
connection could be a bit high, although this really depends on what these
connections are used for. I don't know how to test against that 'gets
flooded' problem though, as that isn't measurable by itself.
And yes I build my own binaries for Subversion and all its dependencies...
All my scripting that I use for that is in
https://sharpsvn.open.collab.net/svn/sharpsvn/trunk/imports (username
'guest', no password). The default build doesn't build httpd, but if you use
a Subversion dev build (copy dev-default.build to a directory one level
above imports) it builds httpd. My scripts should work for VS2005 upto 2015
and require nant, and some python and perl versions. Everything else is
built from the scripts.
These same scripts drive the Subversion and Serf win32 buildbots that I
maintain on ci.apache.org... and they also deliver the SharpSvn and 'Slik
Subversion Client' binaries.
(I currently explicitly don't deliver httpd itself or anything that depends
on that though)
Thanks [ / Groeten ;-)],
Bert
> --
> Jan
RE: No H2 Window updates!
I saw issues on boxes with the default of number of concurrent
streams.
Lowering the default 100 value of the directive H2MaxSessionStreams
solved it.
On Saturday 28/11/2015 at 10:29, Bert Huijben wrote:
-Original Message-
From: Jan Ehrhardt [mailto:php...@ehrhardt.nl]
Sent: vrijdag 27 november 2015 22:35
To: dev@httpd.apache.org
Subject: Re: No H2 Window updates!
Bert Huijben in gmane.comp.apache.devel (Fri, 27 Nov 2015 20:04:14
+0100):
Well. it is not a regression, so can it be a show stopper? ?.
But I would like to see this fixed.
Curious: are you still testing this on Windows? If so, I guess you
compiled your own httpd. I tried to do the same a couple of days ago,
but
ran into problems with Drupal7: the admin menu sometimes showed and
sometimes did not show at all. I could not lay my finger on what went
wrong.
Because I did not have the problems with Apachelounge's 2.4.18-dev at
https://www.apachelounge.com/viewtopic.php?t=6842 I checked out an
earlier
revision of the alpha branch:
| svn co -r 1715218
http://svn.apache.org/repos/asf/httpd/httpd/branches/2.4-http2-alpha
That revision compiled into a httpd with no problems. I am waiting now
for
Stefan Eissing to finish his work on mod_http2.
BTW: did you switch to nghttp2 1.5.0 already?
Hi Jan,
No I didn't switch to nghttp2 1.5.0 yet. Thanks for reminding me to
check if
there is a new version :)
The code I wrote for Serf code doesn't use nghttp2... After reading
the
specs I didn't think building on nghttp2 would win me a lot of time,
and
having multiple separate implementations of the same specification/RFC
in
the open source world would be a good thing. (Almost every recent H2
project
I see builds on top of nghttp2)
It is entirely possible that you hit the same problem as I did. (I'm
actually very surprised that I didn't hit this problem much earlier
on.
There is just one test in the Subversion testsuite that sends more
than 64
Kbyte of request bodies over a single connection... I'll fix that)
But back to that problem... There is an easy workaround, which I used
on the
other side until two days ago: just making the H2 default window big
enough
that you never get near it.
Just configure 'H2WindowSize' to be something like 1 GB and you
probably
never have to think about window updates. (The max allowed value is 2
GB -1)
This windowing allows the server to throttle incoming DATA from the
other
side, so servers like httpd really want to tune this dynamically. Note
that
headers and new requests are not counted to this limit, so on the
server
side it is really just request bodies... (Technically data on already
closed
streams should be counted too... but that is an implementation
detail).
I remember reading that Stefan Eissing is away for some time ('Re:
NOTICE:
Intent to T&R 2.4.18' thread), so perhaps I should spend some time
looking
at this myself.
Disabling that line that explicitly disables Window updates from the
nghttp
library, could be an easy fix... but it might require some
compensating
actions, like lowering the number of supported concurrent streams if
that
comment is still up to date. Allowing up to 100 concurrent streams per
connection could be a bit high, although this really depends on what
these
connections are used for. I don't know how to test against that 'gets
flooded' problem though, as that isn't measurable by itself.
And yes I build my own binaries for Subversion and all its
dependencies...
All my scripting that I use for that is in
https://sharpsvn.open.collab.net/svn/sharpsvn/trunk/imports (username
'guest', no password). The default build doesn't build httpd, but if
you use
a Subversion dev build (copy dev-default.build to a directory one
level
above imports) it builds httpd. My scripts should work for VS2005 upto
2015
and require nant, and some python and perl versions. Everything else
is
built from the scripts.
These same scripts drive the Subversion and Serf win32 buildbots that
I
maintain on ci.apache.org... and they also deliver the SharpSvn and
'Slik
Subversion Client' binaries.
(I currently explicitly don't deliver httpd itself or anything that
depends
on that though)
Thanks [ / Groeten ;-)],
Bert
--
Jan
Re: No H2 Window updates!
I am not really here, but... the window updates are sent out via update_window(), line 1001, h2_session.c. If you do not see any window updates with a client, it may be that the server app you called does not read its input. I have several test cases with uploads and they work with nghttp and curl. Basics of mod_http2 flow control: - the auto udates of nghttp2 are disabled because nghttp2 would continously update the window for the client, letting the client sent more and more - until we run out of memory. - instead, input reads from workers against the h2_mplx io are recorded and lead to regular window update being sent out. So clients can only send more when the data is actually consumed by someone. What is not well tested, I think, is the timeout/cleanup behaviour on this. So, as long as the connection is alive, individual stream could appear to hang infinitely *if* no one ever reads the data. Maybe you see something like this? //Stefan Am 27.11.2015 um 14:23 schrieb Bert Huijben: -Original Message- From: Bert Huijben [mailto:b...@qqmail.nl] Sent: vrijdag 27 november 2015 13:56 To: b...@qqmail.nl Subject: Hi, I finally took the time to diagnose that segfault I had, and I think it points to a serious bug in httpd. To summarize this: I don’t receive window updates. In this specific test we set a very huge amount of small requests (bodies of 95 and 113 bytes), until we get out of the 65535 (or 65536) bytes of window space I get from httpd at the connection level. (Each stream doesn’t get near its limit. I can try if I can receive window updates there… but currently I can’t reproduce ever receiving a window update) Originally this caused a segfault in my code, but I fixed that one. But now I’m just stuck waiting to receive a window update from httpd… My last testing was against 2.4.x (to get the 2.4.18 goodness) And I think the combination of: === h2_session.c around line 707 === /* We need to handle window updates ourself, otherwise we * get flooded by nghttp2. */ nghttp2_option_set_no_auto_window_update(options, 1); And not a single call to nghttp2_submit_window_update() to find, explains the situation. I haven't tried what happens when I disable this auto_window call... but sending window updates is really required by the H2 specs. And I totally understand that this wasn't high priority... I worked around not sending updates in my implementation until yesterday :-) Bert
RE: No H2 Window updates!
> -Original Message- > From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] > Sent: zaterdag 28 november 2015 13:01 > To: dev@httpd.apache.org > Subject: Re: No H2 Window updates! > > I am not really here, but... > > the window updates are sent out via update_window(), line 1001, > h2_session.c. If you do not see any window updates with a client, it may > be that the server app you called does not read its input. I have > several test cases with uploads and they work with nghttp and curl. I do see window updates against other servers. And I'm pretty sure Subversion reads its input... things are committed correctly and the propfinds return their result. In the testcase at hand I'm sending very small requests (+- 100 bytes max), so the per stream window updates are not really used... They get nowhere near the 64K limit and the first data frame has EOS set. Perhaps this also skips sending window updates on the connection level. I do remember testing larger uploads as a single stream... So perhaps it is a combination that makes them not appear for me. Bert
RE: No H2 Window updates!
> -Original Message-
> From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de]
> Sent: zaterdag 28 november 2015 13:01
> To: dev@httpd.apache.org
> Subject: Re: No H2 Window updates!
>
> I am not really here, but...
>
> the window updates are sent out via update_window(), line 1001,
> h2_session.c. If you do not see any window updates with a client, it may
> be that the server app you called does not read its input. I have
> several test cases with uploads and they work with nghttp and curl.
>
> Basics of mod_http2 flow control:
> - the auto udates of nghttp2 are disabled because nghttp2 would
> continously update the window for the client, letting the client sent
> more and more - until we run out of memory.
> - instead, input reads from workers against the h2_mplx io are recorded
> and lead to regular window update being sent out. So clients can only
> send more when the data is actually consumed by someone.
>From h2_session.c around line 1640.
===
switch (status) {
case APR_SUCCESS: /* successful read, reset our idle
timers */
have_read = 1;
wait_micros = 0;
break;
case APR_EAGAIN: /* non-blocking read, nothing
there */
break;
default:
if (APR_STATUS_IS_ETIMEDOUT(status)
|| APR_STATUS_IS_ECONNABORTED(status)
|| APR_STATUS_IS_ECONNRESET(status)
|| APR_STATUS_IS_EOF(status)
|| APR_STATUS_IS_EBADF(status)) {
/* common status for a client that has left */
ap_log_cerror( APLOG_MARK, APLOG_DEBUG, status,
session->c,
"h2_session(%ld): terminating",
session->id);
/* Stolen from mod_reqtimeout to speed up lingering
when
* a read timeout happened.
*/
apr_table_setn(session->c->notes,
"short-lingering-close", "1");
}
else {
/* uncommon status, log on INFO so that we see this
*/
ap_log_cerror( APLOG_MARK, APLOG_INFO, status,
session->c,
APLOGNO(02950)
"h2_session(%ld): error reading,
terminating",
session->id);
}
h2_session_abort(session, status, 0);
goto end_process;
}
===
I'm not familiar enough with differences in bucket handling between serf and
httpd to really make the call, but as the serf buckets were designed by the
same group:
I'm guessing that there might be successful reads with different status
values than just APR_SUCCESS.
In serf I would expect to see an immediate APR_EOF when there is only a
single frame to be read (or perhaps a few intermediate APR_EAGAINS and then
a EOF), which may both imply successful reading 0 or more bytes, followed by
their status code.
Bert
Re: No H2 Window updates!
Steffen in gmane.comp.apache.devel (Sat, 28 Nov 2015 10:44:44 +0100): >I saw issues on boxes with the default of number of concurrent >streams. > >Lowering the default 100 value of the directive H2MaxSessionStreams >solved it. I know that, because it is in this topic: https://www.apachelounge.com/viewtopic.php?p=32186#32186 What is worrying me is that I do not have problems with mod_http2 1.0.5, but mod_http2 1.0.8 in a default installation causes trouble with Drupal7. It would be great if I could test it with mod_http2.so 1.0.8 from a Apachelounge build. -- Jan
RE: No H2 Window updates!
> -Original Message-
> From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de]
> Sent: zaterdag 28 november 2015 13:01
> To: dev@httpd.apache.org
> Subject: Re: No H2 Window updates!
>
> I am not really here, but...
>
> the window updates are sent out via update_window(), line 1001,
> h2_session.c. If you do not see any window updates with a client, it may
> be that the server app you called does not read its input. I have
> several test cases with uploads and they work with nghttp and curl.
In my case it doesn't...
---
if (h2_stream_set_has_open_input(session->streams)) {
/* Check that any pending window updates are sent. */
status = h2_mplx_in_update_windows(session->mplx,
update_window, session);
if (APR_STATUS_IS_EAGAIN(status)) {
status = APR_SUCCESS;
}
else if (status == APR_SUCCESS) {
/* need to flush window updates onto the connection asap
*/
h2_conn_io_flush(&session->io);
}
}
Looks like that ' h2_stream_set_has_open_input()' always returns false for
me, with those tiny requests of only 1 data frame.
The streams are marked as closed the moment a data frame with EOS is
received... that is: before the frame is processed.
In this testcase all data frames have EOS set, so this value would only be
true for me if some other outstanding request did not receive its data yet.
(Which doesn't happen yet as Subversion still assumes http/1 like
processing)
Bert
RE: No H2 Window updates!
> -Original Message-
> From: Bert Huijben [mailto:b...@qqmail.nl]
> Sent: zaterdag 28 november 2015 14:09
> To: stefan.eiss...@greenbytes.de; dev@httpd.apache.org
> Subject: RE: No H2 Window updates!
>
>
>
> > -Original Message-
> > From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de]
> > Sent: zaterdag 28 november 2015 13:01
> > To: dev@httpd.apache.org
> > Subject: Re: No H2 Window updates!
> >
> > I am not really here, but...
> >
> > the window updates are sent out via update_window(), line 1001,
> > h2_session.c. If you do not see any window updates with a client, it may
> > be that the server app you called does not read its input. I have
> > several test cases with uploads and they work with nghttp and curl.
>
> In my case it doesn't...
>
> ---
> if (h2_stream_set_has_open_input(session->streams)) {
> /* Check that any pending window updates are sent. */
> status = h2_mplx_in_update_windows(session->mplx,
> update_window, session);
> if (APR_STATUS_IS_EAGAIN(status)) {
> status = APR_SUCCESS;
> }
> else if (status == APR_SUCCESS) {
> /* need to flush window updates onto the connection
asap
> */
> h2_conn_io_flush(&session->io);
> }
> }
>
>
>
> Looks like that ' h2_stream_set_has_open_input()' always returns false for
> me, with those tiny requests of only 1 data frame.
>
> The streams are marked as closed the moment a data frame with EOS is
> received... that is: before the frame is processed.
>
> In this testcase all data frames have EOS set, so this value would only be
> true for me if some other outstanding request did not receive its data
yet.
> (Which doesn't happen yet as Subversion still assumes http/1 like
> processing)
If I just replace the
if (h2_stream_set_has_open_input(session->streams))
with if(TRUE) {
then I do receive a few window updates... Of a few bytes each though...
I would have expected a few huge returns (Kbyte+) every few requests that
send data.
So over the course of hundreds of requests my connection level window
shrinks to almost zero... at some points I receive a few updates of a few
hundred bytes... I even see one of a single byte.
Data on closed streams must be accounted towards the connection window, so
conditionalizing the whole processing on 'are there any readable streams' is
a clear bug. Skipping all data in the last frame is another one.
Perhaps the code can skip some of the processing in this case, but (as a
client) I would like to see my outgoing window at the connection updated
sooner.
In case of Subversion's real usage, I want to commit potentially hundreds of
MByte, so a connection level window of more than a few bytes would be very
welcome With HTTP/1 we send out the data as fast as we can and the TCP
windowing handles this from the httpd side... Now the http/2 level windowing
should handle this.
Mod_dav and mod_dav_svn are usually fast readers; just limited by trivial
disk io or xml processing in the common operations, so I don't expect real
problems there.
Bert
RE: No H2 Window updates!
> -Original Message- > From: Bert Huijben [mailto:b...@qqmail.nl] > Sent: zaterdag 28 november 2015 14:32 > To: stefan.eiss...@greenbytes.de; dev@httpd.apache.org > Subject: RE: No H2 Window updates! > In case of Subversion's real usage, I want to commit potentially hundreds of > MByte, so a connection level window of more than a few bytes would be > very > welcome With HTTP/1 we send out the data as fast as we can and the TCP > windowing handles this from the httpd side... Now the http/2 level > windowing > should handle this. > > > Mod_dav and mod_dav_svn are usually fast readers; just limited by trivial > disk io or xml processing in the common operations, so I don't expect real > problems there. For my commits against the svn-master.apache.org repository my latency/ping time is +- 142 ms. With the current simple algorithm and a maximum window of 64 Kbyte/connection, I can send a theoretical maximum of 64 Kbyte/142 ms per connection to that server... which would be about 450 Kbyte/s. (=1/0.142*65536/1024) That is < 1/30th of the bandwidth I have at my disposal (+- 150 Mbit). But currently I don't even get anywhere near that as my window continues to shrink because some data is missed in accounting even after my simple patch. For httpd we have to think carefully which algorithm we want to implement here. Preferably the algorithm should be better than TCP could, as we know the specifics of the http algorithm better than plain TCP could for HTTP/1,1. TCP does send a lot of window updates though... Almost every packet does. Solving the really hard problems, like this one, is the reason I didn't think the nghttp2 library would really be the solution for serf. It is nice for such a library to provide a head start on the protocol level, but there is no way a standard library can really solve this scheduling problem in a generic way. (If it could do it, we had used that solution for TCP... and never had to resort to designing a HTTP/2 in the first place). See https://en.wikipedia.org/wiki/TCP_window_scale_option for some introduction on how TCP was extended to work above the limit that currently applies to H2. Dynamic window sizing/scaling will have to be implemented at some point, at both the stream and the connection level... This will involve timing, measuring, etc. etc. Things nghttp2 can't do for us right now. And if I'm right this might take continuous optimizing for years to come. When I connect to sites like google I immediately receive a large connection level window, to allow me to post huge blobs without a delay. (Haven't tested their stream level windows behavior yet) In serf I do something similar... and then apply a bit of throttling at the stream level. I would guess some clients have already implemented some of this, so we might be able to learn from their implementations... Clients will see much more incoming data than servers of course :). Bert
Re: No H2 Window updates!
Ok, thanks. I think I have an idea of what's happening: - on short request bodies, window updates get omitted and gives a shrinking connection window - the window size of the connection itself should be at max Value right from the start I won't be able to do anything about it until later this week, though. //stefan Am 28.11.2015 um 22:29 schrieb Bert Huijben : >> -Original Message- >> From: Bert Huijben [mailto:b...@qqmail.nl] >> Sent: zaterdag 28 november 2015 14:32 >> To: stefan.eiss...@greenbytes.de; dev@httpd.apache.org >> Subject: RE: No H2 Window updates! > > >> In case of Subversion's real usage, I want to commit potentially hundreds > of >> MByte, so a connection level window of more than a few bytes would be >> very >> welcome With HTTP/1 we send out the data as fast as we can and the TCP >> windowing handles this from the httpd side... Now the http/2 level >> windowing >> should handle this. >> >> >> Mod_dav and mod_dav_svn are usually fast readers; just limited by trivial >> disk io or xml processing in the common operations, so I don't expect real >> problems there. > > For my commits against the svn-master.apache.org repository my latency/ping > time is +- 142 ms. > > With the current simple algorithm and a maximum window of 64 > Kbyte/connection, I can send a theoretical maximum of 64 Kbyte/142 ms per > connection to that server... which would be about 450 Kbyte/s. > (=1/0.142*65536/1024) > > That is < 1/30th of the bandwidth I have at my disposal (+- 150 Mbit). > > But currently I don't even get anywhere near that as my window continues to > shrink because some data is missed in accounting even after my simple patch. > > > > For httpd we have to think carefully which algorithm we want to implement > here. Preferably the algorithm should be better than TCP could, as we know > the specifics of the http algorithm better than plain TCP could for > HTTP/1,1. TCP does send a lot of window updates though... Almost every > packet does. > > > Solving the really hard problems, like this one, is the reason I didn't > think the nghttp2 library would really be the solution for serf. > It is nice for such a library to provide a head start on the protocol level, > but there is no way a standard library can really solve this scheduling > problem in a generic way. (If it could do it, we had used that solution for > TCP... and never had to resort to designing a HTTP/2 in the first place). > > See https://en.wikipedia.org/wiki/TCP_window_scale_option for some > introduction on how TCP was extended to work above the limit that currently > applies to H2. > > > Dynamic window sizing/scaling will have to be implemented at some point, at > both the stream and the connection level... This will involve timing, > measuring, etc. etc. Things nghttp2 can't do for us right now. > And if I'm right this might take continuous optimizing for years to come. > > > > When I connect to sites like google I immediately receive a large connection > level window, to allow me to post huge blobs without a delay. (Haven't > tested their stream level windows behavior yet) > In serf I do something similar... and then apply a bit of throttling at the > stream level. > > > I would guess some clients have already implemented some of this, so we > might be able to learn from their implementations... Clients will see much > more incoming data than servers of course :). > > >Bert > >
RE: No H2 Window updates!
> -Original Message- > From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] > Sent: zondag 29 november 2015 09:04 > To: dev@httpd.apache.org > Subject: Re: No H2 Window updates! > > Ok, thanks. I think I have an idea of what's happening: > - on short request bodies, window updates get omitted and gives a shrinking > connection window Yes, this is the problem I'm seeing. I would like to see this fixed for 2.4.18... and I wouldn't be surprised if this also fixes Jan Erhardts problem. Web frameworks are likely to issue many small (<16K) posts to a server, which all fit in a single frame. In my testcase I needed > 500 requests though. > - the window size of the connection itself should be at max Value right from > the start That is one solution. (Google sends one huge window update on the connection directly at the start, after the settings frame) But this really needs a proper design behind it... Optimal windowing is a hard topic. The window needs to be large enough not to slow down the client... but not too big to hog the server (and by that the connection), etc. etc. The problem with a huge connection window up front is that it may be used for a lot of small requests or one large. That large one can easily be throttled at the stream level, while those small ones can't. > > I won't be able to do anything about it until later this week, though. With an increased default window on the httpd site the Subversion test now completes successfully over H2 on 2.4.17 and 2.4.18. It just reports an XPASS in a specific case where we assume that httpd handles a request as invalid because one header is to long with the http/1.1 rules. If this change is expected on the httpd site, the assumption should be fixed in the Subversion testsuite. Thanks, Bert
Re: No H2 Window updates!
Jan Ehrhardt in gmane.comp.apache.devel (Sat, 28 Nov 2015 14:09:07 +0100): >What is worrying me is that I do not have problems with mod_http2 1.0.5, >but mod_http2 1.0.8 in a default installation causes trouble with Drupal7. >It would be great if I could test it with mod_http2.so 1.0.8 from a >Apachelounge build. Apachelounge now has a new build for 2.4.18: https://www.apachelounge.com/viewtopic.php?t=6842 This build has the same problems with the admin manu in Drupal7 as my own build had. The admin menu just does not show up sometimes. Conclusion: there is a regression between mod_http2 1.0.5 and mod_http2 1.0.8. -- Jan
Re: No H2 Window updates!
Bert Huijben in gmane.comp.apache.devel (Sun, 29 Nov 2015 14:48:46 +0100): >> -Original Message- >> From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] >> Sent: zondag 29 november 2015 09:04 >> To: dev@httpd.apache.org >> Subject: Re: No H2 Window updates! >> >> Ok, thanks. I think I have an idea of what's happening: >> - on short request bodies, window updates get omitted and gives a shrinking >> connection window > >Yes, this is the problem I'm seeing. >I would like to see this fixed for 2.4.18... and I wouldn't be surprised if >this also fixes Jan Erhardts problem. > >Web frameworks are likely to issue many small (<16K) posts to a server, >which all fit in a single frame. In my testcase I needed > 500 requests >though. If we are facing the same problem, there is a chance that your problem does not occur with mod_http2 1.0.5. To be precise, I did not have problems with the mod_http2 from revision 1715218 of the 2.4-http2-alpha branch: svn co -r 1715218 \ http://svn.apache.org/repos/asf/httpd/httpd/branches/2.4-http2-alpha I do not have time at the moment to dive deeper into the problem with Drupal 7 and mod_http2 1.0.8, so you are on your own right now. -- Jan
Re: No H2 Window updates!
I'm assuming that the brokenness also shows up on trunk, right? Bert, Jan, can you check if trunk shows the same behavior? I would prefer not hacking away on 2.4 directly and independently. tia!! > On Nov 29, 2015, at 3:03 AM, Stefan Eissing > wrote: > > Ok, thanks. I think I have an idea of what's happening: > - on short request bodies, window updates get omitted and gives a shrinking > connection window > - the window size of the connection itself should be at max Value right from > the start > > I won't be able to do anything about it until later this week, though. > > //stefan > > Am 28.11.2015 um 22:29 schrieb Bert Huijben : > >>> -Original Message- >>> From: Bert Huijben [mailto:b...@qqmail.nl] >>> Sent: zaterdag 28 november 2015 14:32 >>> To: stefan.eiss...@greenbytes.de; dev@httpd.apache.org >>> Subject: RE: No H2 Window updates! >> >> >>> In case of Subversion's real usage, I want to commit potentially hundreds >> of >>> MByte, so a connection level window of more than a few bytes would be >>> very >>> welcome With HTTP/1 we send out the data as fast as we can and the TCP >>> windowing handles this from the httpd side... Now the http/2 level >>> windowing >>> should handle this. >>> >>> >>> Mod_dav and mod_dav_svn are usually fast readers; just limited by trivial >>> disk io or xml processing in the common operations, so I don't expect real >>> problems there. >> >> For my commits against the svn-master.apache.org repository my latency/ping >> time is +- 142 ms. >> >> With the current simple algorithm and a maximum window of 64 >> Kbyte/connection, I can send a theoretical maximum of 64 Kbyte/142 ms per >> connection to that server... which would be about 450 Kbyte/s. >> (=1/0.142*65536/1024) >> >> That is < 1/30th of the bandwidth I have at my disposal (+- 150 Mbit). >> >> But currently I don't even get anywhere near that as my window continues to >> shrink because some data is missed in accounting even after my simple patch. >> >> >> >> For httpd we have to think carefully which algorithm we want to implement >> here. Preferably the algorithm should be better than TCP could, as we know >> the specifics of the http algorithm better than plain TCP could for >> HTTP/1,1. TCP does send a lot of window updates though... Almost every >> packet does. >> >> >> Solving the really hard problems, like this one, is the reason I didn't >> think the nghttp2 library would really be the solution for serf. >> It is nice for such a library to provide a head start on the protocol level, >> but there is no way a standard library can really solve this scheduling >> problem in a generic way. (If it could do it, we had used that solution for >> TCP... and never had to resort to designing a HTTP/2 in the first place). >> >> See https://en.wikipedia.org/wiki/TCP_window_scale_option for some >> introduction on how TCP was extended to work above the limit that currently >> applies to H2. >> >> >> Dynamic window sizing/scaling will have to be implemented at some point, at >> both the stream and the connection level... This will involve timing, >> measuring, etc. etc. Things nghttp2 can't do for us right now. >> And if I'm right this might take continuous optimizing for years to come. >> >> >> >> When I connect to sites like google I immediately receive a large connection >> level window, to allow me to post huge blobs without a delay. (Haven't >> tested their stream level windows behavior yet) >> In serf I do something similar... and then apply a bit of throttling at the >> stream level. >> >> >> I would guess some clients have already implemented some of this, so we >> might be able to learn from their implementations... Clients will see much >> more incoming data than servers of course :). >> >> >> Bert >> >>
Re: No H2 Window updates!
Jim Jagielski in gmane.comp.apache.devel (Mon, 30 Nov 2015 07:24:07 -0500): >I'm assuming that the brokenness also shows up on trunk, >right? > >Bert, Jan, can you check if trunk shows the same behavior? >I would prefer not hacking away on 2.4 directly and independently. I checked out revision 1717334 from trunk and tried to update mod_http2 without updating the other sources. Surprise: Creating library .\Release/mod_http2.lib and object .\Release/mod_http2.exp 1>h2_from_h1.obj : error LNK2019: unresolved external symbol _ap_casecmpstr referenced in function _create_response 1>.\Release\mod_http2.so : fatal error LNK1120: 1 unresolved externals Somebody followed your advice and actually wrote some code! I cannot easily test mod_http2 from trunk, because it involves updating APR as well. -- Jan
Re: No H2 Window updates!
Jim Jagielski in gmane.comp.apache.devel (Mon, 30 Nov 2015 07:24:07 -0500): >I'm assuming that the brokenness also shows up on trunk, >right? > >Bert, Jan, can you check if trunk shows the same behavior? >I would prefer not hacking away on 2.4 directly and independently. FYI: the only differences between trunk and mod_http2 1.0.8 were ap_strcasecmp vs. strcasecmp and 1.0.9-dev vs 1.0.8. -- Jan
Re: No H2 Window updates!
1.0.8 is known problematic, 1.0.5 mod_h2 is known good. I wouldn't expect any change there On Nov 30, 2015 17:41, "Jan Ehrhardt" wrote: > Jim Jagielski in gmane.comp.apache.devel (Mon, 30 Nov 2015 07:24:07 -0500): > >I'm assuming that the brokenness also shows up on trunk, > >right? > > > >Bert, Jan, can you check if trunk shows the same behavior? > >I would prefer not hacking away on 2.4 directly and independently. > > FYI: the only differences between trunk and mod_http2 1.0.8 were > ap_strcasecmp > vs. strcasecmp and 1.0.9-dev vs 1.0.8. > -- > Jan > >
Re: No H2 Window updates!
> On Nov 30, 2015, at 5:53 PM, Jan Ehrhardt wrote: > > Jim Jagielski in gmane.comp.apache.devel (Mon, 30 Nov 2015 07:24:07 -0500): >> I'm assuming that the brokenness also shows up on trunk, >> right? >> >> Bert, Jan, can you check if trunk shows the same behavior? >> I would prefer not hacking away on 2.4 directly and independently. > > I checked out revision 1717334 from trunk and tried to update mod_http2 > without updating the other sources. Surprise: > > Creating library .\Release/mod_http2.lib and object .\Release/mod_http2.exp > 1>h2_from_h1.obj : error LNK2019: unresolved external symbol _ap_casecmpstr > referenced in function _create_response > 1>.\Release\mod_http2.so : fatal error LNK1120: 1 unresolved externals > > Somebody followed your advice and actually wrote some code! I cannot easily > test mod_http2 from trunk, because it involves updating APR as well. Sounds like you needed to do a make clean and rebuild... ap_casecmpstr() is httpd not APR :) Thanks for the test!
Re: No H2 Window updates!
Jim Jagielski in gmane.comp.apache.devel (Tue, 1 Dec 2015 07:38:47 -0500): > >> On Nov 30, 2015, at 5:53 PM, Jan Ehrhardt wrote: >> >> Creating library .\Release/mod_http2.lib and object .\Release/mod_http2.exp >> 1>h2_from_h1.obj : error LNK2019: unresolved external symbol _ap_casecmpstr >> referenced in function _create_response >> 1>.\Release\mod_http2.so : fatal error LNK1120: 1 unresolved externals > >Sounds like you needed to do a make clean and rebuild... ap_casecmpstr() >is httpd not APR :) If only it was that simple on Windows. Replacing all sources of 2.4.x with those from trunk leads to oher errors. For instance: 1>-- Build started: Project: mod_ssl, Configuration: Release Win32 -- 1> mod_ssl.c 1>mod_ssl.c(43): error C2491: 'ssl_run_pre_handshake': definition of dllimport function not allowed 1> ssl_engine_config.c 1> ssl_engine_init.c 1>ssl_engine_init.c(39): error C2491: 'ssl_run_init_server': definition of dllimport function not allowed etc I do not have the time now to solve things like this. -- Jan
Re: No H2 Window updates!
Can you (or someone) provide a test case or explanation of how to recreate the issue? tia
Re: No H2 Window updates!
On Tue, Dec 1, 2015 at 7:15 AM, Jan Ehrhardt wrote: > Jim Jagielski in gmane.comp.apache.devel (Tue, 1 Dec 2015 07:38:47 -0500): > > > >> On Nov 30, 2015, at 5:53 PM, Jan Ehrhardt wrote: > >> > >> Creating library .\Release/mod_http2.lib and object > .\Release/mod_http2.exp > >> 1>h2_from_h1.obj : error LNK2019: unresolved external symbol > _ap_casecmpstr > >> referenced in function _create_response > >> 1>.\Release\mod_http2.so : fatal error LNK1120: 1 unresolved externals > > > >Sounds like you needed to do a make clean and rebuild... ap_casecmpstr() > >is httpd not APR :) > > If only it was that simple on Windows. Replacing all sources of 2.4.x with > those from trunk leads to oher errors. Hi Jan - it isn't possible to build httpd trunk sources using httpd 2.4.x build files, and this goes for windows or unix. How were you trying to build? ... > For instance: > > 1>-- Build started: Project: mod_ssl, Configuration: Release Win32 > -- > 1> mod_ssl.c > 1>mod_ssl.c(43): error C2491: 'ssl_run_pre_handshake': definition of > dllimport > function not allowed > 1> ssl_engine_config.c > 1> ssl_engine_init.c > 1>ssl_engine_init.c(39): error C2491: 'ssl_run_init_server': definition of > dllimport function not allowed > etc > Right now, SSL_DECLARE_EXPORT is defined in both mod_ssl.dsp, from the ancient build schema, and in CMakeFiles.txt, for our new and still evolving cmake-based build schema. It seems like you may have been trying to build mod_ssl in trunk using the mod_ssl.mak file from 2.4.x? Someone can generate you the .mak files for trunk, we just don't check them in during trunk development because they are a more rapidly moving target that doesn't settle down till we are about ready to release trunk as some 2.6 or 3.0 version. That's what the cmake effort is all about. See the README.cmake in the root httpd trunk directory for guidance on how to use that. If that is what you did, then we have a bigger issue to resolve, but I compared the use of SSL_DECLARE_EXPORT to the long-existing DAV_DECLARE_EXPORT and it seems to be consistent. Cheers, Bill
Re: No H2 Window updates!
William A Rowe Jr in gmane.comp.apache.devel (Tue, 1 Dec 2015 08:32:24 -0600): >Hi Jan - it isn't possible to build httpd trunk sources using httpd 2.4.x build >files, and this goes for windows or unix. How were you trying to build? ... That was exactly what I was doing. I did not replace my working *.vcxproj files. But, sorry guys, really no time ATM. I will check in as soon as I have some spare time. -- Jan
Re: No H2 Window updates!
Please find with r1717641 version 1.0.9-DEV of mod_http2 in trunk and branches/2.4.x that fixes the issue of streams with smallish inputs and lost WINDOW_UPDATEs. Please report back if this works for you. I have another time slot on Friday where I can follow up on it. Thanks. //Stefan
Re: No H2 Window updates!
Passes all tests: OSX and CentOS6. > On Dec 2, 2015, at 9:44 AM, Stefan Eissing > wrote: > > Please find with r1717641 version 1.0.9-DEV of mod_http2 in trunk and > branches/2.4.x > that fixes the issue of streams with smallish inputs and lost WINDOW_UPDATEs. > > Please report back if this works for you. I have another time slot on Friday > where I can follow up on it. Thanks. > > //Stefan >
Re: No H2 Window updates!
Stefan Eissing in gmane.comp.apache.devel (Wed, 2 Dec 2015 15:44:53 +0100): >Please find with r1717641 version 1.0.9-DEV of mod_http2 in trunk and >branches/2.4.x >that fixes the issue of streams with smallish inputs and lost WINDOW_UPDATEs. I checked out 2.4.x which gave me revision 1717657. >Please report back if this works for you. I have another time slot on Friday >where I can follow up on it. Thanks. Quick test: it did not solve my problems (and I do not have time slots yet). BTW: it is still called 1.0.8 in 2.4.x: http://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/http2/h2_version.h?view=markup -- Jan
RE: No H2 Window updates!
> -Original Message- > From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] > Sent: woensdag 2 december 2015 15:45 > To: dev@httpd.apache.org > Subject: Re: No H2 Window updates! > > Please find with r1717641 version 1.0.9-DEV of mod_http2 in trunk and > branches/2.4.x > that fixes the issue of streams with smallish inputs and lost > WINDOW_UPDATEs. > > Please report back if this works for you. I have another time slot on Friday > where I can follow up on it. Thanks. I haven't traced for other problems yet, but the current ^/httpd/httpd/branches/2.4.x branch does *NOT* the windowing issue for me. My Subversion testrun still gets stuck on that same 'svnmover' test, unless I set H2WindowSize to a higher value. Bert
Re: No H2 Window updates!
That is unfortunate. I added a test case which reproduced window exhaustion before I fixed it. What sort of requests do still miss window updates? content-length? response codes? thx. Am 04.12.2015 um 02:36 schrieb Bert Huijben : >> -Original Message- >> From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] >> Sent: woensdag 2 december 2015 15:45 >> To: dev@httpd.apache.org >> Subject: Re: No H2 Window updates! >> >> Please find with r1717641 version 1.0.9-DEV of mod_http2 in trunk and >> branches/2.4.x >> that fixes the issue of streams with smallish inputs and lost >> WINDOW_UPDATEs. >> >> Please report back if this works for you. I have another time slot on > Friday >> where I can follow up on it. Thanks. > > I haven't traced for other problems yet, but the current > ^/httpd/httpd/branches/2.4.x branch does *NOT* the windowing issue for me. > > My Subversion testrun still gets stuck on that same 'svnmover' test, unless > I set H2WindowSize to a higher value. > >Bert > >
RE: No H2 Window updates!
> -Original Message- > From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] > Sent: vrijdag 4 december 2015 10:18 > To: dev@httpd.apache.org > Subject: Re: No H2 Window updates! > > That is unfortunate. I added a test case which reproduced window > exhaustion before I fixed it. What sort of requests do still miss window > updates? content-length? response codes? I don't think we send content-length headers... (see older problem). I fixed Subversion to always send Content-Type though. I'm going to add some more tracing to get the details again, but from what I remember is that I had requests with 113 and 128 bytes +- over and over. [The tool experiencing this problem is a tool that tries to replicate some behavior we can't express in Subversion yet. To implement this it asks the server for similar things over and over. If we decide that we go this way we would probably send one advanced request and get one response from the server... But currently it is a nice tool to test the network IO limits :-)] These requests are always 1 HEADERS (/endofheaders) + 1 DATA (/eos) and the response is not much either (Certainly less than 1 Kbyte, so probably just HEADERS+1 DATA as well) -- Ok, added some logging back. I still don't see a lot of window updates, but I do see some. After my connection windows is slightly below 32767... I receive a single connection window update to bring it back to exactly 32767. In all cases in this log this update is < 100 bytes. (1 <= update < 256). But then I don't receive further window updates until my connection window is almost depleted... And then I get another 255 bytes or so. Not enough to get through the test. The log I just used is on: https://lpt1.nl/f/2015/201512-NoWindow.txt (Search for 'DBG: Connection window update' to find the updates between the allocations) Bert
Re: No H2 Window updates!
If you find the time, the lastest v1.0.10 mod_http2 in 2.4.x sets the connection window to max which addresses for me the window starvation issues I was able to reproduce (and put into my test suite). I hope this works for you as well, otherwise I'd need more detailed data on how to reproduce the hanger. //Stefan > Am 04.12.2015 um 10:50 schrieb Bert Huijben : > >> -Original Message- >> From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] >> Sent: vrijdag 4 december 2015 10:18 >> To: dev@httpd.apache.org >> Subject: Re: No H2 Window updates! >> >> That is unfortunate. I added a test case which reproduced window >> exhaustion before I fixed it. What sort of requests do still miss window >> updates? content-length? response codes? > > I don't think we send content-length headers... (see older problem). I fixed > Subversion to always send Content-Type though. > > I'm going to add some more tracing to get the details again, but from what I > remember is that I had requests with 113 and 128 bytes +- over and over. > > [The tool experiencing this problem is a tool that tries to replicate some > behavior we can't express in Subversion yet. To implement this it asks the > server for similar things over and over. If we decide that we go this way we > would probably send one advanced request and get one response from the > server... But currently it is a nice tool to test the network IO limits :-)] > > These requests are always 1 HEADERS (/endofheaders) + 1 DATA (/eos) and the > response is not much either (Certainly less than 1 Kbyte, so probably just > HEADERS+1 DATA as well) > > > -- > Ok, added some logging back. > > I still don't see a lot of window updates, but I do see some. > > After my connection windows is slightly below 32767... I receive a single > connection window update to bring it back to exactly 32767. In all cases in > this log this update is < 100 bytes. (1 <= update < 256). > > But then I don't receive further window updates until my connection window > is almost depleted... And then I get another 255 bytes or so. Not enough to > get through the test. > > The log I just used is on: > https://lpt1.nl/f/2015/201512-NoWindow.txt > (Search for 'DBG: Connection window update' to find the updates between the > allocations) > > Bert >
RE: No H2 Window updates!
> -Original Message- > From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] > Sent: vrijdag 4 december 2015 16:23 > To: dev@httpd.apache.org > Subject: Re: No H2 Window updates! > > If you find the time, the lastest v1.0.10 mod_http2 in 2.4.x sets the > connection window to max which addresses for me the window starvation > issues I was able to reproduce (and put into my test suite). I hope this works > for you as well, otherwise I'd need more detailed data on how to reproduce > the hanger. The latest version gives me immediately a connection window update of the maximum allowed value. But given that the initial connection window was already set via the settings window, adding this value is not allowed. The code has his comment copied from the RFC: /* A sender MUST NOT allow a flow-control window to exceed 2^31-1 octets. If a sender receives a WINDOW_UPDATE that causes a flow- control window to exceed this maximum, it MUST terminate either the stream or the connection, as appropriate. For streams, the sender sends a RST_STREAM with an error code of FLOW_CONTROL_ERROR; for the connection, a GOAWAY frame with an error code of FLOW_CONTROL_ERROR is sent.*/ Serf does exactly this... it terminates the connection. So it gets nowhere near the original point of failure in the test. It fails around the first stream created the test, while the original failure is after thousands of requests. Bert -- With the same logging enabled as on that older url, this is the whole test output now: START: svnmover_tests.py I: CMD: svnadmin.exe create svn-test-work\local_tmp\repos --compatible-version=1.10 --fs-type=fsfs I: I: CMD: svn.exe import -m "Log message for revision 1." svn-test-work\local_tmp\greekfiles https://localhost:7829/svn-test-work/local_tmp/repos --config-dir R:\subversion\tests\cmdline\svn-test-work\local_tmp\config --password rayjandom --no-auth-cache --username jrandom I: CMD: R:\subversion\svn/svn.exe import -m "Log message for revision 1." svn-test-work\local_tmp\greekfiles https://localhost:7829/svn-test-work/local_tmp/repos --config-dir R:\subversion\tests\cmdline\svn-test-work\local_tmp\config --password rayjandom --no-auth-cache --username jrandom exited with 1 I: I: DBG: Allocated 131 bytes for window on stream 0x1 (left: 65404, 65404) I: DBG: Connection window update of 2147483392 to -2147418500 I: ..\..\..\subversion\svn\import-cmd.c:129, I: ..\..\..\subversion\libsvn_client\import.c:868, I: ..\..\..\subversion\libsvn_client\ra.c:509, I: ..\..\..\subversion\libsvn_client\ra.c:488, I: ..\..\..\subversion\libsvn_ra\ra_loader.c:404: (apr_err=SVN_ERR_RA_CANNOT_CREATE_SESSION) I: svn: E170013: Unable to connect to a repository at URL 'https://localhost:7829/svn-test-work/local_tmp/repos' I: ..\..\..\subversion\libsvn_ra_serf\serf.c:603, I: ..\..\..\subversion\libsvn_ra_serf\options.c:538, I: ..\..\..\subversion\libsvn_ra_serf\util.c:1032, I: ..\..\..\subversion\libsvn_ra_serf\util.c:981, I: ..\..\..\subversion\libsvn_ra_serf\util.c:958: (apr_err=120153) I: svn: E120153: Error running context: HTTP2 flow control limits exceeded W: ..\..\..\subversion\svn\import-cmd.c:129, W: ..\..\..\subversion\libsvn_client\import.c:868, W: ..\..\..\subversion\libsvn_client\ra.c:509, W: ..\..\..\subversion\libsvn_client\ra.c:488, W: ..\..\..\subversion\libsvn_ra\ra_loader.c:404: (apr_err=SVN_ERR_RA_CANNOT_CREATE_SESSION) W: svn: E170013: Unable to connect to a repository at URL 'https://localhost:7829/svn-test-work/local_tmp/repos' W: ..\..\..\subversion\libsvn_ra_serf\serf.c:603, W: ..\..\..\subversion\libsvn_ra_serf\options.c:538, W: ..\..\..\subversion\libsvn_ra_serf\util.c:1032, W: ..\..\..\subversion\libsvn_ra_serf\util.c:981, W: ..\..\..\subversion\libsvn_ra_serf\util.c:958: (apr_err=120153) W: svn: E120153: Error running context: HTTP2 flow control limits exceeded END: svnmover_tests.py ELAPSED: svnmover_tests.py 0:00:00.609000
Re: No H2 Window updates!
Stefan Eissing in gmane.comp.apache.devel (Fri, 4 Dec 2015 16:23:19 +0100): >If you find the time, the lastest v1.0.10 mod_http2 in 2.4.x sets the >connection window to max which addresses for me the window starvation >issues I was able to reproduce (and put into my test suite). I hope this >works for you as well, otherwise I'd need more detailed data on how to >reproduce the hanger. 1.0.10 did not solve my problems with the admin menu in Drupal7. Sorry. -- Jan
Re: No H2 Window updates!
The code increases the window by the difference between max and current so that the client has exactly the max value. nghttp2 accepts this on the client side. It rejects any larger value as you described. So we seem to have a difference in calculation between nghttp2 and serf. which values do you see? some data would be helpful here. > Am 04.12.2015 um 18:42 schrieb Bert Huijben : > > > >> -Original Message- >> From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] >> Sent: vrijdag 4 december 2015 16:23 >> To: dev@httpd.apache.org >> Subject: Re: No H2 Window updates! >> >> If you find the time, the lastest v1.0.10 mod_http2 in 2.4.x sets the >> connection window to max which addresses for me the window starvation >> issues I was able to reproduce (and put into my test suite). I hope this > works >> for you as well, otherwise I'd need more detailed data on how to reproduce >> the hanger. > > The latest version gives me immediately a connection window update of the > maximum allowed value. > > But given that the initial connection window was already set via the > settings window, adding this value is not allowed. > > The code has his comment copied from the RFC: > /* A sender MUST NOT allow a flow-control window to exceed 2^31-1 > octets. If a sender receives a WINDOW_UPDATE that causes a flow- > control window to exceed this maximum, it MUST terminate either the > stream or the connection, as appropriate. For streams, the sender > sends a RST_STREAM with an error code of FLOW_CONTROL_ERROR; for > the > connection, a GOAWAY frame with an error code of FLOW_CONTROL_ERROR > is sent.*/ > > Serf does exactly this... it terminates the connection. > > So it gets nowhere near the original point of failure in the test. > It fails around the first stream created the test, while the original > failure is after thousands of requests. > >Bert > > -- > With the same logging enabled as on that older url, this is the whole test > output now: > > > START: svnmover_tests.py > I: CMD: svnadmin.exe create svn-test-work\local_tmp\repos > --compatible-version=1.10 --fs-type=fsfs > I: > I: CMD: svn.exe import -m "Log message for revision 1." > svn-test-work\local_tmp\greekfiles > https://localhost:7829/svn-test-work/local_tmp/repos --config-dir > R:\subversion\tests\cmdline\svn-test-work\local_tmp\config --password > rayjandom --no-auth-cache --username jrandom > I: CMD: R:\subversion\svn/svn.exe import -m "Log message for revision 1." > svn-test-work\local_tmp\greekfiles > https://localhost:7829/svn-test-work/local_tmp/repos --config-dir > R:\subversion\tests\cmdline\svn-test-work\local_tmp\config --password > rayjandom --no-auth-cache --username jrandom exited with 1 > I: > I: DBG: Allocated 131 bytes for window on stream 0x1 (left: 65404, 65404) > I: DBG: Connection window update of 2147483392 to -2147418500 > I: ..\..\..\subversion\svn\import-cmd.c:129, > I: ..\..\..\subversion\libsvn_client\import.c:868, > I: ..\..\..\subversion\libsvn_client\ra.c:509, > I: ..\..\..\subversion\libsvn_client\ra.c:488, > I: ..\..\..\subversion\libsvn_ra\ra_loader.c:404: > (apr_err=SVN_ERR_RA_CANNOT_CREATE_SESSION) > I: svn: E170013: Unable to connect to a repository at URL > 'https://localhost:7829/svn-test-work/local_tmp/repos' > I: ..\..\..\subversion\libsvn_ra_serf\serf.c:603, > I: ..\..\..\subversion\libsvn_ra_serf\options.c:538, > I: ..\..\..\subversion\libsvn_ra_serf\util.c:1032, > I: ..\..\..\subversion\libsvn_ra_serf\util.c:981, > I: ..\..\..\subversion\libsvn_ra_serf\util.c:958: (apr_err=120153) > I: svn: E120153: Error running context: HTTP2 flow control limits exceeded > W: ..\..\..\subversion\svn\import-cmd.c:129, > W: ..\..\..\subversion\libsvn_client\import.c:868, > W: ..\..\..\subversion\libsvn_client\ra.c:509, > W: ..\..\..\subversion\libsvn_client\ra.c:488, > W: ..\..\..\subversion\libsvn_ra\ra_loader.c:404: > (apr_err=SVN_ERR_RA_CANNOT_CREATE_SESSION) > W: svn: E170013: Unable to connect to a repository at URL > 'https://localhost:7829/svn-test-work/local_tmp/repos' > W: ..\..\..\subversion\libsvn_ra_serf\serf.c:603, > W: ..\..\..\subversion\libsvn_ra_serf\options.c:538, > W: ..\..\..\subversion\libsvn_ra_serf\util.c:1032, > W: ..\..\..\subversion\libsvn_ra_serf\util.c:981, > W: ..\..\..\subversion\libsvn_ra_serf\util.c:958: (apr_err=120153) > W: svn: E120153: Error running context: HTTP2 flow control limits exceeded > END: svnmover_tests.py > ELAPSED: svnmover_tests.py 0:00:00.609000 > >
Re: No H2 Window updates!
> Am 04.12.2015 um 20:51 schrieb Jan Ehrhardt : > > Stefan Eissing in gmane.comp.apache.devel (Fri, 4 Dec 2015 16:23:19 > +0100): >> If you find the time, the lastest v1.0.10 mod_http2 in 2.4.x sets the >> connection window to max which addresses for me the window starvation >> issues I was able to reproduce (and put into my test suite). I hope this >> works for you as well, otherwise I'd need more detailed data on how to >> reproduce the hanger. > > 1.0.10 did not solve my problems with the admin menu in Drupal7. Sorry. Do you have any advice how to reproduce the problem, so that I might see what it is? I am not familiar with drupal7, unfortunately.
RE: No H2 Window updates!
> -Original Message- > From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] > Sent: vrijdag 4 december 2015 21:36 > To: dev@httpd.apache.org > Subject: Re: No H2 Window updates! > > The code increases the window by the difference between max and current > so that the client has exactly the max value. nghttp2 accepts this on the client > side. It rejects any larger value as you described. > > So we seem to have a difference in calculation between nghttp2 and serf. > which values do you see? some data would be helpful here. The values are completely in the log file at the bottom of this mail. I open a connection and the server announces a default window of 65535. > > I: DBG: Allocated 131 bytes for window on stream 0x1 (left: 65404, 65404) I send a request with a DATA frame with a payload of 131 bytes, which updates the stream and windows size with -131. So both the stream and connection windows are no longer default, but 65404. Logged for easy reading Then I receive a WINDOW_UPDATE frame > > I: DBG: Connection window update of 2147483392 to -2147418500 Which tries to add 2147483392 to the existing window. Which doesn't fit, because the total outgoing window has to fit in 2^31-1... See the RFC. That is all that happens. (The windowing in the other direction is completely uninteresting here... as it happens completely independent) Bert > > > Am 04.12.2015 um 18:42 schrieb Bert Huijben : > > > > > > > >> -Original Message- > >> From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] > >> Sent: vrijdag 4 december 2015 16:23 > >> To: dev@httpd.apache.org > >> Subject: Re: No H2 Window updates! > >> > >> If you find the time, the lastest v1.0.10 mod_http2 in 2.4.x sets the > >> connection window to max which addresses for me the window > starvation > >> issues I was able to reproduce (and put into my test suite). I hope this > > works > >> for you as well, otherwise I'd need more detailed data on how to > reproduce > >> the hanger. > > > > The latest version gives me immediately a connection window update of > the > > maximum allowed value. > > > > But given that the initial connection window was already set via the > > settings window, adding this value is not allowed. > > > > The code has his comment copied from the RFC: > > /* A sender MUST NOT allow a flow-control window to exceed 2^31-1 > > octets. If a sender receives a WINDOW_UPDATE that causes a flow- > > control window to exceed this maximum, it MUST terminate either the > > stream or the connection, as appropriate. For streams, the sender > > sends a RST_STREAM with an error code of FLOW_CONTROL_ERROR; > for > > the > > connection, a GOAWAY frame with an error code of > FLOW_CONTROL_ERROR > > is sent.*/ > > > > Serf does exactly this... it terminates the connection. > > > > So it gets nowhere near the original point of failure in the test. > > It fails around the first stream created the test, while the original > > failure is after thousands of requests. > > > >Bert > > > > -- > > With the same logging enabled as on that older url, this is the whole test > > output now: > > > > > > START: svnmover_tests.py > > I: CMD: svnadmin.exe create svn-test-work\local_tmp\repos > > --compatible-version=1.10 --fs-type=fsfs > > I: > > I: CMD: svn.exe import -m "Log message for revision 1." > > svn-test-work\local_tmp\greekfiles > > https://localhost:7829/svn-test-work/local_tmp/repos --config-dir > > R:\subversion\tests\cmdline\svn-test-work\local_tmp\config --password > > rayjandom --no-auth-cache --username jrandom > > I: CMD: R:\subversion\svn/svn.exe import -m "Log message for revision 1." > > svn-test-work\local_tmp\greekfiles > > https://localhost:7829/svn-test-work/local_tmp/repos --config-dir > > R:\subversion\tests\cmdline\svn-test-work\local_tmp\config --password > > rayjandom --no-auth-cache --username jrandom exited with 1 > > I: > > I: DBG: Allocated 131 bytes for window on stream 0x1 (left: 65404, 65404) > > I: DBG: Connection window update of 2147483392 to -2147418500 > > I: ..\..\..\subversion\svn\import-cmd.c:129, > > I: ..\..\..\subversion\libsvn_client\import.c:868, > > I: ..\..\..\subversion\libsvn_client\ra.c:509, > > I: ..\..\..\subversion\libsvn_client\ra.c:488, > > I: ..\..\..\subversion\libsvn_ra\ra_loader.c:404: > > (apr_err=SVN_ERR_RA_CANNOT_CREATE_SESSION) > > I: svn: E170013: Unable to
Re: No H2 Window updates!
Stefan Eissing in gmane.comp.apache.devel (Fri, 4 Dec 2015 21:37:46 +0100): >> >> 1.0.10 did not solve my problems with the admin menu in Drupal7. Sorry. > >Do you have any advice how to reproduce the problem, so that I might see >what it is? I am not familiar with drupal7, unfortunately. I finally found out how it goes wrong. The admin menu is a compressed file of 16899 bytes, which should be decoded into something of 216000+ html content. With mod_http2 1.0.7+ Chrome bails out with a ERR_CONTENT_DECODING_FAILED. See the logs: GET /js/admin_menu/cache/53b57aa88f09fd5f75ccfb325815c700 HTTP/1.1" 200 16899 "-" GET /js/admin_menu/cache/53b57aa88f09fd5f75ccfb325815c700 HTTP/2" 200 16899 "-" The second one was directly after restarting the server with mod_http2 1.0.7. mod_http2 1.0.5 does not have this problem. I do not know yet how and where it goes wrong. Something between v1.0.5 and v1.0.7 changed that causes a ERR_CONTENT_DECODING_FAILED. I will try to intercept the compressed content and see if I can make a static page out of it, outside of Drupal and without having to log in as Drupal admin. Stefan: ERR_CONTENT_DECODING_FAILED is the most informative info I can give you at the moment. -- Jan
Re: No H2 Window updates!
Stefan Eissing in gmane.comp.apache.devel (Fri, 4 Dec 2015 21:37:46 +0100): >> 1.0.10 did not solve my problems with the admin menu in Drupal7. Sorry. > >Do you have any advice how to reproduce the problem, so that I might see >what it is? I am not familiar with drupal7, unfortunately. mod_http2-1.0.5 revision 1715371 from the 2.4.x branch was still OK. mod_http2-1.0.7 revision 1716210 from the 2.4.x branch doubled the arguments in the Drupal7 admin menu: Content-Encoding: gzip, gzip This was the cause of my troubles with Drupal7. I rebuilt both mod_https.so versions today with the latest nghttp2 (git head). No other changes. Snippet from my conf file: LoadModule http2_module modules/mod_http2-1.0.5-r1715371.so # bad below #LoadModule http2_module modules/mod_http2-1.0.7-r1716210.so -- Jan
Re: No H2 Window updates!
Jan, if you could find the time to verify that the duplicate headers are gone in the current 2.4.x version, that'd be nice. Thanks! //Stefan > Am 06.12.2015 um 07:21 schrieb Jan Ehrhardt : > > Stefan Eissing in gmane.comp.apache.devel (Fri, 4 Dec 2015 21:37:46 > +0100): >>> 1.0.10 did not solve my problems with the admin menu in Drupal7. Sorry. >> >> Do you have any advice how to reproduce the problem, so that I might see >> what it is? I am not familiar with drupal7, unfortunately. > > mod_http2-1.0.5 revision 1715371 from the 2.4.x branch was still OK. > > mod_http2-1.0.7 revision 1716210 from the 2.4.x branch doubled the > arguments in the Drupal7 admin menu: > Content-Encoding: gzip, gzip > > This was the cause of my troubles with Drupal7. I rebuilt both > mod_https.so versions today with the latest nghttp2 (git head). > No other changes. Snippet from my conf file: > > LoadModule http2_module modules/mod_http2-1.0.5-r1715371.so > # bad below > #LoadModule http2_module modules/mod_http2-1.0.7-r1716210.so > -- > Jan >
Re: No H2 Window updates!
Stefan Eissing in gmane.comp.apache.devel (Mon, 7 Dec 2015 13:58:07 +0100): >if you could find the time to verify that the duplicate headers are gone >in the current 2.4.x version, that'd be nice. Thanks! They are gone! Time to T&R 2.4.18. -- Jan
Re: No H2 Window updates!
On Mon, Dec 7, 2015 at 10:56 AM, Jan Ehrhardt wrote: > Stefan Eissing in gmane.comp.apache.devel (Mon, 7 Dec 2015 13:58:07 > +0100): >>if you could find the time to verify that the duplicate headers are gone >>in the current 2.4.x version, that'd be nice. Thanks! > > They are gone! Time to T&R 2.4.18. Thanks for driving this one Jan.
Re: No H2 Window updates!
Eric Covener in gmane.comp.apache.devel (Mon, 7 Dec 2015 11:03:15 -0500): >On Mon, Dec 7, 2015 at 10:56 AM, Jan Ehrhardt wrote: >> Stefan Eissing in gmane.comp.apache.devel (Mon, 7 Dec 2015 13:58:07 >> +0100): >>>if you could find the time to verify that the duplicate headers are gone >>>in the current 2.4.x version, that'd be nice. Thanks! >> >> They are gone! Time to T&R 2.4.18. > >Thanks for driving this one Jan. My pleasure. To confirm: I have been working on our devserver with mod_http2 1.0.11 a couple of hours now. No problems. -- Jan
RE: No H2 Window updates! (Probably a Serf issue!)
> -Original Message- > From: Bert Huijben [mailto:b...@qqmail.nl] > Sent: vrijdag 4 december 2015 21:45 > To: dev@httpd.apache.org > Subject: RE: No H2 Window updates! > > > > > -Original Message- > > From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] > > Sent: vrijdag 4 december 2015 21:36 > > To: dev@httpd.apache.org > > Subject: Re: No H2 Window updates! > > > > The code increases the window by the difference between max and > current > > so that the client has exactly the max value. nghttp2 accepts this on the > client > > side. It rejects any larger value as you described. > > > > So we seem to have a difference in calculation between nghttp2 and serf. > > which values do you see? some data would be helpful here. > > The values are completely in the log file at the bottom of this mail. > > I open a connection and the server announces a default window of 65535. > > > > I: DBG: Allocated 131 bytes for window on stream 0x1 (left: 65404, > 65404) > I send a request with a DATA frame with a payload of 131 bytes, which > updates the stream and windows size with -131. > > So both the stream and connection windows are no longer default, but > 65404. > Logged for easy reading > > > Then I receive a WINDOW_UPDATE frame > > > I: DBG: Connection window update of 2147483392 to -2147418500 > Which tries to add 2147483392 to the existing window. > > Which doesn't fit, because the total outgoing window has to fit in 2^31-1... > See the RFC. Ok, I think I found a -and provably THE- problem in serf... There is a bug in decoding the window update frames... Reviewing and testing some bits there now. Bert
RE: No H2 Window updates! (Probably a Serf issue!)
> -Original Message- > From: Bert Huijben [mailto:b...@qqmail.nl] > Sent: vrijdag 4 december 2015 22:27 > To: dev@httpd.apache.org > Subject: RE: No H2 Window updates! (Probably a Serf issue!) > > > > > -Original Message- > > From: Bert Huijben [mailto:b...@qqmail.nl] > > Sent: vrijdag 4 december 2015 21:45 > > To: dev@httpd.apache.org > > Subject: RE: No H2 Window updates! > > > > > > > > > -Original Message- > > > From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] > > > Sent: vrijdag 4 december 2015 21:36 > > > To: dev@httpd.apache.org > > > Subject: Re: No H2 Window updates! > > > > > > The code increases the window by the difference between max and > > current > > > so that the client has exactly the max value. nghttp2 accepts this on > the > > client > > > side. It rejects any larger value as you described. > > > > > > So we seem to have a difference in calculation between nghttp2 and serf. > > > which values do you see? some data would be helpful here. > > > > The values are completely in the log file at the bottom of this mail. > > > > I open a connection and the server announces a default window of 65535. > > > > > > I: DBG: Allocated 131 bytes for window on stream 0x1 (left: 65404, > > 65404) > > I send a request with a DATA frame with a payload of 131 bytes, which > > updates the stream and windows size with -131. > > > > So both the stream and connection windows are no longer default, but > > 65404. > > Logged for easy reading > > > > > > Then I receive a WINDOW_UPDATE frame > > > > I: DBG: Connection window update of 2147483392 to -2147418500 > > Which tries to add 2147483392 to the existing window. > > > > Which doesn't fit, because the total outgoing window has to fit in > 2^31-1... > > See the RFC. > > Ok, I think I found a -and provably THE- problem in serf... > > There is a bug in decoding the window update frames... > > > Reviewing and testing some bits there now. >From r1718038 [[ --- serf/trunk/protocols/http2_protocol.c (original) +++ serf/trunk/protocols/http2_protocol.c Fri Dec 4 21:35:11 2015 @@ -619,7 +619,7 @@ http2_handle_stream_window_update(void * window_update = (const void *)data; value = (window_update->v3 << 24) | (window_update->v2 << 16) -| (window_update->v2 << 8) | window_update->v0; +| (window_update->v1 << 8) | window_update->v0; value &= HTTP2_WINDOW_MAX_ALLOWED; /* The highest bit is reserved */ ]] This bug (also copied in another location) also explains the unusually small window updates I received earlier... Most likely I missed a few interesting bits in that second byte of the value, while the small window in httpd kept the highest two bytes at 0. The overflow of the max value is resolved by this patch... and the current 2.4.x code goes through the test now without any windowing delays. You might want to change the initial window back to something like 1MB or 16MB now, to at least test cases where a further update is necessary. I certainly don't get connection window updates after the first one now in my usual testing. I don't test uploading over 2 GB in a usual Subversion test run. Bert
Re: No H2 Window updates! (Probably a Serf issue!)
Thanks for the quick update and the good news. > Am 04.12.2015 um 22:46 schrieb Bert Huijben : > > > >> -Original Message- >> From: Bert Huijben [mailto:b...@qqmail.nl] >> Sent: vrijdag 4 december 2015 22:27 >> To: dev@httpd.apache.org >> Subject: RE: No H2 Window updates! (Probably a Serf issue!) >> >> >> >>> -Original Message- >>> From: Bert Huijben [mailto:b...@qqmail.nl] >>> Sent: vrijdag 4 december 2015 21:45 >>> To: dev@httpd.apache.org >>> Subject: RE: No H2 Window updates! >>> >>> >>> >>>> -Original Message- >>>> From: Stefan Eissing [mailto:stefan.eiss...@greenbytes.de] >>>> Sent: vrijdag 4 december 2015 21:36 >>>> To: dev@httpd.apache.org >>>> Subject: Re: No H2 Window updates! >>>> >>>> The code increases the window by the difference between max and >>> current >>>> so that the client has exactly the max value. nghttp2 accepts this on >> the >>> client >>>> side. It rejects any larger value as you described. >>>> >>>> So we seem to have a difference in calculation between nghttp2 and > serf. >>>> which values do you see? some data would be helpful here. >>> >>> The values are completely in the log file at the bottom of this mail. >>> >>> I open a connection and the server announces a default window of 65535. >>> >>>>> I: DBG: Allocated 131 bytes for window on stream 0x1 (left: 65404, >>> 65404) >>> I send a request with a DATA frame with a payload of 131 bytes, which >>> updates the stream and windows size with -131. >>> >>> So both the stream and connection windows are no longer default, but >>> 65404. >>> Logged for easy reading >>> >>> >>> Then I receive a WINDOW_UPDATE frame >>>>> I: DBG: Connection window update of 2147483392 to -2147418500 >>> Which tries to add 2147483392 to the existing window. >>> >>> Which doesn't fit, because the total outgoing window has to fit in >> 2^31-1... >>> See the RFC. >> >> Ok, I think I found a -and provably THE- problem in serf... >> >> There is a bug in decoding the window update frames... >> >> >> Reviewing and testing some bits there now. > > From r1718038 > > [[ > --- serf/trunk/protocols/http2_protocol.c (original) > +++ serf/trunk/protocols/http2_protocol.c Fri Dec 4 21:35:11 2015 > @@ -619,7 +619,7 @@ http2_handle_stream_window_update(void * > window_update = (const void *)data; > > value = (window_update->v3 << 24) | (window_update->v2 << 16) > -| (window_update->v2 << 8) | window_update->v0; > +| (window_update->v1 << 8) | window_update->v0; > > value &= HTTP2_WINDOW_MAX_ALLOWED; /* The highest bit is reserved */ > ]] > > This bug (also copied in another location) also explains the unusually small > window updates I received earlier... Most likely I missed a few interesting > bits in that second byte of the value, while the small window in httpd kept > the highest two bytes at 0. > > > The overflow of the max value is resolved by this patch... and the current > 2.4.x code goes through the test now without any windowing delays. > > > You might want to change the initial window back to something like 1MB or > 16MB now, to at least test cases where a further update is necessary. > > > I certainly don't get connection window updates after the first one now in > my usual testing. I don't test uploading over 2 GB in a usual Subversion > test run. > > >Bert >
