Just to give some numbers from my laptop, testing against localhost:
httpd mpm worker, OS X, client h2load, transfer rates in GByte/s
Version #requests 2.4.18 2.4.x
1 GB resource
h2c clear 1*1 0.017 1.0
8*1 2.2
h2 tls 1*1 0.430 0.425
8*1 0.75
10 MB resource
h2c clear 1*100 1.0 0.94
8*100 2.4 2.4
h2 tls 1*100 0.425 0.418
8*100 0.8 0.8
There is a clear bug in 2.4.18 for the cleartext case where client/server are
stalling and 0% CPU is used. With TLS or with smaller resources, this is not
observable. This has been fixed in the upcoming 2.4.19 release.
-Stefan
> Am 11.03.2016 um 12:40 schrieb Molina <jose.mol...@cern.ch>:
>
> Hello again,
>
> I installed the newest version now (2.4.18), but I still get the same results.
>
> Answering by parts:
> 1. I do not use SSL
> 2. I have tried now with h2o and nghttpd. Obtained results are explained below
> 3. Increasing or decreasing the number of concurrent streams does not
> significantly modify the transfer time as long as it remains in a decent
> number of files (say, minimum 100)
> 4. Varying the size of each stream did not significantly modify the transfer
> time
>
> I will now explain the results I’ve observed when trying with other
> server/clients:
>
> nghttpd
> I started to suspect this issue is related to the window size, so I run
> nghttp with a window size of 2^16 (64KB, the default one). I got the
> following sample output in the client when downloading a 1GB-file:
>
> [ 12.087] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 12.089] recv DATA frame <length=16383, flags=0x00, stream_id=13>
> [ 12.089] send WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
> (window_size_increment=32767)
> [ 12.089] send WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=13>
> (window_size_increment=32767)
>
> So it seems it’s waiting for window update requests often, which causes a
> huge delay with such a big distance between server and client. The download
> speed was 450KB/s on average.
>
> Then I increased the TCP buffer in both client and server to 2MB. I then
> tried with a HTTP/2 per-connection windows size of 2^21 (2MB) I got the
> following (repeated many times until the file was downloaded):
>
> [ 8.005] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.006] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.095] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.096] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.102] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.128] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.130] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.131] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.134] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.139] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.140] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.142] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.143] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.231] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.232] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.237] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.260] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.266] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.267] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.269] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.276] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.277] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.278] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.280] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.367] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.370] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.371] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.397] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.402] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.403] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.405] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.413] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.414] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.416] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.418] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.502] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.505] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.506] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.511] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.537] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.543] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.544] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.549] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.550] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.551] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.553] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.637] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.641] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.643] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.646] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.668] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.678] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.680] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 8.680] send WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
> (window_size_increment=1048576)
> [ 8.680] send WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=13>
> (window_size_increment=1048576)
>
> Which allows the server to send much more data before waiting for a window
> update request, hence increasing the download speed up to 50MB/s. It
> concretely sends 53 packages of ~16KB, which is ~1MB, which is half of the
> window. I happened the same in the previous case.
>
>
> h2o
> I performed the following test:
>
> Download a big file (1GB): both protocols had the same download speed, which
> was expected.
> Download in parallel 10000 10KB files: in HTTP/1.1 it took 14 seconds, while
> in HTTP/2 it took 7.
>
> Hence in long distances and with many parallel downloads HTTP/2 performed
> better, which is expected.
> However, when performing both tests it was necessary to use the nghttp
> utility to increase the per-connection window size to 16MB and the per-stream
> window size to 1MB, having 16 parallel streams.
>
>
>
> Back to Apache, I tried nghttp wit a per-connection window size of 2^17. Then
> I tried to download the 1GB-file used in the previous test. For my own
> surprise, the server (and client!) hanged at some point, provoking a timeout.
> Here’s the full nghttp log:
>
> [ 0.137] Connected
> [ 0.138] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
> (niv=2)
> [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
> [SETTINGS_INITIAL_WINDOW_SIZE(0x04):131071]
> [ 0.138] send PRIORITY frame <length=5, flags=0x00, stream_id=3>
> (dep_stream_id=0, weight=201, exclusive=0)
> [ 0.138] send PRIORITY frame <length=5, flags=0x00, stream_id=5>
> (dep_stream_id=0, weight=101, exclusive=0)
> [ 0.138] send PRIORITY frame <length=5, flags=0x00, stream_id=7>
> (dep_stream_id=0, weight=1, exclusive=0)
> [ 0.138] send PRIORITY frame <length=5, flags=0x00, stream_id=9>
> (dep_stream_id=7, weight=1, exclusive=0)
> [ 0.138] send PRIORITY frame <length=5, flags=0x00, stream_id=11>
> (dep_stream_id=3, weight=1, exclusive=0)
> [ 0.138] send WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
> (window_size_increment=65536)
> [ 0.138] send HEADERS frame <length=69, flags=0x25, stream_id=13>
> ; END_STREAM | END_HEADERS | PRIORITY
> (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
> ; Open new stream
> :method: GET
> :path: /cvmfs/test.http2/test/1MB/FOO
> :scheme: http
> :authority: hcc-anvil-175-13.unl.edu
> accept: */*
> accept-encoding: gzip, deflate
> user-agent: nghttp2/1.7.1
> [ 0.275] recv SETTINGS frame <length=12, flags=0x00, stream_id=0>
> (niv=2)
> [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):1000]
> [SETTINGS_INITIAL_WINDOW_SIZE(0x04):2097151]
> [ 0.275] recv WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
> (window_size_increment=2145386496)
> [ 0.275] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
> ; ACK
> (niv=0)
> [ 0.275] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
> ; ACK
> (niv=0)
> [ 0.275] recv (stream_id=13) :status: 200
> [ 0.275] recv (stream_id=13) date: Wed, 09 Mar 2016 11:54:44 GMT
> [ 0.275] recv (stream_id=13) server: Apache/2.4.18 (Ubuntu)
> [ 0.276] recv (stream_id=13) accept-ranges: bytes
> [ 0.276] recv (stream_id=13) content-length: 1048576000
> [ 0.276] recv (stream_id=13) cache-control: max-age=259200
> [ 0.276] recv (stream_id=13) expires: Sat, 12 Mar 2016 11:54:44 GMT
> [ 0.276] recv HEADERS frame <length=96, flags=0x04, stream_id=13>
> ; END_HEADERS
> (padlen=0)
> ; First response header
> [ 0.410] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 0.412] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 0.546] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 0.547] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 0.547] send WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
> (window_size_increment=65536)
> [ 0.547] send WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=13>
> (window_size_increment=65536)
> [ 0.548] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 0.680] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 0.681] recv DATA frame <length=16384, flags=0x00, stream_id=13>
> [ 0.683] recv DATA frame <length=16383, flags=0x00, stream_id=13>
> [ 0.683] send WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
> (window_size_increment=65535)
> [ 0.683] send WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=13>
> (window_size_increment=65535)
> [ERROR] Timeout
>
>
>
> And the apache debug log:
>
> [Thu Mar 10 10:07:11.576838 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_conn.c(147): [client 128.141.155.247:51137]
> h2_conn_process start
> [Thu Mar 10 10:07:11.577041 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_conn.c(172): [client 128.141.155.247:51137]
> h2_session(8): starting on 127.0.0.1:80
> [Thu Mar 10 10:07:11.577129 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send SETTINGS[length=12, stream=0]
> [Thu Mar 10 10:07:11.577185 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send WINDOW_UPDATE[length=4, stream=0]
> [Thu Mar 10 10:07:11.577272 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_stream.c(168): [client 128.141.155.247:51137]
> h2_stream(8-13): opened
> [Thu Mar 10 10:07:11.577374 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_stream.c(313): [client 128.141.155.247:51137]
> h2_stream(8-13): scheduled GET
> http://hcc-anvil-175-13.unl.edu/cvmfs/test.http2/test/1MB/FOO
> [Thu Mar 10 10:07:11.577711 2016] [http2:debug] [pid 5363:tid
> 140323217999616] h2_mplx.c(703): [client 128.141.155.247:51137]
> h2_mplx(8-13): write with trailers=no
> [Thu Mar 10 10:07:11.577800 2016] [http2:debug] [pid 5363:tid
> 140323217999616] h2_mplx.c(703): [client 128.141.155.247:51137]
> h2_mplx(8-13): write with trailers=no
> [Thu Mar 10 10:07:11.577853 2016] [http2:debug] [pid 5363:tid
> 140323217999616] h2_mplx.c(749): [client 128.141.155.247:51137]
> h2_mplx(8-13): close with trailers=no
> [Thu Mar 10 10:07:11.577925 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_stream.c(234): [client 128.141.155.247:51137]
> h2_stream(8-13): set_response(200)
> [Thu Mar 10 10:07:11.578000 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(1231): [client 128.141.155.247:51137]
> h2_stream(8-13): submit response 200
> [Thu Mar 10 10:07:11.578039 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send SETTINGS[ack=1, stream=0]
> [Thu Mar 10 10:07:11.578118 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send HEADERS[length=96, hend=1, stream=13, eos=0]
> [Thu Mar 10 10:07:11.578147 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16384, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:11.578171 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16384, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:11.578193 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16384, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:11.578215 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16384, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:11.578237 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16384, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:11.578259 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16384, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:11.578281 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16384, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:11.578416 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16383, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:11.983733 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16384, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:11.983885 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16384, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:11.983978 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16384, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:11.984034 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16384, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:12.119071 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16384, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:12.119156 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16384, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:12.119240 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16384, flags=0, stream=13, padlen=0]
> [Thu Mar 10 10:07:12.119282 2016] [http2:debug] [pid 5363:tid
> 140322990438144] h2_session.c(590): [client 128.141.155.247:51137]
> h2_session(8): frame_send DATA[length=16383, flags=0, stream=13, padlen=0]
>
>
> The timeout I used was 30 seconds, which should be more than enough.
>
> Is there a bug in Apache, or do I have to tune the HTTP/2-related Apache
> configuration?
>
> Cheers,
> José
>
>
>> On 08 Mar 2016, at 14:34, Stefan Eissing <stefan.eiss...@greenbytes.de>
>> wrote:
>>
>> José,
>>
>> there is a newer version 2.4.18 of Apache httpd which is be better suited
>> for performance testing. 2.4.17 was the first release of the module and much
>> has happened since.
>>
>> Test throughput here on localhost, the limit of http/1.1 and http/2 on
>> Apache are several gigabytes/s, without any super special configuration.
>>
>> It is interesting to know how latency affects this and I would like to
>> see results on this.
>>
>> As to the calibration of your test client, I recommend getting alternate
>> figures from servers such has h2o and nghttpd and also using other clients
>> like the h2load which is part of the nghttp2 distribution. That way, you
>> can check where difference in client implementation might affect your
>> performance results.
>>
>> Cheers,
>>
>> Stefan
>>
>>> Am 08.03.2016 um 14:05 schrieb Joshua Marantz <jmara...@google.com.INVALID>:
>>>
>>> That's a great benchmark, José. A couple of questions:
>>> 1. When using http 1.1, is your connection over SSL? I assume that your h2
>>> connection is over ssl, and that might account for a speed penalty.
>>> 2. Have you tried this on other h2 implementations (h2o, nginx)?
>>> 3. Can you take some other data points? If you just have 100 shards rather
>>> than 1000, how does that affect the results?
>>> 4. Can you also vary the size of each shard?
>>>
>>> Publishing your results in a more comprehensive study would be quite
>>> valuable.
>>>
>>> -Josh
>>>
>>> On Tue, Mar 8, 2016 at 6:33 AM, Molina <jose.mol...@cern.ch> wrote:
>>>
>>>> Hello,
>>>>
>>>> I’m testing the HTTP/2 performance to add support to the software I’m
>>>> working for. My benchmark consist on an apache 2.4.17 server with the
>>>> following http2-related configuration:
>>>>
>>>> LoadModule http2_module modules/mod_http2.so
>>>> <IfModule http2_module>
>>>> Protocols h2c http/1.1
>>>> H2WindowSize 1460000000
>>>> H2StreamMaxMemSize 1460000000
>>>> H2MinWorkers 8
>>>> H2Direct on
>>>> LogLevel http2:info
>>>> H2MaxSessionStreams 100
>>>> H2SessionExtraFiles 1000
>>>> H2MaxWorkers 100
>>>> </IfModule>
>>>>
>>>> The rest is simply the default apache configuration.
>>>> My client is a C application that downloads 1024 files of exactly 1MiB
>>>> each, so totally 1GB. When performing the test with normal HTTP/1.1 the
>>>> download speed is around 20MiB/s, while when using HTTP/2 is 1.5MiB/s. The
>>>> client uses exactly one multiplexed connection and works as expected
>>>> Lastly, the client and server are located in center Europe and center USA
>>>> respectively to increase the impact of opening many HTTP/1.1 connections
>>>> with such a high physical distance due to latency.
>>>>
>>>> I tried to increase the Linux window size, as well as the TCP buffer to
>>>> increase the download speed, but without success so far.
>>>>
>>>> Is there some configuration parameter I’m missing to make it perform
>>>> better? Or perhaps the module itself needs an special configuration for
>>>> such a use case?
>>>>
>>>> Thanks a lot in advance. Best regards,
>>>> José Molina
>>>>
>>>>
>>
>
