As another data point, I decided to test a few implementations of http2 downloads on OSX.
Using a Go server with default frame size (16k): Go client: 900 MB/s Java client: 1300 MB/s curl: 1500 MB/s Using a Java server with default frame size (16k): Go client: 670 MB/s Java client: 720 MB/s curl: 800 M/s Using Go server using 256k client max frame size: Go client: 2350 MB/s Java client: 2800 MB/s h2load: 4300 MB/s Using Java server using 256k client max frame size: Go client: 2900 MB/s Java client: 2800 MB/s h2load: 3750 MB/s For h2load, I needed to create a PR to allow the frame size to be set, see https://github.com/nghttp2/nghttp2/pull/1640 > On Nov 10, 2021, at 7:04 PM, robert engels <reng...@ix.netcom.com> wrote: > > No worries. I updated the issue and the CL. I will comment in the CL with a > few more details. > >> On Nov 10, 2021, at 2:30 PM, Andrey T. <xnow4fippy...@sneakemail.com >> <mailto:xnow4fippy...@sneakemail.com>> wrote: >> >> Thank you Robert, >> I somehow missed the reference to the ticket in the first message, sorry >> about that. >> >> As for the CL - I think adding link to the github issue, and add a bit of >> explanation in a commit message would help. >> I added link to your CL to the github issue's discussion, hopefully it will >> bring more attention to it. >> >> A. >> >> On Wednesday, November 10, 2021 at 1:22:42 PM UTC-7 ren...@ix.netcom.com >> <http://ix.netcom.com/> wrote: >> As reported in the OP, the issue was filed long ago >> https://github.com/golang/go/issues/47840 >> <https://github.com/golang/go/issues/47840> >> >> My CL https://go-review.googlesource.com/c/net/+/362834 >> <https://go-review.googlesource.com/c/net/+/362834> is a viable fix (and >> should of been supported originally). >> >> >>> On Nov 10, 2021, at 12:59 PM, Andrey T. <xnow4f...@sneakemail.com >>> <applewebdata://11DC8BA4-6589-42F5-BBFD-BE3933D9DE20>> wrote: >>> >> >>> Fellas, >>> I would say the 5x throughput difference is a serious problem.Would you be >>> kind and open an issue on github about it? >>> Also, the PR that you have might benefit from explanation about what you >>> are trying to solve (and probably link to an issue on github), so it would >>> get more attention. >>> >>> Thanks! >>> >>> Andrey >>> >>> >>> On Tuesday, November 9, 2021 at 4:50:34 PM UTC-7 ren...@ix.netcom.com >>> <http://ix.netcom.com/> wrote: >>> Well, I figured out a way to do it simply. The CL is here >>> https://go-review.googlesource.com/c/net/+/362834 >>> <https://go-review.googlesource.com/c/net/+/362834> >>> >>> The frame size will be used for all connections using that transport, so it >>> is probably better to create a transport specifically for the >>> high-throughput transfers. >>> >>> You can also create perform single shot requests like: >>> >>> if useH2C { >>> rt = &http2.Transport{ >>> AllowHTTP: true, >>> DialTLS: func(network, addr string, cfg *tls.Config) (net.Conn, >>> error) { >>> return dialer.Dial(network, addr) >>> }, >>> MaxFrameSize: 1024*256, >>> } >>> } >>> >>> var body io.ReadCloser = http.NoBody >>> >>> req, err := http.NewRequestWithContext(ctx, "GET", url, body) >>> if err != nil { >>> return err >>> } >>> >>> resp, err := rt.RoundTrip(req) >>> >>> >>>> On Nov 9, 2021, at 3:31 PM, Robert Engels <ren...@ix.netcom.com <>> wrote: >>>> >>>> To be clear, I have no plans to submit a Cl to improve this at this time. >>>> >>>> It would require some api changes to implement properly. >>>> >>>>> On Nov 9, 2021, at 12:19 PM, Kirth Gersen <kirth...@gmail.com <>> wrote: >>>>> >>>>> Great ! >>>>> >>>>> > I made some local mods to the net library, increasing the frame size to >>>>> > 256k, and the http2 performance went from 8Gbps to 38Gbps. >>>>> That is already enormous for us. thx for finding this. >>>>> >>>>> 4 -> Indeed a lot of WINDOW_UPDATE messages are visible when using >>>>> GODEBUG=http2debug=1 >>>>> >>>>> >>>>> On Tuesday, November 9, 2021 at 6:28:16 PM UTC+1 ren...@ix.netcom.com >>>>> <http://ix.netcom.com/> wrote: >>>>> I did a review of the codebase. >>>>> >>>>> Http2 is a multiplexed protocol with independent streams. The Go >>>>> implementation uses a common reader thread/routine to read all of the >>>>> connection content, and then demuxes the streams and passes the data via >>>>> pipes to the stream readers. This multithreaded nature requires the use >>>>> of locks to coordinate. By managing the window size, the connection >>>>> reader should never block writing to a steam buffer - but a stream reader >>>>> may stall waiting for data to arrive - get descheduled - only to be >>>>> quickly rescheduled when reader places more data in the buffer - which is >>>>> inefficient. >>>>> >>>>> Out of the box on my machine, http1 is about 37 Gbps, and http2 is about >>>>> 7 Gbps on my system. >>>>> >>>>> Some things that jump out: >>>>> >>>>> 1. The chunk size is too small. Using 1MB pushed http1 from 37 Gbs to 50 >>>>> Gbps, and http2 to 8 Gbps. >>>>> >>>>> 2. The default buffer in io.Copy() is too small. Use io.CopyBuffer() with >>>>> a larger buffer - I changed to 4MB. This pushed http1 to 55 Gbs, and >>>>> http2 to 8.2. Not a big difference but needed for later. >>>>> >>>>> 3. The http2 receiver frame size of 16k is way too small. There is >>>>> overhead on every frame - the most costly is updating the window. >>>>> >>>>> I made some local mods to the net library, increasing the frame size to >>>>> 256k, and the http2 performance went from 8Gbps to 38Gbps. >>>>> >>>>> 4. I haven’t tracked it down yet, but I don’t think the window size >>>>> update code is not working as intended - it seems to be sending window >>>>> updates (which are expensive due to locks) far too frequently. I think >>>>> this is the area that could use the most improvement - using some >>>>> heuristics there is the possibility to detect the sender rate, and adjust >>>>> the refresh rate (using high/low water marks). >>>>> >>>>> 5. The implementation might need improvements using lock-free structures, >>>>> atomic counters, and busy-waits in order to achieve maximum performance. >>>>> >>>>> So 38Gbps for http2 vs 55 Gbps for http1. Better but still not great. >>>>> Still, with some minor changes, the net package could allow setting of a >>>>> large frame size on a per stream basis - which would enable much higher >>>>> throughput. The gRPC library allows this. >>>>> >>>>> >>>>>> On Nov 8, 2021, at 10:58 AM, Kirth Gersen <kirth...@ <>gmail.com >>>>>> <http://gmail.com/>> wrote: >>>>>> >>>>> >>>>>> http/2 implementation seems ~5x slower in bytes per seconds (when >>>>>> transfer is cpu capped). >>>>>> >>>>>> POC: https://github.com/nspeed-app/http2issue >>>>>> <https://github.com/nspeed-app/http2issue> >>>>>> >>>>>> I submitted an issue about this 3 months ago in the Go Github ( >>>>>> https://github.com/golang/go/issues/47840 >>>>>> <https://github.com/golang/go/issues/47840> ) but first commenter >>>>>> misunderstood it and it got buried (they're probably just swamped with >>>>>> too many open issues (5k+...)). >>>>>> >>>>>> Everything using Golang net/http is impacted, the Caddy web server for >>>>>> instance. >>>>>> >>>>>> I know it probably doesn't matter for most use cases because it's only >>>>>> noticeable with high throughput transfers (>1 Gbps). >>>>>> Most http benchmarks focus on "requests per second" and not "bits per >>>>>> seconds" but this performance matters too sometimes. >>>>>> >>>>>> If anyone with expertise in profiling Go code and good knowledge of the >>>>>> net/http lib internal could take a look. It would be nice to optimize it >>>>>> or at least have an explanation. >>>>>> >>>>>> thx (sorry if wrong group to post this). >>>>>> >>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "golang-nuts" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>>> an email to golang-nuts...@ <>googlegroups.com >>>>>> <http://googlegroups.com/>. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/golang-nuts/89926c2f-ec73-43ad-be49-a8bc76a18345n%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/golang-nuts/89926c2f-ec73-43ad-be49-a8bc76a18345n%40googlegroups.com?utm_medium=email&utm_source=footer>. >>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "golang-nuts" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to golang-nuts...@googlegroups.com <>. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/golang-nuts/7332f727-6716-4c4d-85c5-a86cacd0c89fn%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/golang-nuts/7332f727-6716-4c4d-85c5-a86cacd0c89fn%40googlegroups.com?utm_medium=email&utm_source=footer>. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to golang-nuts...@googlegroups.com >>> <applewebdata://11DC8BA4-6589-42F5-BBFD-BE3933D9DE20>. >> >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/1bfe6aec-abd2-4f63-bf77-bbfa6fd213ban%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/golang-nuts/1bfe6aec-abd2-4f63-bf77-bbfa6fd213ban%40googlegroups.com?utm_medium=email&utm_source=footer>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com >> <mailto:golang-nuts+unsubscr...@googlegroups.com>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/1b63863b-45af-45d0-a885-8716acc65ac7n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/golang-nuts/1b63863b-45af-45d0-a885-8716acc65ac7n%40googlegroups.com?utm_medium=email&utm_source=footer>. > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/4558357B-4022-46DC-8AE6-EBEDDC441B6D%40ix.netcom.com.
