(sorry for the previous reply, wrong account) and sorry I wasn't clear enough: I already use the replace directive for x/net. this works fine for the POC but only when explicitly using a http2 transport. But this doesn't for the net/http because of the h2_bundle thing. see https://www.youtube.com/watch?v=FARQMJndUn0&t=814s So for instance if I change the line golang.org/x/net/http2/transport.go:2418 like you suggested in github it's not working with the default http server unless I rebuild the std lib.
On Friday, November 19, 2021 at 6:34:11 PM UTC+1 ren...@ix.netcom.com wrote: > > Use the replace directive to point the net package to your local copy. > > Much easier to test X changes than stdlib changes. > > On Nov 19, 2021, at 10:58 AM, Kirth Gersen <kirth...@gmail.com> wrote: > > Your CL works well with the POC. > > > Side question not specific to this issue: how to test changes to > golang.org/x/net with net/http ? > The 'h2_bundle' trick with go generate & bundle requires to fork the std > lib too ? > I have a hard time figuring how to do this. I tried with gotip but I get > an error with "gotip generate" (bundle: internal error: package "strings" > without types was imported from "golang.org/x/net/http2") > Any doc/tutorial on how to deal with this 'bundle' trick ? > > thx > Le lundi 15 novembre 2021 à 17:32:48 UTC+1, ren...@ix.netcom.com a écrit : > >> Since http2 multiplexes streams it will delicately affect latency on >> other streams. This is why I suggested using multiple transports - one for >> high throughput transfers and another for lower latency “interactive” >> sessions. >> >> On Nov 15, 2021, at 9:23 AM, Kevin Chowski <ke...@chowski.com> wrote: >> >> These are interesting results, thanks for investigating and sharing >> results! >> >> >> I see that you have mostly been focusing on throughput in your posts, >> have you done testing for latency differences too? >> >> On Saturday, November 13, 2021 at 6:11:40 PM UTC-7 ren...@ix.netcom.com >> wrote: >> >>> As another data point, I decided to test a few implementations of http2 >>> downloads on OSX. >>> >>> Using a Go server with default frame size (16k): >>> >>> Go client: 900 MB/s >>> Java client: 1300 MB/s >>> curl: 1500 MB/s >>> >>> Using a Java server with default frame size (16k): >>> >>> Go client: 670 MB/s >>> Java client: 720 MB/s >>> curl: 800 M/s >>> >>> Using Go server using 256k client max frame size: >>> >>> Go client: 2350 MB/s >>> Java client: 2800 MB/s >>> h2load: 4300 MB/s >>> >>> Using Java server using 256k client max frame size: >>> >>> Go client: 2900 MB/s >>> Java client: 2800 MB/s >>> h2load: 3750 MB/s >>> >>> For h2load, I needed to create a PR to allow the frame size to be set, >>> see https://github.com/nghttp2/nghttp2/pull/1640 >>> >>> >>> On Nov 10, 2021, at 7:04 PM, robert engels <ren...@ix.netcom.com> wrote: >>> >>> No worries. I updated the issue and the CL. I will comment in the CL >>> with a few more details. >>> >>> On Nov 10, 2021, at 2:30 PM, Andrey T. <xnow4f...@sneakemail.com> wrote: >>> >>> Thank you Robert, >>> I somehow missed the reference to the ticket in the first message, sorry >>> about that. >>> >>> As for the CL - I think adding link to the github issue, and add a bit >>> of explanation in a commit message would help. >>> I added link to your CL to the github issue's discussion, hopefully it >>> will bring more attention to it. >>> >>> A. >>> >>> On Wednesday, November 10, 2021 at 1:22:42 PM UTC-7 ren...@ix.netcom.com >>> wrote: >>> >>>> As reported in the OP, the issue was filed long ago >>>> https://github.com/golang/go/issues/47840 >>>> >>>> My CL https://go-review.googlesource.com/c/net/+/362834 is a viable >>>> fix (and should of been supported originally). >>>> >>>> On Nov 10, 2021, at 12:59 PM, Andrey T. <xnow4f...@sneakemail.com> >>>> wrote: >>>> >>>> Fellas, >>>> I would say the 5x throughput difference is a serious problem.Would you >>>> be kind and open an issue on github about it? >>>> Also, the PR that you have might benefit from explanation about what >>>> you are trying to solve (and probably link to an issue on github), so it >>>> would get more attention. >>>> >>>> Thanks! >>>> >>>> Andrey >>>> >>>> >>>> On Tuesday, November 9, 2021 at 4:50:34 PM UTC-7 ren...@ix.netcom.com >>>> wrote: >>>> >>>>> Well, I figured out a way to do it simply. The CL is here >>>>> https://go-review.googlesource.com/c/net/+/362834 >>>>> >>>>> The frame size will be used for all connections using that transport, >>>>> so it is probably better to create a transport specifically for the >>>>> high-throughput transfers. >>>>> >>>>> You can also create perform single shot requests like: >>>>> >>>>> if useH2C { >>>>> rt = &http2.Transport{ >>>>> AllowHTTP: true, >>>>> DialTLS: func(network, addr string, cfg *tls.Config) >>>>> (net.Conn, error) { >>>>> return dialer.Dial(network, addr) >>>>> }, >>>>> MaxFrameSize: 1024*256, >>>>> } >>>>> } >>>>> >>>>> var body io.ReadCloser = http.NoBody >>>>> >>>>> req, err := http.NewRequestWithContext(ctx, "GET", url, body) >>>>> if err != nil { >>>>> return err >>>>> } >>>>> >>>>> resp, err := rt.RoundTrip(req) >>>>> >>>>> >>>>> On Nov 9, 2021, at 3:31 PM, Robert Engels <ren...@ix.netcom.com> >>>>> wrote: >>>>> >>>>> To be clear, I have no plans to submit a Cl to improve this at this >>>>> time. >>>>> >>>>> It would require some api changes to implement properly. >>>>> >>>>> On Nov 9, 2021, at 12:19 PM, Kirth Gersen <kirth...@gmail.com> wrote: >>>>> >>>>> Great ! >>>>> >>>>> > *I made some local mods to the net library, increasing the frame >>>>> size to 256k, and the http2 performance went from 8Gbps to 38Gbps.* >>>>> That is already enormous for us. thx for finding this. >>>>> >>>>> 4 -> Indeed a lot of WINDOW_UPDATE messages are visible when >>>>> using GODEBUG=http2debug=1 >>>>> >>>>> >>>>> On Tuesday, November 9, 2021 at 6:28:16 PM UTC+1 ren...@ix.netcom.com >>>>> wrote: >>>>> >>>>>> I did a review of the codebase. >>>>>> >>>>>> Http2 is a multiplexed protocol with independent streams. The Go >>>>>> implementation uses a common reader thread/routine to read all of the >>>>>> connection content, and then demuxes the streams and passes the data via >>>>>> pipes to the stream readers. This multithreaded nature requires the use >>>>>> of >>>>>> locks to coordinate. By managing the window size, the connection reader >>>>>> should never block writing to a steam buffer - but a stream reader may >>>>>> stall waiting for data to arrive - get descheduled - only to be quickly >>>>>> rescheduled when reader places more data in the buffer - which is >>>>>> inefficient. >>>>>> >>>>>> Out of the box on my machine, http1 is about 37 Gbps, and http2 is >>>>>> about 7 Gbps on my system. >>>>>> >>>>>> Some things that jump out: >>>>>> >>>>>> 1. The chunk size is too small. Using 1MB pushed http1 from 37 Gbs to >>>>>> 50 Gbps, and http2 to 8 Gbps. >>>>>> >>>>>> 2. The default buffer in io.Copy() is too small. Use io.CopyBuffer() >>>>>> with a larger buffer - I changed to 4MB. This pushed http1 to 55 Gbs, >>>>>> and >>>>>> http2 to 8.2. Not a big difference but needed for later. >>>>>> >>>>>> 3. The http2 receiver frame size of 16k is way too small. There is >>>>>> overhead on every frame - the most costly is updating the window. >>>>>> >>>>>> *I made some local mods to the net library, increasing the frame size >>>>>> to 256k, and the http2 performance went from 8Gbps to 38Gbps.* >>>>>> >>>>>> 4. I haven’t tracked it down yet, but I don’t think the window size >>>>>> update code is not working as intended - it seems to be sending window >>>>>> updates (which are expensive due to locks) far too frequently. I think >>>>>> this >>>>>> is the area that could use the most improvement - using some heuristics >>>>>> there is the possibility to detect the sender rate, and adjust the >>>>>> refresh >>>>>> rate (using high/low water marks). >>>>>> >>>>>> 5. The implementation might need improvements using lock-free >>>>>> structures, atomic counters, and busy-waits in order to achieve maximum >>>>>> performance. >>>>>> >>>>>> So 38Gbps for http2 vs 55 Gbps for http1. Better but still not great. >>>>>> Still, with some minor changes, the net package could allow setting of a >>>>>> large frame size on a per stream basis - which would enable much higher >>>>>> throughput. The gRPC library allows this. >>>>>> >>>>>> On Nov 8, 2021, at 10:58 AM, Kirth Gersen <kirth...@gmail.com> wrote: >>>>>> >>>>>> http/2 implementation seems ~5x slower in bytes per seconds (when >>>>>> transfer is cpu capped). >>>>>> >>>>>> POC: https://github.com/nspeed-app/http2issue >>>>>> >>>>>> I submitted an issue about this 3 months ago in the Go Github ( >>>>>> https://github.com/golang/go/issues/47840 ) but first commenter >>>>>> misunderstood it and it got buried (they're probably just swamped with >>>>>> too >>>>>> many open issues (5k+...)). >>>>>> >>>>>> Everything using Golang net/http is impacted, the Caddy web server >>>>>> for instance. >>>>>> >>>>>> I know it probably doesn't matter for most use cases because it's >>>>>> only noticeable with high throughput transfers (>1 Gbps). >>>>>> Most http benchmarks focus on "requests per second" and not "bits per >>>>>> seconds" but this performance matters too sometimes. >>>>>> >>>>>> If anyone with expertise in profiling Go code and good knowledge of >>>>>> the net/http lib internal could take a look. It would be nice to >>>>>> optimize >>>>>> it or at least have an explanation. >>>>>> >>>>>> thx (sorry if wrong group to post this). >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "golang-nuts" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to golang-nuts...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/golang-nuts/89926c2f-ec73-43ad-be49-a8bc76a18345n%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/golang-nuts/89926c2f-ec73-43ad-be49-a8bc76a18345n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "golang-nuts" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to golang-nuts...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/golang-nuts/7332f727-6716-4c4d-85c5-a86cacd0c89fn%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/golang-nuts/7332f727-6716-4c4d-85c5-a86cacd0c89fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "golang-nuts" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to golang-nuts...@googlegroups.com. >>>> >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/golang-nuts/1bfe6aec-abd2-4f63-bf77-bbfa6fd213ban%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/golang-nuts/1bfe6aec-abd2-4f63-bf77-bbfa6fd213ban%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/1b63863b-45af-45d0-a885-8716acc65ac7n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/golang-nuts/1b63863b-45af-45d0-a885-8716acc65ac7n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts...@googlegroups.com. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/3138434b-d480-4473-8b20-2598412a0eden%40googlegroups.com >> >> <https://groups.google.com/d/msgid/golang-nuts/3138434b-d480-4473-8b20-2598412a0eden%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/03b1493a-bf8a-4cd2-b10d-20e2cc57cad9n%40googlegroups.com > > <https://groups.google.com/d/msgid/golang-nuts/03b1493a-bf8a-4cd2-b10d-20e2cc57cad9n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/4b30e2d5-05bb-41bb-9b07-93dd7d3a6727n%40googlegroups.com.
