Just an update with some new numbers, dan's changes brought down the time for a full clone from my home connection down to 5m48s (down from 27m39s previously)
well done dan! --Ray On 2022-06-17 3:45 p.m., Dan McGrath wrote: > Just a heads-up that Ray and I took a packet capture on both ends for me. > > Thanks to his capture, I noticed that the SVN server host was not sending TCP > window scale or SACK options. Ultimately this turned out to be due to that > host using the PF firewall "synproxy" state option, which apparently was > causing the TCP header options to be removed, which limited the window size > to 64k. > > I have corrected this change to match the www host, and we did observe some > improvements that were hidden from me before due to my and the studio IPs > special treatment that bypassed this feature. > > While it should help, I also tuned a few sysctl settings for send and receive > buffers. In addition, I noticed that the SVN server is also not sending > gzip/deflate accept headers for the files. No doubt this is going to be a > factor. However, there was a note from 2016 from myself explaining that > Brecht asked for the SVN compression to be disabled due to some client > problems. So, this will require some research, perhaps next week. > > Anyway, hopefully it helps a bit until I can review some more settings. > > Enjoy your weekend everyone! > > > Dan > > On Fri, Jun 17, 2022, 2:22 PM Ray Molenkamp via Bf-committers > <bf-committers@blender.org> wrote: > > =8<==[Editors note]=8<== > I'll be honest I'm 100% convinced mailinator will ruin > the little formatting I have done, I put a copy of this > email on https://developer.blender.org/P3014. > =8<====8<====8<====8<== > > How much as I like the sentiment of "lets move to git will it > solve all these problems" lets be honest here, git.blender.org > <http://git.blender.org>'s > speed is nothing to write home about either it may or may not be > as glitchy as svn, but it still wouldn't be fast. > > from my 300mbit connection in western Canada > > Receiving objects: 6% (135931/2078159), 33.34 MiB | 458.00 KiB/s > > total time for a clone off git.blender.org <http://git.blender.org> 27m39s > > Cloning off https://github.com/blender/blender.git however > > total time for a clone off github.com <http://github.com> 1m 52s > > Now amount of finger pointing to client settings will convince me > it's a client setting at this point, but I'll be honest, I am all > the way in western Canada and that could definitely be a factor. > > sooooo, lets investigate! > > Let’s spin up an AWS EC2 instance in London, I'd say that be close > enough to Amsterdam, and let’s rule out any other CPU or Bandwidth > related factors as well, I threw a few dollars at it and got a > cn5.18xlarge instance (72 CPU’s 192gb memory, 100gbit ethernet) > overkill to do a git/svn test? yes.. yes indeed > > first to rule out the blender.org <http://blender.org> servers lets grab > a ubuntu iso > off an .nl mirror ( > https://mirror.nl.datapacket.com/ubuntu-releases/22.04/ ) > > 2022-06-17 17:13:21 (232 MB/s) - ‘ubuntu-22.04-desktop-amd64.iso’ saved > [3654957056/3654957056] > > all right bandwidth is NOT going to be an issue on this box! > > On to blender stuff: > > cloning from git.blender.org <http://git.blender.org> 2m32.895s > svn checkout of lib/win64_vc15 2m56.388s (iftop said 65Mbit peak rate) > > yowza! > > All right, so in London, safe to say all is well > > lets move closer to my location, us-east-2 Ohio (same instance type and > os) > > grabbing ubuntu from that same mirror > > 2022-06-17 17:31:00 (29.5 MB/s) - ‘ubuntu-22.04-desktop-amd64.iso’ saved > [3654957056/3654957056] > > all right, that has lost quite a bit of steam, but it's > still nothing to sneeze at, just to be sure lets grab > an iso off the Princeton university mirror 120MB/s ok.. > good enough... > > onto cloning blender > > Receiving objects: 5% (104235/2078173), 25.44 MiB | 692.00 KiB/s > > well... that's not looking promising... lets wait it out > > cloning from git.blender.org <http://git.blender.org> 17m23.449s > > sadface... lets try svn next, i lost patience and did not let > it finish.. iftop reported a top RX of 5.59Mbit though.. > > To summarize: > > Ubuntu iso from mirror.nl.datapacket.com > <http://mirror.nl.datapacket.com> (Speed taken from wget) > My home - western Canada - 1.3 MB/s > AWS - London - 232 MB/S > AWS - Ohio - 29.5 MB/s > > Clone of git.blender.org <http://git.blender.org> (time taken from linux > time command) > My home - western Canada - 27m39 > AWS - London - 2m32 > AWS - Ohio - 17m23 > > SVN Clone: (peak RX bitrate taken from iftop) > My home - western Canada - 1.5 Mbit/s > AWS - London - 64 Mbit/s > AWS - Ohio - 5.59 Mbit/s > > Think the only thing we really can conclude is > that being further from the server is leading > to an "unhappy" time for the developer. Given > the fact the EC2 instances were 100% identical > between London and Ohio, it's unlikely to be a > client configuration issue. > > --Ray > > On 2022-06-17 10:10 a.m., Dan McGrath via Bf-committers wrote: > > Hi, > > > > Really, I would need to see a capture from both sides to dig into it. I > > need to see what is being sent, and what is received on each side. There > > could be some weird firewall stuff happening that is causing > fragmentation > > or blocking TCP options, or window scaling issues, etc. > > > > As for the default settings for tortoisesvn, the connection stuff I was > > referring to was about TCP/IP behaviour, rather than an application > > setting. Either way, I need to see what is actually happening. The fact > > that I can pull 5-8MB/sec (~40mbit) from Canada and not have a single > > interruption is telling. Granted, the host only has around 100 or > 200mbit > > allocated, iirc. Multiply that by a few dozen people and you have a > problem. > > > > For a simpler test, maybe try a large svn check, perhaps some old > FreeBSD > > ports svn repo if you can find one, something also in .NL, and see how > it > > goes, just to help eliminate your PC/firewall. Worst case scenario, you > can > > also poke me in chat and we can try some server side and client side > > captures. > > > > TBH though, 1gig uplink just isn't a whole lot to serve our user base ;) > > > > On Fri, Jun 17, 2022 at 11:45 AM Tom M <letter...@gmail.com> wrote: > > > >> On Fri, Jun 17, 2022 at 5:20 AM Dan McGrath <danmcgrath...@gmail.com> > >> wrote: > >>> Hi Tom, > >>> > >>> Long time no see :) > >>> > >>> I did some tests from home and found that, aside from slow speeds, I > was > >> able to checkout at least 5 or 6 gig of the bf-blender repo without > error > >> or having to retransmit. > >> > >> Yeah, looks like I was mistaking really slow download of the larger > >> (.7 GB) libraries as freezing. I was averaging about .3 MB/s. I used > >> a different SVN client and saw that it was indeed downloading (the > >> sliksvn doesn't provide any feedback on files in progress, only once a > >> file completes) - It was taking about 40 minutes for each of the > >> larger files. It did complete eventually after about 5 hours. > >> > >>> I took the liberty of packet capturing the stream from my PC's POV, > and > >> placed them in the ftp download area[1], if you care to take a look. > >>> The gist is that I was able to get around 5MB/sec for at least 5G of > the > >> checkout, before I cancelled it. > >> Interesting, I usually get up to 3 MB/s download here normally from > >> other sources, but was typically getting .3 MB/s, with occasional > >> bursts of higher speed to a max of .8 MB/s when it wasn't downloading > >> the large files. > >> > >>> Now, my IP is a bit of a special case, so I am wondering if you are > just > >> hitting the server so hard with http requests in separate sockets, or > >> something extreme, that is causing the system to block you for a > period of > >> time. For comparison, my PC only had about 4 sockets open to the > server for > >> the transfer. I would be curious to know if you are doing pipelining, > or > >> sending each request in a separate connection. > >> Whatever the sliksvn and tortoisesvn defaults are, I'd assume they are > >> setup for typical usage. > >> > >>> If you can, the next time it happens, see if you are unable to visit > >> Phabricator or resume the svn checkout process. It may not be a > firewall > >> issue, but without seeing some packet captures, I wouldn't be able to > >> really guess. Ideally, try to capture a few packets, until failure. At > the > >> very least, you will need to snap 20b+20b=40b for IP+TCP headers, but I > >> would suggest a tad higher, maybe 64b. > >>> Cheers, > >>> > >>> Dan > >>> > >>> [1] - https://download.blender.org/ftp/dan/svn-checkout/ > >>> > >>> > >>> > >>> On Thu, Jun 16, 2022 at 2:43 PM Tom M via Bf-committers < > >> bf-committers@blender.org> wrote: > >>>> When checking out the libraries from SVN following the directions on > >>>> > >>>> https://wiki.blender.org/wiki/Building_Blender/Windows > >>>> > >>>> The svn checkout of the libraries fails with significant frequency, > >>>> > >>>> if you google you might stumble across using make svnfix > >>>> > >>>> > https://www.mail-archive.com/bf-blender-cvs@blender.org/msg151996.html > >>>> > >>>> Even after using it, I'm still having to do Ctrl C, and then make > >>>> svnfix repeatedly (it downloads some files, than randomly freezes, > >>>> sometimes after 20-50 files, sometimes after 100's of files). > >>>> > >>>> Git doesn't seem to have any problems, though. > >>>> > >>>> So might we consider migrating the libs to git, rather than using > SVN? > >>>> > >>>> LetterRip > >>>> _______________________________________________ > >>>> Bf-committers mailing list > >>>> Bf-committers@blender.org > >>>> List details, subscription details or unsubscribe: > >>>> https://lists.blender.org/mailman/listinfo/bf-committers > >>> > >>> > >>> -- > >>> Cheers, > >>> Danny > >>> > >>> ---------------------------------------------------------- > >>> Danny McGrath - danmcgrath...@gmail.com > >>> GPG key: EDF6 AFF5 2086 F93A 1F59 36A5 44B6 26F3 6968 71CA > > > _______________________________________________ > Bf-committers mailing list > Bf-committers@blender.org > List details, subscription details or unsubscribe: > https://lists.blender.org/mailman/listinfo/bf-committers > _______________________________________________ Bf-committers mailing list Bf-committers@blender.org List details, subscription details or unsubscribe: https://lists.blender.org/mailman/listinfo/bf-committers
