Bob, Thanks for the detailed response. Since that report things have been quite stable, so the changes you describe have definitely made a difference.
Also, when I do things to/from Savannah, it's across the pond. I expect it to be less instantaneous than when I'm on the same continent. But, I know intuitively what "normal" is for me, and what's not, and I only report when things aren't normal. Anyway, things are just fine at the moment. I really appreciate all that you, and the all the other Savannah Hackers, do to keep the GNU infrastructure up and running. I know it's not trivial. Thanks, Arnold Bob Proulx <[email protected]> wrote: > Hi Arnold, > > > Pushing to and pulling from the Savannah Git repo for gawk (using ssh) > > is running really slow right now. It's 11:00 AM Israel time, equals > > 4:00 AM US East Coast time, if anyone who can do anything about it > > is awake right now. > > Your message was from 29-May and today I am reading through these > messages. Life and time is what keeps everything from happening all > at once. Since 26-May though things have been significantly better. > > On 26-May we activated a new mirror collective to offload the CGIT and > GITWEB traffic from the primary onto the mirrors. > > https://lists.gnu.org/archive/html/savannah-users/2025-05/msg00002.html > > This is not something you need to remember or know about because on > the primary an http redirect was installed that redirects all traffic > for cgit and gitweb over to the mirrors. It will happen to you > whether you know about it or not. The primary is no longer serving > cgit and no longer serving gitweb. It just can't. > > I have been running routine testing and I no longer see showstopper > problems with using git.savannah.gnu.org anymore. I am testing from > my location in Colorado across the Internet to the GNU servers in > Boston. Load on the primary server is hugely reduced. I set up a few > system metric graphs on the top level page to make the system load > visible. > > https://git.savannah.gnu.org/ > > Looking at those graphs when things are bad should show if there is a > loading spike problem. And it could also be the network between. > > Does that mean you are not still seeing problems? No. I believe you! > If you say things are very slow then I am sure they are very slow. > It's just that it took some work and we reduced the average load on > the server from a 24x7 max'd out load average of 38 (it would be > higher if we had more resources there) where things were definitely > having problems and network connections were getting dropped down to a > load of 1-2 where things have been so much better that it was time for > us to stop and breath for a moment. That was a big architectural > change. It really had a big effect. When I poll things now I find > them to be very usable. > > I am not usually looking at the server at the 4am time that you are > reporting problems. That's 2am for me. That's a time when very > typically daily tasks are run. That's a time when very typically > backups are running. I know that one backup is running at about 0700 > hours and another at about 1900 hours with some variation in when the > serialized list of machines gets to it. Additionally another new > backup from a new backup system has been added and I don't know when > that one is triggered. After it has been proven reliable we can > decommission the previous one. > > Additionally the git primary server is a VM running along side 24 > total virtual machines on that kvm hosting system. Load at the same > time on all of the others would need to be examined. Other load that > is invisible will also be affecting the git server. > > There is still a lot of possibilities for problems to be occurring. > > Bob > > Here is some testing that I just performed on the read-only protocols > doing a full depth clone of gawk.git from both the primary and from > the read-only mirror collective. This is cloning from Boston to my > location in Colorado across the Internet. > > primary ssh:// member access > > rwp@madness:~/tmp/junk$ time git clone > ssh://git.savannah.gnu.org/srv/git/gawk.git > Cloning into 'gawk'... > remote: Counting objects: 63581, done. > remote: Compressing objects: 100% (16738/16738), done. > remote: Total 63581 (delta 50154), reused 58199 (delta 46188) > Receiving objects: 100% (63581/63581), 67.99 MiB | 2.63 MiB/s, done. > Resolving deltas: 100% (50154/50154), done. > > real 1m0.322s > user 0m42.613s > sys 0m0.987s > > primary git:// > > rwp@madness:~/tmp/junk$ rm -rf gawk > rwp@madness:~/tmp/junk$ time git clone git://git.savannah.gnu.org/gawk.git > Cloning into 'gawk'... > remote: Counting objects: 63581, done. > remote: Compressing objects: 100% (16738/16738), done. > remote: Total 63581 (delta 50157), reused 58199 (delta 46188) > Receiving objects: 100% (63581/63581), 66.50 MiB | 822.00 KiB/s, done. > Resolving deltas: 100% (50157/50157), done. > > real 2m32.166s > user 0m42.513s > sys 0m0.923s > > mirrored git:// > > rwp@madness:~/tmp/junk$ rm -rf gawk > rwp@madness:~/tmp/junk$ time git clone > git://git.git.savannah.gnu.org/gawk.git > Cloning into 'gawk'... > remote: Enumerating objects: 63581, done. > remote: Counting objects: 100% (63581/63581), done. > remote: Compressing objects: 100% (16738/16738), done. > remote: Total 63581 (delta 50161), reused 58198 (delta 46188), > pack-reused 0 > Receiving objects: 100% (63581/63581), 65.48 MiB | 12.18 MiB/s, done. > Resolving deltas: 100% (50161/50161), done. > > real 0m29.791s > user 0m42.477s > sys 0m0.710s > > primary http:// > > rwp@madness:~/tmp/junk$ rm -rf gawk > rwp@madness:~/tmp/junk$ time git clone > http://git.savannah.gnu.org/git/gawk.git > Cloning into 'gawk'... > remote: Counting objects: 63581, done. > remote: Compressing objects: 100% (16738/16738), done. > remote: Total 63581 (delta 50154), reused 58198 (delta 46188) > Receiving objects: 100% (63581/63581), 67.88 MiB | 1.05 MiB/s, done. > Resolving deltas: 100% (50154/50154), done. > > real 1m35.765s > user 0m42.726s > sys 0m1.073s > > mirrored http:// > > rwp@madness:~/tmp/junk$ rm -rf gawk > rwp@madness:~/tmp/junk$ time git clone > http://http.git.savannah.gnu.org/git/gawk.git > Cloning into 'gawk'... > remote: Enumerating objects: 63581, done. > remote: Counting objects: 100% (63581/63581), done. > remote: Compressing objects: 100% (16738/16738), done. > remote: Total 63581 (delta 50159), reused 58198 (delta 46188), > pack-reused 0 > Receiving objects: 100% (63581/63581), 65.90 MiB | 8.62 MiB/s, done. > Resolving deltas: 100% (50159/50159), done. > > real 0m33.034s > user 0m42.395s > sys 0m0.765s > > primary https:// > > rwp@madness:~/tmp/junk$ rm -rf gawk > rwp@madness:~/tmp/junk$ time git clone > https://git.savannah.gnu.org/git/gawk.git > Cloning into 'gawk'... > remote: Counting objects: 63581, done. > remote: Compressing objects: 100% (16738/16738), done. > remote: Total 63581 (delta 50155), reused 58199 (delta 46188) > Receiving objects: 100% (63581/63581), 67.96 MiB | 839.00 KiB/s, done. > Resolving deltas: 100% (50155/50155), done. > > real 1m55.001s > user 0m42.710s > sys 0m1.051s > > mirrored http:// > > rwp@madness:~/tmp/junk$ rm -rf gawk > rwp@madness:~/tmp/junk$ time git clone > https://https.git.savannah.gnu.org/git/gawk.git > Cloning into 'gawk'... > remote: Enumerating objects: 63581, done. > remote: Counting objects: 100% (63581/63581), done. > remote: Compressing objects: 100% (16738/16738), done. > remote: Total 63581 (delta 50162), reused 58198 (delta 46188), > pack-reused 0 > Receiving objects: 100% (63581/63581), 64.94 MiB | 12.25 MiB/s, done. > Resolving deltas: 100% (50162/50162), done. > > real 0m27.460s > user 0m42.406s > sys 0m0.762s > > Time now: > > rwp@madness:~/tmp/junk$ date -R > Fri, 06 Jun 2025 15:48:54 -0600 > rwp@madness:~/tmp/junk$ date -Ru > Fri, 06 Jun 2025 21:49:38 +0000
