Hi Arnold,
> Pushing to and pulling from the Savannah Git repo for gawk (using ssh)
> is running really slow right now. It's 11:00 AM Israel time, equals
> 4:00 AM US East Coast time, if anyone who can do anything about it
> is awake right now.
Your message was from 29-May and today I am reading through these
messages. Life and time is what keeps everything from happening all
at once. Since 26-May though things have been significantly better.
On 26-May we activated a new mirror collective to offload the CGIT and
GITWEB traffic from the primary onto the mirrors.
https://lists.gnu.org/archive/html/savannah-users/2025-05/msg00002.html
This is not something you need to remember or know about because on
the primary an http redirect was installed that redirects all traffic
for cgit and gitweb over to the mirrors. It will happen to you
whether you know about it or not. The primary is no longer serving
cgit and no longer serving gitweb. It just can't.
I have been running routine testing and I no longer see showstopper
problems with using git.savannah.gnu.org anymore. I am testing from
my location in Colorado across the Internet to the GNU servers in
Boston. Load on the primary server is hugely reduced. I set up a few
system metric graphs on the top level page to make the system load
visible.
https://git.savannah.gnu.org/
Looking at those graphs when things are bad should show if there is a
loading spike problem. And it could also be the network between.
Does that mean you are not still seeing problems? No. I believe you!
If you say things are very slow then I am sure they are very slow.
It's just that it took some work and we reduced the average load on
the server from a 24x7 max'd out load average of 38 (it would be
higher if we had more resources there) where things were definitely
having problems and network connections were getting dropped down to a
load of 1-2 where things have been so much better that it was time for
us to stop and breath for a moment. That was a big architectural
change. It really had a big effect. When I poll things now I find
them to be very usable.
I am not usually looking at the server at the 4am time that you are
reporting problems. That's 2am for me. That's a time when very
typically daily tasks are run. That's a time when very typically
backups are running. I know that one backup is running at about 0700
hours and another at about 1900 hours with some variation in when the
serialized list of machines gets to it. Additionally another new
backup from a new backup system has been added and I don't know when
that one is triggered. After it has been proven reliable we can
decommission the previous one.
Additionally the git primary server is a VM running along side 24
total virtual machines on that kvm hosting system. Load at the same
time on all of the others would need to be examined. Other load that
is invisible will also be affecting the git server.
There is still a lot of possibilities for problems to be occurring.
Bob
Here is some testing that I just performed on the read-only protocols
doing a full depth clone of gawk.git from both the primary and from
the read-only mirror collective. This is cloning from Boston to my
location in Colorado across the Internet.
primary ssh:// member access
rwp@madness:~/tmp/junk$ time git clone
ssh://git.savannah.gnu.org/srv/git/gawk.git
Cloning into 'gawk'...
remote: Counting objects: 63581, done.
remote: Compressing objects: 100% (16738/16738), done.
remote: Total 63581 (delta 50154), reused 58199 (delta 46188)
Receiving objects: 100% (63581/63581), 67.99 MiB | 2.63 MiB/s, done.
Resolving deltas: 100% (50154/50154), done.
real 1m0.322s
user 0m42.613s
sys 0m0.987s
primary git://
rwp@madness:~/tmp/junk$ rm -rf gawk
rwp@madness:~/tmp/junk$ time git clone git://git.savannah.gnu.org/gawk.git
Cloning into 'gawk'...
remote: Counting objects: 63581, done.
remote: Compressing objects: 100% (16738/16738), done.
remote: Total 63581 (delta 50157), reused 58199 (delta 46188)
Receiving objects: 100% (63581/63581), 66.50 MiB | 822.00 KiB/s, done.
Resolving deltas: 100% (50157/50157), done.
real 2m32.166s
user 0m42.513s
sys 0m0.923s
mirrored git://
rwp@madness:~/tmp/junk$ rm -rf gawk
rwp@madness:~/tmp/junk$ time git clone
git://git.git.savannah.gnu.org/gawk.git
Cloning into 'gawk'...
remote: Enumerating objects: 63581, done.
remote: Counting objects: 100% (63581/63581), done.
remote: Compressing objects: 100% (16738/16738), done.
remote: Total 63581 (delta 50161), reused 58198 (delta 46188), pack-reused 0
Receiving objects: 100% (63581/63581), 65.48 MiB | 12.18 MiB/s, done.
Resolving deltas: 100% (50161/50161), done.
real 0m29.791s
user 0m42.477s
sys 0m0.710s
primary http://
rwp@madness:~/tmp/junk$ rm -rf gawk
rwp@madness:~/tmp/junk$ time git clone
http://git.savannah.gnu.org/git/gawk.git
Cloning into 'gawk'...
remote: Counting objects: 63581, done.
remote: Compressing objects: 100% (16738/16738), done.
remote: Total 63581 (delta 50154), reused 58198 (delta 46188)
Receiving objects: 100% (63581/63581), 67.88 MiB | 1.05 MiB/s, done.
Resolving deltas: 100% (50154/50154), done.
real 1m35.765s
user 0m42.726s
sys 0m1.073s
mirrored http://
rwp@madness:~/tmp/junk$ rm -rf gawk
rwp@madness:~/tmp/junk$ time git clone
http://http.git.savannah.gnu.org/git/gawk.git
Cloning into 'gawk'...
remote: Enumerating objects: 63581, done.
remote: Counting objects: 100% (63581/63581), done.
remote: Compressing objects: 100% (16738/16738), done.
remote: Total 63581 (delta 50159), reused 58198 (delta 46188), pack-reused 0
Receiving objects: 100% (63581/63581), 65.90 MiB | 8.62 MiB/s, done.
Resolving deltas: 100% (50159/50159), done.
real 0m33.034s
user 0m42.395s
sys 0m0.765s
primary https://
rwp@madness:~/tmp/junk$ rm -rf gawk
rwp@madness:~/tmp/junk$ time git clone
https://git.savannah.gnu.org/git/gawk.git
Cloning into 'gawk'...
remote: Counting objects: 63581, done.
remote: Compressing objects: 100% (16738/16738), done.
remote: Total 63581 (delta 50155), reused 58199 (delta 46188)
Receiving objects: 100% (63581/63581), 67.96 MiB | 839.00 KiB/s, done.
Resolving deltas: 100% (50155/50155), done.
real 1m55.001s
user 0m42.710s
sys 0m1.051s
mirrored http://
rwp@madness:~/tmp/junk$ rm -rf gawk
rwp@madness:~/tmp/junk$ time git clone
https://https.git.savannah.gnu.org/git/gawk.git
Cloning into 'gawk'...
remote: Enumerating objects: 63581, done.
remote: Counting objects: 100% (63581/63581), done.
remote: Compressing objects: 100% (16738/16738), done.
remote: Total 63581 (delta 50162), reused 58198 (delta 46188), pack-reused 0
Receiving objects: 100% (63581/63581), 64.94 MiB | 12.25 MiB/s, done.
Resolving deltas: 100% (50162/50162), done.
real 0m27.460s
user 0m42.406s
sys 0m0.762s
Time now:
rwp@madness:~/tmp/junk$ date -R
Fri, 06 Jun 2025 15:48:54 -0600
rwp@madness:~/tmp/junk$ date -Ru
Fri, 06 Jun 2025 21:49:38 +0000
