Re: gcc git locked out for hours second day in a row
Hello, Le 12/06/2024 à 16:57, Jakub Jelinek a écrit : On Wed, Jun 12, 2024 at 04:53:38PM +0200, Mikael Morin wrote: Perhaps you could create a mirror version of the repo and do some experiments locally on that to identify where the bottle-neck is coming from? Not sure where to start for that.Are the hooks published somewhere? Yes: https://github.com/AdaCore/git-hooks/tree/master Note, we use some tweaks on top of that, but that is mostly for the release branches and trunk, so it would be interesting to just try to reproduce that with the stock AdaCore git hooks. I have finally taken some time to investigate this hook slowness, and here are my findings. My tests were run with configs commit-extra-checker and commit-email-formatter disabled, and hooks.update-hook set to a minimal script (either "true" or "sleep 1"). With that config, I could not reproduce the slowness pushing to refs/users/mikael/*. The push finishes in less than a minute. However, trying to push to a normal tag, there is some email count check coming into play, and I can reproduce some slowness (details below). This email count check shouldn't happen on the gcc repository in my use case (as email checks don't apply to user references), but the slowness could well happen in other cases than email count check depending on the configuration, as the problem relates to the size of the list of new commits and is not restricted to email count. Anyway, even with email count check triggering, each tag takes less than 2 minutes to be rejected in my test. With 330 tags to process, that would make an upper bound of 11 hours before rejecting the push in my test (I killed it after a few minutes). On the other hand, with the information you gave upthread, the hook on the gcc repository seemed to be still processing the first tag after a few hours (assuming they are processed in alphabetical order, which seems to be the case). So this still doesn't explain what was happening on the gcc repository. Regarding the email count check slowness I mentioned above, I traced it back to the updates.AbstractUpdate class, whose (procedural) new_commits_for_ref attribute is a list of "new" commits, containing both really new commits and commits newly on the branch to be updated, but already known to the repository. For a tag or branch creation, a list of "new on the branch" commits would be huge as everything is new, so parent commits of the oldest "repository-new" commit are not picked up. But in my test the list still amounts to a little less than 80,000 commits, basically what happened on trunk in the last 8 years. Anything that walks such a big list is bound to be slow. To sum up: - The hooks support checking "new on the branch" commits additionally to "new on the repository" commits, and that is a feature, not a bug. - In my use case, that means that the hooks process 80,000 commits, even if only 330 of them are new on the repository. - As the hook is called on a per-reference basis, the same commits would be processed over and over again for every reference in my use case, so the best would be to push them one by one, in order. - I still don't know why it took hours (without even finishing) to process just one tag the other day on the gcc repository. Nikael
Re: gcc git locked out for hours second day in a row
On Wed, 12 Jun 2024 at 14:23, Mikael Morin wrote: > > Le 12/06/2024 à 14:58, Jonathan Wakely a écrit : > > On Wed, 12 Jun 2024 at 13:57, Mikael Morin via Gcc wrote: > >> > >> Le 12/06/2024 à 13:48, Jakub Jelinek a écrit : > >>> Hi! > >>> > >>> Yesterday the gcc git repository was locked for 3 hours > >>> locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167) > >>> 78:06 python hooks/update.py > >>> refs/users/mikael/tags/fortran-dev_merges/r10-1545 > >>> > >>> c2f9fe1d8111b9671bf0aa8362446516fd942f1d > >>> process until overseers killed it but today we have the same > >>> situation for 3 ours and counting again: > >>> locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652) > >>> 78:06 python hooks/update.py refs/users/mikael/tags/toto > >>> > >>> cca005166dba2cefeb51afac3ea629b3972acea3 > >>> > >>> It is possible we have some bug in the git hook scripts, but it would > >>> be helpful trying to understand what exactly you're trying to commit > >>> and why nobody else (at least to my knowledge) has similarly stuck > >>> commits. > >>> > >>> The effect is that nobody can push anything else to gcc git repo > >>> for hours. > >>> > >>>Jakub > >>> > >> Yes, sorry for the inconvenience. > >> I tried pushing a series of tags labeling merge points between the > >> fortran-dev branch and recent years master. > > > > Just pushing tags should not cause a problem, assuming all the commits > > being tagged already exist. What exactly are you pushing? > > > Well, the individual commits to be merged do exist, but the merge points > don't and they are what I'm trying to push. I see. Merge commits are still commits, so they get processed by the hooks. We don't even allow merge commits on trunk or the release branches, so I don't know how the hooks handle them. > To be clear, the branch hasn't seen any update for years, and I'm trying > to reapply what happened on trunk since, in a step-wise manner. With > 300 merges I'm summing up 6 commits of history. Yes, that's going to give the hooks a ton of work to do. So it's probably just the number of commits being pushed to the branch, not the fact they're merge commits (and certainly not the tags referring to any of those commits, which should be very cheap). If it was my branch, I'd be tempted to rebase the fortran_dev branch on trunk, instead of merging years and years of trunk commits into the ancient branch. Or create a new branch, say "fortran_dev_2", from trunk, then merge the fortran_dev branch into that, and push the new branch. That would presumably be a much smaller set of "new" commits relative to trunk, so much less work for the hooks.
Re: gcc git locked out for hours second day in a row
On Wed, Jun 12, 2024 at 04:53:38PM +0200, Mikael Morin wrote: > > Perhaps you could create a mirror version of the repo and do some > > experiments locally on that to identify where the bottle-neck is coming > > from? > > > Not sure where to start for that.Are the hooks published somewhere? Yes: https://github.com/AdaCore/git-hooks/tree/master Note, we use some tweaks on top of that, but that is mostly for the release branches and trunk, so it would be interesting to just try to reproduce that with the stock AdaCore git hooks. Jakub
Re: gcc git locked out for hours second day in a row
Le 12/06/2024 à 16:34, Richard Earnshaw (lists) a écrit : On 12/06/2024 14:23, Mikael Morin via Gcc wrote: Le 12/06/2024 à 14:58, Jonathan Wakely a écrit : On Wed, 12 Jun 2024 at 13:57, Mikael Morin via Gcc wrote: Le 12/06/2024 à 13:48, Jakub Jelinek a écrit : Hi! Yesterday the gcc git repository was locked for 3 hours locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167) 78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 c2f9fe1d8111b9671bf0aa8362446516fd942f1d process until overseers killed it but today we have the same situation for 3 ours and counting again: locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652) 78:06 python hooks/update.py refs/users/mikael/tags/toto cca005166dba2cefeb51afac3ea629b3972acea3 It is possible we have some bug in the git hook scripts, but it would be helpful trying to understand what exactly you're trying to commit and why nobody else (at least to my knowledge) has similarly stuck commits. The effect is that nobody can push anything else to gcc git repo for hours. Jakub Yes, sorry for the inconvenience. I tried pushing a series of tags labeling merge points between the fortran-dev branch and recent years master. Just pushing tags should not cause a problem, assuming all the commits being tagged already exist. What exactly are you pushing? Well, the individual commits to be merged do exist, but the merge points don't and they are what I'm trying to push. To be clear, the branch hasn't seen any update for years, and I'm trying to reapply what happened on trunk since, in a step-wise manner. With 300 merges I'm summing up 6 commits of history. The number of merge points is a bit high (329) but I expected it to be a manageable number. I tried again today with just the most recent merge point, but it got stuck again. I should try with the oldest one, but I'm afraid locking the repository again. I waited for the push to finish for say one hour before killing it yesterday, and no more than 15 minutes today. Unfortunately, killing the process doesn't seem to unlock things on the server side. It may be a misconfiguration on my side, but I have never had this problem before. Sorry again. Perhaps you could create a mirror version of the repo and do some experiments locally on that to identify where the bottle-neck is coming from? Not sure where to start for that.Are the hooks published somewhere?
Re: gcc git locked out for hours second day in a row
On 12/06/2024 14:23, Mikael Morin via Gcc wrote: > Le 12/06/2024 à 14:58, Jonathan Wakely a écrit : >> On Wed, 12 Jun 2024 at 13:57, Mikael Morin via Gcc wrote: >>> >>> Le 12/06/2024 à 13:48, Jakub Jelinek a écrit : Hi! Yesterday the gcc git repository was locked for 3 hours locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167) 78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 c2f9fe1d8111b9671bf0aa8362446516fd942f1d process until overseers killed it but today we have the same situation for 3 ours and counting again: locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652) 78:06 python hooks/update.py refs/users/mikael/tags/toto cca005166dba2cefeb51afac3ea629b3972acea3 It is possible we have some bug in the git hook scripts, but it would be helpful trying to understand what exactly you're trying to commit and why nobody else (at least to my knowledge) has similarly stuck commits. The effect is that nobody can push anything else to gcc git repo for hours. Jakub >>> Yes, sorry for the inconvenience. >>> I tried pushing a series of tags labeling merge points between the >>> fortran-dev branch and recent years master. >> >> Just pushing tags should not cause a problem, assuming all the commits >> being tagged already exist. What exactly are you pushing? >> > Well, the individual commits to be merged do exist, but the merge points > don't and they are what I'm trying to push. > > To be clear, the branch hasn't seen any update for years, and I'm trying to > reapply what happened on trunk since, in a step-wise manner. With 300 merges > I'm summing up 6 commits of history. > >> >>> The number of merge points is a bit high (329) but I expected it to be a >>> manageable number. I tried again today with just the most recent merge >>> point, but it got stuck again. I should try with the oldest one, but >>> I'm afraid locking the repository again. >>> >>> I waited for the push to finish for say one hour before killing it >>> yesterday, and no more than 15 minutes today. Unfortunately, killing >>> the process doesn't seem to unlock things on the server side. >>> >>> It may be a misconfiguration on my side, but I have never had this >>> problem before. >>> >>> Sorry again. >>> >>> > Perhaps you could create a mirror version of the repo and do some experiments locally on that to identify where the bottle-neck is coming from? R.
Re: gcc git locked out for hours second day in a row
Le 12/06/2024 à 14:58, Jonathan Wakely a écrit : On Wed, 12 Jun 2024 at 13:57, Mikael Morin via Gcc wrote: Le 12/06/2024 à 13:48, Jakub Jelinek a écrit : Hi! Yesterday the gcc git repository was locked for 3 hours locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167) 78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 c2f9fe1d8111b9671bf0aa8362446516fd942f1d process until overseers killed it but today we have the same situation for 3 ours and counting again: locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652) 78:06 python hooks/update.py refs/users/mikael/tags/toto cca005166dba2cefeb51afac3ea629b3972acea3 It is possible we have some bug in the git hook scripts, but it would be helpful trying to understand what exactly you're trying to commit and why nobody else (at least to my knowledge) has similarly stuck commits. The effect is that nobody can push anything else to gcc git repo for hours. Jakub Yes, sorry for the inconvenience. I tried pushing a series of tags labeling merge points between the fortran-dev branch and recent years master. Just pushing tags should not cause a problem, assuming all the commits being tagged already exist. What exactly are you pushing? Well, the individual commits to be merged do exist, but the merge points don't and they are what I'm trying to push. To be clear, the branch hasn't seen any update for years, and I'm trying to reapply what happened on trunk since, in a step-wise manner. With 300 merges I'm summing up 6 commits of history. The number of merge points is a bit high (329) but I expected it to be a manageable number. I tried again today with just the most recent merge point, but it got stuck again. I should try with the oldest one, but I'm afraid locking the repository again. I waited for the push to finish for say one hour before killing it yesterday, and no more than 15 minutes today. Unfortunately, killing the process doesn't seem to unlock things on the server side. It may be a misconfiguration on my side, but I have never had this problem before. Sorry again.
Re: gcc git locked out for hours second day in a row
On Wed, 12 Jun 2024 at 13:57, Mikael Morin via Gcc wrote: > > Le 12/06/2024 à 13:48, Jakub Jelinek a écrit : > > Hi! > > > > Yesterday the gcc git repository was locked for 3 hours > > locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167) > > 78:06 python hooks/update.py > > refs/users/mikael/tags/fortran-dev_merges/r10-1545 > > > > c2f9fe1d8111b9671bf0aa8362446516fd942f1d > > process until overseers killed it but today we have the same > > situation for 3 ours and counting again: > > locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652) > > 78:06 python hooks/update.py refs/users/mikael/tags/toto > > > > cca005166dba2cefeb51afac3ea629b3972acea3 > > > > It is possible we have some bug in the git hook scripts, but it would > > be helpful trying to understand what exactly you're trying to commit > > and why nobody else (at least to my knowledge) has similarly stuck commits. > > > > The effect is that nobody can push anything else to gcc git repo > > for hours. > > > > Jakub > > > Yes, sorry for the inconvenience. > I tried pushing a series of tags labeling merge points between the > fortran-dev branch and recent years master. Just pushing tags should not cause a problem, assuming all the commits being tagged already exist. What exactly are you pushing? > The number of merge points is a bit high (329) but I expected it to be a > manageable number. I tried again today with just the most recent merge > point, but it got stuck again. I should try with the oldest one, but > I'm afraid locking the repository again. > > I waited for the push to finish for say one hour before killing it > yesterday, and no more than 15 minutes today. Unfortunately, killing > the process doesn't seem to unlock things on the server side. > > It may be a misconfiguration on my side, but I have never had this > problem before. > > Sorry again. > >
Re: gcc git locked out for hours second day in a row
Le 12/06/2024 à 14:14, Mark Wielaard a écrit : Hi, On Wed, 2024-06-12 at 13:48 +0200, Jakub Jelinek via Gcc wrote: Yesterday the gcc git repository was locked for 3 hours locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167) 78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 c2f9fe1d8111b9671bf0aa8362446516fd942f1d process until overseers killed it but today we have the same situation for 3 ours and counting again: locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652) 78:06 python hooks/update.py refs/users/mikael/tags/toto cca005166dba2cefeb51afac3ea629b3972acea3 Just for the record, I kill the process and removed the git- hooks::update.token.lock file. Sorry that killed mikael's push, but hopefully that makes it possible for others to push again. No problem, I had interrupted the push long before on my side.
Re: gcc git locked out for hours second day in a row
Le 12/06/2024 à 13:48, Jakub Jelinek a écrit : Hi! Yesterday the gcc git repository was locked for 3 hours locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167) 78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 c2f9fe1d8111b9671bf0aa8362446516fd942f1d process until overseers killed it but today we have the same situation for 3 ours and counting again: locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652) 78:06 python hooks/update.py refs/users/mikael/tags/toto cca005166dba2cefeb51afac3ea629b3972acea3 It is possible we have some bug in the git hook scripts, but it would be helpful trying to understand what exactly you're trying to commit and why nobody else (at least to my knowledge) has similarly stuck commits. The effect is that nobody can push anything else to gcc git repo for hours. Jakub Yes, sorry for the inconvenience. I tried pushing a series of tags labeling merge points between the fortran-dev branch and recent years master. The number of merge points is a bit high (329) but I expected it to be a manageable number. I tried again today with just the most recent merge point, but it got stuck again. I should try with the oldest one, but I'm afraid locking the repository again. I waited for the push to finish for say one hour before killing it yesterday, and no more than 15 minutes today. Unfortunately, killing the process doesn't seem to unlock things on the server side. It may be a misconfiguration on my side, but I have never had this problem before. Sorry again.
Re: gcc git locked out for hours second day in a row
Hi, On Wed, 2024-06-12 at 13:48 +0200, Jakub Jelinek via Gcc wrote: > Yesterday the gcc git repository was locked for 3 hours > locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167) > 78:06 python hooks/update.py > refs/users/mikael/tags/fortran-dev_merges/r10-1545 > > c2f9fe1d8111b9671bf0aa8362446516fd942f1d > process until overseers killed it but today we have the same > situation for 3 ours and counting again: > locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652) > 78:06 python hooks/update.py refs/users/mikael/tags/toto > > cca005166dba2cefeb51afac3ea629b3972acea3 Just for the record, I kill the process and removed the git- hooks::update.token.lock file. Sorry that killed mikael's push, but hopefully that makes it possible for others to push again. Cheers, Mark
gcc git locked out for hours second day in a row
Hi! Yesterday the gcc git repository was locked for 3 hours locked by user mikael at 2024-06-11 13:27:44.301067 (pid = 974167) 78:06 python hooks/update.py refs/users/mikael/tags/fortran-dev_merges/r10-1545 c2f9fe1d8111b9671bf0aa8362446516fd942f1d process until overseers killed it but today we have the same situation for 3 ours and counting again: locked by user mikael at 2024-06-12 08:35:48.137564 (pid = 2219652) 78:06 python hooks/update.py refs/users/mikael/tags/toto cca005166dba2cefeb51afac3ea629b3972acea3 It is possible we have some bug in the git hook scripts, but it would be helpful trying to understand what exactly you're trying to commit and why nobody else (at least to my knowledge) has similarly stuck commits. The effect is that nobody can push anything else to gcc git repo for hours. Jakub
