RE: sync bug - corrupted proxy repo
Hi, I am assuming that if the commits must start at least one second apart - then the sync from the post-commit hook would not be able to reach a race condition. Is this a reasonable assumption? No, the bug is worse than that. Suppose there are 3 commits: - At time 12:00:00, a commit starts sync process #1. The sync takes 6 seconds. - At time 12:00:02, a commit starts sync process #2. This blocks due to sync process #1's lock. - At time 12:00:04, a commit starts sync process #3. This blocks due to sync process #1's lock. - At time 12:00:06, sync process #1 finishes. Sync processes #2 and #3 both try to take the lock; due to the bug they may _both_ succeed in taking the lock. Chaos ensues. I suggest you use the flock(1) tool. [1]. This is installed as a standard part of Debian (it's in the util-linux package). Something like this, in your post-commit hook: --- cut here - start --- #!/bin/sh /usr/bin/flock --wait 1200 \ -x /var/lock/svn_sync_lock \ /usr/local/bin/svnsync sync --non-interactive \ http://mirrorserver.example.com/svn --- cut here - end --- You will need to make the /var/lock/svn_sync_lock file and ensure it's writable by the user your post-commit hook is running as. flock is a mature, tested piece of code to handle locking. It will ensure that only one copy of svnsync is running at a time. That way, the race condition in svnsync is avoided. Kind regards, Jon [1] http://www.google.co.uk/search?q=man+flock%281%29 -Original Message- From: Andersen, Krista [mailto:krista.ander...@itg.com] Sent: 15 January 2010 22:29 To: Jon Foster; users@subversion.apache.org Cc: ssi-svn_admin Subject: RE: sync bug - corrupted proxy repo Thank you Jon, for your explanation and workaround. Are there any best practices that we can advise our dev groups to follow to avoid this problem? Otherwise, your suggestions seem to indicate I would have to run the sync on a cronjob and not with the hook script. That is something we would like to avoid. So I have added a start time comparison and sleep in a start-commit hook instead. Do you see any reason why this would cause other problems? I am assuming that if the commits must start at least one second apart - then the sync from the post-commit hook would not be able to reach a race condition. Is this a reasonable assumption? #!/usr/bin/sh # START-COMMIT HOOK # kanderse Jan 13, 2010 # The start-commit hook is invoked before a Subversion txn is created # in the process of doing a commit. # This script checks the start time and compares with the start time # of the previous commit. It will cause a commit to wait one second if # the last commit was started less than one second earlier. # The purpose of this wait is to prevent known issue 3546 [1][2]. # a race condition involving multiple sync processes running at # the same time that result in a corrupted proxy. REPOS=$1 USER=$2 DATE1=`cat /$REPOS/hooks/start-time.txt` # previous start time DATE2=`/usr/local/bin/date +%s` # record current start time echo $DATE2 /$REPOS/hooks/start-time.txt # echo $DATE2 $DATE1 `expr $DATE2 - $DATE1` if [ `expr $DATE2 - $DATE1` -lt 1 ] then sleep 1 # to prevent sync race that results in sync duplication and corrupted proxy fi # All checks passed, so allow the commit. exit 0 Krista Andersen Global Development Infrastructure Investment Technology Group, Inc. 400 Corporate Pointe, 8th Floor Culver City, CA 90230 Direct: 213.270.7570 -Original Message- From: Jon Foster [mailto:jon.fos...@cabot.co.uk] Sent: Wednesday, January 13, 2010 5:13 AM To: Andersen, Krista; users@subversion.apache.org Cc: ssi-svn_admin Subject: RE: sync bug - corrupted proxy repo Hi, Andersen, Krista [mailto:krista.ander...@itg.com] wrote: Twice I have seen one of my proxy repositories become corrupted due to an apparent bug in the svnsync sync process. Has anyone else seen this type of behavior from Subversion? This is probably caused by issue 3546 [1][2]. This is a race condition - if you have several sync processes running at the same time then the mirror can get corrupted. You had three commits which were 1 second apart, so your hook script started 3 copies of svnsync within 2 seconds. I think this is the first practical report of this bug; previous discussion was theoretical. Here is a comparison the output of the svn log -v for the offending revisions (324,325) on both the corrupted and non-corrupted proxy repo. It looks like rev 323 got mirrored twice (as mirror revs 323 and 324), then rev 324 was mirrored (as mirror rev 325). I am a bit concerned about the stability of Subversion since this is the second time in two months that I have had to fix this issue. Is there a patch or something to prevent this in the future? Suggested workaround: Change your hook scripts to use the lockf or lockfile tools[3] to ensure that only one instance of svnsync runs at once. Kind regards, Jon [1] http://mail
RE: sync bug - corrupted proxy repo
Hi, Ryan Schmidt wrote: But Subversion blocks the commit until the post-commit is done. That particular SVN client will be blocked. But if you have two users committing at the same time, or if a user runs svn twice in parallel, then the post-commit hook will be run in parallel. Here's how I tested this. I created a new repository with a post-commit hook that takes 30 seconds to run. I then checked that it works, and that a normal commit took 30 seconds. I then did two commits in parallel, and that took 30 seconds. This shows that the post-commit hook is running in parallel - if it had been run in series, then it would have taken 60 seconds for 2 commits. (I also checked the output of ps and observed the two post-commit processes running). ~$ mkdir svnscratch ~$ cd svnscratch/ ~/svnscratch$ svn --version | head -n1 svn, version 1.6.8 (dev build) ~/svnscratch$ svnadmin create repo ~/svnscratch$ cat repo/hooks/post-commit #! /bin/bash sleep 30 ~/svnscratch$ chmod a+x repo/hooks/post-commit ~/svnscratch$ time repo/hooks/post-commit real0m30.004s user0m0.000s sys 0m0.008s ~/svnscratch$ time svn mkdir -m Test file://`pwd`/repo/trunk Committed revision 1. real0m30.030s user0m0.008s sys 0m0.008s ~/svnscratch$ time ( svn mkdir -m Test file://`pwd`/repo/branches svn mkdir -m Test file://`pwd`/repo/tags ) Committed revision 2. Committed revision 3. real0m30.069s user0m0.004s sys 0m0.020s ~/svnscratch$ Kind regards, Jon ** This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Cabot Communications Ltd. If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone. Cabot Communications Limited Verona House, Filwood Road, Bristol BS16 3RY, UK +44 (0) 1179584232 Co. Registered in England number 02817269 Please contact the sender if you believe you have received this email in error. ** __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __
Re: sync bug - corrupted proxy repo
On Fri, Jan 15, 2010 at 12:08 PM, Jon Foster jon.fos...@cabot.co.uk wrote: Hi, Ryan Schmidt wrote: But Subversion blocks the commit until the post-commit is done. That particular SVN client will be blocked. But if you have two users committing at the same time, or if a user runs svn twice in parallel, then the post-commit hook will be run in parallel. Here's how I tested this. I created a new repository with a post-commit hook that takes 30 seconds to run. I then checked that it works, and that a normal commit took 30 seconds. I then did two commits in parallel, and that took 30 seconds. This shows that the post-commit hook is running in parallel - if it had been run in series, then it would have taken 60 seconds for 2 commits. (I also checked the output of ps and observed the two post-commit processes running). Also, I'm pretty sure that, while the post-commit hook is running for a particular commit, the commit itself is already visible to other users. So, as you would expect from the name *post*-commit hook, the commit itself is already finalized before the post-commit hook starts running. Otherwise, people wouldn't be able to do things like automatically updating a working copy on the server, from within their post-commit hook. The only thing that has to wait on the post-commit hook is that particular svn client that's running the commit (as Jon pointed out). Regards, Johan
Re: sync bug - corrupted proxy repo
On Wed, Jan 13, 2010 at 2:27 PM, Giulio Troccoli giulio.trocc...@uk.linedata.com wrote: From: Jon Foster [mailto:jon.fos...@cabot.co.uk] Sent: 13 January 2010 13:13 To: Andersen, Krista; users@subversion.apache.org Cc: ssi-svn_admin Subject: RE: sync bug - corrupted proxy repo Hi, Andersen, Krista [mailto:krista.ander...@itg.com] wrote: Twice I have seen one of my proxy repositories become corrupted due to an apparent bug in the svnsync sync process. Has anyone else seen this type of behavior from Subversion? This is probably caused by issue 3546 [1][2]. This is a race condition - if you have several sync processes running at the same time then the mirror can get corrupted. You had three commits which were 1 second apart, so your hook script started 3 copies of svnsync within 2 seconds. I think this is the first practical report of this bug; previous discussion was theoretical. Here is a comparison the output of the svn log -v for the offending revisions (324,325) on both the corrupted and non-corrupted proxy repo. It looks like rev 323 got mirrored twice (as mirror revs 323 and 324), then rev 324 was mirrored (as mirror rev 325). I am a bit concerned about the stability of Subversion since this is the second time in two months that I have had to fix this issue. Is there a patch or something to prevent this in the future? Suggested workaround: Change your hook scripts to use the lockf or lockfile tools[3] to ensure that only one instance of svnsync runs at once. Is it not the case that a svn commit cannot start before the post-commit hooks has finished? I am asking becuase I will be implementing a DR system using svnsync, but I am not planning to let the post-commit finish before svnsync has finished (I don't care if it takes a bit longer, I can cope with that and with my users). No, that's definitely not the case. Both pre-commit and post-commit hooks can run simultaneously for multiple commits in parallel. Otherwise, a pre/post-commit hook could end up being a big bottleneck. As a practical example: if your pre-commit hook does validation of properties (e.g. making sure svn:eol-style is set correctly on the right types of files), then for a commit with a lot of files it may be running for multiple seconds (even minutes). In the meantime, other commits can be made without problems, without being bothered by that one commit which takes a long time to pass through pre-commit hook. I saw this in action myself with such a pre-commit hook, after I added some debug logging at the start and the end of the hook. Regards, Johan
Re: sync bug - corrupted proxy repo
On Jan 14, 2010, at 09:06, Johan Corveleyn wrote: Is it not the case that a svn commit cannot start before the post-commit hooks has finished? I am asking becuase I will be implementing a DR system using svnsync, but I am not planning to let the post-commit finish before svnsync has finished (I don't care if it takes a bit longer, I can cope with that and with my users). No, that's definitely not the case. Both pre-commit and post-commit hooks can run simultaneously for multiple commits in parallel. Otherwise, a pre/post-commit hook could end up being a big bottleneck. As a practical example: if your pre-commit hook does validation of properties (e.g. making sure svn:eol-style is set correctly on the right types of files), then for a commit with a lot of files it may be running for multiple seconds (even minutes). In the meantime, other commits can be made without problems, without being bothered by that one commit which takes a long time to pass through pre-commit hook. I saw this in action myself with such a pre-commit hook, after I added some debug logging at the start and the end of the hook. pre-commit hook, I believe you're right. But Subversion blocks the commit until the post-commit is done. Only one post-commit hook will run at a time. Unless you tell it to allow simultaneous runs, by redirecting the hook's stdout and stderr someplace.
RE: sync bug - corrupted proxy repo
Hi, Andersen, Krista [mailto:krista.ander...@itg.com] wrote: Twice I have seen one of my proxy repositories become corrupted due to an apparent bug in the svnsync sync process. Has anyone else seen this type of behavior from Subversion? This is probably caused by issue 3546 [1][2]. This is a race condition - if you have several sync processes running at the same time then the mirror can get corrupted. You had three commits which were 1 second apart, so your hook script started 3 copies of svnsync within 2 seconds. I think this is the first practical report of this bug; previous discussion was theoretical. Here is a comparison the output of the svn log -v for the offending revisions (324,325) on both the corrupted and non-corrupted proxy repo. It looks like rev 323 got mirrored twice (as mirror revs 323 and 324), then rev 324 was mirrored (as mirror rev 325). I am a bit concerned about the stability of Subversion since this is the second time in two months that I have had to fix this issue. Is there a patch or something to prevent this in the future? Suggested workaround: Change your hook scripts to use the lockf or lockfile tools[3] to ensure that only one instance of svnsync runs at once. Kind regards, Jon [1] http://mail-archives.apache.org/mod_mbox/subversion-dev/200911.mbox/%3C2 0091127115356.gc9...@jack.stsp.name%3e [2] http://subversion.tigris.org/issues/show_bug.cgi?id=3546 [3] http://mail-archives.apache.org/mod_mbox/subversion-dev/200911.mbox/%3C2 0091127132659.ge9...@jack.stsp.name%3e ** This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Cabot Communications Ltd. If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone. Cabot Communications Limited Verona House, Filwood Road, Bristol BS16 3RY, UK +44 (0) 1179584232 Co. Registered in England number 02817269 Please contact the sender if you believe you have received this email in error. ** __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __
