differences in dump/load/dump cycle
Hi, I went through a long overdue dump/load cycle of our main repository and am wondering atm about a difference I see when comparing the dump of the repository with the original dump. There are a few (at a guess around 20-50) differences of the following structure (using fc [olddump] [newdump] on Windows): * svn.dump Node-path: XRebirth/tags/XR v3.53 RC2 (final)/src/SDKs/DX9SDK Node-kind: dir Node-action: change Revision-number: 193958 * SVN_NEW.DMP Revision-number: 193958 * svn.dump was generated using svnadmin 1.8.15 (32-bit). The dump was then loaded in a new clean repository using svnadmin 1.8.16 (64-bit). svn_new.dmp was then written using svnadmin 1.8.16 (64-bit) as well. The original repository was created using SVN 1.7. At some point in the past (around 2 years ago) the server was upgraded to SVN 1.8 but the repository was still kept at fsfs format 5. Around a year later the repository was upgraded to SVN 1.8 (fsfs format 6). For the new repository fsfs.conf was modified to enable directory and property deltification. For the DX9SDK directory (which is reported being different in both dumps in some revisions) this was originally using externals and at some point we switched to a direct copy of the folder (not sure whether that's relevant though). Is this difference expected? I remember (and Bert mentioned it too) that there were some cases for different handling of noop-changes. Is that what explains the difference I see here? If so, I take it that's expected and does not result in any difference between the repository states, or does it? JCorvel, would you have an idea? Regards, Stefan smime.p7s Description: S/MIME Cryptographic Signature
Re: differences in dump/load/dump cycle
On Sun, Jul 31, 2016 at 5:55 PM, Stefan wrote: > Hi, > > I went through a long overdue dump/load cycle of our main repository and > am wondering atm about a difference I see when comparing the dump of the > repository with the original dump. > > There are a few (at a guess around 20-50) differences of the following > structure (using fc [olddump] [newdump] on Windows): > > * svn.dump > > Node-path: XRebirth/tags/XR v3.53 RC2 (final)/src/SDKs/DX9SDK > Node-kind: dir > Node-action: change > > > Revision-number: 193958 > * SVN_NEW.DMP > > Revision-number: 193958 > * > > > svn.dump was generated using svnadmin 1.8.15 (32-bit). > > The dump was then loaded in a new clean repository using svnadmin 1.8.16 > (64-bit). svn_new.dmp was then written using svnadmin 1.8.16 (64-bit) as > well. > > The original repository was created using SVN 1.7. At some point in the > past (around 2 years ago) the server was upgraded to SVN 1.8 but the > repository was still kept at fsfs format 5. Around a year later the > repository was upgraded to SVN 1.8 (fsfs format 6). > > For the new repository fsfs.conf was modified to enable directory and > property deltification. > > For the DX9SDK directory (which is reported being different in both > dumps in some revisions) this was originally using externals and at some > point we switched to a direct copy of the folder (not sure whether > that's relevant though). > > Is this difference expected? I remember (and Bert mentioned it too) that > there were some cases for different handling of noop-changes. Is that > what explains the difference I see here? If so, I take it that's > expected and does not result in any difference between the repository > states, or does it? JCorvel, would you have an idea? > It's possible that this is a benign change, with no visible effects. I'm not sure. The problem I ran into with dump was a new bug in 1.9.0 (fixed in 1.9.3 I think). It was with no-op changes to files, not directories. This was IMO definitely a bug, because the effect was visible in the new repository (namely, if you ran 'svn log somepath', where somepath was a file which had such a no-op change (not possible to create with the standard svn client btw, but possible with other tools or from a cvs2svn conversion) in revision R, then revision R would not be listed as part of somepath's history). It's unclear to me if you can see a similar loss of "changed-path / history" association. In my case the change in the dumpfile was a bit different: the Node-path / Node-kind / Node-action lines were still there, but the Text-content lines were gone (if you dumped again from the new repository, the entire block with Node-path etc would be gone). See http://svn.haxx.se/dev/archive-2015-09/0269.shtml for the entire (long) discussion, which lead to the bugfix for 1.9.3. -- Johan
E165001 pre-commit hook failed
Hi all, I've been banging my head against this problem for a day and I need some help. We recently updated the machine hosting our svn repo, and this broke commits using svn+ssh. Here's the setup: * Machine runs debian 3.16.7, svn 1.8.10. * svn runs as user "www-data" (the apache user). * svn+ssh access uses a forced ssh command which sudos to the www-data user and executes an svnserve wrapper. * svnserve wrapper sets umask to 022 and then executes "svnserve -t" with a specific tunnel user. This setup is identical to what it was on the old machine. However, there must be something different about it, because now https commits work fine, but the svn+ssh commits give the error: Transmitting file data .svn: E165001: Commit failed (details follow): svn: E165001: Commit blocked by pre-commit hook (exit code 255) with no output. If I access the repository directly using file:// and sudo to the www-data user when executing svn, commits work fine. This, in combination with the fact that https access works, makes me conclude it is not a permissions or hook problem on any of the files since all these access methods run as the www-data user. Nevertheless, the error comes from the hook, because if I remove the hook file completely, the failure moves to the post-commit hook. It's not a problem finding !#/bin/sh either, because I tried replacing the hook with a compiled C program that just returns 1, and I still got the 255 return code. When I attempt to commit, I can see successful authentication in the syslog, like: sudo: : TTY=unknown ; PWD=/home/ ; USER=www-data ; GROUP=www-data ; COMMAND=/usr/local/bin/svnserve-wrapper sudo: pam_unix(sudo:session): session opened for user www-data by (uid=0) sshd[26903]: Received disconnect from : 11: disconnected by user sshd[26897]: pam_unix(sshd:session): session closed for user The svnserve log file gets this (the name of the repo here is "test") test open 2 cap=(edit-pipeline svndiff1 absent-entries depth mergeinfo log-revprops) / SVN/1.8.10%20(x86_64-pc-linux-gnu) - and then nothing. (I don't know what the "open" command does, it's not included in the list of commands on e.g. http://svnbook.red-bean.com/en/1.8/svn.serverconfig.operational-logging.html ) I've seen some similar reports of this, but no suggestions apart from permissions or corrupted hook files, which this can't be. I don't even know how to proceed with debugging this. Is it possible to see what svnserve attempts to do with the hook file? Since it's spawned on demand, I don't know how to attach to it with a debugger, or where in the source code this error originates. Any ideas would be much appreciated, Regards, /Patrik J.
Re: E165001 pre-commit hook failed
On 7/31/2016 2:30 PM, Patrik Jonsson wrote: Hi all, I've been banging my head against this problem for a day and I need some help. We recently updated the machine hosting our svn repo, and this broke commits using svn+ssh. Here's the setup: * Machine runs debian 3.16.7, svn 1.8.10. * svn runs as user "www-data" (the apache user). * svn+ssh access uses a forced ssh command which sudos to the www-data user and executes an svnserve wrapper. * svnserve wrapper sets umask to 022 and then executes "svnserve -t" with a specific tunnel user. This setup is identical to what it was on the old machine. However, there must be something different about it, because now https commits work fine, but the svn+ssh commits give the error: Transmitting file data .svn: E165001: Commit failed (details follow): svn: E165001: Commit blocked by pre-commit hook (exit code 255) with no output. If I access the repository directly using file:// and sudo to the www-data user when executing svn, commits work fine. This, in combination with the fact that https access works, makes me conclude it is not a permissions or hook problem on any of the files since all these access methods run as the www-data user. Nevertheless, the error comes from the hook, because if I remove the hook file completely, the failure moves to the post-commit hook. It's not a problem finding !#/bin/sh either, because I tried replacing the hook with a compiled C program that just returns 1, and I still got the 255 return code. When I attempt to commit, I can see successful authentication in the syslog, like: sudo: : TTY=unknown ; PWD=/home/ ; USER=www-data ; GROUP=www-data ; COMMAND=/usr/local/bin/svnserve-wrapper sudo: pam_unix(sudo:session): session opened for user www-data by (uid=0) sshd[26903]: Received disconnect from : 11: disconnected by user sshd[26897]: pam_unix(sshd:session): session closed for user The svnserve log file gets this (the name of the repo here is "test") test open 2 cap=(edit-pipeline svndiff1 absent-entries depth mergeinfo log-revprops) / SVN/1.8.10%20(x86_64-pc-linux-gnu) - and then nothing. (I don't know what the "open" command does, it's not included in the list of commands on e.g. http://svnbook.red-bean.com/en/1.8/svn.serverconfig.operational-logging.html) I've seen some similar reports of this, but no suggestions apart from permissions or corrupted hook files, which this can't be. I don't even know how to proceed with debugging this. Is it possible to see what svnserve attempts to do with the hook file? Since it's spawned on demand, I don't know how to attach to it with a debugger, or where in the source code this error originates. Any ideas would be much appreciated, Regards, /Patrik J. Is SELinux enabled on the new server? I've seen some oddball permission problems result when upgrading Linux systems if SELinux is enabled on the new server but not the old. I don't use svnserve, so I can't offer specific advice other than the security context in which the new user runs may be different than that of svnserve, and SELinux may be blocking it. On Red Hat/CentOS, you would look in "/var/log/audit/audit.log" for signs of trouble. I don't know if that is the location of the SELinux log files under Debian. In particular, watch out for files (scripts, configuration files) copied directly from an older server without SELinux into a new server with SELinux. They don't get a context appropriate to the directory in which you put them. I use Apache, and I had to track down these files afterward and fix them one by one - very painful. This isn't just a Subversion problem but is a general Apache problem. If you do have SELinux running, a quick way to determine whether you have a security context problem is to turn SELinux off briefly. -- David Chapman dcchap...@acm.org Chapman Consulting -- San Jose, CA Software Development Done Right. www.chapman-consulting-sj.com
Re: E165001 pre-commit hook failed
On Sun, Jul 31, 2016 at 11:15 PM, David Chapman wrote: > On 7/31/2016 2:30 PM, Patrik Jonsson wrote: >> >> Hi all, >> >> I've been banging my head against this problem for a day and I need some >> help. We recently updated the machine hosting our svn repo, and this broke >> commits using svn+ssh. Here's the setup: >> >> * Machine runs debian 3.16.7, svn 1.8.10. >> * svn runs as user "www-data" (the apache user). >> * svn+ssh access uses a forced ssh command which sudos to the www-data >> user and executes an svnserve wrapper. >> * svnserve wrapper sets umask to 022 and then executes "svnserve -t" with >> a specific tunnel user. >> >> This setup is identical to what it was on the old machine. However, there >> must be something different about it, because now https commits work fine, >> but the svn+ssh commits give the error: >> >> Transmitting file data .svn: E165001: Commit failed (details follow): >> svn: E165001: Commit blocked by pre-commit hook (exit code 255) with no >> output. >> >> If I access the repository directly using file:// and sudo to the www-data >> user when executing svn, commits work fine. This, in combination with the >> fact that https access works, makes me conclude it is not a permissions or >> hook problem on any of the files since all these access methods run as the >> www-data user. Nevertheless, the error comes from the hook, because if I >> remove the hook file completely, the failure moves to the post-commit hook. >> >> It's not a problem finding !#/bin/sh either, because I tried replacing the >> hook with a compiled C program that just returns 1, and I still got the 255 >> return code. >> >> When I attempt to commit, I can see successful authentication in the >> syslog, like: >> >> sudo: : TTY=unknown ; PWD=/home/ ; USER=www-data ; >> GROUP=www-data ; COMMAND=/usr/local/bin/svnserve-wrapper >> sudo: pam_unix(sudo:session): session opened for user www-data by (uid=0) >> sshd[26903]: Received disconnect from : 11: disconnected by user >> sshd[26897]: pam_unix(sshd:session): session closed for user >> >> The svnserve log file gets this (the name of the repo here is "test") >> >> test open 2 cap=(edit-pipeline svndiff1 absent-entries depth >> mergeinfo log-revprops) / SVN/1.8.10%20(x86_64-pc-linux-gnu) - >> >> and then nothing. (I don't know what the "open" command does, it's not >> included in the list of commands on e.g. >> http://svnbook.red-bean.com/en/1.8/svn.serverconfig.operational-logging.html) >> >> I've seen some similar reports of this, but no suggestions apart from >> permissions or corrupted hook files, which this can't be. I don't even know >> how to proceed with debugging this. Is it possible to see what svnserve >> attempts to do with the hook file? Since it's spawned on demand, I don't >> know how to attach to it with a debugger, or where in the source code this >> error originates. >> >> Any ideas would be much appreciated, >> >> Regards, >> >> /Patrik J. >> > > Is SELinux enabled on the new server? I've seen some oddball permission > problems result when upgrading Linux systems if SELinux is enabled on the > new server but not the old. I don't use svnserve, so I can't offer specific > advice other than the security context in which the new user runs may be > different than that of svnserve, and SELinux may be blocking it. On Red > Hat/CentOS, you would look in "/var/log/audit/audit.log" for signs of > trouble. I don't know if that is the location of the SELinux log files > under Debian. > > In particular, watch out for files (scripts, configuration files) copied > directly from an older server without SELinux into a new server with > SELinux. They don't get a context appropriate to the directory in which you > put them. I use Apache, and I had to track down these files afterward and > fix them one by one - very painful. This isn't just a Subversion problem but > is a general Apache problem. > > If you do have SELinux running, a quick way to determine whether you have a > security context problem is to turn SELinux off briefly. Set it to "warning", not completely disabled, or you won'g get logs in /var/log/audit.log.
Re: E165001 pre-commit hook failed
No, SElinux isn't installed. I figured it out, finally. It was pretty obscure: The ssh login is done as user X, which then force-executes the svnserve wrapper using sudo to the www-data user. When svnserve attempts to execute the hook, one of the things done in the call to svn_io_start_cmd3 is to set the path to ".". The apr_proc_create call then attempts to chdir to this path after forking. Since the call originated in a login as user X, the cwd was /home/X. The www-data user, however, did not have permissions to chdir to /home/X, so the chdir(".") call failed with EACCES and the forked process returned -1 (ie 255) and bailed on even attempting to execute the pre-commit hook. I think this is pretty bad in that it fails on "cd .", a completely useless operation, and there is no diagnostic whatsoever that indicates that the cwd has anything to do with anything, since the hook itself is being executed with an absolute path. It should probably be mentioned somewhere that svnserve needs to have access permissions to the cwd even if it doesn't want to do anything with it. Cheers, /Patrik On Sun, Jul 31, 2016 at 5:15 PM, David Chapman wrote: > On 7/31/2016 2:30 PM, Patrik Jonsson wrote: > >> Hi all, >> >> I've been banging my head against this problem for a day and I need some >> help. We recently updated the machine hosting our svn repo, and this broke >> commits using svn+ssh. Here's the setup: >> >> * Machine runs debian 3.16.7, svn 1.8.10. >> * svn runs as user "www-data" (the apache user). >> * svn+ssh access uses a forced ssh command which sudos to the www-data >> user and executes an svnserve wrapper. >> * svnserve wrapper sets umask to 022 and then executes "svnserve -t" >> with a specific tunnel user. >> >> This setup is identical to what it was on the old machine. However, there >> must be something different about it, because now https commits work fine, >> but the svn+ssh commits give the error: >> >> Transmitting file data .svn: E165001: Commit failed (details follow): >> svn: E165001: Commit blocked by pre-commit hook (exit code 255) with no >> output. >> >> If I access the repository directly using file:// and sudo to the >> www-data user when executing svn, commits work fine. This, in combination >> with the fact that https access works, makes me conclude it is not a >> permissions or hook problem on any of the files since all these access >> methods run as the www-data user. Nevertheless, the error comes from the >> hook, because if I remove the hook file completely, the failure moves to >> the post-commit hook. >> >> It's not a problem finding !#/bin/sh either, because I tried replacing >> the hook with a compiled C program that just returns 1, and I still got the >> 255 return code. >> >> When I attempt to commit, I can see successful authentication in the >> syslog, like: >> >> sudo: : TTY=unknown ; PWD=/home/ ; USER=www-data ; >> GROUP=www-data ; COMMAND=/usr/local/bin/svnserve-wrapper >> sudo: pam_unix(sudo:session): session opened for user www-data by (uid=0) >> sshd[26903]: Received disconnect from : 11: disconnected by user >> sshd[26897]: pam_unix(sshd:session): session closed for user >> >> The svnserve log file gets this (the name of the repo here is "test") >> >> test open 2 cap=(edit-pipeline svndiff1 absent-entries depth >> mergeinfo log-revprops) / SVN/1.8.10%20(x86_64-pc-linux-gnu) - >> >> and then nothing. (I don't know what the "open" command does, it's not >> included in the list of commands on e.g. >> http://svnbook.red-bean.com/en/1.8/svn.serverconfig.operational-logging.html >> ) >> >> I've seen some similar reports of this, but no suggestions apart from >> permissions or corrupted hook files, which this can't be. I don't even know >> how to proceed with debugging this. Is it possible to see what svnserve >> attempts to do with the hook file? Since it's spawned on demand, I don't >> know how to attach to it with a debugger, or where in the source code this >> error originates. >> >> Any ideas would be much appreciated, >> >> Regards, >> >> /Patrik J. >> >> > Is SELinux enabled on the new server? I've seen some oddball permission > problems result when upgrading Linux systems if SELinux is enabled on the > new server but not the old. I don't use svnserve, so I can't offer > specific advice other than the security context in which the new user runs > may be different than that of svnserve, and SELinux may be blocking it. On > Red Hat/CentOS, you would look in "/var/log/audit/audit.log" for signs of > trouble. I don't know if that is the location of the SELinux log files > under Debian. > > In particular, watch out for files (scripts, configuration files) copied > directly from an older server without SELinux into a new server with > SELinux. They don't get a context appropriate to the directory in which > you put them. I use Apache, and I had to track down these files afterward > and fix them one by one - very painful. This isn't just a Subversion > pro