Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
On Wed, Mar 28, 2018 at 1:56 AM, Bo Berglund wrote: > I proceeded to sync our repositories. On the 3rd repo I get the > following error at revision 210 (it contains 1290 revisions): > > E:\>svnsync synchronize https://home.mirrordomain.com/svn/cmp > https://masterserver/svn/cmp > Transmitting file data ... 8 long lines of dots > ..svnsync: E120106: ra_serf: The server sent a > truncated HTTP response body. > > The sync was interrupted so I tried restarting and it went directly to > the Transmitting step as shown above. > On my screen there were 8 lines of dots until the error appeared and > it happened at *exactly* the same place for both the original and > restarted sync command. Took a few mnutes to get there. > > What can this be caused by and what can I do to fix it? > > I am running the command on the master server (Windows Server 2016) > towards an overseas server running Ubuntu 16.04 I'd start with going through the apache logs on both servers to look for more information about this error (since "The server sent a truncated HTTP response body." indicates to me that an error should be logged on that server). It's also possible that some other network component has truncated things ... proxies, security content filters, ... -- Johan
Re: issue with relative externals after a rename
On Wed, Mar 28, 2018 at 5:36 AM, Branko Čibej wrote: > On 28.03.2018 01:38, Jonathan Schell wrote: >> I have a tree that looks like: >> >> https://svn/repo/name1/externs/normal_files_here >> >> The folder "externs" has two external properties: >> ../fold1@18 fold1 >> ../fold2/file1.txt@18 file1.txt >> >> These folders and file exist at rev 18. The commit to add the properties to >> "externs" is commit 32. >> >> I then do a rename of "name1" to "name2". >> >> The externs now break, as the item they were pointing to no longer exists. > > Because the default operative revision is HEAD and there's no such > object in HEAD, yes. Are you sure? That's surprising to me, because for normal commands the operative revision always defaults to the peg revision. For example, in [1] I read "Notice that we didn't provide an operative revision this time. That's because when no operative revision is specified, Subversion assumes a default operative revision that's the same as the peg revision." Or is there a different rule for externals definitions? [1] http://svnbook.red-bean.com/nightly/en/svn.advanced.pegrevs.html -- Johan
Re: issue with relative externals after a rename
On Tue, Mar 27, 2018 at 11:36 PM, Branko Čibej wrote: > Because the default operative revision is HEAD and there's no such > object in HEAD, yes. [snip] > You need both peg and operative revisions: > > -r 18 ../fold1 fold1@18 > -r 18 ../fold2/file1.txt@18 file1.txt > > The peg revision tells Subversion which object to look for and the > operative revision tells it which version of that object to use. > And then... On Wed, Mar 28, 2018 at 3:12 AM, Johan Corveleyn wrote: > Are you sure? That's surprising to me, because for normal commands the > operative revision always defaults to the peg revision. > > For example, in [1] I read "Notice that we didn't provide an operative > revision this time. That's because when no operative revision is > specified, Subversion assumes a default operative revision that's the > same as the peg revision." > > Or is there a different rule for externals definitions? > > [1] http://svnbook.red-bean.com/nightly/en/svn.advanced.pegrevs.html > Oops, that makes me a bit nervous about how we've set up our externals. Our requirements are: at any time in the future, if someone checks out code from the past, they should get exactly the same tree as what existed in the past. I assume that this is probably THE #1 use case and desired behavior for externals. Is that correct? If not, is there a good reason to want otherwise? My understanding is that: (1) ^/SVN/path/to/project/tags/specific_tag That will probably work most of the time but if someone commits to that tag, future checkouts of past revisions will not look exactly as they did in the past. (2) -r [number] ^/SVN/path/to/project/tags/specific_tag Subversion first goes to that path, then follows it back in time to the specified revision. So if someone commits to that tag, the future checkout of the past revision will get the past revision. But... if a tag is ever moved, removed, or renamed in the future, (the project is declared defunct and removed, for example), what Subversion finds at that path and tries to follow back in time won't lead to the right place. (3) ^/SVN/path/to/project/tags/specific_tag@[number] Subversion first goes back in time to that revision, then goes to that path. This is where I'm a bit foggy: Once it goes back in time and goes to that path, does it then follow it forward to HEAD? Or does it stay in the past? We've set up all of our externals like (3)... Do we need to go through our repo and change all externals to specify both the peg and operative revision? Thanks, Nathan Hartman
Re: issue with relative externals after a rename
On Wed, Mar 28, 2018 at 10:36 AM, Nathan Hartman wrote: > Our requirements are: at any time in the future, if someone checks > out code from the past, they should get exactly the same tree as what > existed in the past. I assume that this is probably THE #1 use case > and desired behavior for externals. Is that correct? If not, is there > a good reason to want otherwise? Then make tags. Tags need to be locked down, with no commits permitted on top of the tag. > > My understanding is that: > > (1) ^/SVN/path/to/project/tags/specific_tag > > That will probably work most of the time but if someone commits > to that tag, future checkouts of past revisions will not look > exactly as they did in the past. See above. "No commits permitted on top of the tag". > (2) -r [number] ^/SVN/path/to/project/tags/specific_tag > > Subversion first goes to that path, then follows it back in time > to the specified revision. > > So if someone commits to that tag, the future checkout of the > past revision will get the past revision. > > But... if a tag is ever moved, removed, or renamed in the future, > (the project is declared defunct and removed, for example), what > Subversion finds at that path and tries to follow back in time > won't lead to the right place. > > (3) ^/SVN/path/to/project/tags/specific_tag@[number] > > Subversion first goes back in time to that revision, then goes to > that path. > > This is where I'm a bit foggy: Once it goes back in time and goes > to that path, does it then follow it forward to HEAD? Or does it > stay in the past? > > We've set up all of our externals like (3)... Do we need to go > through our repo and change all externals to specify both the peg > and operative revision? > > Thanks, > Nathan Hartman If you're permitting commits on top of tags, probably so.
Re: issue with relative externals after a rename
On Wed, Mar 28, 2018 at 10:59 AM, Nico Kadel-Garcia wrote: > On Wed, Mar 28, 2018 at 10:36 AM, Nathan Hartman > wrote: > > > Our requirements are: at any time in the future, if someone checks > > out code from the past, they should get exactly the same tree as what > > existed in the past. I assume that this is probably THE #1 use case > > and desired behavior for externals. Is that correct? If not, is there > > a good reason to want otherwise? > > Then make tags. Tags need to be locked down, with no commits permitted > on top of the tag. [snip] > > (3) ^/SVN/path/to/project/tags/specific_tag@[number] > > > > Subversion first goes back in time to that revision, then goes to > > that path. > > > > This is where I'm a bit foggy: Once it goes back in time and goes > > to that path, does it then follow it forward to HEAD? Or does it > > stay in the past? > > > > We've set up all of our externals like (3)... Do we need to go > > through our repo and change all externals to specify both the peg > > and operative revision? > > If you're permitting commits on top of tags, probably so. > Yup. Right now tags are not locked down. Perhaps they should be. But we will begin with specifying both the peg and operative revisions and then go from there. Thank you for your input. Thanks also to Branko and thanks to the OP for asking the original question, without which we would not have discovered this about our repo until after things broke later on. :-)
Re: issue with relative externals after a rename
Just curious since I am working on setting up our svn repositories migrated from CVS.How does one "lock down" tags to disallow further commits? In CVS that was built-in but svn works differently... Bo B.
Re: issue with relative externals after a rename
On 3/28/2018 9:37 AM, Bo Berglund wrote: Just curious since I am working on setting up our svn repositories migrated from CVS. How does one "lock down" tags to disallow further commits? In CVS that was built-in but svn works differently... Bo B. Put a check for the commit path(s) in the pre-commit hook. If a tag directory is in the commit list, return an error. http://svnbook.red-bean.com/nightly/en/svn.ref.reposhooks.pre-commit.html -- David Chapman dcchap...@acm.org Chapman Consulting -- San Jose, CA EDA Software Developer, Expert Witness www.chapman-consulting-sj.com 2018 Chair, IEEE Consultants' Network of Silicon Valley
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
On Wed, 28 Mar 2018 09:00:05 +0200, Johan Corveleyn wrote: >> Transmitting file data ... 8 long lines of dots >> ..svnsync: E120106: ra_serf: The server sent a >> truncated HTTP response body. > >I'd start with going through the apache logs on both servers to look >for more information about this error (since "The server sent a >truncated HTTP response body." indicates to me that an error should be >logged on that server). Unfortunately it is not possible for me at the moment since I am travelling... >It's also possible that some other network component has truncated >things ... proxies, security content filters, ... I tested the synchronize function on two repositories and they both triggered this error, but on different revs of course. So I used svnlook on the master server to find out what was actually being transferred for these revs. Turns out to be in both cases an initial import of a big "library" file set. In the case of the cmp repo it involved 9848 files some of which are very big. In the other case it was also an import affecting 3669 files of a cutdown version of the library.. The first import was done by cvs2svn when I converted the CVS repo to SVN, but the other was an import I did when I created a repo for a consultant who should have access only to a subset of our versioned files. So I exported these and then imported the export into the new repository. So I guess that the sheer size of the imports it tries to sync is what kills it, especially when doing it over the Internet QUESTION: - Is there a possibility to dump the source repos, then use the dumpfiles to set up the mirrors and finally tell svnsync to start syncing from the revision that is now the last in the mirror? It would certainly make it more controlled since I could ftp the dump files from the source to the destination and then do the svn operation locally with no network issues like latencies and such. I have read that one must start with the target repo at rev 0, but then I have also read some notes that later versions of svn have improved on this We use 1.9.3 on the mirror server and 1.9.7 on the master server. Grateful for any advice here! -- Bo Berglund Developer in Sweden
Re: Subversion 1.9.7 server on Windows, advice on setting up svnsync?
On Wed, 28 Mar 2018 12:41:49 -0500, Bo Berglund wrote: >QUESTION: >- >Is there a possibility to dump the source repos, then use the >dumpfiles to set up the mirrors and finally tell svnsync to start >syncing from the revision that is now the last in the mirror? It seems like 3 options are described in the svnbook: 1) Initial data transfer using the synchronize function. This is what is not really working in my situation. 2) Use hotcopy to make a backup copy of everything including hooks. The caveat here is that the hotcopy needs to be using the exact same version on both master and mirror. But I have 1.9.7 master on Windows and 1.9.3 mirror on Ubuntu Linux 3) Use the dump + Load sequence to transfer the repo content. Then after configuring all the hook scripts etc on the mirror the synchronize init can be run with a flag to allow a non-empty target repository. Seems like I have to go route #3 -- Bo Berglund Developer in Sweden
Re: issue with relative externals after a rename
On Wed, Mar 28, 2018 at 4:36 PM, Nathan Hartman wrote: > On Tue, Mar 27, 2018 at 11:36 PM, Branko Čibej wrote: >> >> Because the default operative revision is HEAD and there's no such >> object in HEAD, yes. > > > [snip] > >> >> You need both peg and operative revisions: >> >> -r 18 ../fold1 fold1@18 >> -r 18 ../fold2/file1.txt@18 file1.txt >> >> The peg revision tells Subversion which object to look for and the >> operative revision tells it which version of that object to use. > > > And then... > > On Wed, Mar 28, 2018 at 3:12 AM, Johan Corveleyn wrote: >> >> Are you sure? That's surprising to me, because for normal commands the >> operative revision always defaults to the peg revision. >> >> For example, in [1] I read "Notice that we didn't provide an operative >> revision this time. That's because when no operative revision is >> specified, Subversion assumes a default operative revision that's the >> same as the peg revision." >> >> Or is there a different rule for externals definitions? >> >> [1] http://svnbook.red-bean.com/nightly/en/svn.advanced.pegrevs.html > > > Oops, that makes me a bit nervous about how we've set up our > externals. > > Our requirements are: at any time in the future, if someone checks > out code from the past, they should get exactly the same tree as what > existed in the past. I assume that this is probably THE #1 use case > and desired behavior for externals. Is that correct? If not, is there > a good reason to want otherwise? > > My understanding is that: > > (1) ^/SVN/path/to/project/tags/specific_tag > > That will probably work most of the time but if someone commits > to that tag, future checkouts of past revisions will not look > exactly as they did in the past. > > (2) -r [number] ^/SVN/path/to/project/tags/specific_tag > > Subversion first goes to that path, then follows it back in time > to the specified revision. > > So if someone commits to that tag, the future checkout of the > past revision will get the past revision. > > But... if a tag is ever moved, removed, or renamed in the future, > (the project is declared defunct and removed, for example), what > Subversion finds at that path and tries to follow back in time > won't lead to the right place. > > (3) ^/SVN/path/to/project/tags/specific_tag@[number] > > Subversion first goes back in time to that revision, then goes to > that path. > > This is where I'm a bit foggy: Once it goes back in time and goes > to that path, does it then follow it forward to HEAD? Or does it > stay in the past? > > We've set up all of our externals like (3)... Do we need to go > through our repo and change all externals to specify both the peg > and operative revision? No, you do not need to change your externals. I believe your understanding is correct, and I think Branko was incorrect. Indeed, in the absence of an operative revision, the operative revision defaults to the peg revision (so it does not follow it forward to HEAD), also in externals definitions. The peg revision is sufficient to "pin" the externals. I just tested this with a test repository with svn 1.9.4. However, the problem Jonathan saw is, I think, because he renamed a parent path that is outside of the scope of the relative path of the externals definition. Looking at his example: On 28.03.2018 01:38, Jonathan Schell wrote: > I have a tree that looks like: > > https://svn/repo/name1/externs/normal_files_here > > The folder "externs" has two external properties: > ../fold1@18 fold1 > ../fold2/file1.txt@18 file1.txt > > These folders and file exist at rev 18. The commit to add the properties to > "externs" is commit 32. > > I then do a rename of "name1" to "name2". > > The externs now break, as the item they were pointing to no longer exists. I think the externals processing sees "../fold1@18", and apparently it first converts the relative path into an absolute path by looking at HEAD ... so "../fold1" becomes "/name2/fold1". Then it sees the peg revision, so it wants "/name2/fold1@18", which does not exist (at the time of @18, the parent folder was name1). Adding an operative revision "-r18" does not help. I'm not sure if this behaviour is "as designed", or whether it should be considered a bug. I'm inclined to think it's a bug. The "relative url -> absolute url" conversion should probably also take the peg revision into account. Here below is a transcript of the tests I did with my 1.9.4 client, confirming what I wrote above (with some comments prefixed with ###): [[[ C:\svntest>svnadmin create repos C:\svntest>svn co file:///C:/svntest/repos wc Checked out revision 0. C:\svntest>cd wc C:\svntest\wc>mkdir name1\externs name1\fold1 name1\fold2 C:\svntest\wc>echo test > name1\externs\normal_file.txt C:\svntest\wc>echo test > name1\fold1\file_in_fold1.txt C:\svntest\wc>echo test > name1\fold2\file_in_fold2.txt C:\svntest\wc>svn add name1 A name1 A name1\externs A
Is svnadmin load affected by hooks?
When I load a dump into an empty repo, will the operation be affected by the hooks I have already set up for svnsync? I have created dumpfiles for all 9 repos I want to sync, and now I am about to load these into the repos I had prepared for sync when I discovered that network problems interfered. -- Bo Berglund Developer in Sweden
Re: issue with relative externals after a rename
On Wed, Mar 28, 2018 at 7:48 PM Johan Corveleyn wrote: > > No, you do not need to change your externals. I believe your > understanding is correct, and I think Branko was incorrect. Indeed, in > the absence of an operative revision, the operative revision defaults > to the peg revision (so it does not follow it forward to HEAD), also > in externals definitions. The peg revision is sufficient to "pin" the > externals. I just tested this with a test repository with svn 1.9.4. Thank you. That is reassuring. We have not changed anything yet. However, the problem Jonathan saw is, I think, because he renamed a > parent path that is outside of the scope of the relative path of the > externals definition. Looking at his example: Yes that makes sense. Fortunately for us, all of our externals begin with "^/SVN/" so we are unaffected by the relative directory question. I think that when we designed our monorepo we thought that we may want to restructure in the future, e.g. move or rename project directories, combine projects, split projects, declare projects defunct, etc. So we did not rely on the relative placement of projects to each other, but we did use the caret notation without a URL scheme to ensure the repository itself is portable. So, absolute paths within the repository with peg revisions, relative path to the repository itself. I think the externals processing sees "../fold1@18", and apparently it > first converts the relative path into an absolute path by looking at > HEAD ... so "../fold1" becomes "/name2/fold1". Then it sees the peg > revision, so it wants "/name2/fold1@18", which does not exist (at the > time of @18, the parent folder was name1). Adding an operative > revision "-r18" does not help. > > I'm not sure if this behaviour is "as designed", or whether it should > be considered a bug. I'm inclined to think it's a bug. The "relative > url -> absolute url" conversion should probably also take the peg > revision into account. It makes intuitive sense that ../fold1@18 should Do The Right Thing. But if it takes "." and expands it to a full path, then strips off the last component, then applies the peg revision, that will explain the source of the issue. Part of the problem is trying to locate something in space and time starting from a leaf rather than the root. The leaf moves around; the root stays put. With the peg revision you usually know the full path that existed at some point in the past, so Subversion can pull up that revision and traverse down the tree to the correct location. But in this case you're starting with a leaf in some revision so you have to take "." and expand it to the full path in that revision, then traverse its history back to the peg revision to figure out what its full path was back then, and then go on to strip the last component etc. The problem is, what revision do you use to expand "." to the full path? HEAD? BASE? The way I see it, if it's HEAD and you check out an old revision, and things have moved around some more since then, I think you'll encounter the same kind of breakage. Nathan Hartman
