Re: svn 1.8 migration - directory deltification and revprop packing
One advantage of being in a room full of Subversion developers, specifically the guy that implemented all this stuff, is that I can ask him directly about how to respond to this mail. :-) Hopefully I will accurately represent the answers Stefan Fuhrmann just gave me to your questions. On 06/10/2013 03:05 PM, Thomas Harold wrote: a) Why are directory/property deltifications turned off by default? Stefan's jovial first answer was, Because I'm a chicken. :-) Fortunately, he didn't stop there. He explained that there is no known correctness risk -- you're not going to damage your repositories by enabling the feature. But he wanted to phase in the feature to allow time to collect real-world data about the amount of space savings folks are actually observing in their repositories. The feature is on by default in his proposed next version of the FSFS format. b) Is there a global setting file that can be used to enable directory/property deltifications? Or will we have to update the fsfs.conf file for each newly created repository in order to get this feature? In 1.8, you'll need to toggle this for each new repository manually. c) Is it a safe assumption that in order to apply this change to an older repository that we will need a dump/load cycle? Will we need a full dump or will an delta style dump suffice (--deltas option of svnadmin dump command)? Not exactly. You can apply the change to an older repository sitting behind a server still running 1.8, and any new directory/property lists will be stored in a deltified fashion. If you want retroactive deltification of existing data, then yes, you'll need to dump and load your repositories. But as explained in the release notes, you can dump and re-load right back into a previous version of the repository format if you'd prefer to maintain compatibility with older server versions. As for the --deltas option, that has nothing in the world to do with the types of deltas we're discussing here. (As an aside, I would highly recommend that, unless you need your dumpfiles to be smaller, avoid the --deltas option. The performance penalty of using it isn't worth it.) #2 - revision property (revprops) files packing a) Will there be a svnadmin pack command like there was for SVN 1.6? Or will we need to do a full dump/load of the repository to pack the revprops? The existing 'svnadmin pack' command will govern both revision and revprop packing, and will keep the two in sync with each other. 'svnadmin upgrade' will also take the opportunity to synchronize the packing status of the revision properties with that of the revision backing files. b) Does revprop caching only need to be enabled for http/https access and does it have any effect on svn+ssh access? (All of our users currently use svn+ssh access, but we are considering moving to http/https.) Each server has its own cache enable/disable configuration mechanisms, and both benefit from the revprop caching being enabled. -- C. Michael Pilato cmpil...@collab.net CollabNet www.collab.net Enterprise Cloud Development signature.asc Description: OpenPGP digital signature
Re: svn 1.8 migration - directory deltification and revprop packing
On 6/11/2013 8:52 AM, C. Michael Pilato wrote: One advantage of being in a room full of Subversion developers, specifically the guy that implemented all this stuff, is that I can ask him directly about how to respond to this mail. :-) Hopefully I will accurately represent the answers Stefan Fuhrmann just gave me to your questions. Thank you very much. b) Is there a global setting file that can be used to enable directory/property deltifications? Or will we have to update the fsfs.conf file for each newly created repository in order to get this feature? In 1.8, you'll need to toggle this for each new repository manually. I'll cobble something together with grep, sed/awk, and find then to monitor (and update) our fsfs.conf files. As for the --deltas option, that has nothing in the world to do with the types of deltas we're discussing here. (As an aside, I would highly recommend that, unless you need your dumpfiles to be smaller, avoid the --deltas option. The performance penalty of using it isn't worth it.) Right now, the size of our dumpfile directory is 207G, while the hotcopy is only 104G. So the size savings could be big for us. The hotcopy backup is still our preferred solution, with the dump files being a worst-case fallback. #2 - revision property (revprops) files packing a) Will there be a svnadmin pack command like there was for SVN 1.6? Or will we need to do a full dump/load of the repository to pack the revprops? The existing 'svnadmin pack' command will govern both revision and revprop packing, and will keep the two in sync with each other. 'svnadmin upgrade' will also take the opportunity to synchronize the packing status of the revision properties with that of the revision backing files. Thanks, the svn book is light on details of what exactly counts as minimum amount of work needed for svnadmin upgrade.
Re: svn 1.8 migration - directory deltification and revprop packing
On 06/11/2013 04:13 PM, Thomas Harold wrote: Thanks, the svn book is light on details of what exactly counts as minimum amount of work needed for svnadmin upgrade. svnbook-...@red-bean.com would be a great place to report that, though I'd suggest expanding your report with some context. The minimum amount of work for svnadmin upgrade is, in the general case, to run svnadmin upgrade. It's only when you start trying to take advantage of new features provided by the upgraded format that extra work is involved. -- C. Michael Pilato cmpil...@collab.net CollabNet www.collab.net Enterprise Cloud Development signature.asc Description: OpenPGP digital signature
Re: svn 1.8 migration - directory deltification and revprop packing
C. Michael Pilato wrote on Tue, Jun 11, 2013 at 14:52:48 +0200: On 06/10/2013 03:05 PM, Thomas Harold wrote: b) Does revprop caching only need to be enabled for http/https access and does it have any effect on svn+ssh access? (All of our users currently use svn+ssh access, but we are considering moving to http/https.) Each server has its own cache enable/disable configuration mechanisms, and both benefit from the revprop caching being enabled. Note, though, that svn+ssh spawns one (or more) short-lived svnserve instances, and the revprop cache is per-process. Consequently, the cache will be effective for single operations that use a single server connection (such as 'svn checkout') but will not carry through between clients, or even for a single client that closes and re-opens its connection.
Re: svn 1.8 migration - directory deltification and revprop packing
C. Michael Pilato wrote on Tue, Jun 11, 2013 at 17:32:59 +0200: On 06/11/2013 05:14 PM, Daniel Shahaf wrote: C. Michael Pilato wrote on Tue, Jun 11, 2013 at 14:52:48 +0200: As for the --deltas option, that has nothing in the world to do with the types of deltas we're discussing here. (As an aside, I would highly recommend that, unless you need your dumpfiles to be smaller, avoid the --deltas option. The performance penalty of using it isn't worth it.) That's because we store skip-deltas but dumpfiles want deltas against predecessor, right? So, two thoughts: - Is this still a problem with the new max-linear-deltification setting? - Can we teach 'svnadmin dump' to just dump whatever delta is stored in the filesystem, plus the base of that delta? For the common case of loading the entire dump file, it's not really important what the delta base is so long as it's older than the dumped revision. To do so, we'd need to expose the deltas via the libsvn_fs API, which is currently not the case. Deltas in the repository filesystem are an implementation detail of that filesystem. They are not even guaranteed to be implemented in a fashion that is compatible with what is used in the repository dump stream format. I imagined we could have an API similar to svn_fs_try_process_file_contents(): if you have an svn_txdelta()-delta precalculated, hand it, else return SVN_ERR_UNSUPPORTED_FEATURE.
Re: svn 1.8 migration - svnadmin hotcopy
On 6/11/2013 10:20 AM, Stefan Sperling wrote:
On Tue, Jun 11, 2013 at 10:13:15AM -0400, Thomas Harold wrote:
Right now, the size of our dumpfile directory is 207G, while the
hotcopy is only 104G. So the size savings could be big for us. The
hotcopy backup is still our preferred solution, with the dump files
being a worst-case fallback.
Please try the new svnadmin hotcopy --incremental.
It should accelerate your backup process.
Yes, I'm looking forward to that feature in 1.8. We currently tackle
the time issue in two ways:
1) We only svnadmin hotcopy repositories which have changed in the last
N days (typically 3 days). Since we have about 300 repositories
currently, but we don't do work on things in all 300 constantly, this
means we only backup a few dozen repositories each night.
BASE=/var/svn/
DAYS=3
# Directories get randomized with the perl fragment, so that
# they get processed in random order. This makes the backups
# more reliable over the long term in case one directory
# causes problems.
DIRS=`$FIND ${BASE} -maxdepth 3 -name current -mtime -${DAYS} | \
$GREP 'db/current$' | \
$SED 's:/db/current$::' | $SED s:^${BASE}:: | \
perl -MList::Util -e 'print List::Util::shuffle '`
2) We read the svn repositories from one set of spindles and write the
hotcopy to a second spindle set. Even with the 104GB and 300
repositories that we have, this only takes ~37 minutes.
It still takes 4-5 hours to perform the rdiff-backup step that pushes
the hotcopy folder over to our internal backup server, but that's more
because of the tens of thousands of revprops files in some of the
repositories. Which is another feature in 1.8 that I'm looking forward to.
History in subversion
Hi, I'm trying to work with SVN, but coming from ClearCase, I'm lost. It seems that it is not possible to consult the history of the repository like in CC, I can get the history of a file element with svn+annotate, I can use svn+diff on files, But for directories : no annotate, no diff - nothing ? Do I have to check all the repository revision set in order to follow the changes on a directory element ? Regards Olivier
Re: History in subversion
On Tue, Jun 11, 2013 at 3:15 PM, Olivier Antoine oliviera201...@gmail.com wrote: I'm trying to work with SVN, but coming from ClearCase, I'm lost. It seems that it is not possible to consult the history of the repository like in CC, I can get the history of a file element with svn+annotate, I can use svn+diff on files, But for directories : no annotate, no diff - nothing ? Do I have to check all the repository revision set in order to follow the changes on a directory element ? You want svn log. The book has a good explanation: http://svnbook.red-bean.com/en/1.7/svn.tour.history.html -- Thanks Mark Phippard http://markphip.blogspot.com/
Re: History in subversion
I don't think that svn log describes the history of the directory element, Actually I don't see any command that could give the revisions where the directory changed (tree changes), without giving the changes on the files it contents. No svn+annotate on directories. And I don't see any way to apply svn+diff on the directory element only I tried : svn diff --depth empty = but got only properties information. 2013/6/11 Olivier Antoine oliviera201...@gmail.com Hi, I'm trying to work with SVN, but coming from ClearCase, I'm lost. It seems that it is not possible to consult the history of the repository like in CC, I can get the history of a file element with svn+annotate, I can use svn+diff on files, But for directories : no annotate, no diff - nothing ? Do I have to check all the repository revision set in order to follow the changes on a directory element ? Regards Olivier
Re: History in subversion
On Tue, Jun 11, 2013 at 4:02 PM, Olivier Antoine oliviera201...@gmail.com wrote: I don't think that svn log describes the history of the directory element, OK, did not pick up that you were only interested in that information. You are right that SVN will always show the information for directory plus children. The only changes to a directory that SVN tracks are prop changes, so in general the history of a folder is uninteresting. If you wanted the changes to just the directory you would need to use the --verbose option on log which will print the changed paths. You would then have to parse out the interesting revisions by finding the ones where the folder path is included. Actually I don't see any command that could give the revisions where the directory changed (tree changes), without giving the changes on the files it contents. No svn+annotate on directories. And I don't see any way to apply svn+diff on the directory element only I tried : svn diff --depth empty = but got only properties information. That is all you would get for a directory. -- Thanks Mark Phippard http://markphip.blogspot.com/
Re: History in subversion
Thanks for your help, I will try again this. But this is very poor compared to ClearCase. Nobody tried to script that ? It is also possible to read the SVN repository without checkout, there is a way to address an element, something like this : element url@revnumber But it is not possible to use a syntax like this : directory@revnumber/file I tried, it doesn't work. Actually, I just try to analyze all elements, files and directories, contained in a SVN repository. I'd like to be able to parse all the elements - if possible without any checkout (that would be great). Other challenge is : I need to restore a file element that has been removed in a very old revision, and of course I don't know which one. Any search command or script with Subversion ? 2013/6/11 Olivier Antoine oliviera201...@gmail.com Hi, I'm trying to work with SVN, but coming from ClearCase, I'm lost. It seems that it is not possible to consult the history of the repository like in CC, I can get the history of a file element with svn+annotate, I can use svn+diff on files, But for directories : no annotate, no diff - nothing ? Do I have to check all the repository revision set in order to follow the changes on a directory element ? Regards Olivier
Re: History in subversion
On Tue, Jun 11, 2013 at 4:45 PM, Olivier Antoine oliviera201...@gmail.com wrote: Thanks for your help, I will try again this. But this is very poor compared to ClearCase. Nobody tried to script that ? It is open-source, so nothing is stopping someone from implementing this if they find it important. I think most of us do not see a need for it so it has not risen to the level of being someone's itch to scratch. I am sure someone has scripted this but whether they shared it and you can find it are questions I cannot answer. It is also possible to read the SVN repository without checkout, there is a way to address an element, something like this : element url@revnumber But it is not possible to use a syntax like this : directory@revnumber/file I tried, it doesn't work. Have you tried this? http://subversion.apache.org/docs/release-notes/1.6.html#historical-uris Note that would just be for using something like curl or wget. A svn client has always been capable of doing this. Actually, I just try to analyze all elements, files and directories, contained in a SVN repository. I'd like to be able to parse all the elements - if possible without any checkout (that would be great). Other challenge is : I need to restore a file element that has been removed in a very old revision, and of course I don't know which one. Any search command or script with Subversion ? Restoring something that was removed is easy. However, finding the element and when it was deleted is unfortunately not easy. As with the previous example, you would want to run svn log with the --verbose option and parse the output to find the revision when the item was removed. Looking at the history of a parent folder for the item would show when it was deleted, but you would have to search the output or script it. -- Thanks Mark Phippard http://markphip.blogspot.com/
Re: History in subversion
On Tue, Jun 11, 2013 at 3:45 PM, Olivier Antoine oliviera201...@gmail.com wrote: Thanks for your help, I will try again this. But this is very poor compared to ClearCase. Nobody tried to script that ? svn logs will show file/directory additions/deletions in the parent directory, so you should be able to track the history of things that way if you wanted, but what is it that you specifically need to do? Most people would just check out some directory level and diff it against some other revision or a branch or tag. It is also possible to read the SVN repository without checkout, there is a way to address an element, something like this : element url@revnumber But it is not possible to use a syntax like this : directory@revnumber/file I tried, it doesn't work. The revnumber is global to the whole repository, so path/file@revnumber would mean the same thing. Actually, I just try to analyze all elements, files and directories, contained in a SVN repository. I'd like to be able to parse all the elements - if possible without any checkout (that would be great). You need a starting point and you can log backwards. If you are just browsing, tools like tortoise or web interfaces like viewvc or websvn might be easier. Other challenge is : I need to restore a file element that has been removed in a very old revision, and of course I don't know which one. Any search command or script with Subversion ? If you know the parent directory the log will show the deletion and the revision number where it occurred. But if there were many additions/deletions of files of the same name or at different places in the tree it can be hard to find the one you want. -- Les Mikesell lesmikes...@gmail.com
RE: History in subversion
From: Olivier Antoine [mailto:oliviera201...@gmail.com] Sent: Tuesday, June 11, 2013 13:45 To: users@subversion.apache.org Subject: Re: History in subversion Thanks for your help, I will try again this. But this is very poor compared to ClearCase. Nobody tried to script that ? It is also possible to read the SVN repository without checkout, there is a way to address an element, something like this : element url@revnumber But it is not possible to use a syntax like this : directory@revnumber/file I tried, it doesn't work. Actually, I just try to analyze all elements, files and directories, contained in a SVN repository. I'd like to be able to parse all the elements - if possible without any checkout (that would be great). Unlike ClearCase, Subversion does not have elements each with their own history. Rather, the whole repository is versioned with each commit. So, there is no benefit to this syntax: directory@revnumber/file Instead, use: directory/file@revnumber The Revision identifies a snapshot of the whole repository, not just some snapshot of a directory. -Steve Other challenge is : I need to restore a file element that has been removed in a very old revision, and of course I don't know which one. Any search command or script with Subversion ? 2013/6/11 Olivier Antoine oliviera201...@gmail.com Hi, I'm trying to work with SVN, but coming from ClearCase, I'm lost. It seems that it is not possible to consult the history of the repository like in CC, I can get the history of a file element with svn+annotate, I can use svn+diff on files, But for directories : no annotate, no diff - nothing ? Do I have to check all the repository revision set in order to follow the changes on a directory element ? Regards Olivier
RE: History in subversion
From: Olivier Antoine [mailto:oliviera201...@gmail.com] Sent: Tuesday, June 11, 2013 4:45 PM To: users@subversion.apache.org Subject: Re: History in subversion Thanks for your help, I will try again this. But this is very poor compared to ClearCase. Nobody tried to script that ? I used to use ClearCase in a past life (3.0 - 6.0). I haven't missed the ability to diff dirs. You might be stuck on doing things the CC way instead of learning the Subversion paradigms. It's going to be frustrating for a little while (it was for me.) What are you trying to do that requires diff'ing the contents of directories? It is also possible to read the SVN repository without checkout, there is a way to address an element, something like this : element url@revnumber But it is not possible to use a syntax like this : directory@revnumber/file You want to read up on peg revisions: http://svnbook.red-bean.com/en/1.7/svn.advanced.pegrevs.html You don't need to specify the rev for each path in the component, e.g. /view/foo/path@1/foo@2/bar@3/j.java@5. In SVN, the rev number is global, so you just /path/foo/bar/j.java@5. I tried, it doesn't work. Actually, I just try to analyze all elements, files and directories, contained in a SVN repository. I'd like to be able to parse all the elements - if possible without any checkout (that would be great). Use svn export (or for individual files, svn cat) Other challenge is : I need to restore a file element that has been removed in a very old revision, and of course I don't know which one. Any search command or script with Subversion ? You'll need to use a peg revision, e.g. svn copy svn:///path/foo.java@1234 . To find the revision that the file was in can be tricky if you have deleted a parent dir. In the average case, you can run svn log -v -q ^/path/to/branch log.txt. Then search the text file for your missing file to get the revision. You can also use svn log -v -q | egrep '^r|\/foo.java'. Worse case (you deleted a parent dir), you'll need to run the 'svn log' against the root of the repository (svn log -v -q ^/). To re-emphasize, I'm very serious about the need to stop trying to apply CC paradigms to SVN. It's frustrating, and, in my experience, the CC way of doing things didn't provide significant advantages in (or over) SVN.
Re: History in subversion
On Jun 11, 2013, at 15:45, Olivier Antoine wrote: But this is very poor compared to ClearCase. Nobody tried to script that ? Most of us have never seen or used ClearCase. It might help if you provided a specific example of a command that you might run with ClearCase, and the output you would expect to receive from it. Then perhaps we can show you the Subversion way to get that information.
