Re: BDB backend in SVN 1.7.1
Old BDB-backed repositories stored the older revision as fulltext and newer revisions as deltas. Really? Here is a quotation from SVN 1.4.6 libsvn_fs_base/note/structure: At present, Subversion generally stores the youngest strings in fulltext form, and older strings as deltas against them (unless the delta would save no space compared to the fulltext). My own experiments with SVN 1.4 code confirm that. Repositories created with or 'svnadmin upgrade'd by 1.6 and newer reverse this for new revisions of files (while making sure not to introduce a dependency loop in the direction of deltas). http://subversion.apache.org/docs/release-notes/1.6#bdb-forward-deltas On Friday, November 25, 2011 1:08 AM, Vyacheslav Zholudev vyacheslav.zholu...@gmail.com wrote: Hi, how does SVN 1.7.1 store fulltext and deltas in the BDB backend? From some time ago I remember that previous versions of SVN stored almost always a HEAD revision as fulltext, and others as reverse deltas.(except the case when a delta is bigger that fulltext) Was this behavior changed in SVN 1.7? I've looked at the notes about BDB and they don't differ almost at all from SVN 1.4's ones. Of course, I could look into the code more carefully, but my hope was that it wouldn't be a big deal to give me a short answer, if possible. Thanks in advance! Best, Vyacheslav
Re: BDB backend in SVN 1.7.1
Hi Daniel, would it be easy to change the code (I want to do it for my experiments) so that the HEAD (youngest) revisions are stored as fulltexts? Or is it something that was not foreseen by design to easily switch between approaches of representing history information? Thanks, Vyacheslav On Nov 25, 2011, at 11:59 AM, Daniel Shahaf wrote: Vyacheslav Zholudev wrote on Fri, Nov 25, 2011 at 11:13:00 +0100: Old BDB-backed repositories stored the older revision as fulltext and newer revisions as deltas. Really? It seems that I should have swapped older and newer in the quoted sentence. Thanks for catching that. Here is a quotation from SVN 1.4.6 libsvn_fs_base/note/structure: At present, Subversion generally stores the youngest strings in fulltext form, and older strings as deltas against them (unless the delta would save no space compared to the fulltext). My own experiments with SVN 1.4 code confirm that. Repositories created with or 'svnadmin upgrade'd by 1.6 and newer reverse this for new revisions of files (while making sure not to introduce a dependency loop in the direction of deltas). http://subversion.apache.org/docs/release-notes/1.6#bdb-forward-deltas On Friday, November 25, 2011 1:08 AM, Vyacheslav Zholudev vyacheslav.zholu...@gmail.com wrote: Hi, how does SVN 1.7.1 store fulltext and deltas in the BDB backend? From some time ago I remember that previous versions of SVN stored almost always a HEAD revision as fulltext, and others as reverse deltas.(except the case when a delta is bigger that fulltext) Was this behavior changed in SVN 1.7? I've looked at the notes about BDB and they don't differ almost at all from SVN 1.4's ones. Of course, I could look into the code more carefully, but my hope was that it wouldn't be a big deal to give me a short answer, if possible. Thanks in advance! Best, Vyacheslav
Re: BDB backend in SVN 1.7.1
Thanks, Daniel. That's the pointer I needed. However, I didn't understand what LAAEFTR means. Vyacheslav On Nov 25, 2011, at 12:17 PM, Daniel Shahaf wrote: Change SVN_FS_BASE__MIN_FORWARD_DELTAS_FORMAT to be larger than SVN_FS_BASE__FORMAT_NUMBER. Whether repositories created by an svn patched in this way will be interoperable with repositories created by an unpatched svn is LAAEFTR'd. I'd be cautious and change db/fs-type or db/format. Vyacheslav Zholudev wrote on Fri, Nov 25, 2011 at 12:04:22 +0100: Hi Daniel, would it be easy to change the code (I want to do it for my experiments) so that the HEAD (youngest) revisions are stored as fulltexts? Or is it something that was not foreseen by design to easily switch between approaches of representing history information? Thanks, Vyacheslav On Nov 25, 2011, at 11:59 AM, Daniel Shahaf wrote: Vyacheslav Zholudev wrote on Fri, Nov 25, 2011 at 11:13:00 +0100: Old BDB-backed repositories stored the older revision as fulltext and newer revisions as deltas. Really? It seems that I should have swapped older and newer in the quoted sentence. Thanks for catching that. Here is a quotation from SVN 1.4.6 libsvn_fs_base/note/structure: At present, Subversion generally stores the youngest strings in fulltext form, and older strings as deltas against them (unless the delta would save no space compared to the fulltext). My own experiments with SVN 1.4 code confirm that. Repositories created with or 'svnadmin upgrade'd by 1.6 and newer reverse this for new revisions of files (while making sure not to introduce a dependency loop in the direction of deltas). http://subversion.apache.org/docs/release-notes/1.6#bdb-forward-deltas On Friday, November 25, 2011 1:08 AM, Vyacheslav Zholudev vyacheslav.zholu...@gmail.com wrote: Hi, how does SVN 1.7.1 store fulltext and deltas in the BDB backend? From some time ago I remember that previous versions of SVN stored almost always a HEAD revision as fulltext, and others as reverse deltas.(except the case when a delta is bigger that fulltext) Was this behavior changed in SVN 1.7? I've looked at the notes about BDB and they don't differ almost at all from SVN 1.4's ones. Of course, I could look into the code more carefully, but my hope was that it wouldn't be a big deal to give me a short answer, if possible. Thanks in advance! Best, Vyacheslav
Re: BDB backend in SVN 1.7.1
Thanks, I studied math not in English, that's why I didn't know :) I made a simple tests and it seems to work nicely. However, I'm not sure whether it will work with more complicated cases like copying, deleting, etc. Vyacheslav On Nov 25, 2011, at 1:15 PM, Daniel Shahaf wrote: left as an exercise for the reader --- in other words, I was identifying a potential issue and letting the audience figure out the solution for themselves. It's a standard idiom in math textbooks... (and, of course, if you have questions about that interoperability issue, feel free to raise them on this list.) Vyacheslav Zholudev wrote on Fri, Nov 25, 2011 at 13:07:52 +0100: Thanks, Daniel. That's the pointer I needed. However, I didn't understand what LAAEFTR means. Vyacheslav On Nov 25, 2011, at 12:17 PM, Daniel Shahaf wrote: Change SVN_FS_BASE__MIN_FORWARD_DELTAS_FORMAT to be larger than SVN_FS_BASE__FORMAT_NUMBER. Whether repositories created by an svn patched in this way will be interoperable with repositories created by an unpatched svn is LAAEFTR'd. I'd be cautious and change db/fs-type or db/format. Vyacheslav Zholudev wrote on Fri, Nov 25, 2011 at 12:04:22 +0100: Hi Daniel, would it be easy to change the code (I want to do it for my experiments) so that the HEAD (youngest) revisions are stored as fulltexts? Or is it something that was not foreseen by design to easily switch between approaches of representing history information? Thanks, Vyacheslav On Nov 25, 2011, at 11:59 AM, Daniel Shahaf wrote: Vyacheslav Zholudev wrote on Fri, Nov 25, 2011 at 11:13:00 +0100: Old BDB-backed repositories stored the older revision as fulltext and newer revisions as deltas. Really? It seems that I should have swapped older and newer in the quoted sentence. Thanks for catching that. Here is a quotation from SVN 1.4.6 libsvn_fs_base/note/structure: At present, Subversion generally stores the youngest strings in fulltext form, and older strings as deltas against them (unless the delta would save no space compared to the fulltext). My own experiments with SVN 1.4 code confirm that. Repositories created with or 'svnadmin upgrade'd by 1.6 and newer reverse this for new revisions of files (while making sure not to introduce a dependency loop in the direction of deltas). http://subversion.apache.org/docs/release-notes/1.6#bdb-forward-deltas On Friday, November 25, 2011 1:08 AM, Vyacheslav Zholudev vyacheslav.zholu...@gmail.com wrote: Hi, how does SVN 1.7.1 store fulltext and deltas in the BDB backend? From some time ago I remember that previous versions of SVN stored almost always a HEAD revision as fulltext, and others as reverse deltas.(except the case when a delta is bigger that fulltext) Was this behavior changed in SVN 1.7? I've looked at the notes about BDB and they don't differ almost at all from SVN 1.4's ones. Of course, I could look into the code more carefully, but my hope was that it wouldn't be a big deal to give me a short answer, if possible. Thanks in advance! Best, Vyacheslav
Re: BDB backend in SVN 1.7.1
I guess badness can happen only when accessing repositories locally (not via svn:// or http(s)://) with patched and not patched SVN. But usually only one version of SVN is installed on the server side, so that should not be a big problem. However, it's a nice exercise to check. Vyacheslav On Nov 25, 2011, at 1:29 PM, Daniel Shahaf wrote: To clarify, the issues I was concerned about weren't with tree changes (the level of the code dealing with content reps isn't aware of those), but with creating/accessing a single repository sometimes via unmodified svn 1.7.1 libraries and sometimes via forward-delta-patched libraries. The part I left to the readers was determining whther or not Badness will happen in the event of such mixed access. Vyacheslav Zholudev wrote on Fri, Nov 25, 2011 at 13:20:55 +0100: Thanks, I studied math not in English, that's why I didn't know :) I made a simple tests and it seems to work nicely. However, I'm not sure whether it will work with more complicated cases like copying, deleting, etc. Vyacheslav On Nov 25, 2011, at 1:15 PM, Daniel Shahaf wrote: left as an exercise for the reader --- in other words, I was identifying a potential issue and letting the audience figure out the solution for themselves. It's a standard idiom in math textbooks... (and, of course, if you have questions about that interoperability issue, feel free to raise them on this list.) Vyacheslav Zholudev wrote on Fri, Nov 25, 2011 at 13:07:52 +0100: Thanks, Daniel. That's the pointer I needed. However, I didn't understand what LAAEFTR means. Vyacheslav On Nov 25, 2011, at 12:17 PM, Daniel Shahaf wrote: Change SVN_FS_BASE__MIN_FORWARD_DELTAS_FORMAT to be larger than SVN_FS_BASE__FORMAT_NUMBER. Whether repositories created by an svn patched in this way will be interoperable with repositories created by an unpatched svn is LAAEFTR'd. I'd be cautious and change db/fs-type or db/format. Vyacheslav Zholudev wrote on Fri, Nov 25, 2011 at 12:04:22 +0100: Hi Daniel, would it be easy to change the code (I want to do it for my experiments) so that the HEAD (youngest) revisions are stored as fulltexts? Or is it something that was not foreseen by design to easily switch between approaches of representing history information? Thanks, Vyacheslav On Nov 25, 2011, at 11:59 AM, Daniel Shahaf wrote: Vyacheslav Zholudev wrote on Fri, Nov 25, 2011 at 11:13:00 +0100: Old BDB-backed repositories stored the older revision as fulltext and newer revisions as deltas. Really? It seems that I should have swapped older and newer in the quoted sentence. Thanks for catching that. Here is a quotation from SVN 1.4.6 libsvn_fs_base/note/structure: At present, Subversion generally stores the youngest strings in fulltext form, and older strings as deltas against them (unless the delta would save no space compared to the fulltext). My own experiments with SVN 1.4 code confirm that. Repositories created with or 'svnadmin upgrade'd by 1.6 and newer reverse this for new revisions of files (while making sure not to introduce a dependency loop in the direction of deltas). http://subversion.apache.org/docs/release-notes/1.6#bdb-forward-deltas On Friday, November 25, 2011 1:08 AM, Vyacheslav Zholudev vyacheslav.zholu...@gmail.com wrote: Hi, how does SVN 1.7.1 store fulltext and deltas in the BDB backend? From some time ago I remember that previous versions of SVN stored almost always a HEAD revision as fulltext, and others as reverse deltas.(except the case when a delta is bigger that fulltext) Was this behavior changed in SVN 1.7? I've looked at the notes about BDB and they don't differ almost at all from SVN 1.4's ones. Of course, I could look into the code more carefully, but my hope was that it wouldn't be a big deal to give me a short answer, if possible. Thanks in advance! Best, Vyacheslav
Re: BDB backend in SVN 1.7.1
On Nov 25, 2011, at 2:02 PM, Stefan Sperling wrote: On Fri, Nov 25, 2011 at 01:38:44PM +0100, Vyacheslav Zholudev wrote: I guess badness can happen only when accessing repositories locally (not via svn:// or http(s)://) with patched and not patched SVN. But usually only one version of SVN is installed on the server side, so that should not be a big problem. However, it's a nice exercise to check. Out of curiosity, what is the reason for this entire exercise? Is it purely academic or is there a use case where storing the HEAD revision as full-text is critical? The reason for this is that I had a project based on SVN 1.4 which stored HEAD revisions of XML files in Berkeley DB XML - a native XML database-, instead of BDB. (reverted deltas as well as normal files kept on living in BDB). In this way I could also query HEAD XML files directly from SVN repository and access them via BDB XML API - thus it was a nice combination of SVN and XML-database. Recently I've ported my project to SVN 1.7 and due to forward deltas it didn't work as before -- only old revisions of XML files were in BDB XML. But now I changed SVN_FS_BASE__MIN_FORWARD_DELTAS_FORMAT to a higher number, and my project works again as expected. Vyacheslav
BDB backend in SVN 1.7.1
Hi, how does SVN 1.7.1 store fulltext and deltas in the BDB backend? From some time ago I remember that previous versions of SVN stored almost always a HEAD revision as fulltext, and others as reverse deltas.(except the case when a delta is bigger that fulltext) Was this behavior changed in SVN 1.7? I've looked at the notes about BDB and they don't differ almost at all from SVN 1.4's ones. Of course, I could look into the code more carefully, but my hope was that it wouldn't be a big deal to give me a short answer, if possible. Thanks in advance! Best, Vyacheslav
Re: svnadmin load is slow for many small revisions
We are not going to fix any speed issues in 1.5. Yes, I understand that. My hope was that it's not a direct SVN issue, but some problem of incorrect linking libraries or the like. That said, why are you so sure you must use exactly 1.4? Subversion 1.7 can create, read, and write 1.4 repositories. And there have been significant performance improvements in the past four years... I just have my own project based on the code of SVN 1.4 and porting to 1.7 may take considerable efforts. I post this message just in a hope that somebody could have any ideas why slowdown started to appear on a new version of linux distribution. Vyacheslav Vyacheslav Zholudev wrote on Tue, Nov 15, 2011 at 23:29:09 +0100: Hi all, After transition from debian lenny to squeeze I recompiled my version of SVN 1.4 (yes, I do need exactly 1.4). I tried to load a dump with many 1 file revisions into the BDB-based repository. First 300 revisions are committed really fast, like 10 revisions per second. After 1000 revisions, it is about 1-2 seconds per one revision. And performance keeps degrading. I'm curious what that might be? I tried to compile against different versions of APR (from 1.3 to 1.5) with BDB 4.8. I used libtool 1.5 since autogen.sh does not work with libtool 2.2. May this be somehow related to my issue? Maybe some compilations flags are wrong for a new system, or some libraries that I'm linking against (e.g. neon)? I have no real clue what that might be, so I will highly appreciate if somebody can give me any pointers, even vague ones or possible hints to try, since I really need SVN 1.4 for my other projects. P.S. CPU usage is very high, waiting for IO was about high (that what I would expect) and decreases to 0% over the time. P.P.S. I also tried to compile in maintainer mode and without, with preinstalled APR, and by putting them into source tree, with mod_dav and without. Still the same... Thanks a lot in advance, Vyacheslav Best, Vyacheslav
svnadmin load is slow for many small revisions
Hi all, After transition from debian lenny to squeeze I recompiled my version of SVN 1.4 (yes, I do need exactly 1.4). I tried to load a dump with many 1 file revisions into the BDB-based repository. First 300 revisions are committed really fast, like 10 revisions per second. After 1000 revisions, it is about 1-2 seconds per one revision. And performance keeps degrading. I'm curious what that might be? I tried to compile against different versions of APR (from 1.3 to 1.5) with BDB 4.8. I used libtool 1.5 since autogen.sh does not work with libtool 2.2. May this be somehow related to my issue? Maybe some compilations flags are wrong for a new system, or some libraries that I'm linking against (e.g. neon)? I have no real clue what that might be, so I will highly appreciate if somebody can give me any pointers, even vague ones or possible hints to try, since I really need SVN 1.4 for my other projects. P.S. CPU usage is very high, waiting for IO was about high (that what I would expect) and decreases to 0% over the time. P.P.S. I also tried to compile in maintainer mode and without, with preinstalled APR, and by putting them into source tree, with mod_dav and without. Still the same... Thanks a lot in advance, Vyacheslav
