Re: svnserve with SASL and two Kerberos realms
I think this is the issue I have hit: http://subversion.tigris.org/issues/show_bug.cgi?id=3394 Does anyone know if there is any progress with this problem? Victor Sudakov wrote: > Dear Colleagues: > > I have two Kerberos realms: SIBPTUS.RU and SIBPTUS.TOMSK.RU with > mutual trust. > > svnserve is configured to use Kerberos: > > [general] > anon-access = none > auth-access = write > realm = SIBPTUS.RU > #realm = SIBPTUS.TOMSK.RU > #realm = GSS_C_NO_NAME > #realm = GSS_C_NO_CREDENTIAL > [sasl] > use-sasl = true > > If I uncomment the 'realm = SIBPTUS.TOMSK.RU' line, svnserve does not > authenticate users from the SIBPTUS.RU realm, and vice versa: > > svn: E170013: Unable to connect to a repository at URL 'XX > svn: E170001: Authentication error from server: SASL(-5): bad protocol / > cancel: security flags do not match required > > Can I configure svnserve/SASL to authenticate clients from both > realms? It would be great if svnserve considers j...@sibptus.ru and > j...@sibptus.tomsk.ru different users (from the point of view of > logging etc). > > I have tried GSS_C_NO_NAME and GSS_C_NO_CREDENTIAL as realm names, > without any success. > > I am using this setup (two realms) very successfully with sshd (via > the ~/.k5login mechanism) and with the squid kerberos helper which > does not care about the realm and just passes user@REALM to squid > itself. Only svnserve seems to be a problem. > > Thanks in advance for any input. > > -- > Victor Sudakov, VAS4-RIPE, VAS47-RIPN > sip:suda...@sibptus.tomsk.ru -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
svnserve with SASL and two Kerberos realms
Dear Colleagues: I have two Kerberos realms: SIBPTUS.RU and SIBPTUS.TOMSK.RU with mutual trust. svnserve is configured to use Kerberos: [general] anon-access = none auth-access = write realm = SIBPTUS.RU #realm = SIBPTUS.TOMSK.RU #realm = GSS_C_NO_NAME #realm = GSS_C_NO_CREDENTIAL [sasl] use-sasl = true If I uncomment the 'realm = SIBPTUS.TOMSK.RU' line, svnserve does not authenticate users from the SIBPTUS.RU realm, and vice versa: svn: E170013: Unable to connect to a repository at URL 'XX svn: E170001: Authentication error from server: SASL(-5): bad protocol / cancel: security flags do not match required Can I configure svnserve/SASL to authenticate clients from both realms? It would be great if svnserve considers j...@sibptus.ru and j...@sibptus.tomsk.ru different users (from the point of view of logging etc). I have tried GSS_C_NO_NAME and GSS_C_NO_CREDENTIAL as realm names, without any success. I am using this setup (two realms) very successfully with sshd (via the ~/.k5login mechanism) and with the squid kerberos helper which does not care about the realm and just passes user@REALM to squid itself. Only svnserve seems to be a problem. Thanks in advance for any input. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: Cannot load partial dump
Daniel Shahaf wrote: > > $ svnadmin dump repos/configs |\ > > svndumpfilter --drop-empty-revs include /cisco/trunk/caracal-confg \ > > > tmp/test1.svn > > [copious output deleted] > > > > $ svnadmin create test2 > > Run 'svn mkdir --parents file://$(pwd)/test2/cisco/trunk' at this point. > > > $ svnadmin load test2 < tmp/test1.svn Thanks a lot for your help! -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Cannot load partial dump
Dear Colleagues, I need to export one file's history to another repository. Here is what I do and what error I get: $ svnadmin dump repos/configs |\ svndumpfilter --drop-empty-revs include /cisco/trunk/caracal-confg \ > tmp/test1.svn [copious output deleted] $ svnadmin create test2 $ svnadmin load test2 < tmp/test1.svn <<< Started new transaction, based on original revision 540 * editing path : cisco/trunk/caracal-confg ...svnadmin: E160013: * File not found: transaction '0-0', path * '/cisco/trunk/caracal-confg' What am I doing wrong? If I dump/load the complete repository (without svndumpfilter), the operation is successful. But I don't need all the files from the old repository. Any advice is greatly appreciated. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
how can it merge revisions committed before the branch was forked?
Colleagues, I created a branch by running "svn copy trunk branches/FOO" in my working copy and committed the new files, which created revison 3053. Then I modified some files in both locations. When I run "svn mergeinfo" in branches/FOO, it shows the following revisions for merging: r3041 r3051 r3054 r3055 Why does it suggest merging revisions which were committed before branches/FOO was even forked? When I tried to merge nonetheless, some weird conflicts arose. Have I done something wrong? -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: SVN hosting with WebSVN?
I know about http://www.svnhostingcomparison.com/ , but I was asking for some personal experience. If you personally use an SVN hosting with the desired features, please let me know. Arwin Arni Nandagopal wrote: > Try http://www.svnhostingcomparison.com/ > > -Original Message- > From: Victor Sudakov [mailto:suda...@sibptus.tomsk.ru] > Sent: Friday, January 20, 2012 5:54 PM > To: users@subversion.apache.org > Subject: SVN hosting with WebSVN? > > Colleagues, > > It can be sort of a FAQ, but I could not find it in the archives. > > Can you recommend a free or inexpensive SVN hosting with the following > features: > > 1. The repository should be accessible for anonymous checkout. > > 2. The repository should be accessible for browsing via WebSVN or a > similar Web frontend. > > 3. There should be no restriction on the contents of the repository (I > plan to keep and publish texts and articles there, not software code). > > If you happen to know about such a service (except setting it up for > myself), please share. TIA. > > -- > Victor Sudakov, VAS4-RIPE, VAS47-RIPN > sip:suda...@sibptus.tomsk.ru -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
SVN hosting with WebSVN?
Colleagues, It can be sort of a FAQ, but I could not find it in the archives. Can you recommend a free or inexpensive SVN hosting with the following features: 1. The repository should be accessible for anonymous checkout. 2. The repository should be accessible for browsing via WebSVN or a similar Web frontend. 3. There should be no restriction on the contents of the repository (I plan to keep and publish texts and articles there, not software code). If you happen to know about such a service (except setting it up for myself), please share. TIA. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: Combining public and private paths
Stefan Sperling wrote: > > Can two svnserves share one repository? > > Yes. You can run as many server instances as you like, also with > different access methods (e.g. http:// and svn:// at the same time). I have read that different access methods can be used simultaneously. I did not know it was possible for several svnserve processes to access the same repository. That's great news. I think it solves my problem, I will just setup another svnserve process for anonymous users listening on a different TCP port, with a special --config-file. But wait. How do I setup per repository authz-db together with a global svnserve.conf? My svnserve serves several repositories. > > > There will be no data corruption, will there? > > In general, no. There are some multi-access problems with BDB-based > repositories running into issues with berkeleyDB where you might have I use FSFS. In fact, I have hated BDB for many years since I ran slapd and especially spamprobe with the BDB backend. > to unwedge repositories using svnadmin recover: > http://subversion.apache.org/faq.html#stuck-bdb-repos > > However, these days FSFS-based repositories are the default and > they don't have that issue. Do they have some locking mechanism? -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: Combining public and private paths
Can two svnserves share one repository? There will be no data corruption, will there? Daniel Shahaf wrote: > Workaround: you could run two svnserves with different configs, one > allowing only anonymous access and only only authenticated access. > > I know httpd has the problem you're describing, I don't recall previous > reports of it with svnserve. > > Victor Sudakov wrote on Thu, Feb 10, 2011 at 21:14:24 +0600: > > == conf/authz: > > [/] > > @noc = rw > > > > [/foo] > > $anonymous = r > > $authenticated = rw > > > > does not work. A valid user from the noc group receives the following reply: > > > > $ svn diff -c2237 www.txt > > svn: Unreadable path encountered; access denied > > It would be relevant to know www.txt absolute path. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: Combining public and private paths
What the "anon-access = none" option does is remove the ANONYMOUS mech from the list of SASL mechs offered by svnserve (I see this in tcpflow). If this mech is present in the mech list, the svn client does not bother to authenticate even if a valid Kerberos ticket is available. If the svn client had an option to enforce authentication even if offered the ANONYMOUS mech by the server, the problem would be solved IMHO. Which boils down to another problem I stated here about SASL mech selection: http://tinyurl.com/4ntesca John Conrad wrote: > For what it's worth, I have run into the same problem and the only > solution I have found is to switch to a different access method. As > best as I can tell svnserve is simply not an option when trying to set > up a repository with path based authentication when select areas are > flagged inaccessible to anonymous users. I have recently switched from > a svnserve to apache based setup and using the exact same authz-db > file, svnserve failed to return "svn log" results for protected paths > while apache worked correctly. > > The below issue on the SVN tracker I think refers to this issue and it > has been open since Oct. 2009: > http://subversion.tigris.org/issues/show_bug.cgi?id=3516 > > Anyway, I could be totally wrong here, but I do not think what you > want to do is possible with svnserve. I hope I am mistaken, but if > not, sorry to be the bearer of bad news. > > On Thu, Feb 10, 2011 at 9:30 PM, Victor Sudakov > wrote: > > The problem is probably in the following. When anon-access is other > > than "none", svnserve does not request authentication for some > > important operations like "svn log", and I have found no way to force > > it to request authentication. This effectively breaks path based > > authorization. > > > > I have found some tricky solutions for the http access method (like > > defining two aliases for the same repository), but none for the > > svnserve method. Any help? > > > > Victor Sudakov wrote: > >> > >> I am trying to setup the following policy: a private repository with > >> some public paths. Is such configuration supported at all? > >> > >> The following configuration: > >> > >> == conf/svnserve.conf: > >> anon-access = read > >> auth-access = write > >> authz-db = authz > >> > >> == conf/authz: > >> [/] > >> @noc = rw > >> > >> [/foo] > >> $anonymous = r > >> $authenticated = rw > >> > >> does not work. A valid user from the noc group receives the following > >> reply: > >> > >> $ svn diff -c2237 www.txt > >> svn: Unreadable path encountered; access denied > >> > >> If I change "anon-access = read" to "anon-access = none", it begins to > >> work for the valid user, but there is no anonymous access to anyone > >> even to svn://myserver/foo despite the "$anonymous = r" clause. > >> > >> What am I doing wrong? -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: diff-cmd =
Ryan Schmidt wrote: [dd] > > This looks like http://subversion.tigris.org/issues/show_bug.cgi?id=2930 > > i.e., you've found the current status quo; there is no better method > currently available. > I have always been told that I am a good tester (i.e. have the ability to come across bugs), but my rate of tripping on all the subversion rakes is alarming. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
diff-cmd =
Colleagues, I like very much the default output of the FreeBSD diff program. So I prefer using "svn diff --diff-cmd=/usr/bin/diff -x --normal" instead of just "svn diff". However, when I put the line "diff-cmd = /usr/bin/diff -x --normal" into ~/.subversion/config, I get the following error message: exec of '/usr/bin/diff -x --normal' failed: No such file or directorysvn: '/usr/bin/diff -x --normal' returned 255 I guess it is trying to exec '/usr/bin/diff -x --normal' as a whole which is kind of expected. For the present, I have created a shell alias for the "svn diff" command, but I am curious what the correct ~/.subversion/config syntax is to achieve what I want? -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: Combining public and private paths
The problem is probably in the following. When anon-access is other than "none", svnserve does not request authentication for some important operations like "svn log", and I have found no way to force it to request authentication. This effectively breaks path based authorization. I have found some tricky solutions for the http access method (like defining two aliases for the same repository), but none for the svnserve method. Any help? Victor Sudakov wrote: > > I am trying to setup the following policy: a private repository with > some public paths. Is such configuration supported at all? > > The following configuration: > > == conf/svnserve.conf: > anon-access = read > auth-access = write > authz-db = authz > > == conf/authz: > [/] > @noc = rw > > [/foo] > $anonymous = r > $authenticated = rw > > does not work. A valid user from the noc group receives the following reply: > > $ svn diff -c2237 www.txt > svn: Unreadable path encountered; access denied > > If I change "anon-access = read" to "anon-access = none", it begins to > work for the valid user, but there is no anonymous access to anyone > even to svn://myserver/foo despite the "$anonymous = r" clause. > > What am I doing wrong? > > -- > Victor Sudakov, VAS4-RIPE, VAS47-RIPN > sip:suda...@sibptus.tomsk.ru -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Combining public and private paths
Dear Colleagues, I am trying to setup the following policy: a private repository with some public paths. Is such configuration supported at all? The following configuration: == conf/svnserve.conf: anon-access = read auth-access = write authz-db = authz == conf/authz: [/] @noc = rw [/foo] $anonymous = r $authenticated = rw does not work. A valid user from the noc group receives the following reply: $ svn diff -c2237 www.txt svn: Unreadable path encountered; access denied If I change "anon-access = read" to "anon-access = none", it begins to work for the valid user, but there is no anonymous access to anyone even to svn://myserver/foo despite the "$anonymous = r" clause. What am I doing wrong? -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: Betr.: Re: "svnadmin load" a huge file
Stefan Sperling wrote: > > After the 15000th commit, the size of the repository on disk is 5.5G > > with the working directory size being 120M. Besides, after several > > thousand commits to this directory SVN slows down considerably. This > > must be some design flaw (or peculiarity if you like) of SVN. > > Probably related to the way directories are represented in the repository. > See http://svn.haxx.se/dev/archive-2011-02/0007.shtml > and also http://svn.haxx.se/dev/archive-2011-02/0014.shtml for some hints > to how this currently works. BTW I use the FSFS backend if it makes any difference. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: Betr.: Re: "svnadmin load" a huge file
Les Mikesell wrote: > >On Tue, Feb 08, 2011 at 11:32:47PM +0600, Victor Sudakov wrote: > >>After the 15000th commit, the size of the repository on disk is 5.5G > >>with the working directory size being 120M. Besides, after several > >>thousand commits to this directory SVN slows down considerably. This > >>must be some design flaw (or peculiarity if you like) of SVN. > > > >Probably related to the way directories are represented in the repository. > >See http://svn.haxx.se/dev/archive-2011-02/0007.shtml > >and also http://svn.haxx.se/dev/archive-2011-02/0014.shtml for some hints > >to how this currently works. > > I'd expect even local operations like the compare against the pristine > versions to decide what to commit to become slow when you put many > thousands of files in one directory because most filesystems aren't good > at that either (although they make fake it with caching). It's one of > those "if it hurts, don't do it" things. I did not know it would hurt until I tried to migrate this particular repository from CVS to SVN. FreeBSD by itself handles large directories very well due to its dirhash feature. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: Betr.: Re: "svnadmin load" a huge file
Johan Corveleyn wrote:
[dd]
> But that doesn't explain why the resulting repository is so large
> (compared to the original CVS repository). Sure, there might be memory
> usage problems in dump/load (it uses more memory than the resulting
> repository uses diskspace), but I think there is more going on.
>
> That's why I'm guessing on rev files being large (and the
> corresponding memory structures) because of the amount of dir entries
> in each revision. I'm not that intimately familiar with how this is
> all represented, and how the rev files are structured and all that, so
> I'm just guessing ... I seem to remember something like this from
> another discussion in the past.
I have created a small testcase script:
#!/bin/sh
for i in `jot 15000`
do
cat > Testfile_${i}.txt << __END__
This is a small test file.
This is a small test file.
This is a small test file.
This is a small test file.
This is a small test file.
This is a small test file.
__END__
svn add Testfile_${i}.txt
svn commit -m "Iteration $i"
done
After the 15000th commit, the size of the repository on disk is 5.5G
with the working directory size being 120M. Besides, after several
thousand commits to this directory SVN slows down considerably. This
must be some design flaw (or peculiarity if you like) of SVN.
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
sip:suda...@sibptus.tomsk.ru
Re: Betr.: Re: "svnadmin load" a huge file
Colleagues, I have finally completed a test cvs2svn conversion on an amd64 system. The peak memory requirement of svnadmin during the conversion was 9796M SIZE, 1880M RES. The resulting SVN repo size is 8.5G on disk. "svnadmin dump --deltas" of this new SVN repo required 6692M SIZE, 2161M RES of memory at its peak. Such memory requirements make this repo completely unusable on i386 systems. The original CVS repo is 59M on disk with 17859 files (including those in the Attic) and total 23911 revisions (in SVN terms). All files are strictly text. Something seems to be very suboptimal either about SVN itself or about the cvs2svn utility. I am especially surprised by the 8.5G size of the resulting SVN repository (though the result of "svnadmin dump --deltas" is 44M). > - Copy your CVS repository (say /myreypository to /myrepositoryconv) > - In the copy move the ,v files into several subdirectories (using the > operating system, not using CVS commands.) > - Convert the directories one at a time and load them into svn. > - Once loaded into svn you can move everything back into one folder > (using svn commands) if desired. Even if I do this, after moving everything back I will not be able to do "svnadmin dump" on an i386 system, perhaps unless I write some script which will iterate and keep track of dumped revision numbers. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: "svnadmin load" a huge file
Kevin Grover wrote: [dd] > 2) Don't use '--dumpfile' on cvs2svn, let cvs2svn load it into a subversion > repo directly. It did not make any difference. Frankly speaking, I would be surprised if it did. Starting Subversion r10773 / 23520 Starting Subversion r10774 / 23520 Starting Subversion r10775 / 23520 ERROR: svnadmin failed with the following output while loading the dumpfile: $ -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: "svnadmin load" a huge file
Les Mikesell wrote: [dd] > > Does it mean that on a 32bit OS I am stuck hopelessly? A dump/load > > cycle will eventually fail as the repository grows beyond a certain > > size? > > A 'real' svnadmin dump would let you specify revision ranges so you > could do it incrementally but cvs2svn doesn't have an equivalent > option other than splitting out directories. Perhaps someone could do > the load on a larger 64-bit machine and dump it back in smaller ranges > if you can't find a better way to split it. I have also noticed that the --deltas option dramatically decreases the dump size (it becomes megabytes instead of gigabytes). Unfortunately cvs2svn cannot do deltas. I will try to load on a 64-bit machine and dump it back with the --deltas option. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: "svnadmin load" a huge file
Johan Corveleyn wrote: > > Like Stephen Connolly suggested a week ago: I think you should take a > look at svndumptool: http://svn.borg.ch/svndumptool/ > > I've never used it myself, but in the README.txt file, there is > mention of a subcommand "split": I am already trying it but it turns out not as easy as it seems. I will share what comes of it. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: "svnadmin load" a huge file
Les Mikesell wrote: > > >>I don't think you are hitting some absolute limit in the software here, > >>just running out of RAM on your particular machine. Can you do the > >>conversion on a machine with more RAM? > > > >I ran "svnadmin load" on a machine with 1 GB RAM and 25 GB swap (added > >so much swap specially for the occasion). svnadmin crashed after > >reaching the SIZE about 2.5 GB. > > > >Is 1 GB RAM and 25 GB swap not enough? > > If it is a 32bit OS, you'll most likely hit a per-process limit at 2 or > 4 gigs. Or maybe some quota setting before that. The more I think about it, the more likely it seems. Does it mean that on a 32bit OS I am stuck hopelessly? A dump/load cycle will eventually fail as the repository grows beyond a certain size? BTW here are the limits for the svn user: $ whoami svn $ limits Resource limits (current): cputime infinity secs filesize infinity kB datasize 524288 kB stacksize 65536 kB coredumpsize infinity kB memoryuseinfinity kB memorylocked infinity kB maxprocesses 5547 openfiles 11095 sbsize infinity bytes vmemoryuse infinity kB pseudo-terminals infinity swapuse infinity kB $ uname -srm FreeBSD 8.1-RELEASE-p2 i386 -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: "svnadmin load" a huge file
Daniel Shahaf wrote: [dd] > > I believe there are known issues with memory usage in svnadmin. See the > issue tracker. Namely? > > I don't know cvs2svn, but it could have a --sharded-output option, so eg > it would produce a dumpfile per 1000 revisions, rather than one huge > dumpfile. cvs2svn-2.3.0_2 does not seem to have such an option: "cvs2svn: error: no such option: --sharded-output" -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: "svnadmin load" a huge file
Les Mikesell wrote: > > > >>>>I migrated a large CVS repository (25-50 GB) to SVN years ago on SVN > >>>>1.3. Our repo had many sections (projects) within it. We had to > >>>>migrate each project independently so that it's team could coordinate > >>>>when they migrated to SVN. As such, I dumped each project when ready > >>>>and then svnadmin loaded each dump into it's own path/root (so as not to > >>>>overwrite anything previously loaded and unrelated to this project's > >>>>import). > >>>> > > > > >It would be fine if the project in question did not contain almost all > >the files in one directory. You may call the layout silly, but CVS does > >not seem to mind. OTOH, I would have distributed the files over > >several subdirectories, but CVS does not handle moving files well. > > > >I wonder if cvs2svn is to blame that it produces a dump svnadmin > >cannot load. Or I am always risking that "svnadmin dump" may one day > >produce a dump a subsequent "svnadmin load" will be unable to swallow? > > > >I mean, if by hook or by crook, by using third party utilities like > >svndumptool, I will eventually be able to convert this project from > >CVS to SVN. Is there a chance that a subsequent dump will be again > >unloadable? > > I don't think you are hitting some absolute limit in the software here, > just running out of RAM on your particular machine. Can you do the > conversion on a machine with more RAM? I ran "svnadmin load" on a machine with 1 GB RAM and 25 GB swap (added so much swap specially for the occasion). svnadmin crashed after reaching the SIZE about 2.5 GB. Is 1 GB RAM and 25 GB swap not enough? I don't know if this gdb output will be useful: (gdb) where #0 0x2841e117 in kill () from /lib/libc.so.7 #1 0x2841e076 in raise () from /lib/libc.so.7 #2 0x2841cc4a in abort () from /lib/libc.so.7 #3 0x28116ec5 in abort_on_pool_failure (retcode=Could not find the frame base f or "abort_on_pool_failure". ) at subversion/libsvn_subr/pool.c:49 #4 0x283095fb in apr_palloc (pool=0xba46d018, in_size=204800) at memory/unix/apr_pools.c:663 #5 0x280ebad3 in svn_txdelta_target_push ( handler=0x280eb140 , handler_baton=0x5b26c058, source=0xba470738, pool=0xba46d018) at subversion/libsvn_delta/text_delta.c:528 #6 0x280d4d62 in svn_fs_fs__set_contents (stream=0xbfbfe7e4, fs=0x28512020, noderev=0x27dea528, pool=0x2852a018) at subversion/libsvn_fs_fs/fs_fs.c:5066 #7 0x280c9d52 in svn_fs_fs__dag_get_edit_stream (contents=0x2852a138, file=0x5b2e61a0, pool=0x2852a018) at subversion/libsvn_fs_fs/dag.c:997 #8 0x280de42e in fs_apply_text (contents_p=0xbfbfe904, root=0x5aef8058, path=0x2852a080 "ns2/trunk/tomsk.ru/SOA", result_checksum=0x2852a110, pool=0x2852a018) at subversion/libsvn_fs_fs/tree.c:2615 #9 0x280bf44e in svn_fs_apply_text (contents_p=0xbfbfe904, root=0x5aef8058, path=0x2852a080 "ns2/trunk/tomsk.ru/SOA", result_checksum=0x2852a0e8 "b03cbddfbc11be113cbf675862eb971e", pool=0x2852a018) at subversion/libsvn_fs/fs-loader.c:1096 ---Type to continue, or q to quit--- -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: "svnadmin load" a huge file
Brian Brophy wrote: > > > >>I migrated a large CVS repository (25-50 GB) to SVN years ago on SVN > >>1.3. Our repo had many sections (projects) within it. We had to > >>migrate each project independently so that it's team could coordinate > >>when they migrated to SVN. As such, I dumped each project when ready > >>and then svnadmin loaded each dump into it's own path/root (so as not to > >>overwrite anything previously loaded and unrelated to this project's > >>import). > >> > >>So, you can do it by controlling which path/portion of CVS you use > >>cvs2vn to create the dump file from. > >> > > > >The CVS repository in question (with the size 54M with 17751 files) is > >exactly one project. It's the history of a geographical DNS zone for > >more than 10 years. > Fair enough, the same pattern is still applicable. For example, in our > CVS repo what separated one "project" from another was basically a > root-level folder. > > In kind, you could similarly use cvs2svn to "chunk/dump" subdirectories > at a time. > > For example, if in CVS you have something like: > /Folder1 > /Folder2 > /Folder3 > > ... you run cvs2svn three times, once for each subdirectory, producing > folder1.dump, folder2.dump, and folder3.dump respectively. > > Then, svnadmin load each individually: It would be fine if the project in question did not contain almost all the files in one directory. You may call the layout silly, but CVS does not seem to mind. OTOH, I would have distributed the files over several subdirectories, but CVS does not handle moving files well. I wonder if cvs2svn is to blame that it produces a dump svnadmin cannot load. Or I am always risking that "svnadmin dump" may one day produce a dump a subsequent "svnadmin load" will be unable to swallow? I mean, if by hook or by crook, by using third party utilities like svndumptool, I will eventually be able to convert this project from CVS to SVN. Is there a chance that a subsequent dump will be again unloadable? -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: "svnadmin load" a huge file
Brian Brophy wrote: > I migrated a large CVS repository (25-50 GB) to SVN years ago on SVN > 1.3. Our repo had many sections (projects) within it. We had to > migrate each project independently so that it's team could coordinate > when they migrated to SVN. As such, I dumped each project when ready > and then svnadmin loaded each dump into it's own path/root (so as not to > overwrite anything previously loaded and unrelated to this project's > import). > > So, you can do it by controlling which path/portion of CVS you use > cvs2vn to create the dump file from. The CVS repository in question (with the size 54M with 17751 files) is exactly one project. It's the history of a geographical DNS zone for more than 10 years. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: "svnadmin load" a huge file
Daniel Shahaf wrote: > Split the dumpfile to smaller dumpfiles How do I do that? I have not found such an option in cvs2svn. I don't mind writing a script if I knew the idea how to split the dump. I haven't found any "svnadmin load" option to import part of a dump either. man what? > or try a newer version of svnadmin. I am using subversion-1.6.15, it seems to be the latest ported to FreeBSD. > > (or dive into the source and help us plug that memory leak --- compile > with APR pool debugging enabled) I will try to do that but unfortunately I need some immediate workaround :( -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
"svnadmin load" a huge file
Colleagues, I have a CVS repository sized 54M with 17751 files. "cvs2svn --dumpfile" produces a dump sized 13G. svnadmin cannot load this dump aborting with an out of memory condition on a FreeBSD 8.1-RELEASE box with 1G of RAM and 2.5G of swap. I really need to convert this repository to SVN. What should i do? Any advice is appreciated. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: sasl mechanisms order
Daniel Shahaf wrote: > > So we seem to have a stalemate situation. The SASL library believes > > the client should select the preferred mechanism, whereas the Subversion > > client relies on "the order suggested by the server". Brilliant. > > It would be more constructive to summarize the problem on the dev@ > list so that you or someone else can resolve it. I have. Please follow up in dev@ -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: sasl mechanisms order
Alec Kloss wrote: [dd] > > As far as I can tell (and as amazing as this sounds), the order of the > offered mechanisms from Cyrus sasl is, by default, the reverse of the > order that the library finds them. This would be, in effect, the > reverse physical directory order of the modules in > /usr/[local]/lib/sasl2/ which you can find with ls -U. I've confirmed > this by making copies and deletes of the .so files in that directory to > rearrange the ordering. The list is reversed from the order they're > found in because mechanism list is a linked list and new entries are > prepened (around server.c:392). Thank you for having found this out. This is truly amazing. This means that if perchance I touch a file in /usr/local/lib/sasl2/, my Kerberos SSO can stop working? > > As the link your provided mentions, Cyrus SASL believes it's the client > that should select the preferred mechanism from the list offered by the > server, not just the first one. So we seem to have a stalemate situation. The SASL library believes the client should select the preferred mechanism, whereas the Subversion client relies on "the order suggested by the server". Brilliant. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
Re: sasl mechanisms order
Daniel Shahaf wrote:
> > > >
> > > > I have the following line in /usr/local/lib/sasl2/svn.conf:
> > > > mech_list: gssapi digest-md5 anonymous
> > > >
> > > > How can I guarantee that the subversion client/server will always use
> > > > GSSAPI before DIGEST-MD5? Or a more generic question, how can I change
> > > > the order of mechanisms if I have to?
> > > >
> > >
> > > Looking at subversion/libsvn_ra_svn/{client.c,cyrus_auth.c}, it seems
> > > that the
> > > following order is used:
> > >
> > > * EXTERNAL (i.e., ssh tunnel)
> > > * ANONYMOUS
> > > * ${server-reported mechanisms, in the order suggested by the server}
> > > * CRAM-MD5 (used via internal_auth.c even if SASL doesn't support it)
> > >
> > > I don't see a knob that lets you manipulate the order.
> >
> > Then how can I manipulate "the order suggested by the server"? The
> > server is svnserve.
> >
>
> Looking in subversion/svnserve/cyrus_auth.c, the list of mechansms is obtained
> directly from SASL:
[dd]
>
> so you'd have to look up in the SASL docs how to configure the ordering of
> mechanisms. (I don't know offhand how to configure that.)
I was unable to find this in the SASL docs. It only says that
mech_list is a "Whitespace separated list of mechanisms to allow (e.g.
'plain otp'). Used to restrict the mechanisms to a subset of the
installed plugins."
While googling I found even such statements as
http://www.techienuggets.com/CommentDetail?tx=188636
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
sip:suda...@sibptus.tomsk.ru
Re: sasl mechanisms order
Daniel Shahaf wrote:
> >
> > I have the following line in /usr/local/lib/sasl2/svn.conf:
> > mech_list: gssapi digest-md5 anonymous
> >
> > How can I guarantee that the subversion client/server will always use
> > GSSAPI before DIGEST-MD5? Or a more generic question, how can I change
> > the order of mechanisms if I have to?
> >
>
> Looking at subversion/libsvn_ra_svn/{client.c,cyrus_auth.c}, it seems that the
> following order is used:
>
> * EXTERNAL (i.e., ssh tunnel)
> * ANONYMOUS
> * ${server-reported mechanisms, in the order suggested by the server}
> * CRAM-MD5 (used via internal_auth.c even if SASL doesn't support it)
>
> I don't see a knob that lets you manipulate the order.
Then how can I manipulate "the order suggested by the server"? The
server is svnserve.
>
> > I have experimented with the order of mechanisms in the mech_list
> > definition, but the result is always the same ( ANONYMOUS GSSAPI
> > DIGEST-MD5 ). It's fine so far, but how can I change the order if
> > needed?
> >
>
> Is your problem that GSSAPI is before/after DIGEST-MD5, or that it is
> before/after ANONYMOUS? These are quite different situations...
Right now GSSAPI comes before DIGEST-MD5 and this is fine with me. I
just don't want this order to change suddenly with a new version of
subversion or cyrus-sasl or something, because it will break SSO.
--
Victor Sudakov, VAS4-RIPE, VAS47-RIPN
sip:suda...@sibptus.tomsk.ru
sasl mechanisms order
Colleagues, I have the following line in /usr/local/lib/sasl2/svn.conf: mech_list: gssapi digest-md5 anonymous How can I guarantee that the subversion client/server will always use GSSAPI before DIGEST-MD5? Or a more generic question, how can I change the order of mechanisms if I have to? I have experimented with the order of mechanisms in the mech_list definition, but the result is always the same ( ANONYMOUS GSSAPI DIGEST-MD5 ). It's fine so far, but how can I change the order if needed? FreeBSD 6.4, subversion-1.6.12 compiled with cyrus-sasl-2.1.23 from ports. Thank you in advance for any input. -- Victor Sudakov, VAS4-RIPE, VAS47-RIPN sip:suda...@sibptus.tomsk.ru
