What's the process to svnsync a larger repository

[Geoff Hoffman]

I'm learning some gotchas with svnsync this week. As per my typical method
of learning I try 3 wrong ways before finally realizing the right way to do
most things.

We have about a 1GB svn repo on Ubuntu at -r 3738, and I'm trying to get it
mirrored on Win 7 running VisualSVN.

I created the mirror repository fine, ran svnsync init and svnsync sync on
it, then as it started going from -r 1, -r 2, -r 3..., I realized it would
take a week or more to fetch all nearly 4K commits.

So I stopped it, deleted it, started over and did an svnadmin dump -->
svnadmin load from the dump -- which ran/imported much faster, but now when
I try to initialize the sync, I get

svnsync: Cannot initialize a repository with content in it

D'OH! I forgot to svnsync init after creating the empty mirror repo. Whoops.

So, my question is, before I whack a perfectly fine "almost mirrored"
repository, can I svn propset -r 0 all the stuff needed by svnsync
manually, to start syncing at -r 3738?

Or can I set up the repo again (svnadmin create immediately followed by
svnsync init) and then svnadmin load (since load from a dump is so much
faster)?

It would seem odd that the only way to mirror a repo is to replay
everything from -r 0.

Thanks for your guidance!

Geoff

-- 


This email, including any attachments, is for the sole use of the intended 
recipient and may contain confidential information. If you are not the 
intended recipient, please immediately notify us by reply email or by 
telephone, delete this email and destroy any copies. Thank you.

Re: What's the process to svnsync a larger repository

[Mark Phippard]

On Fri, Mar 2, 2012 at 10:41 AM, Geoff Hoffman
 wrote:
> I'm learning some gotchas with svnsync this week. As per my typical method
> of learning I try 3 wrong ways before finally realizing the right way to do
> most things.
>
> We have about a 1GB svn repo on Ubuntu at -r 3738, and I'm trying to get it
> mirrored on Win 7 running VisualSVN.
>
> I created the mirror repository fine, ran svnsync init and svnsync sync on
> it, then as it started going from -r 1, -r 2, -r 3..., I realized it would
> take a week or more to fetch all nearly 4K commits.
>
> So I stopped it, deleted it, started over and did an svnadmin dump -->
> svnadmin load from the dump -- which ran/imported much faster, but now when
> I try to initialize the sync, I get
>
> svnsync: Cannot initialize a repository with content in it
>
> D'OH! I forgot to svnsync init after creating the empty mirror repo. Whoops.
>
> So, my question is, before I whack a perfectly fine "almost mirrored"
> repository, can I svn propset -r 0 all the stuff needed by svnsync manually,
> to start syncing at -r 3738?
>
> Or can I set up the repo again (svnadmin create immediately followed by
> svnsync init) and then svnadmin load (since load from a dump is so much
> faster)?
>
> It would seem odd that the only way to mirror a repo is to replay everything
> from -r 0.

This is, in fact, a limitation of svnsync.  You can manually setup the
properties on r0 as one option, or if you have svnsync from 1.7
installed, you can use the new --allow-non-empty option on the init
command and it essentially will do the same thing for you.

-- 
Thanks

Mark Phippard
http://markphip.blogspot.com/

Re: What's the process to svnsync a larger repository

[Geoff Hoffman]

Yeah I was just reading this:

"By default, the aforementioned basic requirements of a mirror are that it
> allows revision property modifications and that it contains no version
> history. However, as of Subversion 1.7, you may now optionally disable the
> verification that the target repository is empty using the
> --allow-non-empty option. While the use of this option should not become
> habitual (as it bypasses a valuable safeguard mechanism), it does aid in
> one very common use-case: initializing a copy of a repository as a mirror
> of the original. This is especially handy when setting up new mirrors of
> repositories which contain a large amount of version history. Rather than
> initialize a brand new repository as a mirror and then syncronize all of
> the history into it, administrators will find it *significantly* faster
> to first make a copy of the mature repository (perhaps using *svnadmin
> hotcopy*) and then use *svnsync initialize --allow-non-empty* to
> initialize that copy as a mirror which is now already up-to-date with the
> original."



Now here's the interesting part. I'm running svn 1.6 on the source repo,
1.7 on the mirror repo... (I think)



Geoff Hoffman
Solutions Architect & LAMP Engineer
_
Twitter: @m2guru  |
@cardinalpath
LinkedIn: www.linkedin.com/in/geoffreydhoffman
Phone: 480.285.1622 x313
Mobile: 480.231.8323
_
www.cardinalpath.com
training.cardinalpath.com

*PublicInsite, VKI Studios, and WebShare have merged to form Cardinal Path*




On Fri, Mar 2, 2012 at 8:45 AM, Mark Phippard  wrote:

> On Fri, Mar 2, 2012 at 10:41 AM, Geoff Hoffman
>  wrote:
> > I'm learning some gotchas with svnsync this week. As per my typical
> method
> > of learning I try 3 wrong ways before finally realizing the right way to
> do
> > most things.
> >
> > We have about a 1GB svn repo on Ubuntu at -r 3738, and I'm trying to get
> it
> > mirrored on Win 7 running VisualSVN.
> >
> > I created the mirror repository fine, ran svnsync init and svnsync sync
> on
> > it, then as it started going from -r 1, -r 2, -r 3..., I realized it
> would
> > take a week or more to fetch all nearly 4K commits.
> >
> > So I stopped it, deleted it, started over and did an svnadmin dump -->
> > svnadmin load from the dump -- which ran/imported much faster, but now
> when
> > I try to initialize the sync, I get
> >
> > svnsync: Cannot initialize a repository with content in it
> >
> > D'OH! I forgot to svnsync init after creating the empty mirror repo.
> Whoops.
> >
> > So, my question is, before I whack a perfectly fine "almost mirrored"
> > repository, can I svn propset -r 0 all the stuff needed by svnsync
> manually,
> > to start syncing at -r 3738?
> >
> > Or can I set up the repo again (svnadmin create immediately followed by
> > svnsync init) and then svnadmin load (since load from a dump is so much
> > faster)?
> >
> > It would seem odd that the only way to mirror a repo is to replay
> everything
> > from -r 0.
>
> This is, in fact, a limitation of svnsync.  You can manually setup the
> properties on r0 as one option, or if you have svnsync from 1.7
> installed, you can use the new --allow-non-empty option on the init
> command and it essentially will do the same thing for you.
>
> --
> Thanks
>
> Mark Phippard
> http://markphip.blogspot.com/
>

-- 


This email, including any attachments, is for the sole use of the intended 
recipient and may contain confidential information. If you are not the 
intended recipient, please immediately notify us by reply email or by 
telephone, delete this email and destroy any copies. Thank you.

Re: What's the process to svnsync a larger repository

[Philip Martin]

Geoff Hoffman  writes:

> So, my question is, before I whack a perfectly fine "almost mirrored"
> repository, can I svn propset -r 0 all the stuff needed by svnsync
> manually, to start syncing at -r 3738?

Yes.  You need --revprop with the propset.

-- 
Philip

Re: What's the process to svnsync a larger repository

[Geoff Hoffman]

Sweet - if that works it'll save a lot of time. Can someone paste an svn
proplist --revprop -r 0 from a mirror repo so I can see all the stuff to
set?


On Fri, Mar 2, 2012 at 8:47 AM, Philip Martin wrote:

> Geoff Hoffman  writes:
>
> > So, my question is, before I whack a perfectly fine "almost mirrored"
> > repository, can I svn propset -r 0 all the stuff needed by svnsync
> > manually, to start syncing at -r 3738?
>
> Yes.  You need --revprop with the propset.
>
> --
> Philip
>

-- 


This email, including any attachments, is for the sole use of the intended 
recipient and may contain confidential information. If you are not the 
intended recipient, please immediately notify us by reply email or by 
telephone, delete this email and destroy any copies. Thank you.

Re: What's the process to svnsync a larger repository

[Mark Phippard]

On Fri, Mar 2, 2012 at 10:47 AM, Geoff Hoffman wrote:

> Yeah I was just reading this:
>
> "By default, the aforementioned basic requirements of a mirror are that it
>> allows revision property modifications and that it contains no version
>> history. However, as of Subversion 1.7, you may now optionally disable the
>> verification that the target repository is empty using the
>> --allow-non-empty option. While the use of this option should not become
>> habitual (as it bypasses a valuable safeguard mechanism), it does aid in
>> one very common use-case: initializing a copy of a repository as a mirror
>> of the original. This is especially handy when setting up new mirrors of
>> repositories which contain a large amount of version history. Rather than
>> initialize a brand new repository as a mirror and then syncronize all of
>> the history into it, administrators will find it *significantly* faster
>> to first make a copy of the mature repository (perhaps using *svnadmin
>> hotcopy*) and then use *svnsync initialize --allow-non-empty* to
>> initialize that copy as a mirror which is now already up-to-date with the
>> original."
>
>
>
> Now here's the interesting part. I'm running svn 1.6 on the source repo,
> 1.7 on the mirror repo... (I think)
>
>
If the svnsync command you use is from 1.7.x it will work.  Does not matter
what version the server you are synching is running.

-- 
Thanks

Mark Phippard
http://markphip.blogspot.com/

Re: What's the process to svnsync a larger repository

[Geoff Hoffman]

On Fri, Mar 2, 2012 at 8:47 AM, Philip Martin wrote:

> Geoff Hoffman  writes:
>
> > So, my question is, before I whack a perfectly fine "almost mirrored"
> > repository, can I svn propset -r 0 all the stuff needed by svnsync
> > manually, to start syncing at -r 3738?
>
> Yes.  You need --revprop with the propset.
>
> --
> Philip
>



I'm going to try setting svn:sync-* props by hand first...

>From this post
http://journal.paul.querna.org/articles/2006/09/14/using-svnsync/
I guess I need to set four properties by hand...

$ svn proplist --revprop -r 0 ${TOREPO}

  svn:sync-from-uuid
  svn:sync-last-merged-rev
  svn:date
  svn:sync-from-url

I retrieved the uuid from the source repo like so

#> svn info http://localhost/svn/source-repo -r 3738
URL: http://localhost/svn/source-repo
Repository Root: http://localhost/svn/source-repo
Repository UUID: 9c96f4c0-7d9a-42f6-b8c8-54e79b961fad
Revision: 3738

Okay, the sync-from-url is easy, last-merged-rev is 3738... what format is
the svn:date in? Anyone know?

-- 


This email, including any attachments, is for the sole use of the intended 
recipient and may contain confidential information. If you are not the 
intended recipient, please immediately notify us by reply email or by 
telephone, delete this email and destroy any copies. Thank you.

Re: What's the process to svnsync a larger repository

[Les Mikesell]

On Fri, Mar 2, 2012 at 9:41 AM, Geoff Hoffman  wrote:
> I'm learning some gotchas with svnsync this week. As per my typical method
> of learning I try 3 wrong ways before finally realizing the right way to do
> most things.
>
> We have about a 1GB svn repo on Ubuntu at -r 3738, and I'm trying to get it
> mirrored on Win 7 running VisualSVN.
>
> I created the mirror repository fine, ran svnsync init and svnsync sync on
> it, then as it started going from -r 1, -r 2, -r 3..., I realized it would
> take a week or more to fetch all nearly 4K commits.

Slow is one thing - but that sounds unreasonable.  Is there some
network issue involved?  Maybe you can svnsync on a nearby machine,
then copy or move the resulting repo once it is caught up.

-- 
   Les Mikesell
lesmikes...@gmail.com

Re: What's the process to svnsync a larger repository

[Geoff Hoffman]

On Fri, Mar 2, 2012 at 9:13 AM, Les Mikesell  wrote:

> On Fri, Mar 2, 2012 at 9:41 AM, Geoff Hoffman 
> wrote:
> > I'm learning some gotchas with svnsync this week. As per my typical
> method
> > of learning I try 3 wrong ways before finally realizing the right way to
> do
> > most things.
> >
> > We have about a 1GB svn repo on Ubuntu at -r 3738, and I'm trying to get
> it
> > mirrored on Win 7 running VisualSVN.
> >
> > I created the mirror repository fine, ran svnsync init and svnsync sync
> on
> > it, then as it started going from -r 1, -r 2, -r 3..., I realized it
> would
> > take a week or more to fetch all nearly 4K commits.
>
> Slow is one thing - but that sounds unreasonable.  Is there some
> network issue involved?  Maybe you can svnsync on a nearby machine,
> then copy or move the resulting repo once it is caught up.
>
> --
>   Les Mikesell
>lesmikes...@gmail.com
>


The speed issue is that I'm storing the mirror repo on a slow computer, on
a USB Drobo that is in turn being backed up to an online backup service.


Good news- Thanks guys, looks like I got it syncing forward from -r 3738!

Here's cmd output from Windows box -- hopefully it helps someone in the
future.


X:\Repositories>svn propset --revprop -r0 svn:sync-last-merged-rev 3738
http://mirror-server/svn/mirror-repo
property 'svn:sync-last-merged-rev' set on repository revision 0

X:\Repositories>svnsync sync http://mirror-server/svn/mirror-repo
svnsync: Destination repository has not been initialized

X:\Repositories>svn propset --revprop -r0 svn:sync-from-uuid
9c96f4c0-7d9a-42f6-
b8c8-54e79b961fad http://mirror-server/svn/mirror-repo
property 'svn:sync-from-uuid' set on repository revision 0

X:\Repositories>svn propset --revprop -r0 svn:sync-from-url
http://source-server/svn/source-repo http://mirror-server/svn/mirror-repo
property 'svn:sync-from-url' set on repository revision 0

X:\Repositories>svnsync sync http://mirrorr-server/svn/mirror-repo
Transmitting file data .
Committed revision 3739.
Copied properties for revision 3739.

-- 


This email, including any attachments, is for the sole use of the intended 
recipient and may contain confidential information. If you are not the 
intended recipient, please immediately notify us by reply email or by 
telephone, delete this email and destroy any copies. Thank you.

Re: What's the process to svnsync a larger repository

[Les Mikesell]

On Fri, Mar 2, 2012 at 10:20 AM, Geoff Hoffman
 wrote:
>
>> > I created the mirror repository fine, ran svnsync init and svnsync sync
>> > on
>> > it, then as it started going from -r 1, -r 2, -r 3..., I realized it
>> > would
>> > take a week or more to fetch all nearly 4K commits.
>>
>> Slow is one thing - but that sounds unreasonable.  Is there some
>> network issue involved?  Maybe you can svnsync on a nearby machine,
>> then copy or move the resulting repo once it is caught up.
>>

> The speed issue is that I'm storing the mirror repo on a slow computer, on a
> USB Drobo that is in turn being backed up to an online backup service.

But you should be able to copy a whole repo, maintaining state as long
as the target is reasonably similar.  I see you have gotten it to work
another way, but you should have been able to do the initial catch-up
sync on something faster, copy that repository where you want it, then
resume the syncs to keep it updated.

-- 
  Les Mikesell
lesmikes...@gmail.com

Re: What's the process to svnsync a larger repository

[Geoff Hoffman]

On Fri, Mar 2, 2012 at 9:25 AM, Les Mikesell  wrote:

> On Fri, Mar 2, 2012 at 10:20 AM, Geoff Hoffman
>  wrote:
> >
> >> > I created the mirror repository fine, ran svnsync init and svnsync
> sync
> >> > on
> >> > it, then as it started going from -r 1, -r 2, -r 3..., I realized it
> >> > would
> >> > take a week or more to fetch all nearly 4K commits.
> >>
> >> Slow is one thing - but that sounds unreasonable.  Is there some
> >> network issue involved?  Maybe you can svnsync on a nearby machine,
> >> then copy or move the resulting repo once it is caught up.
> >>
>
> > The speed issue is that I'm storing the mirror repo on a slow computer,
> on a
> > USB Drobo that is in turn being backed up to an online backup service.
>
> But you should be able to copy a whole repo, maintaining state as long
> as the target is reasonably similar.  I see you have gotten it to work
> another way, but you should have been able to do the initial catch-up
> sync on something faster, copy that repository where you want it, then
> resume the syncs to keep it updated.



This is pretty much exactly what I did, Les.

My other question from my initial post is, should I be doing it in exactly
this order:

// source-server
1. svnadmin dump source-repo > source.dump
2. tgz it and scp it to mirror-server
// mirror-server
3. svnadmin create mirror-repo
4. svnsync init here !! * // this is what I didn't do before*
5. svnadmin load ? or do you have to just let it sync from -r 0

With 1.6x and before, it may be just as easy to issue 3 propsets as above.
I think I'll write a blog post on this. There doesn't seem to be a single
concise procedure outlined anywhere on the net. Although, it's nice to
learn that svn 1.7+ added the --allow-non-empty option.

-- 


This email, including any attachments, is for the sole use of the intended 
recipient and may contain confidential information. If you are not the 
intended recipient, please immediately notify us by reply email or by 
telephone, delete this email and destroy any copies. Thank you.

Re: What's the process to svnsync a larger repository

[Les Mikesell]

On Fri, Mar 2, 2012 at 10:33 AM, Geoff Hoffman
 wrote:
>
>> But you should be able to copy a whole repo, maintaining state as long
>> as the target is reasonably similar.  I see you have gotten it to work
>> another way, but you should have been able to do the initial catch-up
>> sync on something faster, copy that repository where you want it, then
>> resume the syncs to keep it updated.
>
>
> This is pretty much exactly what I did, Les.

I meant, create your mirror initially with svnsync from scratch on a
fast local system instead of using the dump/load.  Then copy the whole
mirror repo to the system/drive where you wanted it, and svnsync
should pick up where it left off assuming the network reference still
works from the new location.

-- 
   Les Mikesell
lesmikes...@gmail.com