Creating and Verifying a Reliable backup
Hello, We are very paranoid about our Subversion repo, notwithstanding the fact that the previous sysadmin didn't back it up. But that's another story. Now I'm here at my job, I've inherited the repo admin duties, and I want to back it up reliably. If we lose it, we're all out of work. My question is: How do I back it up reliably, and verify it so that I can deliver a 100% recovery guarantee to my boss? I have Subversion 1.8.4 on a CentOS 6.3 server, and Tortoise SVN 1.8.11 on Windows 7 clients. I am thinking to do both an svn hotcopy to one directory, and an rsync to another. The svn hotcopy will give me a backup that I'm pretty sure is reliable (see Notes below). Assuming httpd is down and I can guarantee that I am the only person who will be logged into the SVN server, can I expect with 99.9% surety that the svn repos are quiescent? Thanks. -- -Mike Schwager Notes: We're a little worried about svn hotcopy; we ran into a bug that came about under 1.8 when working with older repos; the hotcopy exits with the following error: svnadmin: E22: Serialized hash missing terminator I have compiled subversion-1.9.4 on the server under /opt/subversion-1.9.4. If I run that version of svn hotcopy, it appears to work and svnverify exits successfully. But if I look at all the files under both the original and the hotcopy on one of our repos, I find that a file is missing: repos2/db/rev-prop-atomics.shm . That's probably ok, but still- how do we know the latest hotcopy, and hotcopies of the future, are and will remain 100% bug-free?
Re: Creating and Verifying a Reliable backup
I think there's two questions here: (1) what are general good backup practices, and (2) how to backup svn repos specifically. "If we lose it, we're all out of work." Hopefully your boss recognizes this and has budgeted appropriately. In my experience there is no perfect backup; the best you can do is ever-decreasing odds of a catastrophic failure. Step one would be to run your svn server with some kind of redundant disk configuration. Of course we all know RAID is not backup, but storage is relatively cheap, so why not? I'd then backup to at least two different machines, preferably offsite. Cloud storage is fairly cheap these days as well. So a good scheme might be (at least) one offsite server that you control, and (at least) a second copy with a cloud provider (CrashPlan, BackBlaze, Amazon, DropBox, etc). What we've always done is a simple rsync of the repo tree. Your email made me realize that we could be doing the backup right when someone is committing, and thus ending up with a corrupt repo tree. However, we have some mitigating factors: we don't have just one repo, but literally dozens. And we do backups twice per week, and we keep several months of backups. So my collection of backups probably does have some corrupt repo trees... but given the number of repos we have, plus the fact that the backup jobs run in the middle of the night/weekend, I think the probability is pretty low that I have any significant corruption. As you suggested, if you can make a fancier backup script that shuts down anyone's ability to make changes to the repo while the backup is taking place, that's even better. For my personal svn repos (home hobby projects) I do simple backups with svndump. Lastly, you probably owe it to your company to regularly test your backups to ensure that they are indeed viable. Just like buildings have fire drills, so should sysadmins have DR drills. Hope these suggestions are useful! On Wed, Jun 1, 2016 at 9:58 AM, Michael Schwager wrote: > Hello, > We are very paranoid about our Subversion repo, notwithstanding the fact > that the previous sysadmin didn't back it up. But that's another story. Now > I'm here at my job, I've inherited the repo admin duties, and I want to back > it up reliably. If we lose it, we're all out of work. > > My question is: How do I back it up reliably, and verify it so that I can > deliver a 100% recovery guarantee to my boss? I have Subversion 1.8.4 on a > CentOS 6.3 server, and Tortoise SVN 1.8.11 on Windows 7 clients. > > I am thinking to do both an svn hotcopy to one directory, and an rsync to > another. The svn hotcopy will give me a backup that I'm pretty sure is > reliable (see Notes below). Assuming httpd is down and I can guarantee that > I am the only person who will be logged into the SVN server, can I expect > with 99.9% surety that the svn repos are quiescent? > > Thanks. > -- > -Mike Schwager > > Notes: > > We're a little worried about svn hotcopy; we ran into a bug that came about > under 1.8 when working with older repos; the hotcopy exits with the > following error: > > svnadmin: E22: Serialized hash missing terminator > > I have compiled subversion-1.9.4 on the server under /opt/subversion-1.9.4. > If I run that version of svn hotcopy, it appears to work and svnverify exits > successfully. But if I look at all the files under both the original and the > hotcopy on one of our repos, I find that a file is missing: > repos2/db/rev-prop-atomics.shm . That's probably ok, but still- how do we > know the latest hotcopy, and hotcopies of the future, are and will remain > 100% bug-free?
Re: Creating and Verifying a Reliable backup
Hello Michael, On Wed, Jun 1, 2016 at 5:58 PM, Michael Schwager wrote: > Hello, > We are very paranoid about our Subversion repo, notwithstanding the fact > that the previous sysadmin didn't back it up. But that's another story. Now > I'm here at my job, I've inherited the repo admin duties, and I want to back > it up reliably. If we lose it, we're all out of work. > > My question is: How do I back it up reliably, and verify it so that I can > deliver a 100% recovery guarantee to my boss? I have Subversion 1.8.4 on a > CentOS 6.3 server, and Tortoise SVN 1.8.11 on Windows 7 clients. Don't forget about `svnadmin verify`[1]. It makes sense to run it right now if you've never ran it. ;) And if you want to deliver recovery guarantee close to 100%, consider `svnadmin dump`[2] in addition to hotcopy. > I am thinking to do both an svn hotcopy to one directory, and an rsync to > another. The svn hotcopy will give me a backup that I'm pretty sure is > reliable (see Notes below). Assuming httpd is down and I can guarantee that > I am the only person who will be logged into the SVN server, can I expect > with 99.9% surety that the svn repos are quiescent? I'm not sure that I understand what you mean by "quiescent" here. What's your concern? So you are planning to make a hotcopy and then transfer this hotcopy using rsync to another machine, right? * It's safe to run `svnadmin hotcopy` on a live repository. * And if you'd have to use `rsync` on a *live* repository, run `svnadmin freeze`[3] to prevent concurrent commits to the repo while `rsync` is running. If httpd is disabled and you are the only user logged on, this means that your repositories should be untouched by any other processes at this moment. Unless you have some indexing services that touch the repos (and it is generally recommended to exclude repos from indexing). > Notes: > > We're a little worried about svn hotcopy; we ran into a bug that came about > under 1.8 when working with older repos; the hotcopy exits with the > following error: > > svnadmin: E22: Serialized hash missing terminator It seems to be a bug #4554[4] that has been fixed in Subversion 1.9. `svnadmin hotcopy` could fail on a repository with older FSFS format. > I have compiled subversion-1.9.4 on the server under /opt/subversion-1.9.4. > If I run that version of svn hotcopy, it appears to work and svnverify exits > successfully. But if I look at all the files under both the original and the > hotcopy on one of our repos, I find that a file is missing: > repos2/db/rev-prop-atomics.shm . That's probably ok, but still- how do we > know the latest hotcopy, and hotcopies of the future, are and will remain > 100% bug-free? Use `svnadmin verify` and consider using `svnadmin dump` in addition to hotcopies. Make sure that your Subversion server is on the latest patch update too. [1]: http://svnbook.red-bean.com/en/1.8/svn.ref.svnadmin.c.verify.html [2]: http://svnbook.red-bean.com/en/1.8/svn.ref.svnadmin.c.dump.html [3]: http://svnbook.red-bean.com/en/1.8/svn.ref.svnadmin.c.freeze.html [4]: https://issues.apache.org/jira/browse/SVN-4554 -- With best regards, Pavel Lyalyakin VisualSVN Team
Re: Creating and Verifying a Reliable backup
Thanks Matt. To your point,
> my collection of backups probably does have some corrupt repo trees...
that is really what I'm driving at. The RAID, offsite, number of backups
(nightly in our case), and testing is all covered. In other words, I can
mitigate the effects of failure with all those tried-and-true sysadmin
techniques.
The essence of my question drives to Subversion specifically. I don't want
*any* unknown corrupt Subversion repo backups lying around. Meaning I don't
trust a shotgun approach, where I do enough of them so one is bound to be
good. I'm looking for a precision approach, where I can be reasonably
assured that the techniques I'm doing will provide me with a recoverable
repo at any chosen backup point. ("reasonably" defined as "as close to 100%
as I can get)
Because if it's Thursday, and Wednesday night's repo backup is the one I
need, I don't want to have to report back that it's corrupt and the best I
can do is Tuesday.
On Wed, Jun 1, 2016 at 10:45 AM, Matt Garman
wrote:
> I think there's two questions here: (1) what are general good backup
> practices, and (2) how to backup svn repos specifically.
>
> "If we lose it, we're all out of work." Hopefully your boss
> recognizes this and has budgeted appropriately. In my experience
> there is no perfect backup; the best you can do is ever-decreasing
> odds of a catastrophic failure.
>
> Step one would be to run your svn server with some kind of redundant
> disk configuration. Of course we all know RAID is not backup, but
> storage is relatively cheap, so why not?
>
> I'd then backup to at least two different machines, preferably
> offsite. Cloud storage is fairly cheap these days as well. So a good
> scheme might be (at least) one offsite server that you control, and
> (at least) a second copy with a cloud provider (CrashPlan, BackBlaze,
> Amazon, DropBox, etc).
>
> What we've always done is a simple rsync of the repo tree. Your email
> made me realize that we could be doing the backup right when someone
> is committing, and thus ending up with a corrupt repo tree. However,
> we have some mitigating factors: we don't have just one repo, but
> literally dozens. And we do backups twice per week, and we keep
> several months of backups. So my collection of backups probably does
> have some corrupt repo trees... but given the number of repos we have,
> plus the fact that the backup jobs run in the middle of the
> night/weekend, I think the probability is pretty low that I have any
> significant corruption.
>
> As you suggested, if you can make a fancier backup script that shuts
> down anyone's ability to make changes to the repo while the backup is
> taking place, that's even better.
>
> For my personal svn repos (home hobby projects) I do simple backups
> with svndump.
>
> Lastly, you probably owe it to your company to regularly test your
> backups to ensure that they are indeed viable. Just like buildings
> have fire drills, so should sysadmins have DR drills.
>
> Hope these suggestions are useful!
>
>
>
>
> On Wed, Jun 1, 2016 at 9:58 AM, Michael Schwager
> wrote:
> > Hello,
> > We are very paranoid about our Subversion repo, notwithstanding the fact
> > that the previous sysadmin didn't back it up. But that's another story.
> Now
> > I'm here at my job, I've inherited the repo admin duties, and I want to
> back
> > it up reliably. If we lose it, we're all out of work.
> >
> > My question is: How do I back it up reliably, and verify it so that I can
> > deliver a 100% recovery guarantee to my boss? I have Subversion 1.8.4 on
> a
> > CentOS 6.3 server, and Tortoise SVN 1.8.11 on Windows 7 clients.
> >
> > I am thinking to do both an svn hotcopy to one directory, and an rsync to
> > another. The svn hotcopy will give me a backup that I'm pretty sure is
> > reliable (see Notes below). Assuming httpd is down and I can guarantee
> that
> > I am the only person who will be logged into the SVN server, can I expect
> > with 99.9% surety that the svn repos are quiescent?
> >
> > Thanks.
> > --
> > -Mike Schwager
> >
> > Notes:
> >
> > We're a little worried about svn hotcopy; we ran into a bug that came
> about
> > under 1.8 when working with older repos; the hotcopy exits with the
> > following error:
> >
> > svnadmin: E22: Serialized hash missing terminator
> >
> > I have compiled subversion-1.9.4 on the server under
> /opt/subversion-1.9.4.
> > If I run that version of svn hotcopy, it appears to work and svnverify
> exits
> > successfully. But if I look at all the files under both the original and
> the
> > hotcopy on one of our repos, I find that a file is missing:
> > repos2/db/rev-prop-atomics.shm . That's probably ok, but still- how do we
> > know the latest hotcopy, and hotcopies of the future, are and will remain
> > 100% bug-free?
>
--
-Mike Schwager
Re: Creating and Verifying a Reliable backup
Hello Matt, On Wed, Jun 1, 2016 at 6:45 PM, Matt Garman wrote: > > As you suggested, if you can make a fancier backup script that shuts > down anyone's ability to make changes to the repo while the backup is > taking place, that's even better. There is `svnadmin freeze`[1] for this. And you can use `svnadmin hotcopy`[2] on a live repository without any problems. Why do you use `rsync`? [1]: http://svnbook.red-bean.com/en/1.8/svn.ref.svnadmin.c.freeze.html [2]: http://svnbook.red-bean.com/en/1.8/svn.ref.svnadmin.c.hotcopy.html -- With best regards, Pavel Lyalyakin VisualSVN Team
Re: svn: E155008issue with subversion in webmethods designer
Hello Stefan, On Sun, May 29, 2016 at 11:03 PM, Stefan wrote: > > I'm not 100% sure, but this might be related to issue #4364, which is fixed > in SVN 1.8.8: > lock of deleted pathes magically reappear when pathes are reused > > In this case upgrading to a later SVN version should resolve the issue. > > Alternatively the issue at question might also be #2507 which is not yet > resolved: > 'commit --no-unlock' doesn't remove locks on files deleted > > In this case the workaround might be to ensure in the tools to not use the > --no-unlock flag and/or to ensure that for deleted files the lock is removed > first via svn unlock. How did you determine that this is #4364 or #2507? To me it looks like webmethods is calling some Subversion client and it is trying to perform `svn unlock` operation on a folder. Looks like the error it totally expected in such case. Without some additional description of the case it's impossible to tell what's going on. -- With best regards, Pavel Lyalyakin VisualSVN Team
Re: svn: E155008issue with subversion in webmethods designer
Hello Divvela, On Fri, May 27, 2016 at 9:52 PM, Divvela Chanukya <2chanu...@gmail.com> wrote: > Hi, > > we are configured subversion(1.7.19) with webmethods(9.5) on windows 2012 > r2. when we try to move a service from one package to other package . > getting the below error . Subversion 1.7 is no longer supported. Upgrade to the latest Subversion 1.8 or 1.9 versions. > when we refresh the packages we see the service is copied but not moved > > Error: > > testsvn/new_folder22:new_flowservice1: [ISS.0132.9117] error running > command [svn, unlock, --username, chanu, --password, , > C:\SoftwareAG\IntegrationServer\packages\testsvn\ns\new_folder22\new_flowservice1\flow.xml, > C:\SoftwareAG\IntegrationServer\packages\testsvn\ns\new_folder22\new_flowservice1\node.ndf, > C:\SoftwareAG\IntegrationServer\packages\testsvn\ns\new_folder22\new_flowservice1, > --force]; exit value: 1; output: [svn: E155008: The node > 'C:\SoftwareAG\IntegrationServer\packages\testsvn\ns\new_folder22\new_flowservice1' > is not a file] > > we raised the issues with application vendor (webmethods )they claim that > the issue is with subversion & asked us to rise issue it subversion forum. Did they say anything else? What did you do to reproduce the problem? To me it does not look like a problem in Subversion. At least it does not look so without additional information you could provide us with. Webmethods is calling `svn unlock` on a directory and it is expected to get this error in such case. You can't lock or unlock directories, you can lock files in directories, but directory as a note can't be locked. -- With best regards, Pavel Lyalyakin VisualSVN Team
Re: Creating and Verifying a Reliable backup
On Wed, Jun 1, 2016 at 11:55 AM, Michael Schwager
wrote:
> Thanks Matt. To your point,
>
> > my collection of backups probably does have some corrupt repo trees...
>
> that is really what I'm driving at. The RAID, offsite, number of backups
> (nightly in our case), and testing is all covered. In other words, I can
> mitigate the effects of failure with all those tried-and-true sysadmin
> techniques.
>
> The essence of my question drives to Subversion specifically. I don't want
> *any* unknown corrupt Subversion repo backups lying around. Meaning I don't
> trust a shotgun approach, where I do enough of them so one is bound to be
> good. I'm looking for a precision approach, where I can be reasonably
> assured that the techniques I'm doing will provide me with a recoverable
> repo at any chosen backup point. ("reasonably" defined as "as close to 100%
> as I can get)
>
> Because if it's Thursday, and Wednesday night's repo backup is the one I
> need, I don't want to have to report back that it's corrupt and the best I
> can do is Tuesday.
>
Personally, I think it is overkill to keep a bunch of backups. With the
exception of revision properties, history is immutable so it is not like
someone can delete something where you would need an older backup to find
it.
The best backup option is svnadmin hotcopy. That is the only backup option
that gets all of the data in the repository. dump and svnsync both do not
backup locks or hook scripts.
As of 1.8 I believe, svnadmin hotcopy supports an --incremental option so
that you can also do these backups very fast. I would recommend doing an
--incremental hotcopy from a post-commit hook script so that you always
have a proper backup. You can then periodically take dumps or perform
verify on your hotcopy or do an rsync to maintain some offsite backup etc.
So do an incremental hotcopy via post-commit hook and do a nightly verify
of the hotcopy if you are paranoid. SVN repositories do not suddenly
become corrupt though. The data is only written once, so unless the disk
drive starts losing bits once it is good it stays good. Only a newer
commit can come in that is somehow "corrupt". That said, corrupt
repositories are really rare.
Mark
Re: Creating and Verifying a Reliable backup
On 6/1/2016 4:58 PM, Michael Schwager wrote: Hello, We are very paranoid about our Subversion repo, notwithstanding the fact that the previous sysadmin didn't back it up. But that's another story. Now I'm here at my job, I've inherited the repo admin duties, and I want to back it up reliably. If we lose it, we're all out of work. My question is: How do I back it up reliably, and verify it so that I can deliver a 100% recovery guarantee to my boss? I have Subversion 1.8.4 on a CentOS 6.3 server, and Tortoise SVN 1.8.11 on Windows 7 clients. I am thinking to do both an svn hotcopy to one directory, and an rsync to another. The svn hotcopy will give me a backup that I'm pretty sure is reliable (see Notes below). Assuming httpd is down and I can guarantee that I am the only person who will be logged into the SVN server, can I expect with 99.9% surety that the svn repos are quiescent? Thanks. -- -Mike Schwager Notes: We're a little worried about svn hotcopy; we ran into a bug that came about under 1.8 when working with older repos; the hotcopy exits with the following error: svnadmin: E22: Serialized hash missing terminator As far as I can tell this indicates a problem in the repository you are trying to hotcopy from. Run svnadmin verify on that to get details where the corruption might be located and resolve that (if possible). I have compiled subversion-1.9.4 on the server under /opt/subversion-1.9.4. If I run that version of svn hotcopy, it appears to work and svnverify exits successfully. But if I look at all the files under both the original and the hotcopy on one of our repos, I find that a file is missing: repos2/db/rev-prop-atomics.shm . That's probably ok, but still- how do we know the latest hotcopy, and hotcopies of the future, are and will remain 100% bug-free? To ensure the integrity of a backup close to 100% I go the fail safe way: 1. svnadmin verify to ensure the current repository is in a good state (if there are errors/issues resolve them) 2. svnadmin dump to dump the current repository 3. svnadmin load the dump into a fresh repository 4. svnadmin dump the newly loaded repository 5. compare the first and the last dump 6. run svnadmin verify on the loaded dump to en If both dumps are equal, I'm certain enough the integrity is given. (Note: this implies the same fsfs-format as well as the same server version for the svnadmin calls). This process however only works when you can take down access to a repository completely. Otherwise svnadmin hotcopy would be my choice too. In addition setting up a mirror which is kept in sync using svnsync is also a reasonable measurement to further increase the reliability of an SVN repository IMO. However, it's also utterly vital to keep the server up to date with patch releases, so you are not suffering flaws/bugs which could impact the server side. Some examples for issues which have been fixed a long time ago and are not fixed in 1.8.4 (but might be relevant for your case of ensuring data correctness/integrity and a reliable backup system): 1.8.5: hotcopy: fix hotcopy losing revprop files in packed repos (issue #4448) 1.8.9: svnadmin dump: don't let invalid mergeinfo stop dump 1.8.9: svnrdump load: fix crash when svn:* normalization (issue #4490) 1.8.9: mod_dav_svn: detect out of dateness correctly during commit (issue #4480) 1.8.11: disable revprop caching feature due to cache invalidation problems 1.8.13: svnadmin load: tolerate invalid mergeinfo at r0 (issue #4476) 1.8.13: svnadmin load: strip references to r1 from mergeinfo (issue #4538) 1.8.13: svnsync: strip any r0 references from mergeinfo (issue #4476) 1.8.14: prevent possible repository corruption on power/disk failures 1.8.16: dump: don't write broken dump files in some ambiguously encoded fsfs repositories (issue #4554) If you feel very conservative you might also wanna consider staying at the old stable version for the server (sidenote: even with the server running at 1.8, clients could use svn 1.9), if you don't need bugfixes/improvements of the current stable svn version. While the current stable build (1.9.4) contains fixes for issues still present in 1.8.16 and also delivers new features, it also contains features/improvements which by the nature of software lifecycles have not been tested as long in the wild as the features present in 1.8.16. So one might argue that from the code integrity point of view, 1.8.16 would be the safer choice to go. If you are compiling your own server, be sure to keep the dependencies used also up to date. -- Regards, Stefan Hett
Re: Creating and Verifying a Reliable backup
Hello Michael, On Wed, Jun 1, 2016 at 7:18 PM, Michael Schwager wrote: > Yessity-yes-yes-ye! svnadmin freeze- I missed that one! That's > something I can really sink my teeth into! Awesome, thanks! > > My intention is to perform the following: > * shut down httpd > Then, for each repoX: > * svnadmin verify /home/svn/hotcopy/repoX. If error, warn and exit. > * svnadmin hotcopy into /home/svn/hotcopy/repoX. If error, warn and exit. > * svnadmin verify /home/svn/hotcopy/repoX. If error, warn and exit. These steps don't require you to shut down httpd. > * svnadmin freeze /path/to/repoX -- rsync -av /path/to/repoX > /home/svn/rsync/repoX. > * rsync -c -n -av /path/to/repoX /home/svn/rsync/repoX. (perform a checksum > compare) If error, warn and exit. Why don't you run `rsync` on "/home/svn/hotcopy/repoX"? You won't need to shut down httpd or run `svnadmin freeze` in such case. > I had considered svnadmin dump, but according to the Wisdom on Serverfault, > "[svnadmin dump] also fails to backup things like the repository scripts and > a few other things..." > (http://serverfault.com/questions/540214/is-it-necessary-to-do-an-svnadmin-dump-to-backup-subversion) Yep, this is right. Subversion repository dump streams (the dump generated by `svnadmin dump` or `svnrdump`) contain versioned history only. Read SVNBook[1]. > "One other thing to watch is svnadmin dump doesn't backup locks and hook > scripts." > http://serverfault.com/questions/52252/best-way-to-do-subversion-backups?rq=1 > > ...I've always presumed this meant that the hotcopy did back up locks and > hook scripts... Yes, hotcopy makes full repository copy with locks and hook scripts. Read SVNBook[2]. [1]: http://svnbook.red-bean.com/en/1.8/svn.reposadmin.maint.html#svn.reposadmin.maint.migrate [2]: http://svnbook.red-bean.com/en/1.8/svn.reposadmin.maint.html#svn.reposadmin.maint.backup -- With best regards, Pavel Lyalyakin VisualSVN Team
Re: Creating and Verifying a Reliable backup
On Wed, Jun 1, 2016 at 10:58 AM, Pavel Lyalyakin wrote: > Hello Matt, > > Why do you use `rsync`? > > [1]: http://svnbook.red-bean.com/en/1.8/svn.ref.svnadmin.c.freeze.html > [2]: http://svnbook.red-bean.com/en/1.8/svn.ref.svnadmin.c.hotcopy.html I have no good reason! Backup scripts were setup long ago, and rsync-based backup scripts (in general) are routine for me. I wasn't aware of freeze and hotcopy at the time. But thank you Mike for starting this thread, clearly my scripts are overdue for some improvement! -Matt
Re: Creating and Verifying a Reliable backup
Yessity-yes-yes-ye! svnadmin freeze- I missed that one! That's something I can really sink my teeth into! Awesome, thanks! My intention is to perform the following: * shut down httpd Then, for each repoX: * svnadmin verify /home/svn/hotcopy/repoX. If error, warn and exit. * svnadmin hotcopy into /home/svn/hotcopy/repoX. If error, warn and exit. * svnadmin verify /home/svn/hotcopy/repoX. If error, warn and exit. * svnadmin freeze /path/to/repoX -- rsync -av /path/to/repoX /home/svn/rsync/repoX. * rsync -c -n -av /path/to/repoX /home/svn/rsync/repoX. (perform a checksum compare) If error, warn and exit. I had considered svnadmin dump, but according to the Wisdom on Serverfault, "[svnadmin dump] also fails to backup things like the repository scripts and a few other things..." ( http://serverfault.com/questions/540214/is-it-necessary-to-do-an-svnadmin-dump-to-backup-subversion ) "One other thing to watch is svnadmin dump doesn't backup locks and hook scripts." http://serverfault.com/questions/52252/best-way-to-do-subversion-backups?rq=1 ...I've always presumed this meant that the hotcopy did back up locks and hook scripts... On Wed, Jun 1, 2016 at 10:51 AM, Pavel Lyalyakin < pavel.lyalya...@visualsvn.com> wrote: > Hello Michael, > > On Wed, Jun 1, 2016 at 5:58 PM, Michael Schwager > wrote: > > Hello, > > We are very paranoid about our Subversion repo, notwithstanding the fact > > that the previous sysadmin didn't back it up. But that's another story. > Now > > I'm here at my job, I've inherited the repo admin duties, and I want to > back > > it up reliably. If we lose it, we're all out of work. > > > > My question is: How do I back it up reliably, and verify it so that I can > > deliver a 100% recovery guarantee to my boss? I have Subversion 1.8.4 on > a > > CentOS 6.3 server, and Tortoise SVN 1.8.11 on Windows 7 clients. > > Don't forget about `svnadmin verify`[1]. It makes sense to run it > right now if you've never ran it. ;) > > And if you want to deliver recovery guarantee close to 100%, consider > `svnadmin dump`[2] in addition to hotcopy. > > > I am thinking to do both an svn hotcopy to one directory, and an rsync to > > another. The svn hotcopy will give me a backup that I'm pretty sure is > > reliable (see Notes below). Assuming httpd is down and I can guarantee > that > > I am the only person who will be logged into the SVN server, can I expect > > with 99.9% surety that the svn repos are quiescent? > > I'm not sure that I understand what you mean by "quiescent" here. > What's your concern? > > So you are planning to make a hotcopy and then transfer this hotcopy > using rsync to another machine, right? > > * It's safe to run `svnadmin hotcopy` on a live repository. > > * And if you'd have to use `rsync` on a *live* repository, run > `svnadmin freeze`[3] to prevent concurrent commits to the repo while > `rsync` is running. > > If httpd is disabled and you are the only user logged on, this means that > your repositories should be untouched by any other processes at this > moment. Unless you have some indexing services that touch the repos > (and it is generally recommended to exclude repos from indexing). > > > Notes: > > > > We're a little worried about svn hotcopy; we ran into a bug that came > about > > under 1.8 when working with older repos; the hotcopy exits with the > > following error: > > > > svnadmin: E22: Serialized hash missing terminator > > It seems to be a bug #4554[4] that has been fixed in Subversion 1.9. > `svnadmin hotcopy` could fail on a repository with older FSFS format. > > > I have compiled subversion-1.9.4 on the server under > /opt/subversion-1.9.4. > > If I run that version of svn hotcopy, it appears to work and svnverify > exits > > successfully. But if I look at all the files under both the original and > the > > hotcopy on one of our repos, I find that a file is missing: > > repos2/db/rev-prop-atomics.shm . That's probably ok, but still- how do we > > know the latest hotcopy, and hotcopies of the future, are and will remain > > 100% bug-free? > > Use `svnadmin verify` and consider using `svnadmin dump` in addition > to hotcopies. Make sure that your Subversion server is on the latest > patch update too. > > [1]: http://svnbook.red-bean.com/en/1.8/svn.ref.svnadmin.c.verify.html > [2]: http://svnbook.red-bean.com/en/1.8/svn.ref.svnadmin.c.dump.html > [3]: http://svnbook.red-bean.com/en/1.8/svn.ref.svnadmin.c.freeze.html > [4]: https://issues.apache.org/jira/browse/SVN-4554 > > -- > With best regards, > Pavel Lyalyakin > VisualSVN Team > -- -Mike Schwager
Re: Creating and Verifying a Reliable backup
On Wed, Jun 1, 2016 at 11:20 AM, Stefan Hett wrote: > On 6/1/2016 4:58 PM, Michael Schwager wrote: > > ...we ran into a bug that came about under 1.8 when working with older > repos; the hotcopy exits with the following error: > > svnadmin: E22: Serialized hash missing terminator > > As far as I can tell this indicates a problem in the repository you are > trying to hotcopy from. Run svnadmin verify on that to get details where > the corruption might be located and resolve that (if possible). > No it's a known bug when hotcopying using 1.8.x on a repo started with an old version of subversion, findable via Google. svnadmin verify on the original repo shows clean. -- -Mike Schwager
Re: Creating and Verifying a Reliable backup
On Wed, Jun 1, 2016 at 11:29 AM, Pavel Lyalyakin < pavel.lyalya...@visualsvn.com> wrote: > Hello Michael, > > On Wed, Jun 1, 2016 at 7:18 PM, Michael Schwager > wrote: > ... > > My intention is to perform the following: > ... > > * svnadmin verify /home/svn/hotcopy/repoX. If error, warn and exit. > > * svnadmin hotcopy into /home/svn/hotcopy/repoX. If error, warn and exit. > > * svnadmin verify /home/svn/hotcopy/repoX. If error, warn and exit. > > These steps don't require you to shut down httpd. > Most likely quite true, but it's nighttime and I'm paranoid. I'm looking for guarantees wherever I can find them. and the cost is quite low (ie, I shut down something that nobody should be using anyway). > * svnadmin freeze /path/to/repoX -- rsync -av /path/to/repoX > > /home/svn/rsync/repoX. > > * rsync -c -n -av /path/to/repoX /home/svn/rsync/repoX. (perform a > checksum > > compare) If error, warn and exit. > > Why don't you run `rsync` on "/home/svn/hotcopy/repoX"? > That assumes the hotcopy is correct. At the start of my script, the only thing that I am comfortable assuming is correct is... well, very little. Even the original repo may have a bug lurking in it somewhere. But the closer I can get to the original, presumably good, copy of our code, the happier I am every step along the way. > > ...I've always presumed this meant that the hotcopy did back up locks and > > hook scripts... > > Yes, hotcopy makes full repository copy with locks and hook scripts. > Read SVNBook[2]. > Excellent. Thanks for the confirmation. I note that the svn book does mention that it performs a "fully functional Subversion repository, able to be dropped in as a replacement for your live repository", but as there are caveats with the svndump command that the uninitiated (yours truly) may not be familiar with, it's not entirely clear that hotcopy obviates all of those concerns. Indeed, a hotcopy does not produce the name number of files as an rsync (a file named rev-prop-atomics.shm is missing in one of my hotcopied directories), so the question remained in my mind until now. Thanks again for the help! -- -Mike Schwager
Re: Creating and Verifying a Reliable backup
On Wed, Jun 1, 2016 at 11:20 AM, Stefan Hett wrote: > To ensure the integrity of a backup close to 100% I go the fail safe way: > 2. svnadmin dump to dump the current repository > 3. svnadmin load the dump into a fresh repository > 4. svnadmin dump the newly loaded repository > 5. compare the first and the last dump > I like this suggestion. What I think I will do ultimately is check out the entire repo somewhere, check out the entire hotcopied repo somewhere, and check all the files against one another. -- -Mike Schwager
