Re: SVN backup with lvm snapshots and rsync
On Wed, Feb 15, 2012 at 03:02:04AM +0200, Daniel Shahaf wrote: > Stefan Sperling wrote on Tue, Feb 14, 2012 at 23:19:58 +0100: > > Until then, svnsync or svnadmin dump/load are the only officially > > supported incremental backup solutions. But, as Daniel explained, > > 'rsync' followed by 'svnadmin recover' produces valid copies of > > FSFS repositories, too. > > I didn't say that, Stefan, since it's not true. rsync is not safe if > the SQLite db is being read or written whilst rsync runs. Ah, true. I keep forgetting about the rep cache... I'm sorry if you felt I was mis-attributing this to you. All I can say in my defense is that the whole point of adding incremental hotcopy support (which of course handles the rep-cache properly) was to finally stop worrying and forget about such details :)
Re: svn+ssh vs. https (different access methods)
Hello, Original-Nachricht > What kind of trouble did you have with apache in the reverse-proxy > mode? I've had that working and I think the only issue was that the > relative path had to be the same as the actual target. That is you > can't proxy http:/realhost/svn as http://proxyhost/somewhere/svn. > Also for a straight passthrough you might run stunnel accepting https > from outside your dmz, connecting to http on the inside server. I had a problem using the COPY-Command over the connections established over the reverse-proxy... "Server sent unexpected return value (502 Bad Gateway) in response to COPY request for ..." But today I've found out that there is a solution using Apache 2.2.4 or greater (add "RequestHeader edit Destination ^https http early" to the Subversion-Configuration): http://www.thedumbterminal.co.uk/php/knowledgebase/article/141 Rgds Dieter -- Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
Bug report
Hello, The following exception occurs every time I try to do an SVN update. There is only one file changed (by somebody else), but it is quite large (1.67MB). The exception also occurred using the previous (1.6.x version) of Tortoise. When trying to synchronise with the Eclipse subversion plugin, Eclipse crashes. Kind regards, Stephan Lichtendahl Technolution --- Subversion Exception! --- Subversion encountered a serious problem. Please take the time to report this on the Subversion mailing list with as much information as possible about what you were trying to do. But please first search the mailing list archives for the error message to avoid reporting the same problem repeatedly. You can find the mailing list archives at http://subversion.apache.org/mailing-lists.html Subversion reported the following (you can copy the content of this dialog to the clipboard using Ctrl-C): In file 'D:\Development\SVN\Releases\TortoiseSVN-1.7.4\ext\subversion\subversion\libsvn_delta\text_delta.c' line 728: assertion failed (window->sview_len == 0 || (window->sview_offset >= ab->sbuf_offset && (window->sview_offset + window->sview_len >= ab->sbuf_offset + ab->sbuf_len))) --- OK ---
backups using LVM snapshots and "svnadmin lstxns"
I want to back up a filesystem on a Linux machine, by taking
an LVM snapshot of the filesystem then copying it to a remote
server. However, there are Subversion repositories on the
filesystem, and I'm concerned that there could be a commit
under way to one or more of these repositories at the instant
that I take the snapshot, which could result in my getting a
corrupted SVN repository in the backup.
I think I have a solution to this problem, which I present in
the following PHP code. (It should be intelligible to anyone
who knows a C-like language.) Does anyone know whether the
solution will work?
# code begins here
do {
make_snapshot();
$transaction_found = FALSE;
foreach ($repository_on_snapshot as $dir) {
if(system("svnadmin lstxns $dir") !== '') {
$transaction_found = TRUE;
break;
}
}
} while($transaction_found);
# all the SVN repositories are consistent,
# so continue with the backup...
# code ends here
The SVN book strongly implies that the solution will work,
but I would prefer to have an explicit assurance on this point.
I'm not subscribed to this list, so please copy me on any
replies.
Re: backups using LVM snapshots and "svnadmin lstxns"
Please read the in the archives the thread taking place YESTERDAY that
discusses LVM snapshots and validity. If you still have any questions
after reading it --- do ask.
Cheers,
Daniel
Ronan wrote on Wed, Feb 15, 2012 at 11:35:25 +:
> I want to back up a filesystem on a Linux machine, by taking
> an LVM snapshot of the filesystem then copying it to a remote
> server. However, there are Subversion repositories on the
> filesystem, and I'm concerned that there could be a commit
> under way to one or more of these repositories at the instant
> that I take the snapshot, which could result in my getting a
> corrupted SVN repository in the backup.
>
> I think I have a solution to this problem, which I present in
> the following PHP code. (It should be intelligible to anyone
> who knows a C-like language.) Does anyone know whether the
> solution will work?
>
> # code begins here
>
> do {
> make_snapshot();
>
> $transaction_found = FALSE;
>
> foreach ($repository_on_snapshot as $dir) {
> if(system("svnadmin lstxns $dir") !== '') {
> $transaction_found = TRUE;
> break;
> }
> }
> } while($transaction_found);
>
> # all the SVN repositories are consistent,
> # so continue with the backup...
>
> # code ends here
>
> The SVN book strongly implies that the solution will work,
> but I would prefer to have an explicit assurance on this point.
>
> I'm not subscribed to this list, so please copy me on any
> replies.
Re: SVN backup with lvm snapshots and rsync
On Wed, Feb 15, 2012 at 6:24 AM, Stefan Sperling wrote: > On Wed, Feb 15, 2012 at 03:02:04AM +0200, Daniel Shahaf wrote: >> Stefan Sperling wrote on Tue, Feb 14, 2012 at 23:19:58 +0100: >> > Until then, svnsync or svnadmin dump/load are the only officially >> > supported incremental backup solutions. But, as Daniel explained, >> > 'rsync' followed by 'svnadmin recover' produces valid copies of >> > FSFS repositories, too. >> >> I didn't say that, Stefan, since it's not true. rsync is not safe if >> the SQLite db is being read or written whilst rsync runs. > > Ah, true. I keep forgetting about the rep cache... > I'm sorry if you felt I was mis-attributing this to you. > > All I can say in my defense is that the whole point of adding incremental > hotcopy support (which of course handles the rep-cache properly) was to > finally stop worrying and forget about such details :) Thank you for this advice. From what I gather rep-cache.db, can be regenerated by svn. If I used rsync and excluded the rep-cache.db would I then want to run 'svnadmin recover' on these backup or is rep-cache.db regenerated automatically when the repository is used?
Re: SVN backup with lvm snapshots and rsync
Harry Bullen wrote on Wed, Feb 15, 2012 at 12:03:15 -0500: > On Wed, Feb 15, 2012 at 6:24 AM, Stefan Sperling wrote: > > On Wed, Feb 15, 2012 at 03:02:04AM +0200, Daniel Shahaf wrote: > >> Stefan Sperling wrote on Tue, Feb 14, 2012 at 23:19:58 +0100: > >> > Until then, svnsync or svnadmin dump/load are the only officially > >> > supported incremental backup solutions. But, as Daniel explained, > >> > 'rsync' followed by 'svnadmin recover' produces valid copies of > >> > FSFS repositories, too. > >> > >> I didn't say that, Stefan, since it's not true. rsync is not safe if > >> the SQLite db is being read or written whilst rsync runs. > > > > Ah, true. I keep forgetting about the rep cache... > > I'm sorry if you felt I was mis-attributing this to you. > > > > All I can say in my defense is that the whole point of adding incremental > > hotcopy support (which of course handles the rep-cache properly) was to > > finally stop worrying and forget about such details :) > > Thank you for this advice. From what I gather rep-cache.db, can be > regenerated by svn. If I used rsync and excluded the rep-cache.db > would I then want to run 'svnadmin recover' on these backup or is > rep-cache.db regenerated automatically when the repository is used? Yes and no. The data for new revisions will be written if fsfs.conf is configured to enable the cache. The data for old revisions won't be backfilled, but it's certainly possible to do so (by walking all the node-revs and adding their reps to the table), but I don't know that anyone has actually written the code to do so.
Re: SVN backup with lvm snapshots and rsync
On Wed, Feb 15, 2012 at 12:03:15PM -0500, Harry Bullen wrote: > From what I gather rep-cache.db, can be > regenerated by svn. If I used rsync and excluded the rep-cache.db > would I then want to run 'svnadmin recover' on these backup or is > rep-cache.db regenerated automatically when the repository is used? It will be re-created automatically. But you're losing the benefits of rep-sharing because a fresh rep-cache.db will be created. So you'll start with an empty cache all over again. The cache exists to prevent unnecessary growth of your repository. It does not affect correctness. It prevents duplicate content from being redundantly stored in the repository. E.g. say two people make two independent commits, which each add one file. The file has a different name in either commit but the exact same content in both commits. rep-cache.db would store a checksum of the file's content when the file is first added. Keyed on this checksum is the exact location of the content in the repository. Locating the content without this cache would involve parsing all existing revisions, which is prohibitively expensive. During the second commit which adds the same content the content's checksum will be computed and generate a cache hit. The file added during the second commit will then be made to refer to the content already stored during the first commit. If the rep-cache is cleared between the two commits, there will be a cache miss so the redundancy cannot be detected. The content will be stored redundantly in the newer revision. But it will also cause a new rep-cache entry. So now you're good again until the cache is cleared once more. To work around this limitation you could write a small tool that uses the sqlite API to perform a hotcopy of rep-cache.db and run this tool in addition to rsync (see http://sqlite.org/backup.html). But rather than going through that effort, I would recommend using svnadmin dump/load, or svnsync with file:// URLs, until Subversion 1.8 is released. At which point you can switch over to using "svnadmin hotcopy --incremental", which will copy rep-cache.db via the appropriate sqlite APIs.
Re: predecessor count for the root node-revision is wrong message
On Wed, Feb 8, 2012 at 6:22 PM, Nico Kadel-Garcia wrote:
> On Wed, Feb 8, 2012 at 7:42 PM, Daniel Shahaf wrote:
>> Daniel Shahaf wrote on Thu, Feb 09, 2012 at 01:46:45 +0200:
>>> Jason Wong wrote on Wed, Feb 08, 2012 at 15:32:05 -0800:
>
>>> Get xxd.exe from http://www.vim.org/ and cat.exe and sed.exe from
>>> http://gnuwin32.sf.net (or from Cygwin). Delete from the script the
>>> line that uses the 'head' command.
>>
>> There is a second use of 'head', which you shouldn't delete. So
>> instead, just get head.exe from the same place as the other two, or use
>> the following kind of statement:
>
> Or install CygWin and run the scripts from inside CygWin. This does
> present end-of-line issues, so be very careful about using "svn:eol
> native" properties.
>
>>
>> my $line = do {
>> open FOO, "perl -V 2>&1 |";
>> ;
>> };
>>
>> Lastly, there's a 'sed' invocation that uses single-quoted arguments.
>> All it does is "print the input up to the first empty line" --- feel
>> free to implement it differently. (One way:
>>
>> my @lines = split /\n/, `command | goes | here`;
>> $_ and print or last for @lines;
>>
>> Both of these examples could do with some error checking.)
>>
>> Daniel
>> (yes, there's also a neater way to do this without split(). but it's
>> not a Perl class here)
Hello.
Sorry for the delay. Here is an update of what I have done since
the last time I posted.
I have run "svn log -q ^/" on the respository and it came back with
no missing revisions.
Since I first posted, each of the projects we have tried to build
that had failed have since successfully been built without any changes
on our side.
I was having an issue with converting the script to run in windows as
I was only getting the first line returned so I set up cygwin.
I ran the script against both of the revisions (61815 and 61852) in
mentioned in the Apache error log and the output was the same for
each.
Commands:
dump-noderev.pl /repository
/project/binaries/release/phase1/iteration/81/trunk 61815
dump-noderev.pl /repository
/project/binaries/release/phase1/iteration/81/trunk 61852
Output:
id: 9-45362.0-61242.r61424/0
type: dir
pred: 9-45362.0-60310/0
count: 43
text: 58741 121716266 218 218 74eb31e90880ba1345fc49252ca6efe6
cpath: /project/binaries/release/phase1/iteration/81/trunk
copyfrom: 61423 /project/binaries/release/phase1/iteration/80/trunk
Is this information helpful? Let me know if this tells you anything. Thanks
Jason.
Re: SVN backup with lvm snapshots and rsync
Stefan Sperling writes: > But rather than going through that effort, I would recommend using > svnadmin dump/load, or svnsync with file:// URLs, until Subversion 1.8 > is released. At which point you can switch over to using > "svnadmin hotcopy --incremental", which will copy rep-cache.db via > the appropriate sqlite APIs. Are you suggesting this because you know the LVM snapshot method doesn't work or just because it's a method that avoids non-standard Subversion tools? Running rsync on an LVM snapshot looks like it should work to me. An LVM snapshot freezes the filesystem by diverting all writes into the snapshot area. A Subversion repository in the frozen filesystem may have transactions in progress, Subversion transactions or SQLite transactions, but they won't change while the copy is made. If the copied repository is ever used then these partial transactions will simply be ignored. Provided Subversion gets its fsyncs and writes in the correct order, and I believe it does, this looks like it should work. I'd go as far as saying if the copied repository doesn't work then this is a Subversion bug, or perhaps an LVM/OS bug. -- Philip
Re: SVN backup with lvm snapshots and rsync
On Wed, Feb 15, 2012 at 07:05:08PM +, Philip Martin wrote: > Stefan Sperling writes: > > > But rather than going through that effort, I would recommend using > > svnadmin dump/load, or svnsync with file:// URLs, until Subversion 1.8 > > is released. At which point you can switch over to using > > "svnadmin hotcopy --incremental", which will copy rep-cache.db via > > the appropriate sqlite APIs. > > Are you suggesting this because you know the LVM snapshot method doesn't > work or just because it's a method that avoids non-standard Subversion > tools? I don't know enough about SQlite to judge whether the DB will keep working at any frozen state a snapshot might create. And, yes, in general, using offical Subversion tools for backup purposes has the advantage that they will keep working after an upgrade. No matter what kinds of changes we make to repository fomats, we'll keep the official tools working. That said, I agree that filesystem snapshots should work fine, provided that sqlite databases will cope, too.
Re: SVN backup with lvm snapshots and rsync
On Wed, 15 Feb 2012 20:41:48 +, Stefan Sperling wrote: > I don't know enough about SQlite to judge whether the DB will > keep working at any frozen state a snapshot might create. If it doesn't then it wouldn't be resilient to system crashes either, and *that* wouldn't exactly be a recommendation to use it at all. Andreas -- "Totally trivial. Famous last words." From: Linus Torvalds Date: Fri, 22 Jan 2010 07:29:21 -0800
Re: SVN backup with lvm snapshots and rsync
On Wed, Feb 15, 2012 at 2:05 PM, Philip Martin wrote: > Stefan Sperling writes: > >> But rather than going through that effort, I would recommend using >> svnadmin dump/load, or svnsync with file:// URLs, until Subversion 1.8 >> is released. At which point you can switch over to using >> "svnadmin hotcopy --incremental", which will copy rep-cache.db via >> the appropriate sqlite APIs. > > Are you suggesting this because you know the LVM snapshot method doesn't > work or just because it's a method that avoids non-standard Subversion > tools? > > Running rsync on an LVM snapshot looks like it should work to me. An It'll work great until it doesn't. If the repository is busy in a commit or partial commit at the moment of the snapshot, you're in potentially very deep trouble. You've entered not only "split-brain" territory, where the repositories are out of sync, but partial commit territory. Subversion is careful that its database writes are atomic operations, but you can't assure that if the filesystem is in the midst of being written when snapshotted. You've casually stated "An LVM snapshot freezes the filesystem". No, it freezes the *partition*, and metadata about the blocks used on the raw device, and excludes them from further writing (until the buffer space allocated for LVM overflows. To the casual eye, this looks like freezing the filesystem, but if you look more closely, I think you'll find that anything paged out and not yet written to disk will not be in the snapshot. And modern filesystems do this kind of paging as a matter of course for performance reasons. Unless you do a "sync" command, or various low level flush commands, you don't know that you write to disk has actually made it to the platter. In a lightly loaded environment, middle of the night, no commits are going on? No problem. But in that kind of environment, you should have space to run an rsync based snapshot several times in a row to verify operational consistency and be done with it, and avoid LVM altogether. Robust backup takes thought. In a heavily loaded system that nightly commits regression testing tags or branches? You have a potential adventure.
What does the number in the Repository file "format" represent ?
How can we determine what version of Subversion a Repository currently is. 5 = 1.4 ? or 1.5 ? Need to wire a script to check for older versions as we upgraded the FSFS server from 1.4.x to 1.5.x then 1.6.x phil
Re: SVN backup with lvm snapshots and rsync
On Wed, Feb 15, 2012 at 3:37 PM, Nico Kadel-Garcia wrote: > If the repository is busy in a commit or partial commit at the moment > of the snapshot, you're in potentially very deep trouble. You've > entered not only "split-brain" territory, where the repositories are > out of sync, but partial commit territory. Subversion is careful that > its database writes are atomic operations, but you can't assure that > if the filesystem is in the midst of being written when snapshotted. > Wouldn't this be the same state of the disk if the system crashed at that point? So if you can't expect to recover from that copy you should also be worried about recovering after a power failure. -- Les Mikesell lesmikes...@gmail.com
Re: SVN backup with lvm snapshots and rsync
Nico Kadel-Garcia writes: > Unless you do a "sync" > command, or various low level flush commands, you don't know that you > write to disk has actually made it to the platter. Subversion does that. It uses fsync (plus fsync on directories on Linux) before assuming data is on disk. -- Philip
Re: What does the number in the Repository file "format" represent ?
On Feb 15, 2012, at 15:45, Phil Pinkerton wrote: > How can we determine what version of Subversion a Repository currently is. > > 5 = 1.4 ? or 1.5 ? > > Need to wire a script to check for older versions as we upgraded the FSFS > server from 1.4.x to 1.5.x then 1.6.x http://stackoverflow.com/questions/282460/how-to-find-out-subversion-repository-version#answer-282484 Make sure you distinguish between the two different format files. The format file directly in the repository root is not relevant to this question (AFAIR it has been "5" for a long time); look instead at the format file in the db directory in the repository root. For a newly-created repository under Subversion 1.6.17, format is "5" and db/format is "4 layout sharded 1000".
Re: SVN backup with lvm snapshots and rsync
On Wed, Feb 15, 2012 at 6:19 PM, Philip Martin wrote: > Nico Kadel-Garcia writes: > >> Unless you do a "sync" >> command, or various low level flush commands, you don't know that you >> write to disk has actually made it to the platter. > > Subversion does that. It uses fsync (plus fsync on directories on > Linux) before assuming data is on disk. So the risk is reduced, which is good. It lowers the window of vulnerability (between when a commit is published, and when the fsync occurs). This is actually good progamming. Not everyone who deals with real data is this thoughtful.
Re: SVN backup with lvm snapshots and rsync
Nico Kadel-Garcia wrote on Wed, Feb 15, 2012 at 19:41:57 -0500: > On Wed, Feb 15, 2012 at 6:19 PM, Philip Martin > wrote: > > Nico Kadel-Garcia writes: > > > >> Unless you do a "sync" > >> command, or various low level flush commands, you don't know that you > >> write to disk has actually made it to the platter. > > > > Subversion does that. It uses fsync (plus fsync on directories on > > Linux) before assuming data is on disk. > > So the risk is reduced, which is good. It lowers the window of > vulnerability (between when a commit is published, and when the fsync > occurs). This is actually good progamming. Not everyone who deals with > real data is this thoughtful. There is no "window of vulnerability".
Re: predecessor count for the root node-revision is wrong message
Jason Wong wrote on Wed, Feb 15, 2012 at 10:20:23 -0800:
> On Wed, Feb 8, 2012 at 6:22 PM, Nico Kadel-Garcia wrote:
> > On Wed, Feb 8, 2012 at 7:42 PM, Daniel Shahaf wrote:
> >> Daniel Shahaf wrote on Thu, Feb 09, 2012 at 01:46:45 +0200:
> >>> Jason Wong wrote on Wed, Feb 08, 2012 at 15:32:05 -0800:
> >
> >>> Get xxd.exe from http://www.vim.org/ and cat.exe and sed.exe from
> >>> http://gnuwin32.sf.net (or from Cygwin). Delete from the script the
> >>> line that uses the 'head' command.
> >>
> >> There is a second use of 'head', which you shouldn't delete. So
> >> instead, just get head.exe from the same place as the other two, or use
> >> the following kind of statement:
> >
> > Or install CygWin and run the scripts from inside CygWin. This does
> > present end-of-line issues, so be very careful about using "svn:eol
> > native" properties.
> >
> >>
> >> my $line = do {
> >> open FOO, "perl -V 2>&1 |";
> >> ;
> >> };
> >>
> >> Lastly, there's a 'sed' invocation that uses single-quoted arguments.
> >> All it does is "print the input up to the first empty line" --- feel
> >> free to implement it differently. (One way:
> >>
> >> my @lines = split /\n/, `command | goes | here`;
> >> $_ and print or last for @lines;
> >>
> >> Both of these examples could do with some error checking.)
> >>
> >> Daniel
> >> (yes, there's also a neater way to do this without split(). but it's
> >> not a Perl class here)
>
> Hello.
>
> Sorry for the delay. Here is an update of what I have done since
> the last time I posted.
>
> I have run "svn log -q ^/" on the respository and it came back with
> no missing revisions.
>
I stand corrected, then. I've confirmed on another instance of the bug
that 'svn log -q ^/' does not behave abnormally when the bug is present.
Sorry for the misinformation.
Question to devs: what operation will walk the predecessor links for
the root fspath? (and can therefore be used to identify instances of
the bug)
> Since I first posted, each of the projects we have tried to build
> that had failed have since successfully been built without any changes
> on our side.
>
What is the significance of this? I don't know how your build process
interacts with Subversion.
> I was having an issue with converting the script to run in windows as
> I was only getting the first line returned so I set up cygwin.
>
> I ran the script against both of the revisions (61815 and 61852) in
> mentioned in the Apache error log and the output was the same for
> each.
>
> Commands:
>dump-noderev.pl /repository
> /project/binaries/release/phase1/iteration/81/trunk 61815
>dump-noderev.pl /repository
> /project/binaries/release/phase1/iteration/81/trunk 61852
>
> Output:
>id: 9-45362.0-61242.r61424/0
>type: dir
>pred: 9-45362.0-60310/0
Are you sure that's the value of the pred: field? It contains only
one ".", instead of two.
>count: 43
>text: 58741 121716266 218 218 74eb31e90880ba1345fc49252ca6efe6
>cpath: /project/binaries/release/phase1/iteration/81/trunk
>copyfrom: 61423 /project/binaries/release/phase1/iteration/80/trunk
>
> Is this information helpful? Let me know if this tells you anything. Thanks
>
The fact that the output is identical suggests that the
/project/binaries/release/phase1/iteration/81/trunk tree hasn't changed
between those two revisions (or that there was a directory replace above
it).
However, this is the error you report:
[Tue Jan 31 11:37:23 2012] [error] [client 9.31.13.109] predecessor count
for the root node-revision is wrong: found 61815, committing r61852 [409,
#160004]
The metadata this error complains about will be output by these two
commands:
./dump-noderev.pl /repository / 61851
./dump-noderev.pl /repository / 61852
> Jason.
Cheers,
Daniel
