Re: large number of large binary files in subversion

2011-05-31 Thread Daniel Shahaf
[ old thread, but I'd like to add this warning to the archives: ] > > > > If you ensure that no commits happen during the backup period you > > > > could use rsync instead. > > > > > > It is not safe to rsync live Subversion filesystems. (the result may or > > > may not be corrupt) > > > > That's

RE: large number of large binary files in subversion

2011-05-25 Thread Winston Smith
> Date: Wed, 25 May 2011 12:17:07 +0200 > From: s...@elego.de > To: d...@daniel.shahaf.name > CC: smith_winston_6...@hotmail.com; dev@subversion.apache.org > Subject: Re: large number of large binary files in subversion > > On Wed,

Re: [PATCH] Server-transmitted "final SHA1 checksums" (Was: "large number of large binary files in subversion")

2011-05-25 Thread C. Michael Pilato
On 05/25/2011 10:42 AM, Bert Huijben wrote: > To resolve this and similar problems we defined that we are not allowed > to remove pristines with refcount zero until there are no more workqueue > items, and nobody else has a working copy lock. > > (During update the only reference to a pristine is

RE: [PATCH] Server-transmitted "final SHA1 checksums" (Was: "large number of large binary files in subversion")

2011-05-25 Thread Bert Huijben
> -Original Message- > From: Branko Čibej [mailto:br...@xbc.nu] On Behalf Of Branko Cibej > Sent: dinsdag 24 mei 2011 18:22 > To: dev@subversion.apache.org > Subject: Re: [PATCH] Server-transmitted "final SHA1 checksums" (Was: "large > number of large bin

AW: large number of large binary files in subversion

2011-05-25 Thread Markus Schaber
Hi, > Von: Stefan Sperling [mailto:s...@elego.de] > > > using > > > 'svnadmin hotcopy' to copy your repositories to it once a week or so > > > > Yes, I planned to do that for a read-only backup repository as part of > > various backup schedules (daily, weekly, monthly, yearly). > > Unfortunately

Re: large number of large binary files in subversion

2011-05-25 Thread Stefan Sperling
On Wed, May 25, 2011 at 01:02:28PM +0300, Daniel Shahaf wrote: > Stefan Sperling wrote on Wed, May 25, 2011 at 11:45:21 +0200: > > On Wed, May 25, 2011 at 10:03:16AM +1100, Winston Smith wrote: > > > Yes, I planned to do that for a read-only backup repository as part of > > > various backup schedul

Re: large number of large binary files in subversion

2011-05-25 Thread Daniel Shahaf
Stefan Sperling wrote on Wed, May 25, 2011 at 11:45:21 +0200: > On Wed, May 25, 2011 at 10:03:16AM +1100, Winston Smith wrote: > > Yes, I planned to do that for a read-only backup repository as part of > > various backup schedules (daily, weekly, monthly, yearly). > > Unfortunately there is no inc

Re: large number of large binary files in subversion

2011-05-25 Thread Stefan Sperling
On Wed, May 25, 2011 at 10:03:16AM +1100, Winston Smith wrote: > > > > repo(s) is/are on a permanently mounted USB disk. > > > > The USB stick might give you less i/o throughput than an internal hard > > disk, and it might fail early due to flash wearing out. > > I believe I mentioned a disk, not

Re: [PATCH] Server-transmitted "final SHA1 checksums" (Was: "large number of large binary files in subversion")

2011-05-25 Thread Greg Stein
On May 24, 2011 11:47 AM, "Stefan Sperling" wrote: > > On Tue, May 24, 2011 at 11:30:39AM -0400, C. Michael Pilato wrote: > > I held off on committing my RA work because there were no consumers. But > > I've still considering making the change because a) there's no penalty for > > doing so, and b

Re: [PATCH] Server-transmitted "final SHA1 checksums" (Was: "large number of large binary files in subversion")

2011-05-25 Thread Julian Foad
On Tue, 2011-05-24 at 18:21 +0200, Branko Čibej wrote: > On 24.05.2011 17:53, Arwin Arni wrote: > >> I was hoping to slip in > >> support for avoiding those text transfers altogether where possible. > >> But I > >> ran into the obvious problems with the editor interface. (Also, I > >> became > >>

RE: large number of large binary files in subversion

2011-05-24 Thread Winston Smith
> > repo(s) is/are on a permanently mounted USB disk. > > The USB stick might give you less i/o throughput than an internal hard > disk, and it might fail early due to flash wearing out. I believe I mentioned a disk, not a stick. But thanks anyway... > using > 'svnadmin hotcopy' to copy your rep

Re: [PATCH] Server-transmitted "final SHA1 checksums" (Was: "large number of large binary files in subversion")

2011-05-24 Thread C. Michael Pilato
On 05/24/2011 11:47 AM, Stefan Sperling wrote: > On Tue, May 24, 2011 at 11:30:39AM -0400, C. Michael Pilato wrote: >> I held off on committing my RA work because there were no consumers. But >> I've still considering making the change because a) there's no penalty for >> doing so, and b) maybe if

Re: [PATCH] Server-transmitted "final SHA1 checksums" (Was: "large number of large binary files in subversion")

2011-05-24 Thread Branko Čibej
On 24.05.2011 17:53, Arwin Arni wrote: >> I was hoping to slip in >> support for avoiding those text transfers altogether where possible. >> But I >> ran into the obvious problems with the editor interface. (Also, I >> became >> concerned about the possibility of a race condition in the pristines

Re: [PATCH] Server-transmitted "final SHA1 checksums" (Was: "large number of large binary files in subversion")

2011-05-24 Thread Arwin Arni
On Tuesday 24 May 2011 09:00 PM, C. Michael Pilato wrote: On 05/24/2011 04:23 AM, Stefan Sperling wrote: On Tue, May 24, 2011 at 12:04:26PM +0530, Arwin Arni wrote: Why can't we send the recorded checksum from the server instead of sending the whole file and then calculating it on the client si

Re: [PATCH] Server-transmitted "final SHA1 checksums" (Was: "large number of large binary files in subversion")

2011-05-24 Thread Stefan Sperling
On Tue, May 24, 2011 at 11:30:39AM -0400, C. Michael Pilato wrote: > I held off on committing my RA work because there were no consumers. But > I've still considering making the change because a) there's no penalty for > doing so, and b) maybe if we add the requisite client-side magic in 1.8, 1.7

[PATCH] Server-transmitted "final SHA1 checksums" (Was: "large number of large binary files in subversion")

2011-05-24 Thread C. Michael Pilato
On 05/24/2011 04:23 AM, Stefan Sperling wrote: > On Tue, May 24, 2011 at 12:04:26PM +0530, Arwin Arni wrote: >> Why can't we send the recorded checksum from the server instead of >> sending the whole file and then calculating it on the client side? >> >> If the checksum matches one of the pristine

Re: large number of large binary files in subversion

2011-05-24 Thread Stefan Sperling
On Tue, May 24, 2011 at 06:02:07PM +1100, Winston Smith wrote: > > Folks, > > Thanks for your replies. So, in principle, I should not expect any problems. > The machine would be a decent one-core Athlon3500+ with 2GB RAM, > doing nothing else other than serving bugzilla, reviewboard and mediawik

Re: large number of large binary files in subversion

2011-05-24 Thread Stefan Sperling
On Tue, May 24, 2011 at 12:04:26PM +0530, Arwin Arni wrote: > Why can't we send the recorded checksum from the server instead of > sending the whole file and then calculating it on the client side? > > If the checksum matches one of the pristine files, then use that to > populate the nodes table.

RE: large number of large binary files in subversion

2011-05-24 Thread Winston Smith
corruption. But this does not seem to be of any concern. Thanks again. - Winston > Date: Tue, 24 May 2011 12:04:26 +0530 > From: ar...@collab.net > To: dev@subversion.apache.org > Subject: Re: large number of large binary files in subversion >

Re: large number of large binary files in subversion

2011-05-23 Thread Arwin Arni
On Tuesday 24 May 2011 12:58 AM, Stefan Sperling wrote: On Mon, May 23, 2011 at 11:07:50PM +0400, Konstantin Kolinko wrote: In svn 1.7 there is pristine storage area in the working copy, where all present files are stored by their checksums. If I understand this pristine storage correctly, if yo

RE: large number of large binary files in subversion

2011-05-23 Thread Bob Archer
> On Mon, May 23, 2011 at 3:07 PM, Konstantin Kolinko > wrote: > > 2011/5/23 Mark Phippard : > >> On Mon, May 23, 2011 at 2:24 PM, Daniel Shahaf > wrote: > >>> Bob Archer wrote on Mon, May 23, 2011 at 13:54:59 -0400: > I can't imagine that this would need to move the file over the > network

Re: large number of large binary files in subversion

2011-05-23 Thread Stefan Sperling
On Mon, May 23, 2011 at 11:07:50PM +0400, Konstantin Kolinko wrote: > In svn 1.7 there is pristine storage area in the working copy, where > all present files are stored by their checksums. If I understand this > pristine storage correctly, if you move a file remotely on the server > (svn mv URL UR

Re: large number of large binary files in subversion

2011-05-23 Thread Mark Phippard
On Mon, May 23, 2011 at 3:07 PM, Konstantin Kolinko wrote: > 2011/5/23 Mark Phippard : >> On Mon, May 23, 2011 at 2:24 PM, Daniel Shahaf >> wrote: >>> Bob Archer wrote on Mon, May 23, 2011 at 13:54:59 -0400: I can't imagine that this would need to move the file over the network since i

Re: large number of large binary files in subversion

2011-05-23 Thread Konstantin Kolinko
2011/5/23 Mark Phippard : > On Mon, May 23, 2011 at 2:24 PM, Daniel Shahaf > wrote: >> Bob Archer wrote on Mon, May 23, 2011 at 13:54:59 -0400: >>> I can't imagine that this would need to move the file over the network >>> since it all happens server side. Although, I guess when you do an >>> upd

Re: large number of large binary files in subversion

2011-05-23 Thread Mark Phippard
On Mon, May 23, 2011 at 2:24 PM, Daniel Shahaf wrote: > Bob Archer wrote on Mon, May 23, 2011 at 13:54:59 -0400: >> I can't imagine that this would need to move the file over the network >> since it all happens server side. Although, I guess when you do an >> update it might bring the file down ag

Re: large number of large binary files in subversion

2011-05-23 Thread Daniel Shahaf
Bob Archer wrote on Mon, May 23, 2011 at 13:54:59 -0400: > I can't imagine that this would need to move the file over the network > since it all happens server side. Although, I guess when you do an > update it might bring the file down again rather than doing a local > move/rename. update used t

RE: large number of large binary files in subversion

2011-05-23 Thread Bob Archer
> I keep my personal photos and videos in Subversion. > Total repository size is currently 475 GB. There are > some large (1-4 GB) video files. Everything seems to > work fine. > > Some details: I'm still running Subversion 1.4.6 on the > server (FSFS, CentOS 5, x86_64). Running 1.6.16 > cmdlin

Re: large number of large binary files in subversion

2011-05-23 Thread Ed Price
I keep my personal photos and videos in Subversion. Total repository size is currently 475 GB. There are some large (1-4 GB) video files. Everything seems to work fine. Some details: I'm still running Subversion 1.4.6 on the server (FSFS, CentOS 5, x86_64). Running 1.6.16 cmdline client on Cent

Re: large number of large binary files in subversion

2011-05-23 Thread Julian Foad
On Fri, 2011-05-20, Winston Smith wrote: > Sorry if this is the wrong list, but I'm curious about one thing: > Are the SVN developers aware of any quirks in SVN in regards > to storing a large number (say, 1000) very large binary files > (say, 1GB each)? So, the entire repository would be 1TB of si

large number of large binary files in subversion

2011-05-19 Thread Winston Smith
Hi all, Sorry if this is the wrong list, but I'm curious about one thing: Are the SVN developers aware of any quirks in SVN in regards to storing a large number (say, 1000) very large binary files (say, 1GB each)? So, the entire repository would be 1TB of size, but my concern is not space, but ra