Re: [BackupPC-users] Large directory

2018-07-10 Thread Markus Köberl
On Monday, 9 July 2018 11:09:32 CEST Les Mikesell wrote: > On Mon, Jul 9, 2018 at 9:42 AM, Bowie Bailey wrote: > > > > There was still plenty of free RAM and no swap usage. I know it was > > still doing something because the pool filesystem was slowly growing. I > > could try an strace, but

Re: [BackupPC-users] Large directory

2018-07-09 Thread Michael Stowe
On 2018-07-09 09:21, Bowie Bailey wrote: On 7/9/2018 12:09 PM, Les Mikesell wrote: On Mon, Jul 9, 2018 at 9:42 AM, Bowie Bailey wrote: There was still plenty of free RAM and no swap usage. I know it was still doing something because the pool filesystem was slowly growing. I could try an

Re: [BackupPC-users] Large directory

2018-07-09 Thread Bowie Bailey
On 7/9/2018 12:09 PM, Les Mikesell wrote: > On Mon, Jul 9, 2018 at 9:42 AM, Bowie Bailey wrote: >> There was still plenty of free RAM and no swap usage. I know it was >> still doing something because the pool filesystem was slowly growing. I >> could try an strace, but I'll have to research

Re: [BackupPC-users] Large directory

2018-07-09 Thread Les Mikesell
On Mon, Jul 9, 2018 at 9:42 AM, Bowie Bailey wrote: > > There was still plenty of free RAM and no swap usage. I know it was > still doing something because the pool filesystem was slowly growing. I > could try an strace, but I'll have to research that. I've never used > strace before. > Be

Re: [BackupPC-users] Large directory

2018-07-09 Thread Bowie Bailey
On 7/6/2018 6:18 PM, Les Mikesell wrote: > On Fri, Jul 6, 2018 at 1:38 PM, Bowie Bailey wrote: >> Right now, the only error I've seen is the error that stopped the backup: >> rsync error: error in rsync protocol data stream (code 12) at io.c(1556) >> [generator=3.0.9.12] >> >> The main annoyance

Re: [BackupPC-users] Large directory

2018-07-09 Thread Bowie Bailey
On 7/6/2018 4:14 PM, Carl W. Soderstrom wrote: > On 07/06 02:38 , Bowie Bailey wrote: >> The problem is that the original backup took 2 weeks to fail with no >> indication of problems that I could see... it was just very slow.  I >> posted a previous question about it on this list while it was

Re: [BackupPC-users] Large directory

2018-07-07 Thread G.W. Haywood via BackupPC-users
Hello again, On Sat, 7 Jul 2018, Bowie Bailey wrote: ... The directory is XFS. I have no experience of XFS, but I've read of strangenesses. It's my understanding is that yours is a fairly newly-copied filesystem, and the strangenesses I've read about have been after the filesystems have had

Re: [BackupPC-users] Large directory

2018-07-06 Thread Les Mikesell
On Fri, Jul 6, 2018 at 1:38 PM, Bowie Bailey wrote: > > > Right now, the only error I've seen is the error that stopped the backup: > rsync error: error in rsync protocol data stream (code 12) at io.c(1556) > [generator=3.0.9.12] > > The main annoyance is that I have no way to track progress.

Re: [BackupPC-users] Large directory

2018-07-06 Thread Carl W. Soderstrom
On 07/06 02:38 , Bowie Bailey wrote: > The problem is that the original backup took 2 weeks to fail with no > indication of problems that I could see... it was just very slow.  I > posted a previous question about it on this list while it was running.  > I could not find any bottlenecks or

Re: [BackupPC-users] Large directory

2018-07-06 Thread Bowie Bailey
On 7/6/2018 10:22 AM, G.W. Haywood via BackupPC-users wrote: > Hi there, > > On Fri, 6 Jul 2018, Bowie Bailey wrote: > >> I am trying to backup a large directory tree with BackupPC v4.? This >> directory is 660GB and contains over 25 million files with about 3 >> million hard links.? The initial

Re: [BackupPC-users] Large directory

2018-07-06 Thread Guillermo Rozas
I think the important point here is the number of hard-links, rsync can have problems in those situations because it has to search them all and keep track of them (https://lists.samba.org/archive/rsync/2014-June/029537.html) > > I am trying to backup a large directory tree with BackupPC v4.? This

Re: [BackupPC-users] Large directory

2018-07-06 Thread G.W. Haywood via BackupPC-users
Hi there, On Fri, 6 Jul 2018, Bowie Bailey wrote: I am trying to backup a large directory tree with BackupPC v4.? This directory is 660GB and contains over 25 million files with about 3 million hard links.? The initial backup ran for 2 weeks before dying with an rsync error.? It is showing as

[BackupPC-users] Large directory

2018-07-05 Thread Bowie Bailey
I am trying to backup a large directory tree with BackupPC v4.  This directory is 660GB and contains over 25 million files with about 3 million hard links.  The initial backup ran for 2 weeks before dying with an rsync error.  It is showing as a partial backup, but it doesn't show a file count.

[BackupPC-users] Large directory moved on backuped server, how can avoid the complete re-synchronization

2015-11-19 Thread Yoann David
Hello, mostly everything is in the title. On a target server, we move a quite large directory (90 Go), as the target change in backuppc, it try to re-sync everything (the whole 90Go) not only the difference. Our bandwith between the backuped/target server and backuppc server is low (80ko/s),

Re: [BackupPC-users] Large directory moved on backuped server, how can avoid the complete re-synchronization

2015-11-19 Thread Johan Ehnberg
Hello Yoann, Rsync checksum caching will help in transferring only the changes. Are you using it? Here are the details: http://backuppc.sourceforge.net/faq/BackupPC.html#Rsync-checksum-caching Best regards, Johan Ehnberg On 2015-11-19 13:09, Yoann David wrote: > Hello, > > mostly everything

Re: [BackupPC-users] Large directory moved on backuped server, how can avoid the complete re-synchronization

2015-11-19 Thread Yoann David
Hello Johan, thanks for your answer, unfortunately checksum caching is not configured on my backuppc. With this system rsync/backuppc can detect file moving ? ie : in my case, backuppc wil detect that /var/opt/gitolite/repositories/aa.git folder is now /home/git/repositories/aa.git (the

Re: [BackupPC-users] Large directory moved on backuped server, how can avoid the complete re-synchronization

2015-11-19 Thread Johan Ehnberg
Hello Yoann, Now I understand better what you are attempting. No, unfortunately BackupPC will not detect moves. This is a feature in upcoming version 4. Checksum caching will offload the server, and rsync allows transferring just changes in current versions of BackupPC, but move detetion (or

Re: [BackupPC-users] Large directory moved on backuped server, how can avoid the complete re-synchronization

2015-11-19 Thread Johan Ehnberg
If it is a one-time move, you may be able to do some tricks with the rsync sharename. Or, use a restore tar file to seed the new location. See my post from earlier today for details, I'd be happy to hear your results. You can use archivemount to change the paths of the tar file you create.