[rdiff-backup-users] Slowdown from a few mins to >8 hours backup time due to one large log file

2011-01-31 Thread Patrick Nagel 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

I have rdiff-backup running nightly on my home server. It keeps a versioned
backup of all relevant data on the second HDD.

I didn't monitor the backup too closely, because I didn't get any alarm that
the backup had failed. Today I looked at the logs, and noticed, that the
backup time went up from a few minutes a while back, to >8 hours.

2010-11-23T03:00: ElapsedTime 277.51 (4 minutes 37.51 seconds)
2010-11-24T03:00: ElapsedTime 438.69 (7 minutes 18.69 seconds)
2010-11-25T03:00: ElapsedTime 321.99 (5 minutes 21.99 seconds)
2010-11-26T03:00: ElapsedTime 2848.57 (47 minutes 28.57 seconds)
2010-11-27T03:00: ElapsedTime 3358.68 (55 minutes 58.68 seconds)
2010-11-28T03:00: ElapsedTime 2876.65 (47 minutes 56.65 seconds)
2010-11-29T03:00: ElapsedTime 2834.06 (47 minutes 14.06 seconds)
2010-11-30T03:00: ElapsedTime 3553.56 (59 minutes 13.56 seconds)
2010-12-01T03:00: ElapsedTime 3549.67 (59 minutes 9.67 seconds)
2010-12-02T03:00: ElapsedTime 3555.71 (59 minutes 15.71 seconds)
[...]
2010-12-10T03:00: ElapsedTime 8015.67 (2 hours 13 minutes 35.67 seconds)
2010-12-11T03:00: ElapsedTime 9066.51 (2 hours 31 minutes 6.51 seconds)
[...]
2011-01-30T03:00: ElapsedTime 28665.77 (7 hours 57 minutes 45.77 seconds)
2011-01-31T03:00: ElapsedTime 31093.79 (8 hours 38 minutes 13.79 seconds)
2011-02-01T03:00: ElapsedTime 31489.30 (8 hours 44 minutes 49.30 seconds)

By tracking what rdiff-backup was doing ('strace -p ' on
another terminal), I quickly found out that it spent huge amounts of time
diffing one big file. The file was a log file, written to by a script I
wrote just around the time the backup time started to increase. I guess I
overdid it with the logging a bit, since the log file had 3.9 GB by now.
After removing the oversized log file, the backup times dropped back to
normal again:

2011-02-01T12:22: ElapsedTime 685.87 (11 minutes 25.87 seconds)
2011-02-01T13:57: ElapsedTime 468.25 (7 minutes 48.25 seconds)

Maybe the diff-algorithm used can be improved to not take so long on large
text files, especially since this one only grew into one direction, and the
other content never changed.

In any case, I'm glad the backup doesn't take the whole night any more.

Patrick.

- -- 
Key ID: 0x86E346D4http://patrick-nagel.net/key.asc
Fingerprint: 7745 E1BE FA8B FBAD 76AB 2BFC C981 E686 86E3 46D4
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1HqUAACgkQyYHmhobjRtQi/gCglViupi1j0dvioxbZ3b6vXRSk
vwQAn2VAoZjB9KbKx6iVVnGM6XU2yxcT
=6F9A
-END PGP SIGNATURE-

___
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Re: [rdiff-backup-users] Slowdown from a few mins to >8 hours backup time due to one large log file

2011-02-01 Thread Dominic Raferd 
That seems extraordinarily slow for rdiff-backup to backup a text file 
to a local disk, even a file that is 3.9GB. Was the file changing 
(additional log entries, maybe?) while rdiff-backup was trying to back 
it up?


Dominic

On 01/02/2011 06:33, Patrick Nagel wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

I have rdiff-backup running nightly on my home server. It keeps a versioned
backup of all relevant data on the second HDD.

I didn't monitor the backup too closely, because I didn't get any alarm that
the backup had failed. Today I looked at the logs, and noticed, that the
backup time went up from a few minutes a while back, to>8 hours.

2010-11-23T03:00: ElapsedTime 277.51 (4 minutes 37.51 seconds)
2010-11-24T03:00: ElapsedTime 438.69 (7 minutes 18.69 seconds)
2010-11-25T03:00: ElapsedTime 321.99 (5 minutes 21.99 seconds)
2010-11-26T03:00: ElapsedTime 2848.57 (47 minutes 28.57 seconds)
2010-11-27T03:00: ElapsedTime 3358.68 (55 minutes 58.68 seconds)
2010-11-28T03:00: ElapsedTime 2876.65 (47 minutes 56.65 seconds)
2010-11-29T03:00: ElapsedTime 2834.06 (47 minutes 14.06 seconds)
2010-11-30T03:00: ElapsedTime 3553.56 (59 minutes 13.56 seconds)
2010-12-01T03:00: ElapsedTime 3549.67 (59 minutes 9.67 seconds)
2010-12-02T03:00: ElapsedTime 3555.71 (59 minutes 15.71 seconds)
[...]
2010-12-10T03:00: ElapsedTime 8015.67 (2 hours 13 minutes 35.67 seconds)
2010-12-11T03:00: ElapsedTime 9066.51 (2 hours 31 minutes 6.51 seconds)
[...]
2011-01-30T03:00: ElapsedTime 28665.77 (7 hours 57 minutes 45.77 seconds)
2011-01-31T03:00: ElapsedTime 31093.79 (8 hours 38 minutes 13.79 seconds)
2011-02-01T03:00: ElapsedTime 31489.30 (8 hours 44 minutes 49.30 seconds)

By tracking what rdiff-backup was doing ('strace -p' on
another terminal), I quickly found out that it spent huge amounts of time
diffing one big file. The file was a log file, written to by a script I
wrote just around the time the backup time started to increase. I guess I
overdid it with the logging a bit, since the log file had 3.9 GB by now.
After removing the oversized log file, the backup times dropped back to
normal again:

2011-02-01T12:22: ElapsedTime 685.87 (11 minutes 25.87 seconds)
2011-02-01T13:57: ElapsedTime 468.25 (7 minutes 48.25 seconds)

Maybe the diff-algorithm used can be improved to not take so long on large
text files, especially since this one only grew into one direction, and the
other content never changed.

In any case, I'm glad the backup doesn't take the whole night any more.

Patrick.


___
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Re: [rdiff-backup-users] Slowdown from a few mins to >8 hours backup time due to one large log file

2011-02-01 Thread Patrick Nagel 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Dominic,

On 2011-02-01 17:14, Dominic Raferd wrote:
> That seems extraordinarily slow for rdiff-backup to backup a text file to a
> local disk, even a file that is 3.9GB. Was the file changing (additional log
> entries, maybe?) while rdiff-backup was trying to back it up?

Yes, I think so, and I don't use LVM snapshots on this server. That's
probably the real trigger then.

Patrick.

- -- 
Key ID: 0x86E346D4http://patrick-nagel.net/key.asc
Fingerprint: 7745 E1BE FA8B FBAD 76AB 2BFC C981 E686 86E3 46D4
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1H1XsACgkQyYHmhobjRtQNcgCgvEfaW0KX3+p5AaGHA3tzNs6m
iHoAn0G/Wm48CZDPcvJTMB+/qJrv3Xou
=afkv
-END PGP SIGNATURE-

___
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Re: [rdiff-backup-users] Slowdown from a few mins to >8 hours backup time due to one large log file

2011-02-01 Thread Dominic Raferd 
Yes, backing up from LVM snapshots is the best way (or, for Windows, use 
VSS), but at least you found a workaround...


Dominic
http://www.timedicer.co.uk

On 01/02/2011 09:42, Patrick Nagel wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Dominic,

On 2011-02-01 17:14, Dominic Raferd wrote:

That seems extraordinarily slow for rdiff-backup to backup a text file to a
local disk, even a file that is 3.9GB. Was the file changing (additional log
entries, maybe?) while rdiff-backup was trying to back it up?

Yes, I think so, and I don't use LVM snapshots on this server. That's
probably the real trigger then.

Patrick.


___
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki