Re: rsync many processes and slow backup
>>> Hi all! :-) >>> >>> I have a small rsync script to sync my data to a usb-disk. It works >>> fine, when I start it in console. I get 3 rsync processes (look in >>> top) and the backup takes ~25 min. But, when I add the script in >>> cron to start it at 1am at night it takes 7 - 9 hours and I see up to >>> 180 processes. When I look in top I see a hight load of 60 - 80 and 40 >>> - 60 waits. Why? Can someone explain why it takes so long when it >>> starts with cron? >>> >>> This is my rsync command: rsync -azc --delete "$QUELLORDNER" >>> "$ZIELORDNER" >>> This is the entry in cron (crontab -e): * 2 * * * >>> /root/backupscript/backup.sh >>> Data to sync: 18 Gb, 185.000 files. >>> >>> When I look in the log files I see errors like this: >>> >>> rsync: >>> rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" >>> -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) >>> >>> rsync error: some files/attrs were not transferred (see previousrsync >>> error: >>> >>> some files/attrs were not transferred (see previous errors) (code 23) at >>> >>> main.c(1196) [sender=3.1.2] >>> >>> directory (2) >>> >>> Can you help me to solve this problem? >>> >>> regards, >>> Hannes Hutmacher >>> Hi, I think your crontab entry is wrong. As shown, * 2 * * * /root/backupscript/backup.sh it will start the script *once per minute* between 02:00 and 02:59 -- hence the 180 processes that you see! You probably meant 12 2 * * * /root/backupscript/backup.sh (start once per day at 02:12) or some such thing. HTH, Dave -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many processes and slow backup
I think others have answered your questions. You may want to implement some sort of locking system in the script so that you can only run a single instance of your script at. If you are looking for a backup tool which leverages rsync you may want to take a look at LBackup : http://www.lbackup.org There is even a pre-script you can enable (in the example resources) so that the script can run a maximum of once a day / once a month etc (even if you start it many times from cron. Hope you have it all working now :) Henri >> Hi all! :-) >> >> I have a small rsync script to sync my data to a usb-disk. It works >> fine, when I start it in console. I get 3 rsync processes (look in >> top) and the backup takes ~25 min. But, when I add the script in >> cron to start it at 1am at night it takes 7 - 9 hours and I see up to >> 180 processes. When I look in top I see a hight load of 60 - 80 and 40 >> - 60 waits. Why? Can someone explain why it takes so long when it >> starts with cron? >> >> This is my rsync command: rsync -azc --delete "$QUELLORDNER" >> "$ZIELORDNER" >> This is the entry in cron (crontab -e): * 2 * * * >> /root/backupscript/backup.sh >> Data to sync: 18 Gb, 185.000 files. >> >> When I look in the log files I see errors like this: >> >> rsync: >> rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" >> -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) >> >> rsync error: some files/attrs were not transferred (see previousrsync >> error: >> >> some files/attrs were not transferred (see previous errors) (code 23) at >> >> main.c(1196) [sender=3.1.2] >> >> directory (2) >> >> >> >> Can you help me to solve this problem? >> >> >> >> regards, >> >> Hannes Hutmacher >> >> > A couple of questions: (I am not an expert user.) > > Does your manual job run as the same user (presumably root) as does your > cron job? > > Are you backing up any temporary files that you might be better off > excluding? > > Is anything else running at the same time as your cron job which may be > creating and destroying files in the backup source or target? (E.g. Your > firewall?) > > I'm not very familiar with fuzzy searching, but using that in places > like /var seems odd. > > One of the constant refrains on this list is that using checksums is > almost never a good idea. In combination with fuzzy, it seems even more > tenuous. > > Joe > > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many processes and slow backup
In addition to the cron problems that others have pointed out there are rsync problems... #1 rsync's -c is almost always a bad idea and is extremely slow #2 if rsync isn't networking then -z is a waste of time On 7/12/19 4:56 AM, Hannes Hutmacher via rsync wrote: > Hi all! :-) > > I have a small rsync script to sync my data to a usb-disk. It works > fine, when I start it in console. I get 3 rsync processes (look in top) > and the backup takes ~25 min. But, when I add the script in cron to > start it at 1am at night it takes 7 - 9 hours and I see up to > 180 processes. When I look in top I see a hight load of 60 - 80 and 40 - > 60 waits. Why? Can someone explain why it takes so long when it starts > with cron? > > This is my rsync command: rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" > This is the entry in cron (crontab -e): * 2 * * * > /root/backupscript/backup.sh > Data to sync: 18 Gb, 185.000 files. > > When I look in the log files I see errors like this: > > rsync: > rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" > -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) > > rsync error: some files/attrs were not transferred (see previousrsync error: > > some files/attrs were not transferred (see previous errors) (code 23) at > > main.c(1196) [sender=3.1.2] > > directory (2) > > > > Can you help me to solve this problem? > > > > regards, > > Hannes Hutmacher > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many processes and slow backup
On 7/12/19 4:56 AM, Hannes Hutmacher via rsync wrote: > Hi all! :-) > > I have a small rsync script to sync my data to a usb-disk. It works > fine, when I start it in console. I get 3 rsync processes (look in > top) and the backup takes ~25 min. But, when I add the script in > cron to start it at 1am at night it takes 7 - 9 hours and I see up to > 180 processes. When I look in top I see a hight load of 60 - 80 and 40 > - 60 waits. Why? Can someone explain why it takes so long when it > starts with cron? > > This is my rsync command: rsync -azc --delete "$QUELLORDNER" > "$ZIELORDNER" > This is the entry in cron (crontab -e): * 2 * * * > /root/backupscript/backup.sh > Data to sync: 18 Gb, 185.000 files. > > When I look in the log files I see errors like this: > > rsync: > rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" > -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) > > rsync error: some files/attrs were not transferred (see previousrsync > error: > > some files/attrs were not transferred (see previous errors) (code 23) at > > main.c(1196) [sender=3.1.2] > > directory (2) > > > > Can you help me to solve this problem? > > > > regards, > > Hannes Hutmacher > > A couple of questions: (I am not an expert user.) Does your manual job run as the same user (presumably root) as does your cron job? Are you backing up any temporary files that you might be better off excluding? Is anything else running at the same time as your cron job which may be creating and destroying files in the backup source or target? (E.g. Your firewall?) I'm not very familiar with fuzzy searching, but using that in places like /var seems odd. One of the constant refrains on this list is that using checksums is almost never a good idea. In combination with fuzzy, it seems even more tenuous. Joe -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many processes and slow backup
Hallo Hannes "man 5 crontab" is your friend ;-) Am 12.07.2019 um 10:56 schrieb Hannes Hutmacher via rsync: But, when I add the script in cron to start it at 1am at night it takes 7 - 9 hours and I see up to 180 processes. When I look in top I see a hight load of 60 - 80 and 40 - 60 waits. Why? Can someone explain why it takes so long when it starts with cron? This is my rsync command: rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" This is the entry in cron (crontab -e): * 2 * * * /root/backupscript/backup.sh Data to sync: 18 Gb, 185.000 files. wrong crontab entry: every minute you start a new rsync process ... crontab syntax:|*1 2 3 4 5 command* field allowed values - -- 1 minute 0-59 2 hour 0-23 3 day of month 1-31 4 month 1-12 (or names, see below) 5 day of week 0-7 (0 or 7 is Sun, or use names) command = command to be run Good luck -- Beat | -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many processes and slow backup
At 10:56 12.07.2019, Hannes Hutmacher via rsync wrote: >Hi all! :-) > >I have a small rsync script to sync my data to a usb-disk. It works fine, when >I start it in console. I get 3 rsync processes (look in top) and the backup >takes ~25 min. But, when I add the script in cron to start it at 1am at night >it takes 7 - 9 hours and I see up to 180 processes. When I look in top I see a >hight load of 60 - 80 and 40 - 60 waits. Why? Can someone explain why it takes >so long when it starts with cron? > >This is my rsync command: rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" >This is the entry in cron (crontab -e): * 2 * * * /root/backupscript/backup.sh I'd say this starts rsync on every minute between 2:00 and 3:00. You should change that to e.g. 0 2 ... >Data to sync: 18 Gb, 185.000 files. > >When I look in the log files I see errors like this: > >rsync: rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" >-> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) That's probably because several rsync instances are stomping on each other's feet. bye Fabi -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync many processes and slow backup
Hi all! :-) I have a small rsync script to sync my data to a usb-disk. It works fine, when I start it in console. I get 3 rsync processes (look in top) and the backup takes ~25 min. But, when I add the script in cron to start it at 1am at night it takes 7 - 9 hours and I see up to 180 processes. When I look in top I see a hight load of 60 - 80 and 40 - 60 waits. Why? Can someone explain why it takes so long when it starts with cron? This is my rsync command: rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" This is the entry in cron (crontab -e): * 2 * * * /root/backupscript/backup.sh Data to sync: 18 Gb, 185.000 files. When I look in the log files I see errors like this: rsync: rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) rsync error: some files/attrs were not transferred (see previousrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2] directory (2) Can you help me to solve this problem? regards, Hannes Hutmacher -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
