rsync performance weirdness
hi, i observe some weirdness in rsync file transfer i cannot explain. i'm transferring data from 2 freenas storages to a linux vm with zfsonlinux. "fnask" is older freenas with rsync 3.1.1 running in daemon mode, freenas-bnkw is more recent with rsync 3.1.2, also daemon-mode. on the linux vm where the data is being pulled from the 2 freenas boxes, i look onto io bandwidth being used with "iotop -a". transfer from freenas-bnkw storage is much much slower then from fnask storage. why can i see two rsync processes transferring data from "fnask" but i see only one rsync process with "freenas-bnkw" where the parent process is transferring nothing ? is there a difference in how this is being handled between 3.1.1 and 3.1.2 on the remote side ? is this the reason for the slowness ? regards roland Total DISK READ : 278.72 M/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 97.09 M/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ> DISK WRITE SWAPIN IO COMMAND 32497 be/4 root 261.41 G 0.00 B 0.00 % 12.87 % rsync -av --inplace fnask::backupall_S2V2_xen-sr-rpkn01-nfs/ . 32498 be/4 root 200.56 G 12.88 G 0.00 % 21.41 % rsync -av --inplace fnask::backupall_S2V2_xen-sr-rpkn01-nfs/ . 24005 be/4 root 81.52 G 12.34 G 0.00 % 46.80 % rsync -av --inplace freenas-bnkw::backupall_zfspool_sas_seagate/sr_rpkn02/ . 3599 be/0 root 10.93 G 0.00 B 0.00 % 0.00 % [z_rd_int] 3604 be/0 root 10.84 G 0.00 B 0.00 % 0.00 % [z_rd_int] 3605 be/0 root 10.81 G 0.00 B 0.00 % 0.00 % [z_rd_int] ├─screen,14590 │ └─bash,14591 │ └─rsync,32497 -av --inplace fnask::backupall_S2V2_xen-sr-rpkn01-nfs/ . │ └─rsync,32498 -av --inplace fnask::backupall_S2V2_xen-sr-rpkn01-nfs/ . ├─screen,32261 │ └─bash,32262 │ └─rsync,24004 -av --inplace freenas-bnkw::backupall_zfspool_sas_seagate/sr_rpkn02/ . │ └─rsync,24005 -av --inplace freenas-bnkw::backupall_zfspool_sas_seagate/sr_rpkn02/ . -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many processes and slow backup
In addition to the cron problems that others have pointed out there are rsync problems... #1 rsync's -c is almost always a bad idea and is extremely slow #2 if rsync isn't networking then -z is a waste of time On 7/12/19 4:56 AM, Hannes Hutmacher via rsync wrote: > Hi all! :-) > > I have a small rsync script to sync my data to a usb-disk. It works > fine, when I start it in console. I get 3 rsync processes (look in top) > and the backup takes ~25 min. But, when I add the script in cron to > start it at 1am at night it takes 7 - 9 hours and I see up to > 180 processes. When I look in top I see a hight load of 60 - 80 and 40 - > 60 waits. Why? Can someone explain why it takes so long when it starts > with cron? > > This is my rsync command: rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" > This is the entry in cron (crontab -e): * 2 * * * > /root/backupscript/backup.sh > Data to sync: 18 Gb, 185.000 files. > > When I look in the log files I see errors like this: > > rsync: > rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" > -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) > > rsync error: some files/attrs were not transferred (see previousrsync error: > > some files/attrs were not transferred (see previous errors) (code 23) at > > main.c(1196) [sender=3.1.2] > > directory (2) > > > > Can you help me to solve this problem? > > > > regards, > > Hannes Hutmacher > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many processes and slow backup
On 7/12/19 4:56 AM, Hannes Hutmacher via rsync wrote: > Hi all! :-) > > I have a small rsync script to sync my data to a usb-disk. It works > fine, when I start it in console. I get 3 rsync processes (look in > top) and the backup takes ~25 min. But, when I add the script in > cron to start it at 1am at night it takes 7 - 9 hours and I see up to > 180 processes. When I look in top I see a hight load of 60 - 80 and 40 > - 60 waits. Why? Can someone explain why it takes so long when it > starts with cron? > > This is my rsync command: rsync -azc --delete "$QUELLORDNER" > "$ZIELORDNER" > This is the entry in cron (crontab -e): * 2 * * * > /root/backupscript/backup.sh > Data to sync: 18 Gb, 185.000 files. > > When I look in the log files I see errors like this: > > rsync: > rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" > -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) > > rsync error: some files/attrs were not transferred (see previousrsync > error: > > some files/attrs were not transferred (see previous errors) (code 23) at > > main.c(1196) [sender=3.1.2] > > directory (2) > > > > Can you help me to solve this problem? > > > > regards, > > Hannes Hutmacher > > A couple of questions: (I am not an expert user.) Does your manual job run as the same user (presumably root) as does your cron job? Are you backing up any temporary files that you might be better off excluding? Is anything else running at the same time as your cron job which may be creating and destroying files in the backup source or target? (E.g. Your firewall?) I'm not very familiar with fuzzy searching, but using that in places like /var seems odd. One of the constant refrains on this list is that using checksums is almost never a good idea. In combination with fuzzy, it seems even more tenuous. Joe -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many processes and slow backup
Hallo Hannes "man 5 crontab" is your friend ;-) Am 12.07.2019 um 10:56 schrieb Hannes Hutmacher via rsync: But, when I add the script in cron to start it at 1am at night it takes 7 - 9 hours and I see up to 180 processes. When I look in top I see a hight load of 60 - 80 and 40 - 60 waits. Why? Can someone explain why it takes so long when it starts with cron? This is my rsync command: rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" This is the entry in cron (crontab -e): * 2 * * * /root/backupscript/backup.sh Data to sync: 18 Gb, 185.000 files. wrong crontab entry: every minute you start a new rsync process ... crontab syntax:|*1 2 3 4 5 command* field allowed values - -- 1 minute 0-59 2 hour 0-23 3 day of month 1-31 4 month 1-12 (or names, see below) 5 day of week 0-7 (0 or 7 is Sun, or use names) command = command to be run Good luck -- Beat | -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many processes and slow backup
At 10:56 12.07.2019, Hannes Hutmacher via rsync wrote: >Hi all! :-) > >I have a small rsync script to sync my data to a usb-disk. It works fine, when >I start it in console. I get 3 rsync processes (look in top) and the backup >takes ~25 min. But, when I add the script in cron to start it at 1am at night >it takes 7 - 9 hours and I see up to 180 processes. When I look in top I see a >hight load of 60 - 80 and 40 - 60 waits. Why? Can someone explain why it takes >so long when it starts with cron? > >This is my rsync command: rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" >This is the entry in cron (crontab -e): * 2 * * * /root/backupscript/backup.sh I'd say this starts rsync on every minute between 2:00 and 3:00. You should change that to e.g. 0 2 ... >Data to sync: 18 Gb, 185.000 files. > >When I look in the log files I see errors like this: > >rsync: rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" >-> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) That's probably because several rsync instances are stomping on each other's feet. bye Fabi -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync many processes and slow backup
Hi all! :-) I have a small rsync script to sync my data to a usb-disk. It works fine, when I start it in console. I get 3 rsync processes (look in top) and the backup takes ~25 min. But, when I add the script in cron to start it at 1am at night it takes 7 - 9 hours and I see up to 180 processes. When I look in top I see a hight load of 60 - 80 and 40 - 60 waits. Why? Can someone explain why it takes so long when it starts with cron? This is my rsync command: rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" This is the entry in cron (crontab -e): * 2 * * * /root/backupscript/backup.sh Data to sync: 18 Gb, 185.000 files. When I look in the log files I see errors like this: rsync: rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) rsync error: some files/attrs were not transferred (see previousrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2] directory (2) Can you help me to solve this problem? regards, Hannes Hutmacher -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
