Re: rsync many processes and slow backup

2019-10-15 Thread Dave Gordon via rsync

>>> Hi all! :-)
>>>
>>> I have a small rsync script to sync my data to a usb-disk. It works
>>> fine, when I start it in console. I get 3 rsync processes (look in
>>> top) and the backup takes ~25 min. But, when I add the script in
>>> cron to start it at 1am at night it takes 7 - 9 hours and I see up to
>>> 180 processes. When I look in top I see a hight load of 60 - 80 and 40
>>> - 60 waits. Why? Can someone explain why it takes so long when it
>>> starts with cron?
>>>
>>> This is my rsync command:  rsync -azc --delete "$QUELLORDNER"
>>> "$ZIELORDNER" 
>>> This is the entry in cron (crontab -e): * 2 * * *
>>> /root/backupscript/backup.sh
>>> Data to sync: 18 Gb, 185.000 files.
>>>
>>> When I look in the log files I see errors like this: 
>>>
>>> rsync:
>>> rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c"
>>> -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) 
>>>
>>> rsync error: some files/attrs were not transferred (see previousrsync
>>> error:
>>>
>>> some files/attrs were not transferred (see previous errors) (code 23) at
>>>
>>> main.c(1196) [sender=3.1.2]
>>>
>>> directory (2)
>>>
>>> Can you help me to solve this problem?
>>>
>>> regards,
>>> Hannes Hutmacher
>>>
Hi,

I think your crontab entry is wrong. As shown,

* 2 * * * /root/backupscript/backup.sh

it will start the script *once per minute* between 02:00 and 02:59 --
hence the 180 processes that you see! You probably meant

12 2 * * * /root/backupscript/backup.sh

(start once per day at 02:12) or some such thing.

HTH,
Dave

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync many processes and slow backup

2019-10-09 Thread Henri Shustak via rsync
I think others have answered your questions. You may want to implement some 
sort of locking system in the script so that you can only run a single instance 
of your script at. If you are looking for a backup tool which leverages rsync 
you may want to take a look at LBackup : http://www.lbackup.org

There is even a pre-script you can enable (in the example resources) so that 
the script can run a maximum of once a day / once a month etc (even if you 
start it many times from cron.

Hope you have it all working now :)

Henri


>> Hi all! :-)
>> 
>> I have a small rsync script to sync my data to a usb-disk. It works
>> fine, when I start it in console. I get 3 rsync processes (look in
>> top) and the backup takes ~25 min. But, when I add the script in
>> cron to start it at 1am at night it takes 7 - 9 hours and I see up to
>> 180 processes. When I look in top I see a hight load of 60 - 80 and 40
>> - 60 waits. Why? Can someone explain why it takes so long when it
>> starts with cron?
>> 
>> This is my rsync command:  rsync -azc --delete "$QUELLORDNER"
>> "$ZIELORDNER" 
>> This is the entry in cron (crontab -e): * 2 * * *
>> /root/backupscript/backup.sh
>> Data to sync: 18 Gb, 185.000 files.
>> 
>> When I look in the log files I see errors like this: 
>> 
>> rsync:
>> rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c"
>> -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) 
>> 
>> rsync error: some files/attrs were not transferred (see previousrsync
>> error:
>> 
>> some files/attrs were not transferred (see previous errors) (code 23) at
>> 
>> main.c(1196) [sender=3.1.2]
>> 
>> directory (2)
>> 
>> 
>> 
>> Can you help me to solve this problem?
>> 
>> 
>> 
>> regards,
>> 
>> Hannes Hutmacher
>> 
>> 
> A couple of questions: (I am not an expert user.)
> 
> Does your manual job run as the same user (presumably root) as does your
> cron job?
> 
> Are you backing up any temporary files that you might be better off
> excluding?
> 
> Is anything else running at the same time as your cron job which may be
> creating and destroying files in the backup source or target? (E.g. Your
> firewall?)
> 
> I'm not very familiar with fuzzy searching, but using that in places
> like /var seems odd.
> 
> One of the constant refrains on this list is that using checksums is
> almost never a good idea. In combination with fuzzy, it seems even more
> tenuous.
> 
> Joe
> 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync many processes and slow backup

2019-07-12 Thread Kevin Korb via rsync
In addition to the cron problems that others have pointed out there are
rsync problems...

#1  rsync's -c is almost always a bad idea and is extremely slow
#2  if rsync isn't networking then -z is a waste of time

On 7/12/19 4:56 AM, Hannes Hutmacher via rsync wrote:
> Hi all! :-)
>  
> I have a small rsync script to sync my data to a usb-disk. It works
> fine, when I start it in console. I get 3 rsync processes (look in top)
> and the backup takes ~25 min. But, when I add the script in cron to
> start it at 1am at night it takes 7 - 9 hours and I see up to
> 180 processes. When I look in top I see a hight load of 60 - 80 and 40 -
> 60 waits. Why? Can someone explain why it takes so long when it starts
> with cron?
>  
> This is my rsync command:  rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" 
> This is the entry in cron (crontab -e): * 2 * * *
> /root/backupscript/backup.sh
> Data to sync: 18 Gb, 185.000 files.
>  
> When I look in the log files I see errors like this: 
>  
> rsync:
> rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c"
> -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) 
> 
> rsync error: some files/attrs were not transferred (see previousrsync error:
> 
> some files/attrs were not transferred (see previous errors) (code 23) at
> 
> main.c(1196) [sender=3.1.2]
> 
> directory (2)
> 
>  
> 
> Can you help me to solve this problem?
> 
>  
> 
> regards,
> 
> Hannes Hutmacher
> 
> 

-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb  Phone:(407) 252-6853
Systems Administrator   Internet:
FutureQuest, Inc.   ke...@futurequest.net  (work)
Orlando, Floridak...@sanitarium.net (personal)
Web page:   https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync many processes and slow backup

2019-07-12 Thread Joe via rsync
On 7/12/19 4:56 AM, Hannes Hutmacher via rsync wrote:
> Hi all! :-)
>  
> I have a small rsync script to sync my data to a usb-disk. It works
> fine, when I start it in console. I get 3 rsync processes (look in
> top) and the backup takes ~25 min. But, when I add the script in
> cron to start it at 1am at night it takes 7 - 9 hours and I see up to
> 180 processes. When I look in top I see a hight load of 60 - 80 and 40
> - 60 waits. Why? Can someone explain why it takes so long when it
> starts with cron?
>  
> This is my rsync command:  rsync -azc --delete "$QUELLORDNER"
> "$ZIELORDNER" 
> This is the entry in cron (crontab -e): * 2 * * *
> /root/backupscript/backup.sh
> Data to sync: 18 Gb, 185.000 files.
>  
> When I look in the log files I see errors like this: 
>  
> rsync:
> rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c"
> -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) 
>
> rsync error: some files/attrs were not transferred (see previousrsync
> error:
>
> some files/attrs were not transferred (see previous errors) (code 23) at
>
> main.c(1196) [sender=3.1.2]
>
> directory (2)
>
>  
>
> Can you help me to solve this problem?
>
>  
>
> regards,
>
> Hannes Hutmacher
>
>
A couple of questions: (I am not an expert user.)

Does your manual job run as the same user (presumably root) as does your
cron job?

Are you backing up any temporary files that you might be better off
excluding?

Is anything else running at the same time as your cron job which may be
creating and destroying files in the backup source or target? (E.g. Your
firewall?)

I'm not very familiar with fuzzy searching, but using that in places
like /var seems odd.

One of the constant refrains on this list is that using checksums is
almost never a good idea. In combination with fuzzy, it seems even more
tenuous.

Joe


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync many processes and slow backup

2019-07-12 Thread rsync--- via rsync

Hallo Hannes

"man 5 crontab" is your friend ;-)

Am 12.07.2019 um 10:56 schrieb Hannes Hutmacher via rsync:
But, when I add the script in cron to start it at 1am at night it 
takes 7 - 9 hours and I see up to 180 processes. When I look in top I 
see a hight load of 60 - 80 and 40 - 60 waits. Why? Can someone 
explain why it takes so long when it starts with cron?
This is my rsync command:  rsync -azc --delete "$QUELLORDNER" 
"$ZIELORDNER"
This is the entry in cron (crontab -e): * 2 * * * 
/root/backupscript/backup.sh

Data to sync: 18 Gb, 185.000 files.


wrong crontab entry: every minute you start a new rsync process ...

crontab syntax:|*1 2 3 4 5 command*

          field  allowed values
  -  --
  1   minute 0-59
  2   hour   0-23
  3   day of month   1-31
  4   month  1-12 (or names, see below)
  5   day of week    0-7 (0 or 7 is Sun, or use names)

  command = command to be run

Good luck
-- Beat
|
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


Re: rsync many processes and slow backup

2019-07-12 Thread Fabian Cenedese via rsync
At 10:56 12.07.2019, Hannes Hutmacher via rsync wrote:

>Hi all! :-)
> 
>I have a small rsync script to sync my data to a usb-disk. It works fine, when 
>I start it in console. I get 3 rsync processes (look in top) and the backup 
>takes ~25 min. But, when I add the script in cron to start it at 1am at night 
>it takes 7 - 9 hours and I see up to 180 processes. When I look in top I see a 
>hight load of 60 - 80 and 40 - 60 waits. Why? Can someone explain why it takes 
>so long when it starts with cron?
> 
>This is my rsync command:  rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" 
>This is the entry in cron (crontab -e): * 2 * * * /root/backupscript/backup.sh

I'd say this starts rsync on every minute between 2:00 and 3:00.
You should change that to e.g. 0 2 ...

>Data to sync: 18 Gb, 185.000 files.
> 
>When I look in the log files I see errors like this: 
> 
>rsync: rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" 
>-> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) 

That's probably because several rsync instances are stomping
on each other's feet.

bye  Fabi


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html


rsync many processes and slow backup

2019-07-12 Thread Hannes Hutmacher via rsync
Hi all! :-)

 

I have a small rsync script to sync my data to a usb-disk. It works fine, when I start it in console. I get 3 rsync processes (look in top) and the backup takes ~25 min. But, when I add the script in cron to start it at 1am at night it takes 7 - 9 hours and I see up to 180 processes. When I look in top I see a hight load of 60 - 80 and 40 - 60 waits. Why? Can someone explain why it takes so long when it starts with cron?

 

This is my rsync command:  rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" 

This is the entry in cron (crontab -e): * 2 * * * /root/backupscript/backup.sh

Data to sync: 18 Gb, 185.000 files.

 


When I look in the log files I see errors like this: 

 

rsync: rename "/media/usb/sicherung/var/lib/fail2ban/.fail2ban.sqlite3.JCzY1c" -> "var/lib/fail2ban/fail2ban.sqlite3": No such file or directory (2) 

rsync error: some files/attrs were not transferred (see previousrsync error:

some files/attrs were not transferred (see previous errors) (code 23) at

main.c(1196) [sender=3.1.2]

directory (2)

 

Can you help me to solve this problem?

 

regards,

Hannes Hutmacher


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html