How do you speed up rsync?

Hi All, I have a bash script for synchronizing a flashing drive (target) with my hard drive (source) I take to customer sites (with a read only switch so I don't spread viruses). I currently "rsync" 11 different directories. Each sync line looks like this: rsync -rv --delete $MyCDsSource/Linux

Re: How do you speed up rsync?

Tar ball changes and "just copy"? On Jul 11, 2014 3:58 PM, "ToddAndMargo" wrote: > Hi All, > > I have a bash script for synchronizing a flashing drive (target) > with my hard drive (source) I take to customer sites (with a read > only switch so I don't spread viruses). > > I currently "rsync" 11

Re: How do you speed up rsync?

On 11 July 2014 13:58, ToddAndMargo wrote: > Hi All, > > I have a bash script for synchronizing a flashing drive (target) > with my hard drive (source) I take to customer sites (with a read > only switch so I don't spread viruses). > > I currently "rsync" 11 different directories. Each sync line

Re: How do you speed up rsync?

On Fri, Jul 11, 2014 at 3:58 PM, ToddAndMargo mailto:toddandma...@zoho.com>> wrote: Hi All, I have a bash script for synchronizing a flashing drive (target) with my hard drive (source) I take to customer sites (with a read only switch so I don't spread viruses). I currently

Re: How do you speed up rsync?

The likely culprit is encryption. If this is all on a local network segment and you can forgo the security aspect: -e "ssh -T -c arcfour -o Compression=no -x" ~Steven On Fri, Jul 11, 2014 at 3:58 PM, ToddAndMargo wrote: > Hi All, > > I have a bash script for synchronizing a flashing drive (t

Re: How do you speed up rsync?

rsync be default is using an encryption method most likely more taxing than arcfour. I'd imagine your local disk is unencrypted while you are reading/writing to it On Fri, Jul 11, 2014 at 4:09 PM, ToddAndMargo wrote: > On Fri, Jul 11, 2014 at 3:58 PM, ToddAndMargo >>

Re: How do you speed up rsync?

On Jul 11, 2014 3:58 PM, "ToddAndMargo" mailto:toddandma...@zoho.com>> wrote: Hi All, I have a bash script for synchronizing a flashing drive (target) with my hard drive (source) I take to customer sites (with a read only switch so I don't spread viruses). I currently "rsync

Re: How do you speed up rsync?

How big are the 2 directories? I carry a USB stick with about 15GB of data that I synchronize between 2 computers. Usually, it takes about 2 minutes to sync them. Sometimes I sync it to a netbook and it's noticeably slower. Clearly, hardware. But, usually there are not a lot of changes. If

Re: How do you speed up rsync?

On 07/11/2014 01:04 PM, Stephen John Smoogen wrote: On 11 July 2014 13:58, ToddAndMargo mailto:toddandma...@zoho.com>> wrote: Hi All, I have a bash script for synchronizing a flashing drive (target) with my hard drive (source) I take to customer sites (with a read only switc

Re: How do you speed up rsync?

Well, if you have a lot of changes in these files maybe its kust easier to either: 1 compare on a faster hd and update only those that needed back to usb 2. Utilize svn/git to keep track of changes and again copy only changes to usb On Jul 11, 2014 4:14 PM, "ToddAndMargo" wrote: > On Jul 11, 201

Re: How do you speed up rsync?

My USB stick is formatted in EXT4. I wonder if that makes a difference. I don't recall if rsync checks file timestamps or just the checksum.

Re: How do you speed up rsync?

I'd imagine your local disk is unencrypted while you are reading/writing to it On Fri, Jul 11, 2014 at 4:09 PM, ToddAndMargo mailto:toddandma...@zoho.com>> wrote: On Fri, Jul 11, 2014 at 3:58 PM, ToddAndMargo mailto:toddandma...@zoho.com>

Re: How do you speed up rsync?

On 07/11/2014 01:25 PM, Ken Teh wrote: My USB stick is formatted in EXT4. I wonder if that makes a difference. I don't recall if rsync checks file timestamps or just the checksum. Most of the computers I stick the thing into are Windows, so I am stuck with FAT32. For my own stuff, I wipe an

Re: How do you speed up rsync?

On 11 July 2014 14:26, ToddAndMargo wrote: > >> I'd imagine your local disk is unencrypted while you are reading/writing >>> to it >>> >> > On 07/11/2014 01:11 PM, Steven Miano wrote: > > rsync be default is using an encryption method most likely more taxing > > than arcfour. > > > > Just du

Re: How do you speed up rsync?

you can turn it down: -e "ssh -T -c arcfour -o Compression=no -x" Use -e to specify the remote shell to be ssh, and use -c arcfour to ensure the compression type is arcfour if you wan to strip it down further. It will still be encrypted/secure - but just weakly (think WEP). On Fri, Jul 11, 2

Re: How do you speed up rsync?

On Fri, Jul 11, 2014 at 1:20 PM, ToddAndMargo wrote: > On 07/11/2014 01:04 PM, Stephen John Smoogen wrote: >> >> So rsync is going to have to read every file on target and host and see >> if various things have changed. Not true, at least by default... rsync assumes that if the files' names, time

Re: How do you speed up rsync?

On Fri, Jul 11, 2014 at 1:40 PM, Patrick J. LoPresti wrote: > > Try giving the "--size-only" option to rsync. Better yet, try "--modify-window=1". From the rsync man page: --modify-window When comparing two timestamps, rsync treats the timestamps as being equal if

Re: How do you speed up rsync?

On 07/11/2014 01:40 PM, Patrick J. LoPresti wrote: On Fri, Jul 11, 2014 at 1:20 PM, ToddAndMargo wrote: On 07/11/2014 01:04 PM, Stephen John Smoogen wrote: So rsync is going to have to read every file on target and host and see if various things have changed. Not true, at least by default..

Re: How do you speed up rsync?

On 07/11/2014 01:49 PM, Patrick J. LoPresti wrote: On Fri, Jul 11, 2014 at 1:40 PM, Patrick J. LoPresti wrote: Try giving the "--size-only" option to rsync. Better yet, try "--modify-window=1". From the rsync man page: --modify-window When comparing two timestamps, rsync t

Re: How do you speed up rsync?

On Fri, Jul 11, 2014 at 04:02:09PM -0400, Andrew Z wrote: > > > > ... synchronizing a flashing drive (target) with my hard drive (source) ... > > > > Problem: it is slow -- takes three hours. To help the > > speed issue, I upgraded from USB 2 to USB 3. Backup went > > from 3 hr-15 min to 3 hr-5 m

Re: How do you speed up rsync?

On 07/11/2014 06:12 PM, Konstantin Olchanski wrote: On Fri, Jul 11, 2014 at 04:02:09PM -0400, Andrew Z wrote: ... synchronizing a flashing drive (target) with my hard drive (source) ... Problem: it is slow -- takes three hours. To help the speed issue, I upgraded from USB 2 to USB 3. Backup

Re: How do you speed up rsync?

On Fri, Jul 11, 2014 at 06:52:07PM -0700, ToddAndMargo wrote: > On 07/11/2014 06:12 PM, Konstantin Olchanski wrote: > >On Fri, Jul 11, 2014 at 04:02:09PM -0400, Andrew Z wrote: > >>> > >>>... synchronizing a flashing drive (target) with my hard drive (source) ... > >>> > >>>Problem: it is slow -- t

Re: How do you speed up rsync?

On 07/11/2014 01:49 PM, Patrick J. LoPresti wrote: On Fri, Jul 11, 2014 at 1:40 PM, Patrick J. LoPresti wrote: Try giving the "--size-only" option to rsync. Better yet, try "--modify-window=1". From the rsync man page: --modify-window When comparing two timestamps, rsync t

Re: How do you speed up rsync?

On 07/11/2014 07:25 PM, Konstantin Olchanski wrote: Right, this is almost SSD quality flash, but for some reason, performance for writing small files is much worse compared to SSDs. One would think that one could make a USB flash disk with same performance as SATA flash (SSD), but perhaps nobod

Re: How do you speed up rsync?

On 07/11/2014 12:58 PM, ToddAndMargo wrote: Hi All, I have a bash script for synchronizing a flashing drive (target) with my hard drive (source) I take to customer sites (with a read only switch so I don't spread viruses). I currently "rsync" 11 different directories. Each sync line looks like

Re: How do you speed up rsync?

SCIENTIFIC-LINUX-USERS automatic digest system: > How do you speed up rsync? > Von: > ToddAndMargo > Datum: > 11.07.14 21:58 > > MIME-Version: > 1.0 > Content-Type: > text/plain; charset="ISO-8859-1"; format=flowed > Content-Transfer-Encoding: > 7bit >

Re: How do you speed up rsync?

One more point: googling "rsync time zone fat32" brings up an excellent page - http://sabg.tk/wiki/config:vfat which discusses timezone problems, mount options and the like, and also suggests using ”–modify-window=1” . HTH, Kay smime.p7s Description: S/MIME Cryptographic Signature

Re: How do you speed up rsync?

On 11 July 2014 20:25, Konstantin Olchanski wrote: > On Fri, Jul 11, 2014 at 06:52:07PM -0700, ToddAndMargo wrote: > > On 07/11/2014 06:12 PM, Konstantin Olchanski wrote: > > >On Fri, Jul 11, 2014 at 04:02:09PM -0400, Andrew Z wrote: > > >>> > > >>>... synchronizing a flashing drive (target) with

Re: How do you speed up rsync?

On Fri, Jul 11, 2014 at 10:24 PM, ToddAndMargo wrote: > > Hi Pat, > > --modify-window=1 > 3 hr - 9 sec > > --modify-window=10 > 3 hr - 8 sec > > Rat! I really though this sounded right Oh, well... > Any way to turn of the check sum testing? Well, there is the "--whole-file" option.

Re: How do you speed up rsync?

On 07/12/2014 06:20 PM, Patrick J. LoPresti wrote: On Fri, Jul 11, 2014 at 10:24 PM, ToddAndMargo wrote: Hi Pat, --modify-window=1 3 hr - 9 sec --modify-window=10 3 hr - 8 sec Rat! I really though this sounded right Oh, well... Any way to turn of the check sum testing?

Re: How do you speed up rsync?

Wow I wish I got into this thread earlier I could have explained a lot. I've worked with rsync on a low level for many years and have even debated about writing a C library and possibly a multicast transport layer for it so I know it quiet well. Ive seen a lot of misinformation and guessing in this

Re: How do you speed up rsync?

I guess I don't understand the part about how files can be different sizes on different filesystems. They can obviously use up more or less disk space on different filesystems. For instance, a FAT disk with 32KB clusters will use up a minimum of 32KB even for a 10 byte file. While NTFS will p

Re: How do you speed up rsync?

On Mon, Jul 14, 2014 at 04:33:03PM -0500, Kevin K wrote: > I guess I don't understand the part about how files can be different sizes on > different filesystems. > > They can obviously use up more or less disk space on different filesystems. > For instance, a FAT disk with 32KB clusters will us

Re: How do you speed up rsync?

On Jul 14, 2014, at 4:37 PM, Konstantin Olchanski wrote: > On Mon, Jul 14, 2014 at 04:33:03PM -0500, Kevin K wrote: >> I guess I don't understand the part about how files can be different sizes >> on different filesystems. >> >> They can obviously use up more or less disk space on different fil

Re: How do you speed up rsync?

On Mon, Jul 14, 2014 at 04:51:03PM -0500, Kevin K wrote: > On Jul 14, 2014, at 4:37 PM, Konstantin Olchanski wrote: > > > On Mon, Jul 14, 2014 at 04:33:03PM -0500, Kevin K wrote: > >> I guess I don't understand the part about how files can be different sizes > >> on different filesystems. > >>

Re: How do you speed up rsync?

On 07/11/2014 12:58 PM, ToddAndMargo wrote: Hi All, I have a bash script for synchronizing a flashing drive (target) with my hard drive (source) I take to customer sites (with a read only switch so I don't spread viruses). I currently "rsync" 11 different directories. Each sync line looks like

Re: How do you speed up rsync?

Am 15.07.14 07:08, schrieb SCIENTIFIC-LINUX-USERS automatic digest system: >> >> I wiped the stick (set all the charges to zero) and started over. >> I am using >> >> rsync -rv --delete --modify-window=1 --times --inplace \ >> $MyCDsSource/Linux $MyCDsTarget/.; sync; sync >> >> I will have resu

Re: How do you speed up rsync?

I have a question about your test did you unmount the stick between runs of rsync. if not you may have already had all of the information about the filesystem cached in memory instead of having to search the FAT table for information. this could have a huge effect on the speed of an update. also

Re: How do you speed up rsync?

On Tue, Jul 15, 2014 at 10:44 AM, Paul Robert Marino wrote: > I have a question about your test did you unmount the stick between > runs of rsync. if not you may have already had all of the information > about the filesystem cached in memory instead of having to search the > FAT table for informat

Re: How do you speed up rsync?

On 07/15/2014 10:44 AM, Paul Robert Marino wrote: I have a question about your test did you unmount the stick between runs of rsync. The script automatically does that. Same behavour this morning after a power off of the machine last night. It too about 3 minutes to catch all the updates I ha

Re: How do you speed up rsync?

On 07/15/2014 10:52 AM, Patrick J. LoPresti wrote: On Tue, Jul 15, 2014 at 10:44 AM, Paul Robert Marino wrote: I have a question about your test did you unmount the stick between runs of rsync. if not you may have already had all of the information about the filesystem cached in memory instead

Re: How do you speed up rsync?

the sync command controls the synchronization of cache nothing more. essentially it can force your write cache to push down to the block device it can also be used to clear your cache in ram. On Tue, Jul 15, 2014 at 2:02 PM, ToddAndMargo wrote: > On 07/15/2014 10:52 AM, Patrick J. LoPresti wrote

Re: How do you speed up rsync?

On 07/15/2014 11:02 AM, ToddAndMargo wrote: On 07/15/2014 10:52 AM, Patrick J. LoPresti wrote: On Tue, Jul 15, 2014 at 10:44 AM, Paul Robert Marino wrote: I have a question about your test did you unmount the stick between runs of rsync. if not you may have already had all of the information a

Re: How do you speed up rsync?

On Tue, Jul 15, 2014 at 11:11 AM, ToddAndMargo wrote: > >> Who would this differ from the "sync" command? > > > "who" should have been "how" As someone else said, "sync" just synchronizes the copy on disk with the copy in memory; i.e. it forces a flush of dirty pages to disk. The kernel will stil

Re: How do you speed up rsync?

On 07/15/2014 11:07 AM, Paul Robert Marino wrote: the sync command controls the synchronization of cache nothing more. essentially it can force your write cache to push down to the block device it can also be used to clear your cache in ram. I have noticed that the flash drive will flash for u

Re: How do you speed up rsync?

On 07/15/2014 11:18 AM, Patrick J. LoPresti wrote: On Tue, Jul 15, 2014 at 11:11 AM, ToddAndMargo wrote: Who would this differ from the "sync" command? "who" should have been "how" As someone else said, "sync" just synchronizes the copy on disk with the copy in memory; i.e. it forces a f

Re: How do you speed up rsync?

On 07/11/2014 12:58 PM, ToddAndMargo wrote: Hi All, I have a bash script for synchronizing a flashing drive (target) with my hard drive (source) I take to customer sites (with a read only switch so I don't spread viruses). I currently "rsync" 11 different directories. Each sync line looks like