Re: How do you speed up rsync?
On 07/11/2014 10:56 PM, Steven Haigh wrote: On 12/07/2014 3:49 PM, ToddAndMargo wrote: On 07/11/2014 10:44 PM, Steven Haigh wrote: On 12/07/2014 3:24 PM, ToddAndMargo wrote: On 07/11/2014 01:49 PM, Patrick J. LoPresti wrote: On Fri, Jul 11, 2014 at 1:40 PM, Patrick J. LoPresti wrote: Try giving the "--size-only" option to rsync. Better yet, try "--modify-window=1". From the rsync man page: --modify-window When comparing two timestamps, rsync treats the timestamps as being equal if they differ by no more than the modify-window value. This is normally 0 (for an exact match), but you may find it useful to set this to a larger value in some situations. In particular, when transferring to or from an MS Windows FAT filesystem (which represents times with a 2-second resolution), --modify-window=1 is useful (allowing times to differ by up to 1 second). - Pat Hi Pat, --modify-window=1 3 hr - 9 sec --modify-window=10 3 hr - 8 sec Rat! I really though this sounded right I did notice that the bugger the file (with no changes) the longer it took. So, I think they are still doing check sums. Any way to turn of the check sum testing? Now you're starting to get off task... How can you sync something if you don't know if it matches? Sure, you can only go off timestamps How would I do size and time stamp? Virtually everything I modify will either be a different size and/or get a new time stamp. I can not think of a reason why, in this instance, I'd need to do a check sum. Then you don't want to use rsync - as you're not syncing. rsync will also checksum data AFTER copying to verify the copy was successful. If you don't care about the rest, try the normal cp: cp -apu /path/to/source /path/to/destination Hi Steven, man cp: -u, --update copy only when the SOURCE file is newer than the destination file or when the destination file is missing Hmmm. I think I will give it a try. Not finding one for size is different. Not sure that is an issue though Thank you! -T -- ~~ Computers are like air conditioners. They malfunction when you open windows ~~
Re: How do you speed up rsync?
On 12/07/2014 3:49 PM, ToddAndMargo wrote: > On 07/11/2014 10:44 PM, Steven Haigh wrote: >> On 12/07/2014 3:24 PM, ToddAndMargo wrote: >>> On 07/11/2014 01:49 PM, Patrick J. LoPresti wrote: On Fri, Jul 11, 2014 at 1:40 PM, Patrick J. LoPresti wrote: > > Try giving the "--size-only" option to rsync. Better yet, try "--modify-window=1". From the rsync man page: --modify-window When comparing two timestamps, rsync treats the timestamps as being equal if they differ by no more than the modify-window value. This is normally 0 (for an exact match), but you may find it useful to set this to a larger value in some situations. In particular, when transferring to or from an MS Windows FAT filesystem (which represents times with a 2-second resolution), --modify-window=1 is useful (allowing times to differ by up to 1 second). - Pat >>> >>> Hi Pat, >>> >>> --modify-window=1 >>>3 hr - 9 sec >>> >>> --modify-window=10 >>>3 hr - 8 sec >>> >>> Rat! I really though this sounded right >>> >>> >>> I did notice that the bugger the file (with no changes) >>> the longer it took. So, I think they are still doing >>> check sums. >>> >>> Any way to turn of the check sum testing? >> >> Now you're starting to get off task... How can you sync something if you >> don't know if it matches? Sure, you can only go off timestamps > > How would I do size and time stamp? > > Virtually everything I modify will either be a different size > and/or get a new time stamp. I can not think of a reason > why, in this instance, I'd need to do a check sum. Then you don't want to use rsync - as you're not syncing. rsync will also checksum data AFTER copying to verify the copy was successful. If you don't care about the rest, try the normal cp: cp -apu /path/to/source /path/to/destination -- Steven Haigh Email: net...@crc.id.au Web: http://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299 signature.asc Description: OpenPGP digital signature
Re: How do you speed up rsync?
On 07/11/2014 10:44 PM, Steven Haigh wrote: Hell, get a small laptop HDD and put it in a USB caddy case. Get a small SSD (they're cheap!) and put that in a case... I'd have the same performance issues with this application. Switching from USN 2 to USB 3 did nothing. It is a thought for other apps though. Although I'd worry about cooling. When I do a custom computer with an SSD drive, I always put a fan blowing right on the thing. When it comes to electronics, heat is your enemy
Re: How do you speed up rsync?
On 07/11/2014 10:44 PM, Steven Haigh wrote: On 12/07/2014 3:24 PM, ToddAndMargo wrote: On 07/11/2014 01:49 PM, Patrick J. LoPresti wrote: On Fri, Jul 11, 2014 at 1:40 PM, Patrick J. LoPresti wrote: Try giving the "--size-only" option to rsync. Better yet, try "--modify-window=1". From the rsync man page: --modify-window When comparing two timestamps, rsync treats the timestamps as being equal if they differ by no more than the modify-window value. This is normally 0 (for an exact match), but you may find it useful to set this to a larger value in some situations. In particular, when transferring to or from an MS Windows FAT filesystem (which represents times with a 2-second resolution), --modify-window=1 is useful (allowing times to differ by up to 1 second). - Pat Hi Pat, --modify-window=1 3 hr - 9 sec --modify-window=10 3 hr - 8 sec Rat! I really though this sounded right I did notice that the bugger the file (with no changes) the longer it took. So, I think they are still doing check sums. Any way to turn of the check sum testing? Now you're starting to get off task... How can you sync something if you don't know if it matches? Sure, you can only go off timestamps How would I do size and time stamp? Virtually everything I modify will either be a different size and/or get a new time stamp. I can not think of a reason why, in this instance, I'd need to do a check sum. - but what then? It leaves you with a situation where you may get files that are different and you'll never know. You'd be much better off getting a better flash drive (ie not add hack upon hack that may not help) and fixing the root cause of the problem. Hell, get a small laptop HDD and put it in a USB caddy case. Get a small SSD (they're cheap!) and put that in a case...
Re: How do you speed up rsync?
On 12/07/2014 3:24 PM, ToddAndMargo wrote: > On 07/11/2014 01:49 PM, Patrick J. LoPresti wrote: >> On Fri, Jul 11, 2014 at 1:40 PM, Patrick J. LoPresti >> wrote: >>> >>> Try giving the "--size-only" option to rsync. >> >> Better yet, try "--modify-window=1". From the rsync man page: >> >> --modify-window >>When comparing two timestamps, rsync treats the >> timestamps as >>being equal if they differ by no more than the >> modify-window >>value. This is normally 0 (for an exact match), >> but you may >>find it useful to set this to a larger value in some >> situations. >>In particular, when transferring to or from an MS >> Windows FAT >>filesystem (which represents times with a 2-second >> resolution), >>--modify-window=1 is useful (allowing times to differ >> by up to 1 >>second). >> >> >> - Pat >> > > Hi Pat, > > --modify-window=1 > 3 hr - 9 sec > > --modify-window=10 > 3 hr - 8 sec > > Rat! I really though this sounded right > > > I did notice that the bugger the file (with no changes) > the longer it took. So, I think they are still doing > check sums. > > Any way to turn of the check sum testing? Now you're starting to get off task... How can you sync something if you don't know if it matches? Sure, you can only go off timestamps - but what then? It leaves you with a situation where you may get files that are different and you'll never know. You'd be much better off getting a better flash drive (ie not add hack upon hack that may not help) and fixing the root cause of the problem. Hell, get a small laptop HDD and put it in a USB caddy case. Get a small SSD (they're cheap!) and put that in a case... -- Steven Haigh Email: net...@crc.id.au Web: http://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897 Fax: (03) 8338 0299 signature.asc Description: OpenPGP digital signature
Re: Fwd: New binary package set for EL6 x86_64
* On 2014-07-11 at 16:39 BST, Yasha Karant wrote: > I have not found a pkgsrc RPM that would automatically install and > configure pkgsrc for an EL system. There is none that I am aware of. Setting up a build environment for pkgsrc is outside of the scope of a single RPM. > What is the answer to a fundamental question: > > how secure and authenticated is the pkgsrc repository (non-RPM, but > a repository nonetheless)? As far as the builds go they use the same mechanisms that you quoted - each downloaded distfile is verified for both SHA1 and RMD160 checksums to ensure their integrity. As far as the repository itself, it is secure. The part which is missing which I'd like to address for my other package sets too is that the packages themselves are not currently signed. pkgsrc has infrastructure support for this, but I am missing some bootstrap bits to ensure the packaging tools have the necessary features to support it. > In so as possible, I use SL and related repositories because these > in practice are reasonably secure and authenticated. I do what I > can to avoid using contaminated/compromised sources or executables, > and work as "root" as secure as is practicable. Sure, this is good practise. There is of course an element of trust here, but as a company which relies on community involvement a breach of that trust would be pretty catastrophic, so I will certainly do all I can to ensure it isn't broken. Regards, -- Jonathan Perkin - Joyent, Inc. - www.joyent.com
Re: Fwd: New binary package set for EL6 x86_64
For pkgsrc: http://wiki.netbsd.org/pkgsrc/how_to_use_pkgsrc_on_linux/ From: http://nathanahlstrom.wordpress.com/2013/08/20/netbsd-pkgsrc-on-rhel-6-4/ wget http://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc.tar.gz tar xzvf pkgsrc.tar.gz -C /opt cd /opt/pkgsrc/bootstrap ./bootstrap --prefix /opt/pkg Edit your ~/.bash_profile to look like this: PATH=$PATH:/opt/pkg/bin:/opt/pkg/sbin:$HOME/bin export PATH # run from the command line to update your PATH. . ~/.bash_profile End first quote. Next, I quote, after completing the above steps, an example to get apache22 : Now go into /opt/pkgsrc/www/apache22 and as root (or with sudo) run: bmake install It will download the source from apache.org mirrors, check the distribution integrity, compile it all for you, and install it with sensible defaults into /opt/pkg. When it completes (it could take 10 minutes or so depending on your hardware) all the latest Apache HTTPD server will be installed to /opt/pkg/sbin and /opt/pkg/bin. End second quote I have not found a pkgsrc RPM that would automatically install and configure pkgsrc for an EL system. What is the answer to a fundamental question: how secure and authenticated is the pkgsrc repository (non-RPM, but a repository nonetheless)? In so as possible, I use SL and related repositories because these in practice are reasonably secure and authenticated. I do what I can to avoid using contaminated/compromised sources or executables, and work as "root" as secure as is practicable. Yasha Karant On 07/11/2014 01:42 AM, Jonathan Perkin wrote: * On 2014-07-11 at 09:02 BST, Elias Persson wrote: On 2014-07-10 19:53, Yasha Karant wrote: I received the following email message that does not appear to be posted to the SL list. It's on the list: http://listserv.fnal.gov/scripts/wa.exe?A2=ind1407&L=scientific-linux-users&T=0&P=15184 The weird way it was sent (via another list?) and the fact that the SL lists lack list-id and such probably cause any filter you might have to miss it though. Sorry, my fault. I subscribed to a few different lists which I thought would be interested in this, and then sent one mail which bcc'd them - assuming that the list servers in question would handle the rest. Again, if you have any questions about this package set, I'd be delighted to answer them. I've had a few come in so far, so I'll take the chance to summarise them here: - You can browse the list of packages here: http://pkgsrc.joyent.com/packages/Linux/el6/2014Q2/x86_64/All/ - They aren't in RPM format, but pkgsrc (the system used to build them) does have pluggable backend support, and there was an unfinished GSOC project to implement RPM support a few years back. If someone is interested it would be fantastic to see this finished so we can provide them as RPMs via yum instead. - pkgsrc is branched every 3 months, and from that we generate the binary packages and provide a new package set, so every quarter there is a fresh update of new packages. Cheers,
Re: Fwd: New binary package set for EL6 x86_64
* On 2014-07-11 at 09:02 BST, Elias Persson wrote: > On 2014-07-10 19:53, Yasha Karant wrote: > >I received the following email message that does not appear to be posted > >to the SL list. > > It's on the list: > http://listserv.fnal.gov/scripts/wa.exe?A2=ind1407&L=scientific-linux-users&T=0&P=15184 > > The weird way it was sent (via another list?) and the fact that > the SL lists lack list-id and such probably cause any filter you > might have to miss it though. Sorry, my fault. I subscribed to a few different lists which I thought would be interested in this, and then sent one mail which bcc'd them - assuming that the list servers in question would handle the rest. Again, if you have any questions about this package set, I'd be delighted to answer them. I've had a few come in so far, so I'll take the chance to summarise them here: - You can browse the list of packages here: http://pkgsrc.joyent.com/packages/Linux/el6/2014Q2/x86_64/All/ - They aren't in RPM format, but pkgsrc (the system used to build them) does have pluggable backend support, and there was an unfinished GSOC project to implement RPM support a few years back. If someone is interested it would be fantastic to see this finished so we can provide them as RPMs via yum instead. - pkgsrc is branched every 3 months, and from that we generate the binary packages and provide a new package set, so every quarter there is a fresh update of new packages. Cheers, -- Jonathan Perkin - Joyent, Inc. - www.joyent.com
Re: Fwd: New binary package set for EL6 x86_64
On 2014-07-10 19:53, Yasha Karant wrote: I received the following email message that does not appear to be posted to the SL list. It's on the list: http://listserv.fnal.gov/scripts/wa.exe?A2=ind1407&L=scientific-linux-users&T=0&P=15184 The weird way it was sent (via another list?) and the fact that the SL lists lack list-id and such probably cause any filter you might have to miss it though.
