Re: [gentoo-user] Re: [OT] fast recursive local copy
On Freitag 14 August 2009, Joerg Schilling wrote: > Dirk Heinrichs wrote: > > Am Freitag 14 August 2009 10:50:45 schrieb Joerg Schilling: > > > Note that on Linux you may need to add "-no-fsync" because file I/O is > > > slow on Linux. On Solaris, not using -no-fsync slows things down by > > > aprox. 10% but allows star to grant that everything was really copied > > > to stable storage. On Linux, ot using -no-fsync slows things down by > > > aprox. 400%, this is why I recommend to add "-no-fsync". > > > > This is also quite interesting. Do you have some (links to) recent > > benchmarks which would second that? Could this even be depending on the > > filesystem used on Linux? > > I did this test aprox. 3-4 years ago. You may try to do an own test and > report. > > I did just rerun a test on a recent ubuntu in a VirtualBox environment and > the speedup factor with -no-fsync was 8x. > > Jörg reiser4 on raid5, compression is on, since no barriers, sync mode. Three runs, first without, second with, third without -no-fsync. echo 1 > /proc/sys/vm/drop_caches in between each run. temp was removed and recreated between each run. source and target on different md devices. star: 0 blocks + 96006656 bytes (total of 96006656 bytes = 93756.50k). star -copy -p -xdot -acl -sparse -C /usr/local/portage . temp/ 0,85s user 16,79s system 6% cpu 4:48,77 total star: 0 blocks + 96006656 bytes (total of 96006656 bytes = 93756.50k). star -copy -p -xdot -acl -sparse -no-fsync -C /usr/local/portage . temp/ 0,43s user 3,14s system 24% cpu 14,389 total star: 0 blocks + 96006656 bytes (total of 96006656 bytes = 93756.50k). star -copy -p -xdot -acl -sparse -C /usr/local/portage . temp/ 0,88s user 15,93s system 6% cpu 4:13,76 total but reiser4 is infamous for not loving fsync ;)
Re: [gentoo-user] Re: [OT] fast recursive local copy
On Samstag 15 August 2009, Dirk Heinrichs wrote: > Am Freitag 14 August 2009 22:47:46 schrieb Joerg Schilling: > > Dirk Heinrichs wrote: > > > Am Freitag 14 August 2009 10:50:45 schrieb Joerg Schilling: > > > > Note that on Linux you may need to add "-no-fsync" because file I/O > > > > is slow on Linux. On Solaris, not using -no-fsync slows things down > > > > by aprox. 10% but allows star to grant that everything was really > > > > copied to stable storage. On Linux, ot using -no-fsync slows things > > > > down by aprox. 400%, this is why I recommend to add "-no-fsync". > > > > > > This is also quite interesting. Do you have some (links to) recent > > > benchmarks which would second that? Could this even be depending on the > > > filesystem used on Linux? > > > > I did this test aprox. 3-4 years ago. You may try to do an own test and > > report. > > > > I did just rerun a test on a recent ubuntu in a VirtualBox environment > > and the speedup factor with -no-fsync was 8x. > > OK, here's mine, then. Please note that the test happened on an encrypted > logical volume containing an XFS filesystem. Source and destination > directory are on the same LV. > > # time star -copy -p -xdot -acl -sparse -C /gentoo/overlays/portage . > /gentoo/build/portage > star: 0 blocks + 447532544 bytes (total of 447532544 bytes = 437043.50k). > star -copy -p -xdot -acl -sparse -C /gentoo/overlays/portage . 16,57s > user 60,22s system 11% cpu 11:23,63 total > > # time star -copy -p -xdot -acl -sparse -no-fsync -C > /gentoo/overlays/portage . /gentoo/build/portage > star: 0 blocks + 447532544 bytes (total of 447532544 bytes = 437043.50k). > star -copy -p -xdot -acl -sparse -no-fsync -C /gentoo/overlays/portage . > 12,33s user 35,39s system 10% cpu 7:44,13 total > > Unfortunately, I can't compare with OSol :( > > Bye... > > Dirk and you dropped caches in between?
Re: [gentoo-user] Re: [OT] fast recursive local copy
Am Freitag 14 August 2009 22:47:46 schrieb Joerg Schilling: > Dirk Heinrichs wrote: > > Am Freitag 14 August 2009 10:50:45 schrieb Joerg Schilling: > > > Note that on Linux you may need to add "-no-fsync" because file I/O is > > > slow on Linux. On Solaris, not using -no-fsync slows things down by > > > aprox. 10% but allows star to grant that everything was really copied > > > to stable storage. On Linux, ot using -no-fsync slows things down by > > > aprox. 400%, this is why I recommend to add "-no-fsync". > > > > This is also quite interesting. Do you have some (links to) recent > > benchmarks which would second that? Could this even be depending on the > > filesystem used on Linux? > > I did this test aprox. 3-4 years ago. You may try to do an own test and > report. > > I did just rerun a test on a recent ubuntu in a VirtualBox environment and > the speedup factor with -no-fsync was 8x. OK, here's mine, then. Please note that the test happened on an encrypted logical volume containing an XFS filesystem. Source and destination directory are on the same LV. # time star -copy -p -xdot -acl -sparse -C /gentoo/overlays/portage . /gentoo/build/portage star: 0 blocks + 447532544 bytes (total of 447532544 bytes = 437043.50k). star -copy -p -xdot -acl -sparse -C /gentoo/overlays/portage . 16,57s user 60,22s system 11% cpu 11:23,63 total # time star -copy -p -xdot -acl -sparse -no-fsync -C /gentoo/overlays/portage . /gentoo/build/portage star: 0 blocks + 447532544 bytes (total of 447532544 bytes = 437043.50k). star -copy -p -xdot -acl -sparse -no-fsync -C /gentoo/overlays/portage . 12,33s user 35,39s system 10% cpu 7:44,13 total Unfortunately, I can't compare with OSol :( Bye... Dirk signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Re: [OT] fast recursive local copy
Dirk Heinrichs wrote: > Am Freitag 14 August 2009 10:50:45 schrieb Joerg Schilling: > > > Note that on Linux you may need to add "-no-fsync" because file I/O is slow > > on Linux. On Solaris, not using -no-fsync slows things down by aprox. 10% > > but allows star to grant that everything was really copied to stable > > storage. On Linux, ot using -no-fsync slows things down by aprox. 400%, > > this is why I recommend to add "-no-fsync". > > This is also quite interesting. Do you have some (links to) recent benchmarks > which would second that? Could this even be depending on the filesystem used > on > Linux? I did this test aprox. 3-4 years ago. You may try to do an own test and report. I did just rerun a test on a recent ubuntu in a VirtualBox environment and the speedup factor with -no-fsync was 8x. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Re: [gentoo-user] Re: [OT] fast recursive local copy
Am Freitag 14 August 2009 10:50:45 schrieb Joerg Schilling: > Note that on Linux you may need to add "-no-fsync" because file I/O is slow > on Linux. On Solaris, not using -no-fsync slows things down by aprox. 10% > but allows star to grant that everything was really copied to stable > storage. On Linux, ot using -no-fsync slows things down by aprox. 400%, > this is why I recommend to add "-no-fsync". This is also quite interesting. Do you have some (links to) recent benchmarks which would second that? Could this even be depending on the filesystem used on Linux? Bye... Dirk signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Re: [OT] fast recursive local copy
Am Freitag 14 August 2009 18:11:34 schrieb Joerg Schilling: > star by default only overwrites a file if it is older than the file that is > going to be extracted. Thanks. Will definitely try it out on next occasion. Bye... Dirk signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Re: [OT] fast recursive local copy
Dirk Heinrichs wrote: > Am Freitag 14 August 2009 10:50:45 schrieb Joerg Schilling: > > > The fastest method for copying directory trees (typicalls 30% faster than > > any other known method) is to use star: > > > > star -copy -p -xdot -acl -sparse -C . > > That's a really nice one. However, does it also handle the "update " > case mentioned above? star by default only overwrites a file if it is older than the file that is going to be extracted. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Re: [gentoo-user] Re: [OT] fast recursive local copy
Am Freitag 14 August 2009 10:50:45 schrieb Joerg Schilling: > The fastest method for copying directory trees (typicalls 30% faster than > any other known method) is to use star: > > star -copy -p -xdot -acl -sparse -C . That's a really nice one. However, does it also handle the "update " case mentioned above? Bye... Dirk signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Re: [OT] fast recursive local copy
Nikos Chantziaras wrote: > On 08/14/2009 05:47 AM, meino.cra...@gmx.de wrote: > > Hi, > > > > I am looking for a faster way to do a > > > > cp -a r > > > > locally on one machine with one harddisk inside. > > > > Is there a neat trick to accomplish this faster than > > good old cp? > > Nope. Some people like to use pipes in hope to speed it up, something like: > >tar -c | tar -xC > > but this isn't really faster and fscks up sparse files. Pipes definitely do not speed up things, they slow things down as pipes introduce a significant system overhead. The fastest method for copying directory trees (typicalls 30% faster than any other known method) is to use star: star -copy -p -xdot -acl -sparse -C . Do not forget the '.' ! Note that on Linux you may _need_ to add "-no-fsync" because file I/O is slow on Linux. On Solaris, not using -no-fsync slows things down by aprox. 10% but allows star to grant that everything was really copied to stable storage. On Linux, ot using -no-fsync slows things down by aprox. 400%, this is why I recommend to add "-no-fsync". BTW: other programs behave like star "-no-fsync" by default. It may help to speed up things (in case you have enough RAM) to add: fs=128M Use no more than 1/2 of the physical RAM as FIFO size. Star implements the most effective way since more than 20 years. The idea is to use a FIFO made of shared memory and to let star fork by default into two processes that work independently. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Re: [gentoo-user] Re: [OT] fast recursive local copy
On Freitag 14 August 2009, Nikos Chantziaras wrote: > On 08/14/2009 05:47 AM, meino.cra...@gmx.de wrote: > > Hi, > > > > I am looking for a faster way to do a > > > > cp -a r > > > > locally on one machine with one harddisk inside. > > > > Is there a neat trick to accomplish this faster than > > good old cp? > > Nope. Some people like to use pipes in hope to speed it up, something > like: > >tar -c | tar -xC > > but this isn't really faster and fscks up sparse files. > > But if already contains some files from , then rsync > would be faster than cp. If not, stick with cp. use cp with -u and files already existing are skipped.
