Re: rsync of big image files from virtual hosts takes lot of time
>This is the output with --stats for a 50GB image. >Appended also the output from time. >You see it's horrible slow. Please give also the source and destination parameters from your rsync command. These define whether it's a local sync or not (for rsync), the hardware setup doesn't really matter. bye Fabi -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync of big image files from virtual hosts takes lot of time
Your rsync must be a bit old. The current --stats output is about twice as long as that. Anyway, there isn't that much change in the file. If you were using rsync over ssh it would probably take around 30 minutes. I would guess cp would take longer but hard to say how much longer but it would be less than your rsync is taking. All that time is probably being wasted on rsync reading the target file over the network to find out what is different about it. On 10/21/2018 08:35 AM, Lentes, Bernd via rsync wrote: > > > - Am 20. Okt 2018 um 20:05 schrieb rsync ML rsync@lists.samba.org: > >> I don't see any --stats output. It will tell you exactly how much data >> was involved. >> >> A local copy means that rsync isn't networking. Rsync speeds things up >> by running on 2 separate computers and communicating using the rsync >> protocol. If you are running rsync on a single computer with the source >> being a mounted SAN volume and the target being a mounted cifs share >> then as far as rsync is concerned that is a local copy. >> >> So, either get the cifs server to speak rsync (either over ssh or with >> rsyncd) or just use cp. Only copying the changes isn't all that helpful >> when rsync has to read the whole target file over the network just to >> find out what is different about it. That is why --whole-file is >> implied when rsync isn't networking. >> >> On 10/20/2018 02:00 PM, Lentes, Bernd via rsync wrote: > > This is the output with --stats for a 50GB image. > Appended also the output from time. > You see it's horrible slow. > I will try cp. > > Literal data: 187,432,960 bytes > Matched data: 53,499,658,240 bytes > File list size: 0 > File list generation time: 0.001 seconds > File list transfer time: 0.000 seconds > Total bytes sent: 189,111,658 > Total bytes received: 3,277,035 > > sent 189,111,658 bytes received 3,277,035 bytes 6,493.15 bytes/sec > total size is 53,687,091,200 speedup is 279.06 > > real493m49.055s > user15m28.933s > sys 26m13.445s > > Bernd > > > > Helmholtz Zentrum Muenchen > Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) > Ingolstaedter Landstr. 1 > 85764 Neuherberg > www.helmholtz-muenchen.de > Aufsichtsratsvorsitzende: NN > Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter > Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich > Bassler, Dr. rer. nat. Alfons Enhsen > Registergericht: Amtsgericht Muenchen HRB 6466 > USt-IdNr: DE 129521671 > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync of big image files from virtual hosts takes lot of time
- Am 20. Okt 2018 um 20:05 schrieb rsync ML rsync@lists.samba.org: > I don't see any --stats output. It will tell you exactly how much data > was involved. > > A local copy means that rsync isn't networking. Rsync speeds things up > by running on 2 separate computers and communicating using the rsync > protocol. If you are running rsync on a single computer with the source > being a mounted SAN volume and the target being a mounted cifs share > then as far as rsync is concerned that is a local copy. > > So, either get the cifs server to speak rsync (either over ssh or with > rsyncd) or just use cp. Only copying the changes isn't all that helpful > when rsync has to read the whole target file over the network just to > find out what is different about it. That is why --whole-file is > implied when rsync isn't networking. > > On 10/20/2018 02:00 PM, Lentes, Bernd via rsync wrote: This is the output with --stats for a 50GB image. Appended also the output from time. You see it's horrible slow. I will try cp. Literal data: 187,432,960 bytes Matched data: 53,499,658,240 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 189,111,658 Total bytes received: 3,277,035 sent 189,111,658 bytes received 3,277,035 bytes 6,493.15 bytes/sec total size is 53,687,091,200 speedup is 279.06 real493m49.055s user15m28.933s sys 26m13.445s Bernd Helmholtz Zentrum Muenchen Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) Ingolstaedter Landstr. 1 85764 Neuherberg www.helmholtz-muenchen.de Aufsichtsratsvorsitzende: NN Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Dr. rer. nat. Alfons Enhsen Registergericht: Amtsgericht Muenchen HRB 6466 USt-IdNr: DE 129521671 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync of big image files from virtual hosts takes lot of time
I don't see any --stats output. It will tell you exactly how much data was involved. A local copy means that rsync isn't networking. Rsync speeds things up by running on 2 separate computers and communicating using the rsync protocol. If you are running rsync on a single computer with the source being a mounted SAN volume and the target being a mounted cifs share then as far as rsync is concerned that is a local copy. So, either get the cifs server to speak rsync (either over ssh or with rsyncd) or just use cp. Only copying the changes isn't all that helpful when rsync has to read the whole target file over the network just to find out what is different about it. That is why --whole-file is implied when rsync isn't networking. On 10/20/2018 02:00 PM, Lentes, Bernd via rsync wrote: > > > - Am 20. Okt 2018 um 18:56 schrieb rsync ML rsync@lists.samba.org: > >> First, add --stats to find out how much is being transferred. Second, >> if these are fixed size or sparse image files then --sparse will be a >> big help. Finally, if they are not fixed size or sparse then --inplace >> is probably making things worse. >> >> --no-whole-file is the default btw unless you are doing a local copy. >> If you are doing a local copy then just use cp as it is much faster at that. >> > > This is what i just got rsyncing a 300GB file: > > sending incremental file list > > mausdb.raw > > sent 1,195,156,775 bytes received 19,873,319 bytes 67,721.77 bytes/sec > total size is 289,408,024,064 speedup is 238.19 > > real299m1.693s > user68m12.737s > sys 7m56.389s > (i put time before the command). > > The files are fixed size, no sparse files. > What do you mean with a "local copy" ? Source and Destination on the same > disk ? > In my case the source reside on a SAN attached via FC, the destination is a > cifs server. > > Bernd > > > Helmholtz Zentrum Muenchen > Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) > Ingolstaedter Landstr. 1 > 85764 Neuherberg > www.helmholtz-muenchen.de > Aufsichtsratsvorsitzende: NN > Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter > Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich > Bassler, Dr. rer. nat. Alfons Enhsen > Registergericht: Amtsgericht Muenchen HRB 6466 > USt-IdNr: DE 129521671 > > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync of big image files from virtual hosts takes lot of time
- Am 20. Okt 2018 um 18:56 schrieb rsync ML rsync@lists.samba.org: > First, add --stats to find out how much is being transferred. Second, > if these are fixed size or sparse image files then --sparse will be a > big help. Finally, if they are not fixed size or sparse then --inplace > is probably making things worse. > > --no-whole-file is the default btw unless you are doing a local copy. > If you are doing a local copy then just use cp as it is much faster at that. > This is what i just got rsyncing a 300GB file: sending incremental file list mausdb.raw sent 1,195,156,775 bytes received 19,873,319 bytes 67,721.77 bytes/sec total size is 289,408,024,064 speedup is 238.19 real299m1.693s user68m12.737s sys 7m56.389s (i put time before the command). The files are fixed size, no sparse files. What do you mean with a "local copy" ? Source and Destination on the same disk ? In my case the source reside on a SAN attached via FC, the destination is a cifs server. Bernd Helmholtz Zentrum Muenchen Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) Ingolstaedter Landstr. 1 85764 Neuherberg www.helmholtz-muenchen.de Aufsichtsratsvorsitzende: NN Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Dr. rer. nat. Alfons Enhsen Registergericht: Amtsgericht Muenchen HRB 6466 USt-IdNr: DE 129521671 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync of big image files from virtual hosts takes lot of time
First, add --stats to find out how much is being transferred. Second, if these are fixed size or sparse image files then --sparse will be a big help. Finally, if they are not fixed size or sparse then --inplace is probably making things worse. --no-whole-file is the default btw unless you are doing a local copy. If you are doing a local copy then just use cp as it is much faster at that. On 10/20/2018 12:36 PM, Lentes, Bernd via rsync wrote: > Hi, > > i'd like to rsync some image files from virtual hosts each night. The image > files are between 50GB and 400GB big. > I thought rsync would be the appropriate solution because it just transfers > the differences, not the whole file. > But nevertheless rsync takes hours. > The images aren't much modified over the day, i would expect just some tens > of megabyte needed to be transfered. > > I have the following command line: > rsync -av --no-whole-file --inplace source destination. > > Is that ok ? Any suggestions ? > > I skipped the check with the hash values before the transfer because creating > a hash value from a 400GB file takes lng. > But the man pages mentione that nevertheless AFTER the transfer a hash value > is created on both sides to check that the files are equal. > Is it possible to switch the after-transfer check off ? > > I have SLES 12 SP3 and rsync 3.1.0 > > Thanks. > > Bernd > -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: https://sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., signature.asc Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync of big image files from virtual hosts takes lot of time
Hi, i'd like to rsync some image files from virtual hosts each night. The image files are between 50GB and 400GB big. I thought rsync would be the appropriate solution because it just transfers the differences, not the whole file. But nevertheless rsync takes hours. The images aren't much modified over the day, i would expect just some tens of megabyte needed to be transfered. I have the following command line: rsync -av --no-whole-file --inplace source destination. Is that ok ? Any suggestions ? I skipped the check with the hash values before the transfer because creating a hash value from a 400GB file takes lng. But the man pages mentione that nevertheless AFTER the transfer a hash value is created on both sides to check that the files are equal. Is it possible to switch the after-transfer check off ? I have SLES 12 SP3 and rsync 3.1.0 Thanks. Bernd -- Bernd Lentes Systemadministration Institut für Entwicklungsgenetik Gebäude 35.34 - Raum 208 HelmholtzZentrum münchen [ mailto:bernd.len...@helmholtz-muenchen.de | bernd.len...@helmholtz-muenchen.de ] phone: +49 89 3187 1241 fax: +49 89 3187 2294 [ http://www.helmholtz-muenchen.de/idg | http://www.helmholtz-muenchen.de/idg ] wer Fehler macht kann etwas lernen wer nichts macht kann auch nichts lernen Helmholtz Zentrum Muenchen Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH) Ingolstaedter Landstr. 1 85764 Neuherberg www.helmholtz-muenchen.de Aufsichtsratsvorsitzende: NN Stellv.Aufsichtsratsvorsitzender: MinDirig. Dr. Manfred Wolter Geschaeftsfuehrer: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Dr. rer. nat. Alfons Enhsen Registergericht: Amtsgericht Muenchen HRB 6466 USt-IdNr: DE 129521671 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
