Re: Max filesize for rsync?
On Apr 12, Wayne Davison wrote: | On Tue, Apr 12, 2005 at 12:46:55PM -0400, Christophe Kalt wrote: | > FWIW, I just upgraded to 2.6.4, and that has solved a problem | > i'd been having for a few weeks where 2.6.3 repeatedly failed | > to synchronize a 5GB file. Don't remember seeing anything in | > the NEWS or other making me think upgrading would help, but | > it did. | | Perhaps one of these? | | - Fixed a potential protocol-corrupting bug where the generator | could merge a message from the receiver into the middle of a | multiplexed packet of data if only part of that data had been | written out to the socket when the message from the generator | arrived. | | - Fixed a bug that would sometimes surface when using --compress | and sending a file with a block-size larger than 64K (either | manually specified, or computed due to the file being really | large). Prior versions of rsync would sometimes fail to | decompress the data properly, and thus the transferred file | would fail its verification. This last one sounds right as i typically got verification failed messages. (The files did look identical.) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
Actually, it said that the file was too big, so it wasn't a question of performance, it just wouldn't do it. I set my AIX environment to allow me to have files of unlimited size, rsync still said it was too big. >>> Shachar Shemesh <[EMAIL PROTECTED]> 04/14/05 1:28 AM >>> Jeff Schoby wrote: >Well, I got -further- by changing the fsize= to -1 in >/etc/security/limits on my AIX boxes, >but rsync ultimately still did not like my 15GB file I wanted to >transfer. > > What does "doesn't like" mean? Does it freeze with too much CPU usage? >Had to resort to good ol' plain vanilla ftp. > > How long does transferring the file via ftp take? A 15GB file puts a load of 194% on the hash table. This means that, *on average*, each lookup will have to scan two areas. I'm not sure whether that should have created a huge slowdown or not. Try rsyncing that file with the "--block-size=524288" (i.e. - a 0.5MB block size), and please report whether rsync's behavior had improved, and in particular, how does it rate against vanilla ftp. Thanks, Shachar -- Shachar Shemesh Lingnu Open Source Consulting ltd. Have you backed up today's work? http://www.lingnu.com/backup.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
Jeff Schoby wrote: Well, I got -further- by changing the fsize= to -1 in /etc/security/limits on my AIX boxes, but rsync ultimately still did not like my 15GB file I wanted to transfer. What does "doesn't like" mean? Does it freeze with too much CPU usage? Had to resort to good ol' plain vanilla ftp. How long does transferring the file via ftp take? A 15GB file puts a load of 194% on the hash table. This means that, *on average*, each lookup will have to scan two areas. I'm not sure whether that should have created a huge slowdown or not. Try rsyncing that file with the "--block-size=524288" (i.e. - a 0.5MB block size), and please report whether rsync's behavior had improved, and in particular, how does it rate against vanilla ftp. Thanks, Shachar -- Shachar Shemesh Lingnu Open Source Consulting ltd. Have you backed up today's work? http://www.lingnu.com/backup.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
[EMAIL PROTECTED] wrote: I would be very happy to test any patches. (Assorted RedHat/Fedora i386) (Assume I am a total newbie, much safer that way) A few very large files regularly rsync'd in production. Seems like it sometimes gets somewhat stuck in the middle of something large. (The rsync is mostly staging area to staging area. Plenty of redundancy, so I'm unlikely to get hurt if I'm aware of problems. The way the targets are used, I will know about problems before any real damage is done.) The more important of the transfers are over occasionally very bad internet connections, so I'm pretty much in the situation of something to gain, nothing to lose. -rw-rw1 27 27 1187270120 Apr 13 03:24 /home/rsync-sjs/mysql/sjs/dwf.MYD -rw-rw1 27 27 1098515060 Apr 8 07:34 /home/rsync-sjs/mysql/srvs/dwf.MYD -rw-rw1 mysqlmysql840374964 Apr 12 12:59 /home/rsync-2bb/mysql/srv/dwf.MYD -rw-rw1 mysqlmysql520216980 Apr 12 20:54 /home/rsync-2bb/mysql/map/dwf.MYD -rw-rw1 mysqlmysql221876208 Apr 11 14:21 /home/rsync-2bb/mysql/ecp/dwf.MYD Actually, rsync's implementation only starts to give suboptimal results when we pass the 2.5GB area. Up until there one should see no noticeable difference in performance (well, my patch can allow using somewhat less memory for those cases, but no one would really miss 256KB of RAM these days, and I'm not sure how much impact the "CPU cache" effect is going to have in rsync's case. I guess that some effect will be seen anyways, but not as noticeable). The performance bottleneck is due to hash table buckets load. Optimal load, as taught at computer science, is 80% (or 0.8). Your smallest file puts a load of 23%, while your largest file has a load of 53%. You shouldn't see any problems when using rsync (at least, not the type of problem I'm talking about). Shachar -- Shachar Shemesh Lingnu Open Source Consulting ltd. Have you backed up today's work? http://www.lingnu.com/backup.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
Well, I got -further- by changing the fsize= to -1 in /etc/security/limits on my AIX boxes, but rsync ultimately still did not like my 15GB file I wanted to transfer. Had to resort to good ol' plain vanilla ftp. >>> Shachar Shemesh <[EMAIL PROTECTED]> 04/13/05 2:52 PM >>> Jeff Schoby wrote: >What the maximum filesize rsync can transfer? > >I'm trying to rsync one of my servers to another but the rsync is >croaking on a file that's barely 1GB. > >Tips, hints, suggestions? > >rsync server is AIX 4.3.3 ML11 - rsync 2.6.3 > >rsync client is AIX 5.3 ML1 - rsync 2.6.4 > >Thanks >-Jeff > > Please note that, all file size OS limitations aside, rsync has suboptimal performance for too big files. When I get around to it I'll try to create a patch, but in the mean while too big files will have too many non-real hash table collisions, and may become extremely slow. If you run across this problem, please post on list, as we need someone to experience this problem in order to try and fix it. Shachar -- Shachar Shemesh Lingnu Open Source Consulting ltd. Have you backed up today's work? http://www.lingnu.com/backup.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Max filesize for rsync?
I would be very happy to test any patches. (Assorted RedHat/Fedora i386) (Assume I am a total newbie, much safer that way) A few very large files regularly rsync'd in production. Seems like it sometimes gets somewhat stuck in the middle of something large. (The rsync is mostly staging area to staging area. Plenty of redundancy, so I'm unlikely to get hurt if I'm aware of problems. The way the targets are used, I will know about problems before any real damage is done.) The more important of the transfers are over occasionally very bad internet connections, so I'm pretty much in the situation of something to gain, nothing to lose. -rw-rw1 27 27 1187270120 Apr 13 03:24 /home/rsync-sjs/mysql/sjs/dwf.MYD -rw-rw1 27 27 1098515060 Apr 8 07:34 /home/rsync-sjs/mysql/srvs/dwf.MYD -rw-rw1 mysqlmysql840374964 Apr 12 12:59 /home/rsync-2bb/mysql/srv/dwf.MYD -rw-rw1 mysqlmysql520216980 Apr 12 20:54 /home/rsync-2bb/mysql/map/dwf.MYD -rw-rw1 mysqlmysql221876208 Apr 11 14:21 /home/rsync-2bb/mysql/ecp/dwf.MYD Scripts are along the lines of: #!/bin/sh # staged backup time rsync -a /home/rsync-sjs/* /home/rsync-sjs-1bk/ time sync -avPz --timeout= --password-file=/etc/rsync.secrets/rsync-sjs \ [EMAIL PROTECTED]::rsync-sjs/* /home/rsync-sjs/ date; echo rsync-sjs -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Shachar Shemesh Sent: Wednesday, April 13, 2005 2:53 PM To: Jeff Schoby Cc: rsync@lists.samba.org Subject: Re: Max filesize for rsync? Jeff Schoby wrote: >What the maximum filesize rsync can transfer? > >I'm trying to rsync one of my servers to another but the rsync is >croaking on a file that's barely 1GB. > >Tips, hints, suggestions? > >rsync server is AIX 4.3.3 ML11 - rsync 2.6.3 > >rsync client is AIX 5.3 ML1 - rsync 2.6.4 > >Thanks >-Jeff > > Please note that, all file size OS limitations aside, rsync has suboptimal performance for too big files. When I get around to it I'll try to create a patch, but in the mean while too big files will have too many non-real hash table collisions, and may become extremely slow. If you run across this problem, please post on list, as we need someone to experience this problem in order to try and fix it. Shachar -- Shachar Shemesh Lingnu Open Source Consulting ltd. Have you backed up today's work? http://www.lingnu.com/backup.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
Jeff Schoby wrote: What the maximum filesize rsync can transfer? I'm trying to rsync one of my servers to another but the rsync is croaking on a file that's barely 1GB. Tips, hints, suggestions? rsync server is AIX 4.3.3 ML11 - rsync 2.6.3 rsync client is AIX 5.3 ML1 - rsync 2.6.4 Thanks -Jeff Please note that, all file size OS limitations aside, rsync has suboptimal performance for too big files. When I get around to it I'll try to create a patch, but in the mean while too big files will have too many non-real hash table collisions, and may become extremely slow. If you run across this problem, please post on list, as we need someone to experience this problem in order to try and fix it. Shachar -- Shachar Shemesh Lingnu Open Source Consulting ltd. Have you backed up today's work? http://www.lingnu.com/backup.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
On 4/12/05, Jeff Schoby wrote: > I did what steve suggested here, on both machines. I didn't use any > arguments for configure... just did ./configure Odd, I thought we had it nailed on that one. Try the following on each system using the new rsync you just built with the mkstemp use commented out: 1) Create a large file, preferably a sparse file 2) Try to rsync it locally For example: $ echo > smallfile $ dd if=smallfile of=sparsefile_01 bs=1024 count=1 seek=2097153 $ rsync -avvvS sparsefile_01 sparsefile_02 If this works, then the problem may be in your network transport (e.g. ssh). -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
On 4/12/05, Jeff Schoby wrote: > I did what steve suggested here, on both machines. I didn't use any > arguments for configure... just did ./configure Odd, I thought we had it nailed on that one. Try the following on each system using the new rsync you just built with the mkstemp use commented out: 1) Create a large file, preferably a sparse file 2) Try to rsync it locally For example: $ echo > smallfile $ dd if=smallfile of=sparsefile_01 bs=1024 count=1 seek=2097153 $ rsync -avvvS sparsefile_01 sparsefile_02 If this works, then the problem may be in your network transport (e.g. ssh). -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync? (solved?)
I found a usenet article which had some info in it that may help other AIX users who are having issues. in /etc/security/limits Check all the stanzas for fsize = it will probably have some number in it like 2018261 That is what is limiting your filesize. Change fsize = -1 logout and back in. You should be able to use rsync to copy large files. Seems to be working in my case -- Jeff Schoby Unix/Network Admin City of Columbia, Missouri 573.874.6320 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
I did what steve suggested here, on both machines. I didn't use any arguments for configure... just did ./configure rsync -Caupvvvz server::myfiles /usr/mydir Here's what happens when it gets to the one file that it says is too big: pathto/myfile recv mapped pathto/myfile of size 1073741312 rsync: write failed on "/usr/mydir/pathto/myfile": File too large (27) rsync error: error in file IO (code 11) at receiver.c (290) _exit_cleanup(code=11, file=receiver.c, line=290): about to call exit(11) rsync: connection unexpectedly closed (1347054 bytes received so far) [generator] rsync: error in rsync protocol data stream (code 12) at io.c(420) _exit_cleanup(code=12, file=io.c, line=420): about to call exit(12) >>> "Steve Bonds" <[EMAIL PROTECTED]> 04/12/05 12:13 PM >>> On Apr 12, 2005 10:04 AM, Jeff Schoby wrote: > How do I build rsync for large file support..I'm going to need > 4GB > files thru rsync. > > It basically stops with an error message of "File too large"...I can > re-run rsync later with the -vvv flag and post the results tomorrow. It sounds like your version of AIX has similar problems to HP-UX. You will need to re-compile rsync doing something like this: 1) run ./configure with whatever options you normally use 2) edit the config.h file and comment out the line "#define HAVE_SECURE_MKSTEMP 1" 3) continue normally (e.g. "make", "make test", etc.) Here's an older list article that describes the fix: http://lists.samba.org/archive/rsync/2004-July/010199.html -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
On Apr 12, 2005 10:04 AM, Jeff Schoby wrote: > How do I build rsync for large file support..I'm going to need > 4GB > files thru rsync. > > It basically stops with an error message of "File too large"...I can > re-run rsync later with the -vvv flag and post the results tomorrow. It sounds like your version of AIX has similar problems to HP-UX. You will need to re-compile rsync doing something like this: 1) run ./configure with whatever options you normally use 2) edit the config.h file and comment out the line "#define HAVE_SECURE_MKSTEMP 1" 3) continue normally (e.g. "make", "make test", etc.) Here's an older list article that describes the fix: http://lists.samba.org/archive/rsync/2004-July/010199.html -- Steve -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
How do I build rsync for large file support..I'm going to need > 4GB files thru rsync. It basically stops with an error message of "File too large"...I can re-run rsync later with the -vvv flag and post the results tomorrow. >>> Paul Slootman <[EMAIL PROTECTED]> 04/12/05 11:36 AM >>> On Tue 12 Apr 2005, Jeff Schoby wrote: > What the maximum filesize rsync can transfer? 2GB (4GB?) should always be possible, when built with the appropriate options for large file support >4GB files are no problem. > I'm trying to rsync one of my servers to another but the rsync is > croaking on a file that's barely 1GB. Well, it shouldn't... Please define "croaking"? Any diagnostic messages? Try running with -vvv Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
On Tue, Apr 12, 2005 at 12:46:55PM -0400, Christophe Kalt wrote: > FWIW, I just upgraded to 2.6.4, and that has solved a problem > i'd been having for a few weeks where 2.6.3 repeatedly failed > to synchronize a 5GB file. Don't remember seeing anything in > the NEWS or other making me think upgrading would help, but > it did. Perhaps one of these? - Fixed a potential protocol-corrupting bug where the generator could merge a message from the receiver into the middle of a multiplexed packet of data if only part of that data had been written out to the socket when the message from the generator arrived. - Fixed a bug that would sometimes surface when using --compress and sending a file with a block-size larger than 64K (either manually specified, or computed due to the file being really large). Prior versions of rsync would sometimes fail to decompress the data properly, and thus the transferred file would fail its verification. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
On Tue 12 Apr 2005, Christophe Kalt wrote: > > FWIW, I just upgraded to 2.6.4, and that has solved a problem > i'd been having for a few weeks where 2.6.3 repeatedly failed > to synchronize a 5GB file. Don't remember seeing anything in > the NEWS or other making me think upgrading would help, but > it did. Curious, I too have had reports of things that previously mysteriously failed, now started working with 2.6.4 :-) Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
On Apr 12, Jeff Schoby wrote: | What the maximum filesize rsync can transfer? Depends on whether you have LFS (large file support) enabled (at compile time). | I'm trying to rsync one of my servers to another but the rsync is | croaking on a file that's barely 1GB. That should be no problem. | Tips, hints, suggestions? Define "croaking", give us some error message. | rsync server is AIX 4.3.3 ML11 - rsync 2.6.3 | | rsync client is AIX 5.3 ML1 - rsync 2.6.4 FWIW, I just upgraded to 2.6.4, and that has solved a problem i'd been having for a few weeks where 2.6.3 repeatedly failed to synchronize a 5GB file. Don't remember seeing anything in the NEWS or other making me think upgrading would help, but it did. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Max filesize for rsync?
On Tue 12 Apr 2005, Jeff Schoby wrote: > What the maximum filesize rsync can transfer? 2GB (4GB?) should always be possible, when built with the appropriate options for large file support >4GB files are no problem. > I'm trying to rsync one of my servers to another but the rsync is > croaking on a file that's barely 1GB. Well, it shouldn't... Please define "croaking"? Any diagnostic messages? Try running with -vvv Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Max filesize for rsync?
What the maximum filesize rsync can transfer? I'm trying to rsync one of my servers to another but the rsync is croaking on a file that's barely 1GB. Tips, hints, suggestions? rsync server is AIX 4.3.3 ML11 - rsync 2.6.3 rsync client is AIX 5.3 ML1 - rsync 2.6.4 Thanks -Jeff -- Jeff Schoby Unix/Network Admin City of Columbia, Missouri 573.874.6320 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
