Re: opendir(somedir/somefile): Not enough space -- why?
Even though it was a pain in the *ss, I broke up the transfer into several chunks (logically organized by sub-directories of the source) and wrote a script to batch the job. It worked, better and faster than via NFS. Mounting the source directory via NFS is not a solution that makes one feel good - it ended up with errors. _ Bill On Fri, 24 Jan 2003 11:54:13 -0700 Bill Geddes [EMAIL PROTECTED] wrote: I am attempting to use rsync to copy a large filesystem from an HP-UX server to a Linux server with more than enough filespace. This operation fails. A small directory from the same HP-UX server can be transfered just as expected. The HP-UX server is the source. It has 1Gb RAM - the output of bdf for the volume the source files is on is: Filesystem kbytesused avail %used Mounted on /dev/vg02/apps 6144 52722896 8652256 86% /apps On the target Linux server, I have 1Gb RAM and 1TB of free space: Filesystem 1k-blocks Used Available Use% Mounted on /dev/tcvg1/tcvol11007898916 33408 956667148 1% /vol1 I have apps configured properly as a service in the rsyncd.conf file. I started rsync in daemon mode on the HP-UX server, and from the Linux server I invoked: rsync --numeric-ids -vvva root@hpuxsrv::apps /vol1/asic_apps/ The file list get transferred, but then I just get repeated lines of errors: recv_file_name(some_dir/fm_v200209) opendir(some_dir/fm_v200209): Not enough space I have tried with --blocking-io and --no-blocking-io, with --bwlimitXXX. Same problem each time. Any insight is appreciated. -- Bill Geddes [EMAIL PROTECTED] -- UNIX is user friendly, it's just picky about who its friends are. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- Bill Geddes [EMAIL PROTECTED] -- UNIX is user friendly, it's just picky about who its friends are. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
opendir(somedir/somefile): Not enough space -- why?
I am attempting to use rsync to copy a large filesystem from an HP-UX server to a Linux server with more than enough filespace. This operation fails. A small directory from the same HP-UX server can be transfered just as expected. The HP-UX server is the source. It has 1Gb RAM - the output of bdf for the volume the source files is on is: Filesystem kbytesused avail %used Mounted on /dev/vg02/apps 6144 52722896 8652256 86% /apps On the target Linux server, I have 1Gb RAM and 1TB of free space: Filesystem 1k-blocks Used Available Use% Mounted on /dev/tcvg1/tcvol11007898916 33408 956667148 1% /vol1 I have apps configured properly as a service in the rsyncd.conf file. I started rsync in daemon mode on the HP-UX server, and from the Linux server I invoked: rsync --numeric-ids -vvva root@hpuxsrv::apps /vol1/asic_apps/ The file list get transferred, but then I just get repeated lines of errors: recv_file_name(some_dir/fm_v200209) opendir(some_dir/fm_v200209): Not enough space I have tried with --blocking-io and --no-blocking-io, with --bwlimitXXX. Same problem each time. Any insight is appreciated. -- Bill Geddes [EMAIL PROTECTED] -- UNIX is user friendly, it's just picky about who its friends are. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: opendir(somedir/somefile): Not enough space -- why?
More error data: The usage I have described returns an error code of 12 - Error in rsync protocol data stream Also, on the tail end of the std out: opendir(somedir): Not enough space done somedir/ somedir/.somefile is uptodate somedir/somefile.HTML ERROR: out of memory in map_ptr rsync error: error allocating core memory buffers (code 22) at util.c(232) somedir/doc/somefile/ rsync: read error: Connection reset by peer rsync error: error in rsync protocol data stream (code 12) at io.c(162) I know that rsync works only as well as the NIC card drivers in use. I hope that this is not another problem like that. On Fri, 24 Jan 2003 11:54:13 -0700 Bill Geddes [EMAIL PROTECTED] wrote: I am attempting to use rsync to copy a large filesystem from an HP-UX server to a Linux server with more than enough filespace. This operation fails. A small directory from the same HP-UX server can be transfered just as expected. The HP-UX server is the source. It has 1Gb RAM - the output of bdf for the volume the source files is on is: Filesystem kbytesused avail %used Mounted on /dev/vg02/apps 6144 52722896 8652256 86% /apps On the target Linux server, I have 1Gb RAM and 1TB of free space: Filesystem 1k-blocks Used Available Use% Mounted on /dev/tcvg1/tcvol11007898916 33408 956667148 1% /vol1 I have apps configured properly as a service in the rsyncd.conf file. I started rsync in daemon mode on the HP-UX server, and from the Linux server I invoked: rsync --numeric-ids -vvva root@hpuxsrv::apps /vol1/asic_apps/ The file list get transferred, but then I just get repeated lines of errors: recv_file_name(some_dir/fm_v200209) opendir(some_dir/fm_v200209): Not enough space I have tried with --blocking-io and --no-blocking-io, with --bwlimitXXX. Same problem each time. Any insight is appreciated. -- Bill Geddes [EMAIL PROTECTED] -- UNIX is user friendly, it's just picky about who its friends are. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- Bill Geddes [EMAIL PROTECTED] -- UNIX is user friendly, it's just picky about who its friends are. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: opendir(somedir/somefile): Not enough space -- why?
On Fri, Jan 24, 2003 at 02:20:33PM -0700, Bill Geddes wrote: More error data: The usage I have described returns an error code of 12 - Error in rsync protocol data stream Also, on the tail end of the std out: opendir(somedir): Not enough space done somedir/ somedir/.somefile is uptodate somedir/somefile.HTML ERROR: out of memory in map_ptr rsync error: error allocating core memory buffers (code 22) at util.c(232) somedir/doc/somefile/ rsync: read error: Connection reset by peer rsync error: error in rsync protocol data stream (code 12) at io.c(162) malloc has failed. You're out of memory. rsync can be a memory pig. Upgrading to cvs or 2.5.6pre* _may_ help. It is probably necessary to break the job up. A better gauge of the memory requirements of an rsync transfer would be df -i or find . | wc -l -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: opendir(somedir/somefile): Not enough space -- why?
Am I running out of memory on the source server? Seems perhaps more plausible than on the client - it has nothing going on and 1Gb RAM. The server has rsync version 2.5.5. The client has 2.5.6. I am currently mounting the server directory via NFS, and using rsync to copy from the mounted directory to the target. Slow, but it is running! On Fri, 24 Jan 2003 14:52:10 -0800 jw schultz [EMAIL PROTECTED] wrote: On Fri, Jan 24, 2003 at 02:20:33PM -0700, Bill Geddes wrote: More error data: The usage I have described returns an error code of 12 - Error in rsync protocol data stream Also, on the tail end of the std out: opendir(somedir): Not enough space done somedir/ somedir/.somefile is uptodate somedir/somefile.HTML ERROR: out of memory in map_ptr rsync error: error allocating core memory buffers (code 22) at util.c(232) somedir/doc/somefile/ rsync: read error: Connection reset by peer rsync error: error in rsync protocol data stream (code 12) at io.c(162) malloc has failed. You're out of memory. rsync can be a memory pig. Upgrading to cvs or 2.5.6pre* _may_ help. It is probably necessary to break the job up. A better gauge of the memory requirements of an rsync transfer would be df -i or find . | wc -l -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- Bill Geddes [EMAIL PROTECTED] -- UNIX is user friendly, it's just picky about who its friends are. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: opendir(somedir/somefile): Not enough space -- why?
On Fri, Jan 24, 2003 at 04:49:20PM -0700, Bill Geddes wrote: Am I running out of memory on the source server? Seems perhaps more plausible than on the client - it has nothing going on and 1Gb RAM. The server has rsync version 2.5.5. The client has 2.5.6. Yes, it looks like the sender is running out of memory. Break the job up. I am currently mounting the server directory via NFS, and using rsync to copy from the mounted directory to the target. Slow, but it is running! If you continue over NFS be sure you do checksums of everything afterward. NFS has (or at least used to) a measurable error rate. On Fri, 24 Jan 2003 14:52:10 -0800 jw schultz [EMAIL PROTECTED] wrote: On Fri, Jan 24, 2003 at 02:20:33PM -0700, Bill Geddes wrote: More error data: The usage I have described returns an error code of 12 - Error in rsync protocol data stream Also, on the tail end of the std out: opendir(somedir): Not enough space done somedir/ somedir/.somefile is uptodate somedir/somefile.HTML ERROR: out of memory in map_ptr rsync error: error allocating core memory buffers (code 22) at util.c(232) somedir/doc/somefile/ rsync: read error: Connection reset by peer rsync error: error in rsync protocol data stream (code 12) at io.c(162) malloc has failed. You're out of memory. rsync can be a memory pig. Upgrading to cvs or 2.5.6pre* _may_ help. It is probably necessary to break the job up. A better gauge of the memory requirements of an rsync transfer would be df -i or find . | wc -l -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: opendir(somedir/somefile): Not enough space -- why?
On Fri, Jan 24, 2003 at 04:02:28PM -0800, jw schultz wrote: On Fri, Jan 24, 2003 at 04:49:20PM -0700, Bill Geddes wrote: Am I running out of memory on the source server? Seems perhaps more plausible than on the client - it has nothing going on and 1Gb RAM. The server has rsync version 2.5.5. The client has 2.5.6. Yes, it looks like the sender is running out of memory. Break the job up. The important factor is the number of files. Rsync uses up some memory for every file it looks at in a run. That's why JW said to break it up into smaller jobs. I am currently mounting the server directory via NFS, and using rsync to copy from the mounted directory to the target. Slow, but it is running! If you continue over NFS be sure you do checksums of everything afterward. NFS has (or at least used to) a measurable error rate. Rsync is optimized for a slow network and high speed disk access. If you're doing NFS over a slow network, that looks to rsync as a slow disk. If you can, it's probably better to run rsync directly on the server. - Dave -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
