Re: Cannot Allocate Memory
On Tue, 7 Jun 2005, Max Kipness [EMAIL PROTECTED] wrote:
I've just upgraded to 2.6.5.
On a new set of servers I'm trying to sync I seem to be running into a log
of memory allocation errors.
With some of the servers, I start the rsync session, and it counts so many
files and then stops. At that point I'm unable to control-c out of the
process. From another session I can't issue a kill command either...very
strange.
A control-C might not always work, but a kill -9 {PID} should. If it
doesn't, there may be some OS-related problem in your brand-new servers.
On other servers, the first few files/directories issue a 'cannot allocate
memory', but strangly enough, it continues on.
If that's an actual quote of the error message, it's not from rsync.
Rsync uses ERROR: out of memory in {NameOfRoutine}, and then exits.
Does this simply mean that the 256mb memory isn't sufficient? I am trying
to sync a very large folder structure.
At an average of 100 bytes per file, you could do 1 million files in
100 MB, so unless you have a real whopper, 256 MB should be just fine
especially if you also have some swap space.
Is there anyway around this using tmp disk space or something until I can
get more memory?
Using swap should effectively use temp disk space.
What are your ulimit values? If you are running as root it shouldn't matter,
but small memory limits might trigger a problem.
But the overall symptoms imply a deeper problem, IMO.
John
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: problem syncing files on RAM in embedded device
On Fri, 10 Jun 2005, Vijay Ram.C [EMAIL PROTECTED] wrote: i am working on an networking product which has a 32 MB flash memory and a 64MB RAM. i need to sync a file named appl.gz which is of size 3.1MB from a unit with ip 192.168.10.1 to a unit with ip 192.168.10.2. the rsync is called from a C code which does a fork-execve operation. the command line options being passed to rsync is as below: command: rsync --stats -aI --temp-dir=/tmp/ 192.168.10.1::path/appl.gz /config/appl.gz in the above command tmp/ is a part of the filesystem on the RAM space while /config/ is mounted onto a flash space. the above operation is done in two phases. one is a direct function call which does the above operation and the second is in a while loop which keeps doing the operation every one hour. For first phase an error is thrown as below. write failed on appl.gz: Success unexpected EOF in read_timeout I understand your rationale for using an old version, but you are not likely to get much help on such an old version. For example, the above bogus Success message was fixed in 2.6.0. When a write() call returned with less than the requested bytes being written (a partially successful write), the write_file routine returned immediately instead of adjusting the buffer pointer and length and retrying until it got a -1 returned or successfully wrote the entire buffer. See write_file() in fileio.c if you want to try to port the fix to your version. after giving the above error the, syncing of files terminates. From the second phase onwards, the syncing of the same file with the same options as given above works well. That's odd. Is it actually ever rsyncing a new version of the file? Since you are using -I, only the file size will be checked. On experimenting, the same code worked well when done with the --temp-dir=/config/ i.e, temporary redirected to flash ie, where the destination is to be present. The platform being used in the product is monta vista linux and. i have compiled the source code of version 2.4.8 and ported on the device. there is also enough space on the flash as well as the RAM for the temp file to be made. I read in some old mails in the mailing list which said that this error The unexpected EOF in read_timeout error? It may be related to the previous bogus error. comes when there is no data on the socket. But on certain experimentation, i tried syncing another file of size less that 512KB instead of the above mentioned file. At that time everything went well and the syncing was successfull. The main reason for me going with this old version of rsync is the space taken by the rsync binary. Can anybody help me figure out the reason for the failure on the first go.Is there any extra care that should be taken which am missing when redirecting temporary file to RAM space?? None that I'm aware of. There is an extra whole-file copy that rsync has to do every time since an intra-partition rename won't work. Are you using temp space out of concern that there won't be room on the destination for the file and it's temporarily-created replacement? Otherwise, creating the temp file on the destination is more efficient since only a simple rename is needed. John -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2793] New: xattr support fails with multiples attributes
https://bugzilla.samba.org/show_bug.cgi?id=2793 Summary: xattr support fails with multiples attributes Product: rsync Version: 2.6.5 Platform: All OS/Version: Linux Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: [EMAIL PROTECTED] ReportedBy: [EMAIL PROTECTED] QAContact: [EMAIL PROTECTED] When I try to use -X option with a patched rsync (using XFS as file system), I obtain this when a file have multiple user attributes: [EMAIL PROTECTED] data]# echo test test_pierre/test.file [EMAIL PROTECTED] data]# setfattr -n user.test1 -v testval1 test_pierre/test.file [EMAIL PROTECTED] data]# setfattr -n user.test2 -v testval2 test_pierre/test.file [EMAIL PROTECTED] data]# getfattr test_pierre/test.file # file: test_pierre/test.file user.test1 user.test2 [EMAIL PROTECTED] data]# rsync -AavX -P -W --numeric-ids --delete --force test_pierre/ gargamel:/data/test_pierre/ building file list ... 17608 files to consider ./ test.file 5 100%0.00kB/s0:00:00 (1, 0.0% of 17608) logs_test/ .test.file.cAW83c: rsync_xal_set: lsetxattr system.posix_acl_default failed: Not a directory sent 559013 bytes received 78 bytes 65775.41 bytes/sec total size is 5611168219 speedup is 10036.23 rsync error: some files could not be transferred (code 23) at main.c(791) File is transmitted, but without its attributes. If I remove an attribute, transmit seems to works fine... [EMAIL PROTECTED] data]# setfattr -x user.test1 test_pierre/test.file [EMAIL PROTECTED] data]# getfattr test_pierre/test.file # file: test_pierre/test.file user.test2 [EMAIL PROTECTED] data]# rsync -AavX -P -W --numeric-ids --delete --force test_pierre/ gargamel:/data/test_pierre/ building file list ... 17608 files to consider ./ test.file 5 100%0.00kB/s0:00:00 (1, 0.0% of 17608) logs_test/ sent 559013 bytes received 78 bytes 58851.68 bytes/sec total size is 5611168219 speedup is 10036.23 ...but the attribute value and name is not ever kept in the other side : [EMAIL PROTECTED] test_pierre]# getfattr -n user.test4 test.file3 # file: test.file3 user.test4=testval4 [EMAIL PROTECTED] data]# rsync -AavX -P -W --numeric-ids --delete --force test_pierre/ gargamel:/data/test_pierre/ building file list ... 17608 files to consider test.file3 5 100%0.00kB/s0:00:00 (1, 0.0% of 17608) logs_test/ sent 559008 bytes received 72 bytes 65774.12 bytes/sec total size is 5611168219 speedup is 10036.43 [EMAIL PROTECTED] test_pierre]# getfattr test.file3 # file: test.file3 user.ARTE [EMAIL PROTECTED] test_pierre]# getfattr -n user.ARTE test.file3 # file: test.file3 user.ARTE=h [EMAIL PROTECTED] test_pierre]# getfattr -n user.test4 test.file3 test.file3: user.test4: No such attribute I set an attribute named ARTE for an other file in an other sync before. I can't understand why it uses it now. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: restoring incremental backups
On 2005-06-12 13:39:46 +0200, Erik Romijn wrote: A bit complicated, but my problem is this: I want to restore the system as it was on friday. Use rsnapshot Best Martin -- http://www.tm.oneiros.de -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2784] rsync gives following error: buffer overflow in receive_file_entry
https://bugzilla.samba.org/show_bug.cgi?id=2784 --- Additional Comments From [EMAIL PROTECTED] 2005-06-13 11:56 --- disregard my last comment, the code does act as you suggest when not using the ssh as the transport layer. Are there any known work arounds for this behavior? -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2784] rsync gives following error: buffer overflow in receive_file_entry
https://bugzilla.samba.org/show_bug.cgi?id=2784 [EMAIL PROTECTED] changed: What|Removed |Added Status|ASSIGNED|RESOLVED Resolution||FIXED --- Additional Comments From [EMAIL PROTECTED] 2005-06-13 13:28 --- I made additional mods to the way the script uses ssh as the protocl and it worked thanks for the help. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2795] New: tardet of symlinks not correct on destination
https://bugzilla.samba.org/show_bug.cgi?id=2795 Summary: tardet of symlinks not correct on destination Product: rsync Version: 2.6.3 Platform: Sparc OS/Version: Solaris Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: [EMAIL PROTECTED] ReportedBy: [EMAIL PROTECTED] QAContact: [EMAIL PROTECTED] Source: relative.test - ../../../tmp/dantest symlink2.test - /tmp/dantest Destination: relative.test - tmp/dantest symlink2.test - tmp/dantest RSYNC CODE: rsync -vlrz -e ssh --rsync-path=$RPATH rsyncd --daemon --config=$CONFIGFILE $SOURCE [EMAIL PROTECTED]::$MODULE MODULE Can someone explain why this occurs, and if there is something that I can do to fix this? -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 2795] tardet of symlinks not correct on destination
https://bugzilla.samba.org/show_bug.cgi?id=2795 [EMAIL PROTECTED] changed: What|Removed |Added Status|NEW |RESOLVED Resolution||WONTFIX --- Additional Comments From [EMAIL PROTECTED] 2005-06-13 15:04 --- This is what daemon mode does to prevent links from going outside the root of the module's path when chroot isn't enabled. If you want symlinks to be preserved exactly as specified on the sending side, turn chroot on in the daemon's config. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --fuzzy, enhancement idea
On Fri, Jun 10, 2005 at 04:15:42PM -0400, Moshe Jacobson wrote: Instead, I think would be nice to add a flag e.g. --detect-moved-dirs, which would compare the file listing for each side, and look for whole directory trees that seem to be moved since last rsync. This is the enhancement suggested in bug 2294: https://bugzilla.samba.org/show_bug.cgi?id=2294 I don't know of anyone currently working on it, but we're at least thinking about it. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: handling duplicate names deterministically and adding alternative checksum algorithms
On Thu, Jun 09, 2005 at 05:24:45PM -0600, Andrew Shewmaker wrote: Our current practices would benefit if rsync were enhanced to handle duplicate names deterministically as described in the todo list. One proposed solution to this was to use a functioned called mergesort(). A patch was provided that includes a compatibility function for systems that don't include mergesort(): http://lists.samba.org/archive/rsync/2004-August/010398.html we would also be interested in the ability to optionally use a stronger checksum algorithm. I just put a diff into the patches dir that implements the --md5 option (extracted and cleaned up from a larger patch set that was sent to me by Wolfgang Neuman): http://rsync.samba.org/ftp/unpacked/rsync/patches/md5.diff As you'd expect, this tells rsync to use MD5 checksums instead of MD4. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync/patches
Date: Tue Jun 14 02:52:08 2005 Author: wayned Update of /data/cvs/rsync/patches In directory dp.samba.org:/tmp/cvs-serv19572 Added Files: md5.diff Log Message: Extracted the --md5 option from Wolfgang Neuman's patch contribution and cleaned it up a bit. Revisions: md5.diffNONE = 1.1 http://www.samba.org/cgi-bin/cvsweb/rsync/patches/md5.diff?rev=1.1 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
