Re: Other possible solutions to: rsync memory usage, paid feature request
On Wed 06 Jul 2005, David Favro wrote: 1) Free: break your rsync's into several executions rather than one huge one. Do several sub-directory trees, each separately. If your data [...] 2) Cheap: buy more swap space. These days random-access magnetic [...] 4) Expensive: buy more solid-state memory. Possibly still cheaper than [...] None of these proposals would have helped when I wanten to move two year's worth of Debian archive images to another system using rsync. The Debian archive is currently around 88.000 files (at least what we mirror of it). Every day a snapshot is taken; common files are hardlinked across days. This means an incredible amount of directory entries and hunderds of thousands of distinct files. Doing 1) was not feasible, as that would result in very many hardlinks being lost and files effectively duplicated, leading to wasted space. Doing 2) was tried (actually: creating swap files on disk), but then we ran into the virtual address limitations of the 32-bit system: 3GB wasn't enough by far. Doing 3) would have the same problem as 2). Going to a 64-bit system might have helped, but I think that the memory usage would have exceeded what's reasonable in solid-state memory, and using swap would have slowed it all down horribly as the lists in memory are apparently transversed quite regularly. As it is, it took a couple of days before the virtual memory limit was reached... I ended up rsyncing the days separately, and using a perl program to build a tree of md5sums which were hardlinks to the corresponding files. With each new directory the md5sums could be compared and hardlinks recreated. However, I would *love* to see rsync be more memory-efficient... Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync memory usage, paid feature request
On Wed, Jul 06, 2005 at 01:13:23AM -0500, Matthew S. Hallacy wrote: Unfortunately there's no indication of who needs a spare week of coding time, or how much a week would cost. That's a really old comment, so I'm not sure if it was written by Martin Pool or Dave Dykstra or someone else. I'm also fairly sure that a week of time would not suffice to solve this problem in rsync, but you may wish to try to contact one of those guys and see if they wrote the comment in question and what they were thinking of as a solution. Quite a while back I looked into the changes necessary to have rsync perform a more incremental update of a hierarchy of files, and they required very extensive changes to the protocol. I coded up a test-bed application which I jokingly named rZync. This program got to the stage of being able to transfer and update hierarchies of files, but I never took it to the next stage of re-examining the design and coding up a next-gen version of rsync. (The test-bed doesn't have enough features to be a full replacement of rsync for most people, since options like --delete and --hard-links aren't implemented.) Working on this is actually something that I'd like to revisit in the near future. If you had any interest in helping to fund this, please get in touch with me. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Fichier supprim és entre temps
On Wed, Jul 06, 2005 at 04:20:51PM +0200, Philippe BEAU wrote: Is it possible to have the backup marked right even if somes files vanished ? You simply need to tell the caller of rsync that an exit code of 24 is OK. If the backup program doesn't have a way to specify this, you could use a shell-script wrapper to perform this function: #!/bin/sh rsync $@ e=$? if test $e = 24; then exit 0 fi exit $e Then, use the name of that script instead of rsync as your backup program. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Customer Services Representatives needed ASAP
Welcome to one of the most powerful income opportunities available all over the world. Customer Services Representatives needed ASAP! As an Customer Services Representatives count Representative you will.. · Participate in a training program which will develop your professional business practices · Thrive in a culture with a sales focus that provides multiple sales tools and lead sources · Be part of a team where your performance directly impacts financial results · Enjoy a salaried position with the opportunity for quarterly sales incentive plan bonuses Requirements: · A strong commitment to self-motivation · PC skill proficiency · College education considered, but not required · Relocation not required What we offer you: · Flexibility of hours · Part Time · Daytime and Evening Hours · Professional management team with a strong support system We sell Web Site Design, Flash, 3D modeling, Branding, Graphic Design, Interface, Multimedia, Print Production, Banners and a wide range of web programming and software development (anything from simple applications to full-scale network projects including database planning, development and promotion). We are always looking for the best people to join our growing team. As a diversified financial services company, we have a wide range of exciting career opportunities throughout. If you are interested in our job offer please reply your CV. We look forward with great anticipation to working with you. Best regards, Worotel Communications Company Employment Bureau -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Virus detected]
Sophos Plc MailMonitor for Domino/D R1.0(4.003c) Server: Benno Grummel --- Your email contained infected attachment(s). For advice consult your system administrator --- Mail-Info From: rsync@lists.samba.org To:[EMAIL PROTECTED] Rec.: [EMAIL PROTECTED] Date: 08.07.2005 01:11:37 Subject: Mail System Error - Returned Mail --- File: [attachment.zi]State: [file contains virus] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
CVS update: rsync/zlib
Date: Thu Jul 7 20:16:41 2005 Author: wayned Update of /data/cvs/rsync/zlib In directory dp.samba.org:/tmp/cvs-serv19837 Modified Files: trees.c Log Message: Changed Overflow back to overflow. Revisions: trees.c 1.5 = 1.6 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/trees.c?r1=1.5r2=1.6 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/zlib
Date: Thu Jul 7 20:17:35 2005 Author: wayned Update of /data/cvs/rsync/zlib In directory dp.samba.org:/tmp/cvs-serv20197 Modified Files: deflate.c Log Message: Use a define to handle conflicting function read_buf instead of renaming it in the code. Revisions: deflate.c 1.7 = 1.8 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/deflate.c?r1=1.7r2=1.8 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/zlib
Date: Thu Jul 7 20:37:31 2005 Author: wayned Update of /data/cvs/rsync/zlib In directory dp.samba.org:/tmp/cvs-serv30848 Modified Files: inflate.c Log Message: One more #undef BAD for AIX. Revisions: inflate.c 1.4 = 1.5 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/inflate.c?r1=1.4r2=1.5 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/zlib
Date: Thu Jul 7 20:48:02 2005 Author: wayned Update of /data/cvs/rsync/zlib In directory dp.samba.org:/tmp/cvs-serv3440 Modified Files: ChangeLog README adler32.c crc32.c deflate.c deflate.h inffast.c inffast.h inffixed.h inflate.c inftrees.c inftrees.h trees.c zconf.h zlib.h zutil.c zutil.h Added Files: compress.c crc32.h inflate.h Removed Files: Makefile infblock.c infblock.h infcodes.c infcodes.h infutil.c infutil.h Log Message: Upgraded to version 1.2.2. Revisions: ChangeLog 1.3 = 1.4 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/ChangeLog?r1=1.3r2=1.4 Makefile1.3 = NONE http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/Makefile?rev=1.3 README 1.3 = 1.4 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/README?r1=1.3r2=1.4 adler32.c 1.3 = 1.4 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/adler32.c?r1=1.3r2=1.4 compress.c NONE = 1.1 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/compress.c?rev=1.1 crc32.c 1.3 = 1.4 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/crc32.c?r1=1.3r2=1.4 crc32.h NONE = 1.1 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/crc32.h?rev=1.1 deflate.c 1.8 = 1.9 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/deflate.c?r1=1.8r2=1.9 deflate.h 1.4 = 1.5 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/deflate.h?r1=1.4r2=1.5 infblock.c 1.8 = NONE http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/infblock.c?rev=1.8 infblock.h 1.2 = NONE http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/infblock.h?rev=1.2 infcodes.c 1.3 = NONE http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/infcodes.c?rev=1.3 infcodes.h 1.2 = NONE http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/infcodes.h?rev=1.2 inffast.c 1.2 = 1.3 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/inffast.c?r1=1.2r2=1.3 inffast.h 1.2 = 1.3 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/inffast.h?r1=1.2r2=1.3 inffixed.h 1.2 = 1.3 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/inffixed.h?r1=1.2r2=1.3 inflate.c 1.5 = 1.6 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/inflate.c?r1=1.5r2=1.6 inflate.h NONE = 1.1 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/inflate.h?rev=1.1 inftrees.c 1.4 = 1.5 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/inftrees.c?r1=1.4r2=1.5 inftrees.h 1.2 = 1.3 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/inftrees.h?r1=1.2r2=1.3 infutil.c 1.2 = NONE http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/infutil.c?rev=1.2 infutil.h 1.5 = NONE http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/infutil.h?rev=1.5 trees.c 1.6 = 1.7 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/trees.c?r1=1.6r2=1.7 zconf.h 1.4 = 1.5 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/zconf.h?r1=1.4r2=1.5 zlib.h 1.4 = 1.5 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/zlib.h?r1=1.4r2=1.5 zutil.c 1.6 = 1.7 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/zutil.c?r1=1.6r2=1.7 zutil.h 1.6 = 1.7 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/zutil.h?r1=1.6r2=1.7 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/zlib
Date: Thu Jul 7 21:14:29 2005 Author: wayned Update of /data/cvs/rsync/zlib In directory dp.samba.org:/tmp/cvs-serv16001 Modified Files: inftrees.c Log Message: Fix latest zlib security exploit (CAN-2005-2096). Revisions: inftrees.c 1.5 = 1.6 http://www.samba.org/cgi-bin/cvsweb/rsync/zlib/inftrees.c?r1=1.5r2=1.6 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Thu Jul 7 21:19:14 2005 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv21217 Modified Files: NEWS Log Message: Fixed a typo. Revisions: NEWS1.311 = 1.312 http://www.samba.org/cgi-bin/cvsweb/rsync/NEWS?r1=1.311r2=1.312 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Thu Jul 7 21:33:40 2005 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv28790 Modified Files: rsync.yo rsync.1 Log Message: - Attempt to make the discussion of host-specifying args less confusing. - Fixed the names of the uid and gid daemon options in a sentence. Revisions: rsync.yo1.286 = 1.287 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.yo?r1=1.286r2=1.287 rsync.1 1.297 = 1.298 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.1?r1=1.297r2=1.298 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Thu Jul 7 22:50:12 2005 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv5641 Modified Files: NEWS Log Message: Mentioned the zlib changes and tweaked a couple sentences. Revisions: NEWS1.312 = 1.313 http://www.samba.org/cgi-bin/cvsweb/rsync/NEWS?r1=1.312r2=1.313 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync
Date: Thu Jul 7 22:53:59 2005 Author: wayned Update of /data/cvs/rsync In directory dp.samba.org:/tmp/cvs-serv7886 Modified Files: configure configure.in rsync.1 rsync.yo rsyncd.conf.5 rsyncd.conf.yo Log Message: Preparing for release of 2.6.6pre1 Revisions: configure 1.216 = 1.217 http://www.samba.org/cgi-bin/cvsweb/rsync/configure?r1=1.216r2=1.217 configure.in1.224 = 1.225 http://www.samba.org/cgi-bin/cvsweb/rsync/configure.in?r1=1.224r2=1.225 rsync.1 1.298 = 1.299 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.1?r1=1.298r2=1.299 rsync.yo1.287 = 1.288 http://www.samba.org/cgi-bin/cvsweb/rsync/rsync.yo?r1=1.287r2=1.288 rsyncd.conf.5 1.102 = 1.103 http://www.samba.org/cgi-bin/cvsweb/rsync/rsyncd.conf.5?r1=1.102r2=1.103 rsyncd.conf.yo 1.106 = 1.107 http://www.samba.org/cgi-bin/cvsweb/rsync/rsyncd.conf.yo?r1=1.106r2=1.107 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/packaging/lsb
Date: Thu Jul 7 22:53:59 2005 Author: wayned Update of /data/cvs/rsync/packaging/lsb In directory dp.samba.org:/tmp/cvs-serv7886/packaging/lsb Modified Files: rsync.spec Log Message: Preparing for release of 2.6.6pre1 Revisions: rsync.spec 1.27 = 1.28 http://www.samba.org/cgi-bin/cvsweb/rsync/packaging/lsb/rsync.spec?r1=1.27r2=1.28 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsyncweb
Date: Thu Jul 7 23:14:25 2005 Author: wayned Update of /data/cvs/rsyncweb In directory dp.samba.org:/tmp/cvs-serv19355 Modified Files: index.html Log Message: Mention 2.6.6pre1. Revisions: index.html 1.52 = 1.53 http://www.samba.org/cgi-bin/cvsweb/rsyncweb/index.html?r1=1.52r2=1.53 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/patches
Date: Thu Jul 7 23:16:04 2005 Author: wayned Update of /data/cvs/rsync/patches In directory dp.samba.org:/tmp/cvs-serv21771 Modified Files: acls.diff xattrs.diff Log Message: Fixed failing hunks. Revisions: acls.diff 1.56 = 1.57 http://www.samba.org/cgi-bin/cvsweb/rsync/patches/acls.diff?r1=1.56r2=1.57 xattrs.diff 1.4 = 1.5 http://www.samba.org/cgi-bin/cvsweb/rsync/patches/xattrs.diff?r1=1.4r2=1.5 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
CVS update: rsync/patches
Date: Thu Jul 7 23:16:54 2005 Author: wayned Update of /data/cvs/rsync/patches In directory dp.samba.org:/tmp/cvs-serv22408 Modified Files: ODBC-dblog.diff md5.diff Log Message: Got rid of some patch fuzz. Revisions: ODBC-dblog.diff 1.40 = 1.41 http://www.samba.org/cgi-bin/cvsweb/rsync/patches/ODBC-dblog.diff?r1=1.40r2=1.41 md5.diff1.1 = 1.2 http://www.samba.org/cgi-bin/cvsweb/rsync/patches/md5.diff?r1=1.1r2=1.2 ___ rsync-cvs mailing list rsync-cvs@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync-cvs
