Re: LFS support on AIX 4.3.3
On Fri, Mar 16, 2001 at 04:39:18PM +1100, Martin Pool wrote: > On 15 Mar 2001, "Willeat, Todd" <[EMAIL PROTECTED]> wrote: > > > >http://www.rs6000.ibm.com/doc_link/en_US/a_doc_lib/aixprggd/genprogc/prg_lr_files.htm > > explains how to enable large file support on AIX. The fillowing line > > was added to the config.h file: > > > > #define _LARGE_FILES 1 > > I committed this to rsync.h (v1.98), so it should be in 2.4.7. It's > unconditionally defined. If somebody on AIX could build from anoncvs > and confirm that it works that would be great. (Mail me if you would > like help with CVS.) ICK! Please undo your change. Defining _LARGE_FILES as is done in rsync.h means everyone will be impacted (Solaris, HP-UX, IRIX, Linux, FreeBSD, Tru64 UNIX, ...). Discussion has already occurred on this list about LFS support: http://lists.samba.org/pipermail/rsync/2000-May/002182.html http://lists.samba.org/pipermail/rsync/2000-May/002199.html If the current configure test is broken for AIX, the autoconf test should be fixed. Also, Andrew is probably going to move away from the explicit use of 64-bit data types (like off64_t). Paul Eggert has written an autoconf macro, AC_SYS_LARGEFILE, which is now a *standard* part of autoconf in CVS, that detects what to add to CFLAGS and LDFLAGS to get LFS support on your OS. I think this is the direction we should head. -- albert chin ([EMAIL PROTECTED])
Re: LFS support on AIX 4.3.3
On Sat, Mar 17, 2001 at 11:22:31AM +1100, Martin Pool wrote: > On 16 Mar 2001, [EMAIL PROTECTED] wrote: > > > ICK! Please undo your change. Defining _LARGE_FILES as is done in > > rsync.h means everyone will be impacted (Solaris, HP-UX, IRIX, Linux, > > FreeBSD, Tru64 UNIX, ...). Discussion has already occurred on this > > list about LFS support: > > OK, I'll take it out. Thankyou for catching this. Great! > > If the current configure test is broken for AIX, the autoconf test > > should be fixed. Also, Andrew is probably going to move away from the > > explicit use of 64-bit data types (like off64_t). Paul Eggert has > > written an autoconf macro, AC_SYS_LARGEFILE, which is now a *standard* > > part of autoconf in CVS, that detects what to add to CFLAGS and > > LDFLAGS to get LFS support on your OS. I think this is the direction > > we should head. > > That sounds like a more clean solution. So, that will just > automatically make off_t be as long as possible? As long as possible to support LFS, which might not be "as long as possible". -- albert chin ([EMAIL PROTECTED])
Re: Re(2): rsync fails
On Thu, May 17, 2001 at 04:18:57PM -0700, Bob Foster wrote: > Excellent! Thanks! You solved my problem, maybe I can solve yours. If > you have root access on the remote server, try linking from /usr/bin/rsync > to /usr/local/bin/rsync. Here's the command: > > ln -s /usr/local/bin/rsync /usr/bin/rsync > > Now, can someone tell us why the remote shell isn't seeing /usr/local/bin? > As far as I can tell, it's in the path. Either rsh or ssh into the machine as the user you're doing the rsync as and 'echo $PATH'. This is the path used to find rsync. BTW, modifying the value of RSYNC_NAME in rsync.h to the path of rsync on the remote machine will work around --rsync-path=[path to rsync]. Would it be worthwhile to have a configure option to change the default? -- albert chin ([EMAIL PROTECTED])
Re: Does rsync close open files?
On Fri, Jun 01, 2001 at 08:44:18AM -0500, Dave Dykstra wrote: > On Thu, May 31, 2001 at 03:29:42PM -0700, Adam McKenna wrote: > > The Problem: > > > > Since the script runs once a minute, it is possible for the rsync to start > > while a file is still being transferred to us. What appears to happen at > > this time, is that something happens and the file doesn't finish transferring > > properly. > > > > So, is there any way to have rsync skip certain files, if they are not > > complete? There doesn't appear to be any obvious way to do this. Could > > the rsync process be the reason that these transfers are not finishing > > correctly? > > There is really no way around that problem with rsync. Many other people > have tried to do similar things and the wisdom on the mailing list has > always been that rsync via cron is the wrong tool for applications that > have data that changes frequently and needs to be replicated very soon > after it changes. You need something more like a distributed database or a > replicating filesystem, or at the very least call rsync in a synchronous > manner right after a file has been changed and before the file can change > again. This is possible if rsync had an option to *lock* a file before trying to transfer. You would need sftp to lock the file as it was receiving. With this, rsync could skip over the "locked" files and a rerun later would retry them. Having rsync do locking would impact performance but it has advantages (on the client and server). -- albert chin ([EMAIL PROTECTED])
CVS aclocal.m4 patch
The AC_SYS_LARGEFILE macro in CVS is out of date. Patch below brings it up-to-date. BTW, this macro is in autoconf 2.50 now. Also, any reason you use both getconf() and AC_SYS_LARGEFILE? I'd recommend using the latter to replace the former. -- albert chin ([EMAIL PROTECTED]) -- snip snip Index: aclocal.m4 === RCS file: /cvsroot/rsync/aclocal.m4,v retrieving revision 1.2 diff -u -3 -p -r1.2 aclocal.m4 --- aclocal.m4 17 Mar 2001 01:06:34 - 1.2 +++ aclocal.m4 10 Jun 2001 07:47:11 - @@ -22,7 +22,7 @@ AC_DEFUN(AC_VALIDATE_CACHE_SYSTEM_TYPE, ac_cv_target_system_type="$target" ]) -#serial 12 +#serial 19 dnl By default, many hosts won't let programs access large files; dnl one must use special compiler options to get large-file access to work. @@ -35,7 +35,14 @@ dnl Internal subroutine of AC_SYS_LARGEF dnl AC_SYS_LARGEFILE_TEST_INCLUDES AC_DEFUN(AC_SYS_LARGEFILE_TEST_INCLUDES, [[#include -int a[(off_t) 9223372036854775807 == 9223372036854775807 ? 1 : -1]; +/* Check that off_t can represent 2**63 - 1 correctly. + We can't simply "#define LARGE_OFF_T 9223372036854775807", + since some C++ compilers masquerading as C compilers + incorrectly reject 9223372036854775807. */ +# define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62)) +int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721 + && LARGE_OFF_T % 2147483647 == 1) + ? 1 : -1]; ]]) dnl Internal subroutine of AC_SYS_LARGEFILE. @@ -43,23 +50,21 @@ dnl AC_SYS_LARGEFILE_MACRO_VALUE(C-MACRO AC_DEFUN(AC_SYS_LARGEFILE_MACRO_VALUE, [AC_CACHE_CHECK([for $1 value needed for large files], $3, [$3=no - AC_TRY_COMPILE(AC_SYS_LARGEFILE_TEST_INCLUDES -$5 -, -[$6], -, -[AC_TRY_COMPILE([#define $1 $2] -AC_SYS_LARGEFILE_TEST_INCLUDES -$5 - , - [$6], - [$3=$2])])]) + AC_TRY_COMPILE([$5], + [$6], + , + [AC_TRY_COMPILE([#define $1 $2] +[$5] + , + [$6], + [$3=$2])])]) if test "[$]$3" != no; then AC_DEFINE_UNQUOTED([$1], [$]$3, [$4]) fi]) AC_DEFUN(AC_SYS_LARGEFILE, - [AC_ARG_ENABLE(largefile, + [AC_REQUIRE([AC_PROG_CC]) + AC_ARG_ENABLE(largefile, [ --disable-largefile omit support for large files]) if test "$enable_largefile" != no; then @@ -67,14 +72,14 @@ AC_DEFUN(AC_SYS_LARGEFILE, ac_cv_sys_largefile_CC, [ac_cv_sys_largefile_CC=no if test "$GCC" != yes; then - # IRIX 6.2 and later do not support large files by default, - # so use the C compiler's -n32 option if that helps. - AC_TRY_COMPILE(AC_SYS_LARGEFILE_TEST_INCLUDES, , , -[ac_save_CC="$CC" - CC="$CC -n32" - AC_TRY_COMPILE(AC_SYS_LARGEFILE_TEST_INCLUDES, , - ac_cv_sys_largefile_CC=' -n32') - CC="$ac_save_CC"]) + # IRIX 6.2 and later do not support large files by default, + # so use the C compiler's -n32 option if that helps. + AC_TRY_COMPILE(AC_SYS_LARGEFILE_TEST_INCLUDES, , , + [ac_save_CC="$CC" +CC="$CC -n32" +AC_TRY_COMPILE(AC_SYS_LARGEFILE_TEST_INCLUDES, , + ac_cv_sys_largefile_CC=' -n32') +CC="$ac_save_CC"]) fi]) if test "$ac_cv_sys_largefile_CC" != no; then CC="$CC$ac_cv_sys_largefile_CC" @@ -82,17 +87,30 @@ AC_DEFUN(AC_SYS_LARGEFILE, AC_SYS_LARGEFILE_MACRO_VALUE(_FILE_OFFSET_BITS, 64, ac_cv_sys_file_offset_bits, - [Number of bits in a file offset, on hosts where this is settable.]) - AC_SYS_LARGEFILE_MACRO_VALUE(_LARGEFILE_SOURCE, 1, - ac_cv_sys_largefile_source, - [Define to make ftello visible on some hosts (e.g. HP-UX 10.20).], - [#include ], [return !ftello;]) + [Number of bits in a file offset, on hosts where this is settable.], + AC_SYS_LARGEFILE_TEST_INCLUDES) AC_SYS_LARGEFILE_MACRO_VALUE(_LARGE_FILES, 1, ac_cv_sys_large_files, - [Define for large files, on AIX-style hosts.]) - AC_SYS_LARGEFILE_MACRO_VALUE(_XOPEN_SOURCE, 500, - ac_cv_sys_xopen_source, - [Define to make ftello visible on some hosts (e.g. glibc 2.1.3).], - [#include ], [return !ftello;]) + [Define for large files, on AIX-style hosts.], + AC_SYS_LARGEFILE_TEST_INCLUDES) fi ]) + +AC_DEFUN(AC_FUNC_FSEEKO, + [AC_SYS_LARGEFILE_MACRO_VALUE(_LARGEFILE_SOURCE, 1, + ac_cv_sys_largefile_source, + [Define to make fseeko visible on some hosts (e.g. glibc 2.2).], + [#include ], [return !fseeko;]) + # We used to try defining _XOPEN_SOURCE=500 too, to work aroun
New --with-rsync-name option
Patch to set RSYNC_NAME in rsync.h. It adds a new option to configure,
--with-rsync-name, that takes a string that is used to replace
RSYNC_NAME. The default is "rsync".
--
albert chin ([EMAIL PROTECTED])
-- snip snip
Index: configure.in
===
RCS file: /cvsroot/rsync/configure.in,v
retrieving revision 1.72
diff -u -3 -p -r1.72 configure.in
--- configure.in7 May 2001 06:59:37 - 1.72
+++ configure.in10 Jun 2001 07:48:04 -
@@ -38,6 +38,14 @@ fi
AC_ARG_WITH(included-popt,
[ --with-included-poptuse bundled popt library, not from system])
+AC_ARG_WITH(rsync-name,
+ [ --with-rsync-name=NAME set rsync path used on server to NAME
+ (default: "rsync")],[
+ RSYNC_NAME="$with_rsync_name"
+ ],[
+ RSYNC_NAME="rsync"])
+AC_SUBST(RSYNC_NAME)
+
dnl Checks for programs.
AC_PROG_CC
AC_PROG_INSTALL
@@ -340,4 +348,4 @@ AC_SUBST(OBJ_RESTORE)
AC_SUBST(CC_SHOBJ_FLAG)
AC_SUBST(BUILD_POPT)
-AC_OUTPUT(Makefile lib/dummy zlib/dummy)
+AC_OUTPUT(Makefile lib/dummy zlib/dummy rsync.h)
--- rsync.h Sun Jun 10 02:47:57 2001
+++ /dev/null Sat Nov 11 12:59:35 2000
@@ -1,522 +0,0 @@
-/*
- Copyright (C) by Andrew Tridgell 1996, 2000
- Copyright (C) Paul Mackerras 1996
- Copyright (C) 2001 by Martin Pool <[EMAIL PROTECTED]>
-
- This program is free software; you can redistribute it and/or modify
- it under the terms of the GNU General Public License as published by
- the Free Software Foundation; either version 2 of the License, or
- (at your option) any later version.
-
- This program is distributed in the hope that it will be useful,
- but WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- GNU General Public License for more details.
-
- You should have received a copy of the GNU General Public License
- along with this program; if not, write to the Free Software
- Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
-*/
-
-#define False 0
-#define True 1
-
-#define BLOCK_SIZE 700
-#define RSYNC_RSH_ENV "RSYNC_RSH"
-
-#define RSYNC_NAME "rsync"
-#define RSYNCD_CONF "/etc/rsyncd.conf"
-
-#define DEFAULT_LOCK_FILE "/var/run/rsyncd.lock"
-#define URL_PREFIX "rsync://"
-
-#define BACKUP_SUFFIX "~"
-
-/* a non-zero CHAR_OFFSET makes the rolling sum stronger, but is
- incompatible with older versions :-( */
-#define CHAR_OFFSET 0
-
-
-#define FLAG_DELETE (1<<0)
-#define SAME_MODE (1<<1)
-#define SAME_RDEV (1<<2)
-#define SAME_UID (1<<3)
-#define SAME_GID (1<<4)
-#define SAME_DIR (1<<5)
-#define SAME_NAME SAME_DIR
-#define LONG_NAME (1<<6)
-#define SAME_TIME (1<<7)
-
-/* update this if you make incompatible changes */
-#define PROTOCOL_VERSION 24
-#define MIN_PROTOCOL_VERSION 15
-#define MAX_PROTOCOL_VERSION 30
-
-#define RSYNC_PORT 873
-
-#define SPARSE_WRITE_SIZE (1024)
-#define WRITE_SIZE (32*1024)
-#define CHUNK_SIZE (32*1024)
-#define MAX_MAP_SIZE (256*1024)
-#define IO_BUFFER_SIZE (4092)
-
-#define MAX_ARGS 1000
-
-#define MPLEX_BASE 7
-
-/* Log values. I *think* what these mean is: FLOG goes to the server
- * logfile; FERROR and FINFO try to end up on the client, with
- * different levels of filtering. */
-enum logcode {FNONE=0, FERROR=1, FINFO=2, FLOG=3 };
-
-#include "errcode.h"
-
-#include "config.h"
-
-#if HAVE_REMSH
-#define RSYNC_RSH "remsh"
-#else
-#define RSYNC_RSH "rsh"
-#endif
-
-#include
-
-#ifdef HAVE_UNISTD_H
-#include
-#endif
-#include
-#include
-
-#ifdef HAVE_SYS_PARAM_H
-#include
-#endif
-
-#ifdef HAVE_STDLIB_H
-#include
-#endif
-
-#ifdef HAVE_SYS_SOCKET_H
-#include
-#endif
-
-#ifdef HAVE_STRING_H
-#include
-#endif
-
-#ifdef HAVE_MALLOC_H
-#include
-#endif
-
-#ifdef TIME_WITH_SYS_TIME
-#include
-#include
-#else
-#ifdef HAVE_SYS_TIME_H
-#include
-#else
-#include
-#endif
-#endif
-
-#ifdef HAVE_FCNTL_H
-#include
-#else
-#ifdef HAVE_SYS_FCNTL_H
-#include
-#endif
-#endif
-
-#include
-
-#ifdef HAVE_SYS_IOCTL_H
-#include
-#endif
-
-#ifdef HAVE_SYS_FILIO_H
-#include
-#endif
-
-#include
-#ifdef HAVE_SYS_WAIT_H
-#include
-#endif
-#ifdef HAVE_CTYPE_H
-#include
-#endif
-#ifdef HAVE_GRP_H
-#include
-#endif
-#include
-
-#ifdef HAVE_UTIME_H
-#include
-#endif
-
-#ifdef HAVE_SYS_SELECT_H
-#include
-#endif
-
-#ifdef HAVE_SYS_MODE_H
-/* apparently AIX needs this for S_ISLNK */
-#ifndef S_ISLNK
-#include
-#endif
-#endif
-
-#ifdef HAVE_FNMATCH
-#include
-#else
-#include "lib/fnmatch.h"
-#endif
-
-#ifdef HAVE_GLOB_H
-#include
-#endif
-
-/* these are needed for the uid/gid mapping code */
-#include
-#include
-
-#include
-#include
-#include
-#include
-#include
-#include
-
-#if HAVE_DIRENT_H
-# include
-#else
-# define
Re: New --with-rsync-name option
On Mon, Jun 11, 2001 at 10:19:26AM -0500, Dave Dykstra wrote: > On Sun, Jun 10, 2001 at 03:01:26AM -0500, [EMAIL PROTECTED] wrote: > > Patch to set RSYNC_NAME in rsync.h. It adds a new option to configure, > > --with-rsync-name, that takes a string that is used to replace > > RSYNC_NAME. The default is "rsync". > > I kind of hate to pre-process all of rsync.h for just that one variable. > Isn't there some way to get autoheader to stick it into config.h.in? > Alternatively it could go into a -D as part of @CFLAGS@ in Makefile.in > but I'm not sure I care for that much either. How about we #ifndef RSYNC_NAME in rsync.h and add -DRSYNC_NAME to @CPPFLAGS@? The one caveat to adding it to @CPPFLAGS@ rather than in rsync.h is that for your $PATH variable, you'll need to quote it. Not so if you have RSYNC_NAME in config.h or rsync.h. -- albert chin ([EMAIL PROTECTED])
Re: New --with-rsync-name option
On Tue, Jun 12, 2001 at 02:34:15PM -0500, Dave Dykstra wrote: > Below is the patch. I checked it in to the rsync CVS. > > - Dave Dykstra > > > *** configure.in.OTue Jun 12 14:22:15 2001 > --- configure.in Tue Jun 12 14:22:44 2001 > *** > *** 38,43 > --- 38,50 > AC_ARG_WITH(included-popt, > [ --with-included-poptuse bundled popt library, not from system]) > > + AC_ARG_WITH(rsync-path, > + [ --with-rsync-path=PATH set rsync path used on server to PATH (default: >"rsync")], > + [ RSYNC_PATH="$with_rsync_name" ], > + [ RSYNC_PATH="rsync" ]) > + RSYNC_PATH="\"$RSYNC_PATH\"" > + AC_DEFINE_UNQUOTED(RSYNC_PATH, $RSYNC_PATH) This should be: AC_DEFINE_UNQUOTED(RSYNC_PATH, $RSYNC_PATH, [Path to rsync on server]) > *** acconfig.h.O Tue Jun 12 14:22:15 2001 > --- acconfig.hTue Jun 12 14:22:44 2001 > *** > *** 15,18 > #undef HAVE_SOCKETPAIR > #undef HAVE_SECURE_MKSTEMP > #undef HAVE_C99_VSNPRINTF > ! > --- 15,18 > #undef HAVE_SOCKETPAIR > #undef HAVE_SECURE_MKSTEMP > #undef HAVE_C99_VSNPRINTF > ! #undef RSYNC_PATH No need for this if you use the 3-argument version of AC_DEFINE_UNQUOTED. -- albert chin ([EMAIL PROTECTED])
Re: rsync gives ld.so.1 error
On Wed, Jul 18, 2001 at 03:42:11PM -0700, Sudarshan Ramaswamy wrote: > Hi > > I am new to rsync. I have installed rsyn2.4.6 in my home directory. I > can rsh to any machine without any passwd as we have NIS . > > rsync works perfectly on my local machine ie if > > I rsync within the same machine it works ok. But when I rsync from a > differnt machine or to a different machine it gives me the following > error > > my homedir = HOMEDIR > > machine1:/HOMEDIR/rsync246/bin/rsync /x/*.c machine2:/y gives . > building file list ... done > ld.so.1: rsync: fatal: relocation error: symbol not found: stat64: > referenced in rsync > unexpected tag 101 I take it this is Solaris? What this means is that your rsync binary was compiled on Solaris 2.6, 7, or 8 and you're trying to run it on a Solaris 2.5.1 machine (which does have stat64 in libc). Is this what is happening? -- albert chin ([EMAIL PROTECTED])
Re: File merging
On Thu, Sep 13, 2001 at 07:28:46PM -0800, Britton wrote: > On Thu, 13 Sep 2001, Andy Goth wrote: > > > Can rsync be made to merge files? For instance, could I synchronize mboxes > > between multiple computers, even though they're all changing? > > Not if they are changing simultaneously. You can't do that at all, > without a distributed filesystem with locking. Not really. If rsync did fcntl locking on the client/server and the mail system used the same locking mechanism, you could pull it off. -- albert chin ([EMAIL PROTECTED])
efficient file appends
Hi. When I discovered rsync, it immediately became one of my most indispensable utilities. It's a real godsend on bandwidth-limited links, especially digital cellular. It works remarkably well in the general case, but I think the algorithm could be improved for one very important special case. Many (or even most) of the updated files I transfer with rsync change only by stuff being appending to the end. Examples of such files include system logs and (especially) email archives in mbox format. Rsync correctly handles these files, of course, but I think it could do so more efficiently. Right now, the receiver sends back a list of checksums for the blocks it has, and this checksum list can grow quite long when the file is large. I often see transfers of large mailboxes where the appendage of one small email message to the sender's copy results in a reverse transfer of checksum blocks that is much larger than the new message. It seems to me that this situation is common enough that the rsync protocol should look for it as a special case. Once the protocol has determined from differing timestamps and/or lengths that a file needs to be synchronized, the receiver should return a hash (and length) of its copy of the entire file to the sender. The sender then computes the hash for the corresponding leading segment of its copy. If they match, the sender simply sends the newly appended data and instructs the receiver to append it to its copy. I just joined this list, and I couldn't find any obvious discussion of this issue in the archives. My apologies if it has already been discussed. Phil Karn
Re: efficient file appends
>While potentially a useful option, you wouldn't want the protocol to >automatically always check for it, since it would preclude rsync on This extension need not break any existing mechanism; if the hash of the receiver's copy of the file doesn't match the start of the sender's file, the protocol would continue as before. >So at best, you'd only want to enable this option if the only thing >for the entire set of files in a given run were files known to expand >this way. That's certainly reasonable, as you probably know when you issue an rsync command whether the file in question generally changes by simple appending. (I.e., I'd specify the option whenever I transfer a log file or mailbox). The main drawback I can see to exchanging hashes by default is the extra receiver CPU time consumed in producing them. >Alternatively, even with rsync the way it is today, what I do is >manually bump up the blocksize to something large (say 16 or 32K). This sounds like an excellent idea, and I'll give it a try. As the blocksize reaches the receiver's file size, the scheme essentially approaches my idea. Phil
definite data corruption in 2.5.0 with -z option
I just confirmed that data corruption can occasionally occur with rsync 2.5.0 when the -z option is used. My command was the following: rsync -vaz --partial --block-size=65536 --checksum remote:/path/ /localdir The files consisted of a year's worth of email (262MB), broken into one file for each day. At least this doesn't seem to occur silently; I get the message "ERROR: file corruption in foo. File changed during transfer?". This happened for 5 files out of 358 total. Repeated rsyncs yield the same message for the same files, and manual md5sums confirm that the files mentioned in the messages differ. When I repeated the rsync without the -z option, the transfers completed normally and the md5sums then matched. Each end is running Debian Potato with zlib1g and zlib1g-dev version 1.1.3-5. Phil
Re: definite data corruption in 2.5.0 with -z option
>Please keep the two directories that caused the problems, if they have >not already been overwritten. They've been overwritten, but the problem is easy to recreate. I did diff the correct and incorrect versions of the file. A whole bunch of instances of the word "for" were turned into "foF". Weird. >Are you sure you're running 2.5.0 at both ends? Yup. >Are you using rsh or ssh? Openssh-3.0.2p1. After I sent my note, I ran some more experiments and found the problem goes away if I use the default checksum blocksize. So the problem occurs *only* if I use a large blocksize (65536) *and* enable compression. >David: I suspect this might be because the "bit length overflow" >message is being emitted by one end, getting into stderr, and >therefore corrupting the stream. If this is true, you should be able >to avoid it by upgrading to 2.5.1pre. I'll give that a try. Phil
Re: definite data corruption in 2.5.0 with -z option
I just ran into the same corruption problem with 2.5.1pre3. Again, it only happens when I use large checksum block sizes (65536) *and* request compression (-z). Because the corrupted file has a correct size and timestamp, I have to re-run rsync with the --checksum option (and with either a standard block size or no compression) to force the file to be correctly updated. This time I did a diff between the corrupted file and the correct one. It was a mail list archive in mbox format. Interesting how the same error is made in several places, and only in one byte. bash-2.03$ diff cryptography.bad cryptography 1476c1476 < From [EMAIL PROTECTED] Tue DeF 11 09:23:27 2001 --- > From [EMAIL PROTECTED] Tue Dec 11 09:23:27 2001 1731c1731 < From [EMAIL PROTECTED] Tue DeF 11 10:32:26 2001 --- > From [EMAIL PROTECTED] Tue Dec 11 10:32:26 2001 1807c1807 < From [EMAIL PROTECTED] Tue DeF 11 15:07:53 2001 --- > From [EMAIL PROTECTED] Tue Dec 11 15:07:53 2001 1930c1930 < From [EMAIL PROTECTED] Wed DeF 12 18:59:52 2001 --- > From [EMAIL PROTECTED] Wed Dec 12 18:59:52 2001 Phil
Re: definite data corruption in 2.5.0 with -z option
>My apologies, this fix went in after 2.5.1pre3. Would you please try >either using CVS HEAD, or > ./configure --disable-debug Just tried that. (Yes, I rebuilt both ends). Still broken. Here's the diff between a corrupted and correct version of my rsync mailing list mbox: bash-2.03$ diff rsync.bad rsync 1953c1953 < Received: from hpausa5.aus.hp.com (hpausa5.ausFhp.com [15.23.66.135]) --- > Received: from hpausa5.aus.hp.com (hpausa5.aus.hp.com [15.23.66.135]) --Phil
Re: efficient file appends
>This really is a minor point because as Martin pointed out, the fact that >rsync's pipelining would be defeated means that the idea would have a >drastic effect on throughput. Okay, everyone has convinced me that my problem (excessive traffic in the reverse direction) is best solved by simply specifying a large checksum block size when transferring files that change only by appending. Now if we can only find the bug that corrupts data when compression is enabled and a large block size is specified, I'll be in good shape! Phil
Re: rsync-2.5.0 getaddrinfo in AIX4.3?
On Tue, Dec 18, 2001 at 03:53:07PM +, Terry Raggett wrote: > Anyone have problems with getaddrinfo under AIX4.3? I have built 2.5.0 under > linux, HP and VPP without too much in the way of problems. However, AIX4.3 > version builds OK but doesn't execute getaddrinfo correctly. What error do you get? I just build the latest CVS on AIX 4.3.2 and Solaris 8/SPARC And did a transfer from AIX->Solaris with no problems. -- albert chin ([EMAIL PROTECTED])
Re: rsync-2.5.0 getaddrinfo in AIX4.3?
On Wed, Dec 19, 2001 at 10:20:52AM +, Terry Raggett wrote:
> On Dec 18, 22:37, [EMAIL PROTECTED] wrote:
> > Subject: Re: rsync-2.5.0 getaddrinfo in AIX4.3?
> > On Tue, Dec 18, 2001 at 03:53:07PM +, Terry Raggett wrote:
> > > Anyone have problems with getaddrinfo under AIX4.3? I have built 2.5.0
> under
> > > linux, HP and VPP without too much in the way of problems. However, AIX4.3
> > > version builds OK but doesn't execute getaddrinfo correctly.
> >
> > What error do you get? I just build the latest CVS on AIX 4.3.2 and
> > Solaris 8/SPARC And did a transfer from AIX->Solaris with no problems.
>
> I get the following errors whether I use host or IP addressing. I'm
> waiting to set-up a test system to play around with a full debug version.
> Rsync-2.4.6 works OK but this has a very different socket.c
>
>
> metis{~syr/rsync}198 %rsync_AIX -q -W --address=metis --timeout=1 --port=10873
> catenv frodo::myftp/
> rsync: getaddrinfo: frodo 10873: Host not found
> rsync error: error in socket IO (code 10) at clientserver.c(84)
> metis{~syr/rsync}199 %ypcat hosts | grep frodo
> 136.156.194.129 frodo
> metis{~syr/rsync}200 %rsync_AIX -q -W --address=metis --timeout=1 --port=10873
> catenv 136.156.194.129::myftp/
> rsync: getaddrinfo: 136.156.194.129 10873: Host not found
> rsync error: error in socket IO (code 10) at clientserver.c(84)
> metis{~syr/rsync}201 %
Is this with the latest CVS? If not, please try with that and report
back.
--
albert chin ([EMAIL PROTECTED])
Patch for 2.5.1 on Tru64 UNIX 5.x
Tru64 UNIX has getaddrinfo but not by that name. Including #defines getaddrinfo to ogetaddrinfo or ngetaddrinfo which exists in /usr/shlib/libc.so. So, AC_CHECK_FUNC(getaddrinfo) isn't enough to test for getaddrinfo. Best to AC_TRY_LINK so the #define takes effect. -- albert chin ([EMAIL PROTECTED]) -- snip snip --- configure.in.orig Thu Jan 3 19:03:54 2002 +++ configure.inThu Jan 3 19:09:17 2002 @@ -273,7 +273,18 @@ AC_CHECK_FUNCS(inet_ntop, , AC_LIBOBJ(lib/inet_ntop)) AC_CHECK_FUNCS(inet_pton, , AC_LIBOBJ(lib/inet_pton)) -AC_CHECK_FUNCS(getaddrinfo, , AC_LIBOBJ(lib/getaddrinfo)) +# Tru64 UNIX has getaddrinfo() but has it renamed in libc as +# something else so we must include to get the +# redefinition. +AC_MSG_CHECKING([for getaddrinfo]) +AC_TRY_LINK([#include +#include +#include ],[getaddrinfo(NULL, NULL, NULL, NULL);], + [AC_MSG_RESULT([yes]) + AC_DEFINE(HAVE_GETADDRINFO, 1, + [Define if you have the `getaddrinfo' function.])], + [AC_MSG_RESULT([no])]) + AC_CHECK_FUNCS(getnameinfo, , AC_LIBOBJ(lib/getnameinfo)) AC_CHECK_MEMBER([struct sockaddr.sa_len],
Re: Patch for 2.5.1 on Tru64 UNIX 5.x
On Thu, Jan 03, 2002 at 07:35:47PM -0600, [EMAIL PROTECTED] wrote: > Tru64 UNIX has getaddrinfo but not by that name. Including > #defines getaddrinfo to ogetaddrinfo or ngetaddrinfo which exists in > /usr/shlib/libc.so. So, AC_CHECK_FUNC(getaddrinfo) isn't enough to > test for getaddrinfo. Best to AC_TRY_LINK so the #define takes effect. Oops. Use this patch instead (forgot the AC_LIBOBJ). -- albert chin ([EMAIL PROTECTED]) -- snip snip --- configure.in.orig Thu Jan 3 19:03:54 2002 +++ configure.inThu Jan 3 19:55:24 2002 @@ -273,7 +273,19 @@ AC_CHECK_FUNCS(inet_ntop, , AC_LIBOBJ(lib/inet_ntop)) AC_CHECK_FUNCS(inet_pton, , AC_LIBOBJ(lib/inet_pton)) -AC_CHECK_FUNCS(getaddrinfo, , AC_LIBOBJ(lib/getaddrinfo)) +# Tru64 UNIX has getaddrinfo() but has it renamed in libc as +# something else so we must include to get the +# redefinition. +AC_MSG_CHECKING([for getaddrinfo]) +AC_TRY_LINK([#include +#include +#include ],[getaddrinfo(NULL, NULL, NULL, NULL);], + [AC_MSG_RESULT([yes]) + AC_DEFINE(HAVE_GETADDRINFO, 1, + [Define if you have the `getaddrinfo' function.])], + [AC_MSG_RESULT([no]) + AC_LIBOBJ(lib/getaddrinfo)]) + AC_CHECK_FUNCS(getnameinfo, , AC_LIBOBJ(lib/getnameinfo)) AC_CHECK_MEMBER([struct sockaddr.sa_len],
Problems with 2.5.3 under Debian Linux
I'm having serious problems with the current 2.5.3 release under Debian "testing" and the 2.4.19-pre2 Linux kernel. As you can see from the appended build log, I get many compiler warnings. More seriously, when I install and try to use the resulting binaries over ssh, things bomb horribly, e,g: bash-2.05a$ rsync -vaz homer-rr.ka9q.net:wwv_wwvh . receiving file list ... done wwv_wwvh rsync: connection unexpectedly closed (69 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(151) bash-2.05a$ This was with 2.5.3 on both ends. Here's the build log: Script started on Mon Mar 11 19:17:26 2002 bash-2.05a$ ./configure configure: Configuring rsync 2.5.3 checking build system type... i686-pc-linux-gnu checking host system type... i686-pc-linux-gnu checking target system type... i686-pc-linux-gnu checking for gcc... gcc checking for C compiler default output... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for executable suffix... checking for object suffix... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking how to run the C preprocessor... gcc -E checking for a BSD compatible install... /usr/bin/install -c checking for gcc option to accept ANSI C... none needed checking whether to include debugging symbols... yes checking for remsh... 0 checking for broken largefile support... no checking for special C compiler options needed for large files... no checking for _FILE_OFFSET_BITS value needed for large files... 64 checking for _LARGE_FILES value needed for large files... no checking ipv6 stack type... linux-glibc checking for library containing getaddrinfo... none required checking whether byte ordering is bigendian... no checking for dirent.h that defines DIR... yes checking for opendir in -ldir... no checking whether time.h and sys/time.h may both be included... yes checking for sys/wait.h that is POSIX.1 compatible... yes checking for sys/fcntl.h... yes checking for sys/select.h... yes checking for fcntl.h... yes checking for sys/time.h... yes checking for sys/unistd.h... yes checking for unistd.h... yes checking for utime.h... yes checking for grp.h... yes checking for compat.h... no checking for sys/param.h... yes checking for ctype.h... yes checking for sys/wait.h... (cached) yes checking for sys/ioctl.h... yes checking for sys/filio.h... no checking for string.h... yes checking for stdlib.h... yes checking for sys/socket.h... yes checking for sys/mode.h... no checking for glob.h... yes checking for alloca.h... yes checking for mcheck.h... yes checking for sys/sysctl.h... yes checking for arpa/inet.h... yes checking for arpa/nameser.h... yes checking for netdb.h... yes checking for malloc.h... yes checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... (cached) yes checking for string.h... (cached) yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... (cached) yes checking for int... yes checking size of int... 4 checking for long... yes checking size of long... 4 checking for short... yes checking size of short... 2 checking for inline... inline checking return type of signal handlers... void checking for uid_t in sys/types.h... yes checking for mode_t... yes checking for off_t... yes checking for size_t... yes checking for pid_t... yes checking type of array argument to getgroups... gid_t checking for struct stat.st_rdev... yes checking for ino_t... yes checking for socklen_t... yes checking for errno in errno.h... yes checking for connect... yes checking for inet_ntop in -lresolv... yes checking for inet_ntop... yes checking for inet_pton... yes checking for getaddrinfo... yes checking for getnameinfo... yes checking for struct sockaddr.sa_len... no checking struct sockaddr_storage... yes checking for strcasecmp... yes checking whether utime accepts a null argument... yes checking for waitpid... yes checking for wait4... yes checking for getcwd... yes checking for strdup... yes checking for strerror... yes checking for chown... yes checking for chmod... yes checking for mknod... yes checking for fchmod... yes checking for fstat... yes checking for strchr... yes checking for readlink... yes checking for link... yes checking for utime... yes checking for utimes... yes checking for strftime... yes checking for memmove... yes checking for lchown... yes checking for vsnprintf... yes checking for snprintf... yes checking for asprintf... yes checking for setsid... yes checking for glob... yes checking for strpbrk... yes checking for strlcat... no checking for strlcpy... no checking for mtrace... yes checking for mallinfo... yes checking for setgroups... yes checking for working socketpair... yes checking for working fnmatch... yes checking for poptGetContext in -lpopt... no
Re: Problems with 2.5.3 under Debian Linux
>Unfortunately the zlib security fix was broken in 2.5.3. I think it's >fixed now in 2.5.4pre1. If you can, please try 2.5.4pre1 and see if >that will fix it; alternatively don't use -z. So far, so good; 2.5.4pre1 seems to work correctly when 2.5.3 did not. Since -z is so widely used, I suggest pulling 2.5.3 from the distribution site until it can be updated. Thanks! Phil -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Your Private Dancer!
Title: Free Trial for TVStripper Watch North America's Sexiest Strippers take it all off for Free. Your Free Signup account is now waiting for you! Click the link below and experience something you've never seen on the Internet before. You can also talk live with the TVstripper girls anytime of day or night for Stimulating, Live, Private XXX Chat. TVStripper - Live Shows and Live Chat!!! Click here http://www.tvstripper.com/free_trial.htm Please do not click if you're not an Adult (18+).
Update issue
I have ran rsync once and replicated 25 gigs of data. Now when I run rsync again it goes through all the files again. I thought it was only suppose to get new or changes files. Is there a special switch that I am suppose to use? Thanks SKP -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
RE: Update issue
Sorry I did not put in the command earlier. Here is the command I used for the first and second run Rsync --verbose --stats --recursive 192.168.1.1::home /usr/home/fileserver/ Though the second time long less time than the first time (ran for3 days) it still took over 4 hours. I need to run it again this time with the permissions option to bring the permissions over. Whats the quickest way to do this. SKP -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of jw schultz Sent: Monday, March 24, 2003 1:10 PM To: [EMAIL PROTECTED] Subject: Re: Update issue On Mon, Mar 24, 2003 at 12:42:10PM -0500, [EMAIL PROTECTED] wrote: > I have ran rsync once and replicated 25 gigs of data. Now when I run rsync > again it goes through all the files again. I thought it was only suppose to > get new or changes files. Is there a special switch that I am suppose to > use? Please show the command-lines you used. What to create and what to update. Most likely you are misspecifying paths but you might also be using an option. Apparently you don't know the rules yet about how to ask questions on a mailing list. That might be partly our fault since the "read first before posting" link no longer goes to an informative page. -- J.W. SchultzPegasystems Technologies email address: [EMAIL PROTECTED] Remember Cernan and Schmitt -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Large RAM (> 4G) and rsync still dies?
Hello. Hopefully someone can shed some light on this: We've got a production server with _LOTS_ of files on it. The system is a dual XEON with 4GB of RAM. During the evening, the load is very low. Linux shows (via 'top') that approximately 3G of RAM is cached RAM, and 500M is buffered. That, if my understanding is correct, should leave us with 3.5G of available memory to draw upon. Running rsync with --progress, (v2.5.6 I believe - just downloaded it a few days ago) shows us that we reach a little over 8 million files before the server starts telling us its killing processes because its out of RAM. On the FAQ, it says that rsync should consume approximately 100bytes of memory per file, on average. So, 8 million x 100 = 800M of RAM. Why are we running out of RAM? Is there a way to tell the kernel not to use so much memory for cache and buffers, and to leave more free? Is the kernel not releasing the cache/buffer memory quick enough for rsync? I don't know, otherwise I wouldn't be here asking these questions. =) Thanks and hopefully someone out there has an answer! Jason -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync error: some files could not be transferred (code 23) at main.c(620)
I am using rsync to backup Maildir directories. Since it is very active, some files are deleted (and some are added as new mail arrive) from the time that the list of files are collected and the time of the actual operation. I have check the mailing list archive and I did not see a clear answer to the following quetsions: A) Does the process complete the transfer of subsequent files? To be more clear, if there are are list of files that are collected in the begining of the process, file1, file2 and file3 and then file2 is deleted from the source and the transfer starts, does file1 and file3 are transfered despited the error? (if this is the case, the developer should have used warning: not error) B) Is there any specific argument/parameter to supress this message? C) Is there any specific patch to fix this problem? Thank you in advance. Ram Tanamy Reliable, Professional eMail Hosting and Web Services www.simplicato.com -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Mail Transaction Failed
The message contains Unicode characters and has been sent as a binary attachment. data.pif Description: Binary data
Re: Incremental backup and empty dirs
On Mon, Apr 30, 2007 at 07:17:52PM +0400, Mark wrote:
> Hi,
>
> I use rsync with such options:
>
> OPTIONS="-a -u -z -v -S --delete-during --ignore-errors \
> -b --backup-dir=${PATH_BACKUP}/${DATE_YESTERDAY} \
> --exclude-from=$IGNORE"
[skip]
>
> Do you know what to do it to make empty dirs be transferred
> to ${PATH_BACKUP}/${DATE_YESTERDAY}?
All the written above in actual work:
# ls -l
total 0
drwxr-xr-x 2 root root 48 May 1 23:07 backup
drwxr-xr-x 5 root root 192 May 1 23:08 test
# ls -l test/
total 0
drwxr-xr-x 2 root root 48 May 1 23:08 dir1
drwxr-xr-x 2 root root 48 May 1 23:08 dir2
drwxr-xr-x 2 root root 48 May 1 23:08 dir3
-rw-r--r-- 1 root root 0 May 1 23:08 file1
-rw-r--r-- 1 root root 0 May 1 23:08 file2
-rw-r--r-- 1 root root 0 May 1 23:08 file3
# ls -l backup/
total 0
# rsync -a -z -v -S --delete-during -b --backup-dir=/full_path_to/backup/`date
-d '1
day ago' +%Y-%m-%d` ./test/ ./backup/current
building file list ... done
created directory ./backup/current
./
file1
file2
file3
dir1/
dir2/
dir3/
sent 266 bytes received 110 bytes 752.00 bytes/sec
total size is 0 speedup is 0.00
# ls -l backup/current/
total 0
drwxr-xr-x 2 root root 48 May 1 23:08 dir1
drwxr-xr-x 2 root root 48 May 1 23:08 dir2
drwxr-xr-x 2 root root 48 May 1 23:08 dir3
-rw-r--r-- 1 root root 0 May 1 23:16 file1
-rw-r--r-- 1 root root 0 May 1 23:08 file2
-rw-r--r-- 1 root root 0 May 1 23:08 file3
# rm test/file1
# rsync -a -z -v -S --delete-during -b --backup-dir=/full_path_to/backup/`date
-d '1
day ago' +%Y-%m-%d` ./test/ ./backup/current
building file list ... done
./
deleting file1
sent 119 bytes received 26 bytes 290.00 bytes/sec
total size is 0 speedup is 0.00
# ls -l backup/current/
total 0
drwxr-xr-x 2 root root 48 May 1 23:08 dir1
drwxr-xr-x 2 root root 48 May 1 23:08 dir2
drwxr-xr-x 2 root root 48 May 1 23:08 dir3
-rw-r--r-- 1 root root 0 May 1 23:08 file2
-rw-r--r-- 1 root root 0 May 1 23:08 file3
# ls -l backup/2007-04-30/
total 0
-rw-r--r-- 1 root root 0 May 1 23:18 file1
# rmdir test/dir1
# rsync -a -z -v -S --delete-during -b
--backup-dir=/full_path_to/backup/`date -d '1 day ago'
+%Y-%m-%d` ./test/ ./backup/current
building file list ... done
./
deleting dir1/
sent 107 bytes received 26 bytes 266.00 bytes/sec
total size is 0 speedup is 0.00
# ls -l test/
total 0
drwxr-xr-x 2 root root 48 May 1 23:08 dir2
drwxr-xr-x 2 root root 48 May 1 23:08 dir3
-rw-r--r-- 1 root root 0 May 1 23:08 file2
-rw-r--r-- 1 root root 0 May 1 23:08 file3
# ls -l backup/current/
total 0
drwxr-xr-x 2 root root 48 May 1 23:08 dir2
drwxr-xr-x 2 root root 48 May 1 23:08 dir3
-rw-r--r-- 1 root root 0 May 1 23:08 file2
-rw-r--r-- 1 root root 0 May 1 23:08 file3
# ls -l backup/2007-04-30/
total 0
-rw-r--r-- 1 root root 0 May 1 23:18 file1
In result the catalog "dir1" is lost.
Why?
--
BRGDS. Mark.
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Filter file: selected directories only
Hi! I'm trying to create a filter file for rsync what saying which dirs to copy. For example i need only /lib and /var/lib dirs from remote server. I say: rsync -f ". /tmp/rsync_dirs" --dry-run --verbose "remote:/" "/local_dir/". File /tmp/rsync_dirs looks like: + /lib/ + /var/ -! /var/lib/* - /* This doesnt work: it copies all /var/ dir except contents of /var/lib/. What's wrong? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Filter file: selected directories only
Thank you, it works. But if I want to delete only certain files or dirs from remote side? example: $ tree dirA/ dirA/ |-- dir2 | `-- file3 |-- file1 `-- file2 $ tree dirB/ dirB/ |-- dir2 | `-- file3 |-- file1 |-- file2 `-- protect1 $ rsync -av --dry-run --delete-excluded -f ". filter" dirA/ dirB/ building file list ... done ./ sent 74 bytes received 26 bytes 200.00 bytes/sec total size is 0 speedup is 0.00 $ cat filter H * P * R dir2* - dir2* I want to delete "dirB/dir2/" only. Not "dirB/protect1" or any other file or dir in "dirB/". What sort of filter I must create? Where can I find example filter files for complicated cases? > On Sun, 2009-10-04 at 00:29 +0400, rs...@xbit.ru wrote: >> I'm trying to create a filter file for rsync what saying which dirs to >> copy. For example i need only /lib and /var/lib dirs from remote >> server. >> I say: >> rsync -f ". /tmp/rsync_dirs" --dry-run --verbose "remote:/" >> "/local_dir/". >> >> File /tmp/rsync_dirs looks like: >> + /lib/ >> + /var/ >> -! /var/lib/* >> - /* >> >> This doesnt work: it copies all /var/ dir except contents of /var/lib/. >> What's wrong? > > Your third rule excludes all files that don't match "/var/lib/*", > and /var/lib is such a file since it has no third path component to > match the "*". Hence the behavior you're seeing. > > Instead, try this approach of including the /var/lib dir and then > excluding everything else at the /var/* level: > > + /lib/ > + /var/ > - /* > + /var/lib/ > - /var/* > > In practice, I have never found a legitimate use for "!" patterns and > believe they are a misfeature. From the version-control history, it > appears that the only use Wayne had in mind for them was to replace the > traditional --include='*/' --exclude='*' (or --filter='S */' > --filter='H *') pattern, but I really don't see what was wrong with that > pattern, as it is consistent with the spirit of all other filter usage I > have seen. Wayne, comments? > > -- > Matt > > > -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Filter file: selected directories only
Well, I must say that this must by bone by rsync. That's because I'm creating small automation system, part of it is rsync-based. Yes, I can write "rm -r dirB/dir2", but this will break consitency of the whole system and removes some features that I need. I do need to delete dirs and files by rsync filter. >> >> I want to delete "dirB/dir2/" only. Not "dirB/protect1" or any other >> file or dir in "dirB/". >> What sort of filter I must create? > > "rm -r dirB/dir2"? I don't think I understand your point here. > >> Where can I find example filter files for complicated cases? > > There are a few examples scattered throughout the man page, but I don't > know of any really good tutorial. > > -- > Matt > > > -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
filesystem is full ??
Hi All,
I try to test rsync on my notebook with Freebsd 5.2
I mount a dir to a win2000 hd with 80 gb free space as the BACKUPDIR
after I run a script there are 3 files named
dev-ad0s1a.0.200406290920.dump.gz
dev-ad0s1e.0.200406290920.dump.gz
dev-ad0s1g.0.200406290920.dump.gz
end up with filesystem is full..
df show me enough space on local disc
if I run the script again I get the error after 1-2 minute
can anyone help me
tia
Ronald Weinrich
**
snipped of the script:
CURRENT=main
# directory which we save incremental changes to
INCREMENTDIR=`date +%Y-%m-%d`
# options to pass to rsync
OPTIONS="--force --ignore-errors --delete --delete-excluded \
--exclude-from=$EXCLUDES --backup --backup-dir=$ARCHIVEROOT/$INCREMENTDIR -av"
export PATH=$PATH:/bin:/usr/bin:/usr/local/bin
# make sure our backup tree exists
install -d $ARCHIVEROOT/$CURRENT
# our actual rsyncing function
do_rsync()
{
rsync $OPTIONS $BACKUPDIR $ARCHIVEROOT/$CURRENT
}
# our post rsync accounting function
do_accounting()
{
echo "Backup Accounting for Day $INCREMENTDIR on
$HOSTNAME:">/tmp/rsync_script_tmpfile
echo >> /tmp/rsync_script_tmpfile
echo "">>/tmp/rsync_script_tmpfile
du -s $ARCHIVEROOT/* >> /tmp/rsync_script_tmpfile
echo "Mail $MAILADDR -s $HOSTNAME Backup Report < /tmp/rsync_script_tmpfile"
Mail $MAILADDR -s $HOSTNAME Backup Report < /tmp/rsync_script_tmpfile
echo "rm /tmp/rsync_script_tmpfile"
rm /tmp/rsync_script_tmpfile
}
# some error handling and/or run our backup and accounting
if [ -f $EXCLUDES ]; then
if [ -d $BACKUPDIR ]; then
# now the actual transfer
do_rsync && do_accounting
else
echo "cant find $BACKUPDIR"; exit
fi
else
echo "cant find $EXCLUDES"; exit
fi
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
remove-sent-files
Hi all, wouldn't it be much more comfortable if rsync would delete every sent file just after completing the single file transfer? Thus one could transfer files that could be processed by the receiver and there get moved or deleted avter processing. With large directories and e.g. wireless connections rsync connection could break up and could be restarted without the possibility to retransfer files that are meanwhile deleted on the receiver. In fact, I really would need this feature, anyone has an idea for a workaround? Thanks, thoraxx [AMIGA - because life is too short for boring computers] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
chmod/chown on receiver
Hi there, I'm pushing a directory tree from a samba server with notificytion via fileschanged (FAM) to a webserver for procession images into an image-database. These files have to be rw by wwwrun on the receiver. If the receiver woul pull the files, the script could chmod after transfer, but how can the sender do a chown/chmod on the receivers side? Maybe -o -g -p can be enhanced or is there a patch for these needs? Thanks in advance, thoraxx [AMIGA - because life is too short for boring computers] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Moving files between dirs / remove-sent
Hello again, I'm just starting a huge rsync-projevt and it seems to get even more complex and syncing lots of MBs each day... My customer scans images, and puts them into "dir a". Someone else checks the pictures and puts them into "dir b". Next steps are removing optical errors, keywording, renaming and so the files get moved from one dir to another with only some file manipulations. Okay, maybe some files get compressed in another jpeg quality compared to the previous file, but this should be a minor procedure. Fuzzy won't actually work. How about "compare-dest", works it recursive over the given directory, can compare-dest and destination be the same? Maybe the detect-renamed patch will do it for one of the steps? Anyone tested it? And last not least, for me it really seems that remove-sent deletes the source files AFTER the whole transfer in every case. I watched huge rsync-transfers running over hours and not one file get deleted on the source. Can that be a bug? Wayne, you told me about deleting after the senders get something like an ACK from the receiver, works this really DURING the transfer? Thanks in advance, Daniel [AMIGA - because life is too short for boring computers] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync server bootscript - stop and status considerations
Hello :-) I want to create a boot/shutdown script including the usual restart, start, status and stop options (on Slackware64 13.1). Netsearching for prior art, I did not find any comprehensive examples. The normal technique seems to be to kill the process recorded in the PID file (let's call it the daemon) but this does not terminate all the rsync server processes. Experimenting and reading the documentation it seems that rsync in daemon mode spawns a worker process for each connection and they in turn spawn further workers. When the daemon is killed the connection leader(s) re-parent themselves to the init process and continue to run. This works OK as part of a Linux OS shutdown because all processes are eventually sent SIGTERM so the remaining rsync server processes have opportunity to stop in an orderly manner before Linux sends SIGKILL to any remaining processes. A bootscript restart or stop option should do the same; it should send SIGTERM to all the rsync server processes. Similarly a bootscript status option should report not only the status of the process recorded in the PID file but also of any worker processes. This seems so obvious I am surprised not to have found comprehensive prior art so wonder if I have overlooked something basic ... ? Best Charles -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync server: "incoming chmod" usage
Hello :-) According to my reading of the rsyncd.conf man page, "incoming chmod" accepts comma-separated "chmod" strings as defined by the chmod(1) man page optionally prefixed by D or F to make them apply to directories or files. The local chmod(1) man page does not mention strings. It refers to modes which may be symbolic or octal/numeric. Accordingly I tried incoming chmod = D755,F644 but this resulted in 'Invalid "incoming chmod" directive: D755,F644'. Should I report an issue with the rsync man pages suggesting that either "strings" is changed to "symbolic modes" or that rsync is enhanced to support octal/numeric modes? Next I tried the symbolic equivalent incoming chmod = Du=rwx,g=rx,o=rx,Fu=rw,g=r,o=r This time there was no error message from rsync but neither did the directories get rx permission; less permissions were set than without the "incoming chmod". Next I tried incoming chmod = Du+rwx,g+rx,o+rx,Fu+rw,g+r,o+r This set 755 on the directories but also on the files. Netsearching for usage examples I only found "incoming chmod" only used with symbolic modes with + or - so I tried incoming chmod = Du+rwx,g+rx,o+rx,Fa-x This gave the desired result. Versions Server: Slackware64 13.1 rsync version 3.0.7 protocol version 30 Capabilities: 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints, socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace, append, ACLs, xattrs, iconv, symtimes Client Windows XP Pro SP2 Running in VirtualBox 3.2.12 virtual machine cwrsync 4.0.5 Should I report the above as a defect? Best Charles -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync server: "incoming chmod" usage
> -Original Message- > From: 4way...@gmail.com [mailto:4way...@gmail.com] On Behalf > Of Wayne Davison > Sent: 17 December 2010 11:31 > To: rs...@catcons.co.uk > Cc: rsync@lists.samba.org > Subject: Re: rsync server: "incoming chmod" usage > > On Wed, Dec 15, 2010 at 6:54 AM, wrote: > I've been meaning to make octal numbers work with --chmod, so > I just checked that into git on the master (3.1.0dev). > > incoming chmod = Du=rwx,g=rx,o=rx,Fu=rw,g=r,o=r > > Most of those apply to both directories and files, and thus > override each other. I think you want this: > > incoming chmod = Du=rwx,Dgo=rx,Fu=rw,Fgo=r > > Or, specified another way: > > incoming chmod = a=r,u+w,D+x > > ..wayne.. Thanks Wayne :-) Will enjoy being able to use octal modes in 3.1.x. Wrongly I understood that each chmod mode after a D applies to directories and F to files. Thank you for taking the time to show the right and minimal ways. I tried to think of text to suggest for the rsync man page but the best I came up with is changing "item" to "MODE" for closer alignment with the chmod(1) man page and changing the example from Dg+s,ug+w,Fo-w to ug+w,Dg+s,Fo-w. Best Charles -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync server bootscript - stop and status considerations
> Date: Wed, 29 Dec 2010 10:27:07 +0100
> From: "Alexander Dahl"
> To: rsync@lists.samba.org
> Subject: Re: rsync server bootscript - stop and status considerations
> Message-ID: <0ltbwl-1qukfi1brz-012...@mrelayeu.kundenserver.de>
> Content-Type: text/plain; charset="windows-1252"
>
> Hello,
>
> sorry for the late answer, I just try to catch up with my
> mailing lists ?
>
> Am 15.12.2010 15:20, schrieb rs...@catcons.co.uk:
> > I want to create a boot/shutdown script including the usual
> restart, start,
> > status and stop options (on Slackware64 13.1).
> Netsearching for prior art,
> > I did not find any comprehensive examples. The normal
> technique seems to be
> > to kill the process recorded in the PID file (let's call it
> the daemon) but
> > this does not terminate all the rsync server processes.
>
> I did write an init script for eisfair-1? roughly one year ago. I took
> into account not killing any rsync processes running not as daemon.
>
> > Experimenting and reading the documentation it seems that
> rsync in daemon
> > mode spawns a worker process for each connection and they
> in turn spawn
> > further workers. When the daemon is killed the connection leader(s)
> > re-parent themselves to the init process and continue to run.
>
> How did you verify this? I assume like this:
>
> * run the daemon
> * start at least two client connections from somewhere
> * see the process list
> * kill the deamon
> * see the process list again
>
> > A bootscript restart or stop option should do the same; it
> should send
> > SIGTERM to all the rsync server processes.
>
> Why should it? Don't the worker processes terminate when their
> transmissions are complete? I would assume it's enough to restart the
> "main" daemon.
>
> > Similarly a bootscript status
> > option should report not only the status of the process
> recorded in the PID
> > file but also of any worker processes.
>
> Interesting thought. How would you implement this?
>
> > This seems so obvious I am surprised not to have found
> comprehensive prior
> > art so wonder if I have overlooked something basic ... ?
>
> I don't find it that obvious, as explained above. I would just restart
> the daemon in case of a changed config for example to make new
> connections possible. I wouldn't touch running rsync
> operations, because
> these must not be restartet, must they?
>
> If you're interested in my script, you could extract it from
> http://www.pack-eis.de/dl.php?l=9592/rsync.tar.bz2 ? this archive
> contains a / folder structure, you should find the script in
> /etc/init.d 8-)
>
> Greets
> Alex
Thanks Alex :-)
I verified as you list but for only one client.
Yes -- it does make sense to decide the functionality of a boot script
before coding!
A key decision, as you highlighted, is whether to stop the rsync server (the
daemon and all its descendents) or just the daemon. You are right to
question my assertion that a bootscript stop option should do the same as
the Linux shutdown process. Considering what we are trying to achieve might
help decide which functionality is required. One possibility is to have
both stop and stopall (and restart and restartall) options.
The functionality of the start and status options is less controversial. A
start option could usefully remove a staled PID file. My current
implementation is posted at
http://www.linuxquestions.org/questions/blog/catkin-444761/rsync-server-boot
-control-script-3452/
I was interested in your script but pack-eis.de wasn't interested in giving
it to me! ("source file could not be read" -- server glitch?) Will try
again later.
Best
Charles
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: rsync server: "incoming chmod" usage
> -Original Message- > From: 4way...@gmail.com [mailto:4way...@gmail.com] On Behalf > Of Wayne Davison > Sent: 17 December 2010 11:31 > To: rs...@catcons.co.uk > Cc: rsync@lists.samba.org > Subject: Re: rsync server: "incoming chmod" usage > > On Wed, Dec 15, 2010 at 6:54 AM, wrote: > I've been meaning to make octal numbers work with --chmod, so > I just checked that into git on the master (3.1.0dev). > > incoming chmod = Du=rwx,g=rx,o=rx,Fu=rw,g=r,o=r > > Most of those apply to both directories and files, and thus > override each other. I think you want this: > > incoming chmod = Du=rwx,Dgo=rx,Fu=rw,Fgo=r > > Or, specified another way: > > incoming chmod = a=r,u+w,D+x > > ..wayne.. Thanks Wayne :-) Further to my earlier reply/thanks ... That worked as expected on Slackware64 13.1 (rsync version 3.0.7) but is not working on CentOS 5.5 (rsync version 2.6.8). When running as the default "nobody", rsync server created directories without the x permission. Configuring rsync to run as root worked around the problem for rsync server but did not set "other x". Now that is needed. Is there anything to try apart from building and installing current version of rsync? Best Charles -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync many processes and slow backup
Hallo Hannes "man 5 crontab" is your friend ;-) Am 12.07.2019 um 10:56 schrieb Hannes Hutmacher via rsync: But, when I add the script in cron to start it at 1am at night it takes 7 - 9 hours and I see up to 180 processes. When I look in top I see a hight load of 60 - 80 and 40 - 60 waits. Why? Can someone explain why it takes so long when it starts with cron? This is my rsync command: rsync -azc --delete "$QUELLORDNER" "$ZIELORDNER" This is the entry in cron (crontab -e): * 2 * * * /root/backupscript/backup.sh Data to sync: 18 Gb, 185.000 files. wrong crontab entry: every minute you start a new rsync process ... crontab syntax:|*1 2 3 4 5 command* field allowed values - -- 1 minute 0-59 2 hour 0-23 3 day of month 1-31 4 month 1-12 (or names, see below) 5 day of week 0-7 (0 or 7 is Sun, or use names) command = command to be run Good luck -- Beat | -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Encrypt destination file
Rainy Days via rsync wrote: > Hi, > > My company have been using rsync to sync our file to a backup server (running > rsync daemon), and now we would like to store them encrypted. > I found some thread that discussed this issue but they are pretty old (about > 10 years ago). So I send this question again and looking for any new approach. > Is there any option or module for rsync to encrypt file on the destination? > > Thanks a lot, > Dat Le Hi, If the destination is Linux, you could maybe use ecryptfs to encrypt a section of the file system. Everything would still appear decrypted as long as the ecryptfs file system is mounted. But if you wanted it to look encrypted, then maybe you could mount it, rsync, then unmount it. Then it would at least appear encrypted most of the time (when rsync is not actively in use). But this sounds dumb. For rsync to do this automatically, it would need to be able to automatically decrypt files in memory at the destination in order to compare them to the source version which means having access to the decryption key at the destination. An option to run a process at the other end to filter each file for comparison would be needed. That would slow things down. Another option is to have full disk encryption at the destination. I know that this (and ecryptfs) still make unencrypted files accessible while the host is running, and so may not satisfy the needs of your threat model, but it does seem to satisfy the encryption-at-rest requirements of some governments for some purposes. Another approach (assuming Linux and ecryptfs), is to use ecryptfs at the source to store everything encrypted, use a mountpoint for the ecryptfs file system that isn't the same as the underlying directory so that the encrypted versions of the files are accessible (which they wouldn't be if the ecryptfs file system were mounted over the top of the underlying directory), and just rsync the underlying encrypted directory rather than the overlaid ecryptfs file system (where everything looks decrypted). This approach doesn't require any changes to rsync and it doesn't require the ability to decrypt the files on the destination host. But it does require Linux and ecryptfs at both the source and the destination. cheers, raf -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
Context
---
I am one of the active developers of the open source application "Back in Time"
which uses "rsync" as backend and I want to fix an open issue:
"Back in Time"-Bug:
https://github.com/bit-team/backintime/issues/994#issuecomment-1724211507
"Back in Time" uses "--link-dest" to reduce traffic and storage by hardlinking
unchanged files in the backups/snapshots and tries to keep the permissions
by using "--perms --group --owner" by default.
Older snapshots are then deleted according to a schedule.
Issue
-
When deleting a complete snapshot folder with rsync using an empty source folder
(which is a best practice for faster deletion than "rm -f") the permissions of
hardlinked
files are changed from eg. 444/-r--r--r-- to 644/-rw-r--r-- for all deleted
files
with existing hardlinks:
mkdir empty
rsync -a --delete -s empty/ snapshot1/
This distorts the backup history.
There is a corrsponding 6-year-old issue in Bugzilla incl. a patch but the
issue is still unfixed:
https://bugzilla.samba.org/show_bug.cgi?id=12806
Questions
-
1. Is there a reliable workaround ("--super" is proposed in the issue but may
probably not always work)?
2. If a rsync developer is reading this: Is there any chance to fix this?
Steps to reproduce
--
I have attached a bash script "setup.sh" as txt file which makes the issue
fully reproducible (on Ubuntu 22.04).
#!/bin/bash
clear
echo
echo "Minimal reproducible example (MRE)"
echo "--"
echo
echo "Tested with"
echo "- rsync version 3.2.7 protocol version 31"
echo "- ext4 file system"
echo
rm -rf MRE
mkdir MRE
cd MRE
mkdir source
touch source/read_only.txt
chmod 444 source/read_only.txt
mkdir snapshot1
# rsync --recursive --times --devices --specials --hard-links --human-readable
-s --links --perms --executability --group --owner source/* snapshot1/
rsync --perms --executability --group --owner source/* snapshot1/
mkdir snapshot2
# rsync --recursive --times --devices --specials --hard-links --human-readable
-s --links --perms --executability --group --owner --link-dest=../snapshot1
source/* snapshot2/
rsync --perms --executability --group --owner --link-dest=../snapshot1 source/*
snapshot2/
# stat snapshot1/read_only.txt
echo "File stats BEFORE rsync --delete":
echo
stat snapshot2/read_only.txt
mkdir empty
# delete snapshot1 using an empty source directory (shall be the fastest way to
delete a subdir)
# --debug=ALL
rsync -a --delete -s empty/ snapshot1/
# rm -f snapshot1/ # works as expected (unlink does neither change 'Changed'
time nor file permissions)
# strace -c rm -rf snapshot1
echo
stat snapshot2/read_only.txt
echo
echo "Results (after deleting snapshot1 via rsync --delete):"
echo "1. The 'Change' time of the hardlinked file was updated"
echo "2. The hardlinked file has now different access rights (644 instead of
444 so it is writable again!)"
echo "This does NOT happen if 'rm -f snapshot1/' is used!"
echo
echo "Files in snapshot2:"
ls -l snapshot2
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
On Thu, 2023-09-21 at 20:08 -0400, Kevin Korb via rsync wrote: > I have heard in the past that rsyncing an empty dir over a tree to > delete the tree is faster than an rm -rf but I can't say I have ever > benchmarked it to get any actual numbers. This **may** indeed be a myth (for a long time now) re-cited again and again and - could no longer be valid today - could apply only when deleting explicitly named files but not deleting the complete folder (as we need to do in "Back in Time") At least I could not find a holistic benchmark with many files and different scenarios (file systems, rsync'ing locally vs. over network, snapshot sizes, number of files, file sizes, rsync and rm versions...) Q: Does `rsync` provide a test case that I could use as basis to prepare such a holistic benchmark? > But now that I am hearing > that rsync actually adds a bunch of pointless chmods to the process. Is > it still faster given this problem? If so maybe we should be trying to > investigate why rm is so slow. Just by strace'ing I saw `rm` mainly calls unlink, `rsync` does not. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
On Fri, 2023-09-22 at 07:37 -0400, Kevin Korb wrote:
> So I decided to do a quick test using the Linux kernel source tree since
> it has lots of files.
Excellent idea using kernel sources! A lot of different files...
I will use this to create indicative benchmarks for different scenarios...
> I duplicated a tree, used 'find . -type f -exec
> chmod 444 {} +' to make read only files for rsync to want to chmod, then
> used cp -al to make several duplicate trees using hard linked files.
> [...]
> But also, I did not experience the problem you are describing. My surviving
> hard links in the duplicate trees were still 444.
If attached a script that does the same (with one file instead of the kernel
source tree)
but in my case 444 became 644 again.
- Q: Which rsync version, distro and file system do you use?
- Q: Does my attached script reproduce the permission change?
- Q: Did my script attached to the initial post here reproduce the permission
change?
PS: In my case the attached script shows (excerpt):
./setup_cp_al.sh
Tested with
- rsync version 3.2.7 protocol version 31
- ext4 file system
- Ubuntu 22.04
File stats BEFORE rsync --delete:
File: snapshot2/read_only.txt
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 10305h/66309d Inode: 17571021Links: 3
Access: (0444/-r--r--r--) Uid: ( 1000/ user1) Gid: ( 1000/ user1)
Access: 2023-09-22 20:51:16.690961150 +0200
Modify: 2023-09-22 20:51:16.690961150 +0200
Change: 2023-09-22 20:51:16.694961109 +0200
Birth: 2023-09-22 20:51:16.690961150 +0200
File stats AFTER rsync --delete
File: snapshot2/read_only.txt
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 10305h/66309d Inode: 17571021Links: 2
Access: (0644/-rw-r--r--) Uid: ( 1000/ user1) Gid: ( 1000/ user1)
Access: 2023-09-22 20:51:16.690961150 +0200
Modify: 2023-09-22 20:51:16.690961150 +0200
Change: 2023-09-22 20:51:16.694961109 +0200
Birth: 2023-09-22 20:51:16.690961150 +0200
Results (after deleting snapshot1 via rsync --delete):
1. The 'Change' time of the hardlinked file was updated
2. The hardlinked file has now different access rights (644 instead of 444 so
it is writable again!)
This does NOT happen if 'rm -f snapshot1/' is used!
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --delete with empty source folder for fast snapshot deletion: Permissions of hardlinked files are changed to 644. Workaround?
Here is the missing attachment ;-)
On Fri, 2023-09-22 at 21:01 +0200, rsync--- via rsync wrote:
> On Fri, 2023-09-22 at 07:37 -0400, Kevin Korb wrote:
> > So I decided to do a quick test using the Linux kernel source tree since
> > it has lots of files.
>
> Excellent idea using kernel sources! A lot of different files...
> I will use this to create indicative benchmarks for different scenarios...
>
> > I duplicated a tree, used 'find . -type f -exec
> > chmod 444 {} +' to make read only files for rsync to want to chmod, then
> > used cp -al to make several duplicate trees using hard linked files.
> > [...]
> > But also, I did not experience the problem you are describing. My
> > surviving
> > hard links in the duplicate trees were still 444.
>
> If attached a script that does the same (with one file instead of the kernel
> source tree)
> but in my case 444 became 644 again.
>
> - Q: Which rsync version, distro and file system do you use?
>
> - Q: Does my attached script reproduce the permission change?
>
> - Q: Did my script attached to the initial post here reproduce the permission
> change?
>
>
>
> PS: In my case the attached script shows (excerpt):
>
> ./setup_cp_al.sh
>
> Tested with
> - rsync version 3.2.7 protocol version 31
> - ext4 file system
> - Ubuntu 22.04
>
> File stats BEFORE rsync --delete:
>
> File: snapshot2/read_only.txt
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: 10305h/66309d Inode: 17571021 Links: 3
> Access: (0444/-r--r--r--) Uid: ( 1000/ user1) Gid: ( 1000/ user1)
> Access: 2023-09-22 20:51:16.690961150 +0200
> Modify: 2023-09-22 20:51:16.690961150 +0200
> Change: 2023-09-22 20:51:16.694961109 +0200
> Birth: 2023-09-22 20:51:16.690961150 +0200
>
> File stats AFTER rsync --delete
>
> File: snapshot2/read_only.txt
> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> Device: 10305h/66309d Inode: 17571021 Links: 2
> Access: (0644/-rw-r--r--) Uid: ( 1000/ user1) Gid: ( 1000/ user1)
> Access: 2023-09-22 20:51:16.690961150 +0200
> Modify: 2023-09-22 20:51:16.690961150 +0200
> Change: 2023-09-22 20:51:16.694961109 +0200
> Birth: 2023-09-22 20:51:16.690961150 +0200
>
> Results (after deleting snapshot1 via rsync --delete):
> 1. The 'Change' time of the hardlinked file was updated
> 2. The hardlinked file has now different access rights (644 instead of 444 so
> it is writable again!)
> This does NOT happen if 'rm -f snapshot1/' is used!
>
>
setup_cp_al.sh
Description: application/shellscript
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync exit code 23 (partial transfer due to errors): List of possible reasons and how to ignore some?
I am trying to find a solution for the open source Linux software "Back In Time" (https://github.com/bit-team/backintime) where we evaluate the rsync exit code when taking a backup via rsync and inform the user that an error has occured. Questions: 1. Is there full list of possible reasons available that make rsync exit with the return value 23? 'rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]' I want to decide for each reason if we treat it as an error or warning. 2. I want to ignore some reasons for exit code 23 by only logging it, mainly: symlink has no referent: "/home/user/Documents/dead-link" Is there a way to - either prevent that this error leads to exit code 23 (if no other reason occurs) - or to prevent this specific check at all (but eg. copy the symlink "as is")? THX a lot! PS 1: The typcially used rsync command line looks like this: rsync --recursive --times --devices --specials --hard-links --human-readable -s --copy-links --acls --perms --executability --group --owner -- info=progress2 --no-inc-recursive -l --delete --delete-excluded -v -i --out-format=BACKINTIME: %i %n%L --link-dest=../../20231214-143359-584/backup -- chmod=Du+wx --exclude=/home/username/temp/testBAK_profil1 --exclude=/home/username/.local/share/backintime --exclude=.local/share/backintime/mnt -- include=/home/username/Documents/ --include=/home/username/ --include=/home/ --include=/home/username/temp/deleted_folder/ -- include=/home/username/temp/ --include=/home/username/.mozilla/ --exclude=.gvfs --exclude=.cache/* --exclude=.thumbnails* -- exclude=.local/share/[Tt]rash* --exclude=*.backup* --exclude=*~ --exclude=.dropbox* --exclude=/proc/* --exclude=/sys/* --exclude=/dev/* -- exclude=/run/* --exclude=/etc/mtab --exclude=/var/cache/apt/archives/*.deb --exclude=lost+found/* --exclude=/tmp/* --exclude=/var/tmp/* -- exclude=/var/backups/* --exclude=.Private --exclude=lock --exclude=root_only_file.txt --include=/home/username/Documents/** -- include=/home/username/temp/deleted_folder/** --include=/home/username/.mozilla/** --exclude=* / /home/username/temp/testBAK_profil1/backintime/computer1/username/1/new_snapshot/backup PS 2: You can find more details or even answer in our Github issue too: https://github.com/bit-team/backintime/issues/1587 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync exit code 23 (partial transfer due to errors): List of possible reasons and how to ignore some?
On Thu, 2023-12-14 at 14:09 -0500, Kevin Korb wrote:
> Unfortunately, exit 23 litterally just means something else went wrong
> and might have scrolled off of the screen if you have rsync listing
> files (--verbose or --itemize_changes). Essentially, it is anything
> that doesn't have its own exit code. I just ignore it. The most common
> reaosn is file vanished.
THX for sharing your experiences and knowledge :-)
I have just tried to "reverse engineer" the possible reasons from the source
code
and have found 21 reasons that I hope will never happen ;-)
So ignoring (or treating as a warning only) sound as best option so far.
-
Looking into the rsync source code I can see only one location where exit code
23 is set:
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/errcode.h#L42
#define RERR_PARTIAL23 /* partial transfer */
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/log.c#L97
{ RERR_PARTIAL, "some files/attrs were not transferred (see previous
errors)" },
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/cleanup.c#L217-L218
if (io_error & IOERR_GENERAL || got_xfer_error)
exit_code = RERR_PARTIAL;
So the question is which reasons cause
- IOERR_GENERAL
- got_xfer_error
to be true?
IOERR_GENERAL is set for different reasons (first line is the log output format
string):
1. receive_sums failed [what is that at all?]:
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L345-L348
2. send_files failed to open %s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L358-L362
3. fstat failed
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L373C33-L373C45
4. read errors mapping %s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/sender.c#L433-L436
5. change_dir %s failed
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L366-L369
6. skipping overly long name: %s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1245-L1247
7. symlink has no referent: %s
See the source code comments there when symlinks are checked:
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1271C1-L1282C28
8. readlink_stat(%s) failed
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1293-L1295
9. skipping file with bogus (zero) st_mode: %s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1298-L1302
10. skipping symlink with 0-length value: %s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1565-L1566
11. [%s] cannot convert filename: %s (%s)
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1595-L1601
12. [%s] cannot convert symlink data for: %s (%s)
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1609-L1614
13. get_acl(fname, &sx) < 0 // with no explicit error message!
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1628-L1632
14. get_xattr(fname, &sx) < 0 // with no explicit error message!
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1637-L1642
15. link_stat %s failed
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1809-L1811
16. opendir %s failed
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1841-L1843
17. filename overflows max-path len by %u: %s/%s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1863-L1871
18. cannot send file with empty name in %s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1873-L1877
19. readdir(%s)
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L1886-L1889
20. link_stat %s failed
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/flist.c#L2396-L2399
21. cannot add local filter rules in long-named directory: %s
https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/exclude.c#L815-L818
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync error messages always in English (= never translated)?
I want to recognize and handle some rsync error messages in my log files (containing also the --itemize-changes output) on different computers with different language/locale settings. Can I rely on rsync to create only English error messages to have a stable pattern to recognize? PS: In the source code at https://github.com/WayneD/rsync I could not see any indication of message translations... -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [PATCH] Consider nanoseconds when quick-checking for unchanged files
> Date: Wed, 20 Jan 2016 23:04:20 -0800 > From: Wayne Davison > On Thu, Dec 25, 2014 at 2:48 AM, Ingo Br=C3=BCckl wrote: > > On systems using nanoseconds differences should be taken into > > consideration. > The problem is that if you transfer from a filesystem that has nanoseconds > to one that does not support it, rsync would consider most of the files to > be constantly different, since the nanosecond values would only match if > the source file happened to have 0 nanoseconds. So, the logic has to be > improved to somehow detect such a case and treat the truncated values as > equal. One possible improvement would be to skip the nanosecond check if > the destination file has a nanosecond value of 0. That could possibly be > improved if we figure out if a particular device ID supports nanoseconds > somehow. I have a potential heuristic in mind that I can code up and see > how it works. Here's one idea, and note an important issue with ns times and --link-dest: (a) For each end, see if any of the files being considered already have nonzero nanosecond parts. If so, then that end of the transfer supports nanosecond timing. (b) If the sending filesystem appears to have nonzero ns parts, and the receiving filesystem appears to have all-zero ns parts (including any directories under consideration), the receiver may still support ns times, but have been synchronized from a filesystem that didn't. We don't want to perpetuate that on the -next- sync, however, so we can't just disallow ns times on the receiver, or we'll never try them again. (c) In case (b) above, therefore, if any file to be transmitted has a ns time, transfer it and then immediately check the received file's timestamp. If its ns time is still zero, then the receiving filesystem doesn't support it, so disable ns times during the transfer. If it's nonzero, then enable. (I am eliding the pipelining that happens during an actual rsync; that may have to be dealt with somehow.) Also, check the directory mod time, and see if -that's- now nonzero; you have a very small chance of it being zero if ns times are supported, and you can check for being in or near that window. And the first time it's nonzero in this filesystem, you know it'll work for everything else in this fs.* * "This filesystem" assumes either that you can detect mountpoints, or that the heuristic should be applied per-directory, and that no directory has a single-file mountpoint that doesn't support it, etc. I assume rsync must already have some sort of logic like this for dealing with xattr support per-fs, etc. If this is flaky to do, then you might need --[no]ns-timing switches to force rsync to do the right thing without complaining on every single file if it guesses wrong. I don't know if the rsync protocol is flexible enough to dynamically enable or disable this capability partway through a transfer. If it isn't, then there's an even more hackish approach, which is to add, and unconditionally attempt to honor, a --ns-times-valid sort of switch. Users can then use the heuristics above in a dummy transfer to know whether to set that switch for the real transfer. (Or they may know out-of-band that their FS supports ns times.) But I'd think such a switch and workaround should be last resorts. I would really like to see ns times supported. I use dirvish to back up filesystems, which uses rsync, and if I ever have to restore any files from that (which I do more often due to accidentally deleting or bashing a file than due to media failure), I lose the ns timestamps, and they're sometimes extremely valuable forensically when I'm trying to debug something else. Having them be 0 when I thought they shouldn't be has more than once cost me time until I realized that I'm looking at files that were rsync'ed from another host (either to duplicate a setup, or from a backup) and rsync didn't preserve the ns times. Unfortunately, of course, if rsync gets fixed now, it -will- consider every single backed-up file in my dirvish vaults to be "new" and will insanely bloat the vault (doubling its size) the next time it runs, and then I'll have to tell faster-dupemerge how to re-merge all that stuff, too. (After all, even if the file contents haven't changed, its metadata has, so --link-dest is required to create a new copy of the file rather than hardlink to one with a different timestamp.) What I'd -really- like is for some sane interaction with --link-dest as well (which probably requires another switch, alas), which basically says "a change from ns-0 to ns-other with no other changes to the file is considered the same file---update the timestamp to the new ns time but don't break the hardlink", with a wa
Re: [Bug 11521] rsync does not use high-resolution timestamps to determine file differences
[Included text copied from the commit.] > Date: Sun, 24 Jan 2016 19:52:49 + > From: samba-b...@samba.org > Auto-Submitted: auto-generated > https://bugzilla.samba.org/show_bug.cgi?id=11521 > Wayne Davison changed: >What|Removed |Added > > Status|NEW |RESOLVED > Resolution|--- |FIXED > --- Comment #3 from Wayne Davison --- > The latest git version has an option that lets you choose to include > nanoseconds in comparisons if you want them. Having it on by default would > likely cause far too many headaches for various backup solutions that use an > older filesystem (e.g. ext3) that doesn't support nanoseconds. Thanks for the patch! Just FYI, this comment is true but incomplete---the scenario I was describing was straight ext4-to-ext4 copies and/or backups. The timestamp problem I currently see with those is because rsync was throwing away the nanosecond information until this patch, even though both ends supported it. Anyone who's used dirvish, or presumably similar tools such as rsnapshot, from and to ext4 or other ns-supporting filesystems, will be bitten by the problem of non-ns vs ns timestamps bloating backups and breaking hardlinks, either when they manually use --modify-window=-1, or when this becomes the default. I'd still love to see either some in-rsync workaround that can be left in place*, or (second best) some clever by-hand one-time workaround that uses rsync just once to update all those timestamps while not breaking the hardlinks (presumably -not- using --link-dest for those), or (third best) some not-rsync-at-all solution that basically does a giant ls at the source and a giant touch at the destination. Yes, I know that none of those can fix up backed-up files that are no longer in the same place in the source; those at least won't bloat/ unhardlink later backups. This solution is likely going to have to be reinvented repeatedly by people running backups, so working out the right way to do it and then putting it somewhere other rsync users will find it is likely a useful exercise. Before I solve it by hand for myself, it'd be useful to know if it's likely rsync will come up with some way to solve it for me. * E.g., (a) "If modify-window is -1, and --link-dest is in use, and everything else about the file or directory matches -except- the timestamp, then update the timestamp," and (b) "Provide a switch to turn off this heuristic after I'm sure my backups are okay." (The intent of (b) is to catch later slight changes of timestamp but actually record them as -separate files-, which is important so older snapshots don't magically change out from under you if something updates a timestamp by a fraction of a second without otherwise changing anything, which does happen and can occasionally be -very- important to know about when tracking down issues.) -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Bug 11521] rsync does not use high-resolution timestamps to determine file differences
> Date: Sun, 24 Jan 2016 15:43:20 -0800
> From: Wayne Davison
A couple questions below; please bear with me.
> No, if you do a ext4 -> ext4 copy, rsync has set the matching ns info for
> transferred files since 3.1.0. There was a case prior to rsync 3.1.2 where
> a brand-new file transferred in the same second it was created wouldn't
get
> the right ns value because rsync was optimizing away the time-set if the
> file's mod-time matched in the integer part (3.1.2 fixed that).
Oh, I see what happened. My problem is that no Ubuntu LTS before
14.04 had rsync 3.1.0 or newer, and the original capability took more
than four years to make it into a released rsync version, if I'm
reading the release notes correctly.* Unfortunately, that means
the vast majority of my machine base predates the fix, including the
machine hosting the backups. I can obviously install newer rsyncs,
but that gives me a big installed base of pre-fix data that I'm going
to have to fix, and no more rsync security updates unless I track them
manually. Yet I'd rather do this now, so I'm future-proofed, than
be badly surprised some years down the road when this rsync behavior
becomes the default, and to keep the problem from continuing to get worse.
* I may have tried 3.1.0 at some point and then realized the problems
it'd give me for backups and didn't install it everywhere, pending
a better fix; this is starting to ring bells. I'm really surprised
that the initial patch of Sep 7, 2009 never made it into a released
rsync until 3.1.0 of Sep 28 2013; that's a four-year delay, and
explains why I obviously never tried this out until perhaps an
experiment with 3.1.0, and no doubt I didn't want to run a private
version that wouldn't get security updates. Ubuntu rsync versions:
10.04 has 3.0.7 proto 30; 12.04 has 3.0.9 p 30; 14.04 has 3.1.0 p 31.)
So what it looks like is that the capability to transfer ns times at
all existed in CVS but not released since 2009, in released since
2013, and in an Ubuntu LTS since 2014. And the current patch -seems-
to be an optimization that avoids -comparison- if the ns times match,
but that only affects speed---it doesn't change what gets written in
any event, just how fast. Right? But actually I think it changes
behavior besides that---see my test case below.
> Beginning with this patch you can run rsync -aiv --checksum -@-1 and have
> it fix the full mod-time on any matching files it finds. But for most
newer
> backups, the ns time will already be set correctly (as long as it was
> created using a new enough rsync and protocol 31). If someone has a large
> link-dest hierarchy that predates 3.1.0, then you could be sharing
> hard-linked matching files from back before the ns info was included (the
> older files would all have 0 for the ns value).
Wow, that -c really hurts. If one wanted to live dangerously---with
the assumption that two files that otherwise match in all metadata
(including obviously length :) but whose timestamps differ in that one
has integer seconds and the other has the same integer seconds but
also nanoseconds, can rsync readjust the dates, without doing a full
checksum? If not, I may write such a tool, or do it the (very) slow
way and have rsync re-checksum a few terabytes of my backups... :)
[Might find some bitrot that way, of course.]
Also, I actually -can't- use that command to fix my snapshots,
because (if I understand correctly), it will -alter- my existing
snapshots to match the -current- contents of files, destroying them
---I'll no longer be able to go back in time to a previous version.
I only want to update ns times on files in the older snapshots
if and only if changing integer times to ns times would be the
only modification. I think rsync -ac -@-1 will do far more, yes?
As for -@-1, that introduces a surprising change in behavior when I
try it. I'm unsure if it's intended, though I think it is. But it
will -definitely- break my hardlinks and bloat the backups if I try
it without readjusting the dates in --link-dest directories (e.g.,
previous snapshots). I find that specifying -@-1 copies the ns
timestamp from the source to the destination even if the --link-dest
directory has an integer timestamp, and so I assume this is part of
the purpose of the patch? Not just an optimization, but a change in
the way --link-dest might work. Observe:
22:57:22 ~$ mkdir T
22:57:25 ~$ cd T
22:57:26 ~/T$ mkdir 1 2 3 4 5 6
22:57:30 ~/T$ lat() { ls -alF -i --full-time "$@"; }
22:57:49 ~/T$ touch 1/foo
22:57:53 ~/T$ ln 1/foo 1/bar
22:57:56 ~/T$ lat */*
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/bar
1321175 -rw-r--r-- 2 user user 0 2016-01-24 22:57:53.013689572 -0500 1/foo
22:57:59 ~/T$ rsync -aviH 1/ 2/
sending incremental file list
[PATCH] Don't fail if current directory doesn't exists
It's possible to have situation when current directory is not needed for
rsync operation (using absolute path on lazy mounts / being in deleted
directory).
rsync should not fail in such cases
---
util.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/util.c b/util.c
index 8723248f..33107867 100644
--- a/util.c
+++ b/util.c
@@ -1090,8 +1090,8 @@ int change_dir(const char *dir, int set_path_only)
if (!initialised) {
initialised = 1;
if (getcwd(curr_dir, sizeof curr_dir - 1) == NULL) {
- rsyserr(FERROR, errno, "getcwd()");
- exit_cleanup(RERR_FILESELECT);
+ // current dir is unavailable using "/" instead
+ strcpy(curr_dir, "/");
}
curr_dir_len = strlen(curr_dir);
}
--
2.14.3
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
copy to destination only new files
I'm using rsync to copy files from a source folder to a destination folder. It works fine. But, to save some time, what options should I use to only update or copy (to the destination) files that are new and/or have a new time stamp on them? Here's the code I use below (pretty sure the -n means dry-run) rsync -r -n -t -v --progress -s '/Desktop/Book' '/media/Chip/Book June' Thanks, Peter OpenPGP_0x19B6B68457884FFB.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: copy to destination only new files
You're probably right. I've spent too much time on this and it's not working so.I quit. *From:* Robin Lee Powell [mailto:rlpow...@digitalkingdom.org] *Subject:* copy to destination only new files *Date:* Saturday, June 24, 2023 at 1:14 AM *To:* . *Cc:* rsync@lists.samba.org That's how rsync operates normally; you shouldn't have to do anything to get that behaviour. That's kind of the whole thing it's for. If you're *not* seeing that behaviour, something weird is going on. Possibly one side or the other is recording timestamps incorrectly. On Fri, Jun 23, 2023 at 12:57:02PM -0400, . via rsync wrote: I'm using rsync to copy files from a source folder to a destination folder. It works fine. But, to save some time, what options should I use to only update or copy (to the destination) files that are new and/or have a new time stamp on them? Here's the code I use below (pretty sure the -n means dry-run) rsync -r -n -t -v --progress -s '/Desktop/Book' '/media/Chip/Book June' Thanks, Peter -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options:https://lists.samba.org/mailman/listinfo/rsync Before posting, read:http://www.catb.org/~esr/faqs/smart-questions.html OpenPGP_0x19B6B68457884FFB.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Diff between using "-i" and not using "-i"
Hi, Version: sync version 3.3.0 protocol version 31 If I do a I get: = sending incremental file list var/log/journal/ sent 263,31K bytes received 1,14K bytes 528,90K bytes/sec total size is 294,92M speedup is 1.115,21 (DRY RUN) = including option <-i>: rsync -avhHXASDin --del /mnt/ /foo/mnt/ I get: = sending incremental file list .fx usr/bin/ping .d...a. var/log/journal/ sent 263,31K bytes received 1,14K bytes 528,90K bytes/sec total size is 294,92M speedup is 1.115,21 (DRY RUN) = How is that, that "usr/bin/ping" is not included / included in results? That is no feature, is it? Yours lopiuh -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Diff between using "-i" and not using "-i"
Hi, thanks for the information. Shall I file a bug report at https://github.com/RsyncProject/rsync/issues ? Yours lopiuh Am Mo., 3. Juni 2024 um 15:57 Uhr schrieb Paul Slootman via rsync : > > On Mon 03 Jun 2024, Kevin Korb via rsync wrote: > > > It appears that xattr changes (that is what the x mens) never made it into > > the verbose output. I would call that a bug but I would rather it be fixed > > by making -v include -i. > > I think -v only shows when file data is transferred (or files / > directories created / deleted). > Just a permission change is also not shown with -v, but is with -i. > > > Paul > > -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
error in rsync protocol data stream (code 12)
Hello! A good day to you all. I got some error messages from rsync: rsync: error writing 4092 unbuffered bytes - exiting: Connection reset by peerrsync: read error: Connection reset by peerrsync error: error in rsync protocol data stream (code 12) at io.c(162)rsync error: error in rsync protocol data stream (code 12) at io.c(463) ...used the stock rsync from the rh7.0 distro. Comments are more than welcome. Thanks! Do You Yahoo!? Yahoo! Finance - Get real-time stock quotes
rsync over ssh without giving the user access to a shell?
What is the best way to give a remote user the ability to use rsync via ssh without also giving them shell access? I want the rsync user to authenticate via ssh. I want to protect their rsync session with ssh. However, I do not want to give them access to a shell on the Linux server where rsyncd is running. Any recommendations would be appreciated. chongo (Landon Curt Noll) /\oo/\ -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
writefd_unbuffered failed
Hello! A good day to you all. i've installed rsync-2.5.6 on a Red Hat 9.0 box(server) and am using rsync 2.5.6 on a Red Hat 6.2 box(client). In the middle of the rsync transfer, i get the error on the client saying that the server unexpectedly close the connection. Upon checking the logs on the rsync server, i found this entry: Oct 10 14:39:38 linux01 rsyncd[9266]: rsync: writefd_unbuffered failed to write 4096 bytes: phase "unknown": Connection reset by peer Any idea what this means? The client is running rsync with the -ar --partial --delete options. Thanks! - Do you Yahoo!? The New Yahoo! Shopping - with improved product search -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: writefd_unbuffered failed
Hello, Yes, both client and server are clean...but i think i know what the problem is now. Server is using an ext3 fs, the client ext2 -- and trying to rsync a >2gb file over to the client B). i'll compile in ext3 support and try again. Thanks! Jim Salter <[EMAIL PROTECTED]> wrote: > In the middle of the rsync transfer, i get the error on the client saying > that the server unexpectedly close the connection. Upon checking the > logs on the rsync server, i found this entry: > > Oct 10 14:39:38 linux01 rsyncd[9266]: rsync: writefd_unbuffered failed to write 4096 bytes: phase "unknown": Connection reset by peer > > Any idea what this means? The client is running rsync with the -ar --partial --delete options. I've gotten similar errors when I had a corrupt filesystem on the server side. Have you fsck'ed both boxes? -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html - Do you Yahoo!? The New Yahoo! Shopping - with improved product search -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[rsync-announce] rsync security advisory
rsync 2.5.6 security advisory - December 4th 2003 Background -- The rsync team has received evidence that a vulnerability in rsync was recently used in combination with a Linux kernel vulnerability to compromise the security of a public rsync server. While the forensic evidence we have is incomplete, we have pieced together the most likely way that this attack was conducted and we are releasing this advisory as a result of our investigations to date. Our conclusions are that: - rsync version 2.5.6 contains a heap overflow vulnerability that can be used to remotely run arbitrary code. - While this heap overflow vulnerability could not be used by itself to obtain root access on a rsync server, it could be used in combination with the recently announced brk vulnerability in the Linux kernel to produce a full remote compromise. - The server that was compromised was using a non-default rsyncd.conf option "use chroot = no". The use of this option made the attack on the compromised server considerably easier. A successful attack is almost certainly still possible without this option, but it would be much more difficult. Please note that this vulnerability only affects the use of rsync as a "rsync server". To see if you are running a rsync server you should use the netstat command to see if you are listening on TCP port 873. If you are not listening on TCP port 873 then you are not running a rsync server. New rsync release - In response we have released a new version of rsync, version 2.5.7. This is based on the current stable 2.5.6 release with only the changes necessary to prevent this heap overflow vulnerability. There are no new features in this release. We recommend that anyone running a rsync server take the following steps: 1) update to rsync version 2.5.7 immediately 2) if you are running a Linux kernel prior to version 2.4.23 then you should upgrade your kernel immediately. Note that some distribution vendors may have patched versions of the 2.4.x series kernel that fix the brk vulnerability in versions before 2.4.23. Check with your vendor security site to ensure that you are not vulnerable to the brk problem. 3) review your /etc/rsyncd.conf configuration file. If you are using the option "use chroot = no" then remove that line or change it to "use chroot = yes". If you find that you need that option for your rsync service then you should disable your rsync service until you have discussed a workaround with the rsync maintainers on the rsync mailing list. The disabling of the chroot option should not be needed for any normal rsync server. The patches and full source for rsync version 2.5.7 are available from http://rsync.samba.org/ and mirror sites. We expect that vendors will produce updated packages for their distributions shortly. Credits --- The rsync team would like to thank the following individuals for their assistance in investigating this vulnerability and producing this response: * Timo Sirainen * Mike Warfield * Paul Russell * Andrea Barisani Regards, The rsync team signature.asc Description: Digital signature ___ rsync-announce mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-announce
[rsync-announce] rsync 2.6.0pre2 released
I've released rsync 2.6.0pre2. This should be the final pre-release before 2.6.0 goes final (which is scheduled to happen on January 1st). If you'd like to assist in making the final release smoother, please try this out and let me know if you encounter any problems: http://samba.org/ftp/rsync/preview/rsync-2.6.0pre2.tar.gz http://samba.org/ftp/rsync/preview/rsync-2.6.0pre2.tar.gz.sig Thanks, ..wayne.. pgp0.pgp Description: PGP signature ___ rsync-announce mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-announce
[rsync-announce] rsync 2.6.0 (final) released
I've released version 2.6.0 of rsync. Two important things to note
in the new release:
1. The default remote shell is now "ssh" unless you tell
the configure you want to use something else.
2. Some bug fixes in the include/exclude code, while making
things work properly, have resulted in some user-visible
changes for certain wildcard strings. Read the BUG FIXES
below to see if any of these changes apply to you.
Download links:
http://samba.org/ftp/rsync/rsync-2.6.0.tar.gz
http://samba.org/ftp/rsync/rsync-2.6.0.tar.gz.sig
http://samba.org/ftp/rsync/rsync-2.5.7-2.6.0.diffs.gz
http://samba.org/ftp/rsync/rsync-2.5.7-2.6.0.diffs.gz.sig
Here is the "NEWS" file for the release:
NEWS for rsync 2.6.0 (1 Jan 2004)
Protocol: 27 (changed)
Changes since 2.5.7:
ENHANCEMENTS:
* "ssh" is now the default remote shell for rsync. If you want to
change this, configure like this: "./configure --with-rsh=rsh".
* Added --files-from, --no-relative, --no-implied-dirs, and --from0.
Note that --from0 affects the line-ending character for all the
files read by the --*-from options. (Wayne Davison)
* Length of csum2 is now per-file starting with protocol version
27. (J.W. Schultz)
* Per-file dynamic block size is now sqrt(file length). The
per-file checksum size is determined according to an algorithm
provided by Donovan Baarda which reduces the probability of rsync
algorithm corrupting data and falling back using the whole md4
checksums. (J.W. Schultz, Donovan Baarda)
* The --stats option no longer includes the (debug) malloc summary
unless the verbose option was specified at least twice.
* Added a new error/warning code for when files vanish from the
sending side. Made vanished source files not interfere with the
file-deletion pass when --delete-after was specified.
* Various trailing-info sections are now preceded by a newline.
BUG FIXES:
* Fixed several exclude/include matching bugs when using wild-cards.
This has a several user-visible effects, all of which make the
matching more consistent and intuitive. This should hopefully not
cause anyone problems since it makes the matching work more like
what people are expecting. (Wayne Davison)
- A pattern with a "**" no longer causes a "*" to match slashes.
For example, with "/*/foo/**", "foo" must be 2 levels deep.
- "**/foo" now matches at the base of the transfer (i.e. /foo).
- An non-anchored wildcard term floats to match beyond the base of
the transfer. E.g. "CVS/R*" matches at the end of the path,
just like the non-wildcard term "CVS/Root" does.
- Including a "**" in the match term causes it to be matched
against the entire path, not just the name portion, even if
there aren't any interior slashes in the term. E.g. "foo**bar"
would exclude "/path/foo-bar" (just like before) as well as
"/foo-path/baz-bar" (unlike before).
* The exclude list specified in the daemon's config file is now
properly applied to the pulled items no matter how deep the
user's file-args are in the source tree. (Wayne Davison)
* For protocol version >= 27, mdfour_tail() is called when the
block size (including checksum_seed) is a multiple of 64.
Previously it was not called, giving the wrong MD4 checksum.
(Craig Barratt)
* For protocol version >= 27, a 64 bit bit counter is used in
mdfour.c as required by the RFC. Previously only a 32 bit bit
counter was used, causing incorrect MD4 file checksums for
file sizes >= 512MB - 4. (Craig Barratt)
* Fixed a crash bug when interacting with older rsync versions and
multiple files of the same name are destined for the same dir.
(Wayne Davison)
* Keep tmp names from overflowing MAXPATHLEN.
* Make --link-dest honor the absence of -p, -o, and -g.
* Made rsync treat a trailing slash in the destination in a more
consistent manner.
* Fixed file I/O error detection. (John Van Essen)
* Fixed bogus "malformed address {hostname}" message in rsyncd log
when checking IP address against hostnames from "hosts allow"
and "hosts deny" parameters in config file.
* Print heap statistics when verbose >= 2 instead of when >= 1.
* Fixed a compression (-z) bug when syncing a mostly-matching file
that contains already-compressed data. (Yasuoka Masahiko and
Wayne Davison)
* Fixed a bug in the --backup code that could cause deleted files
to not get backed up.
* When the backup code makes new directories, create them with mode
0
Gallery photos
I love meeting new people and making new friends. I am a Mary Kay Beauty Consultant. I am married to a wonderful man. We have no children, exept for a minature schnauzer that thinks he is a child. Looking forward to meeting you. password for archive: 35730 <> -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: make errors, rsync 2.6.9 with acls patch
just for anyone's interest, in context -- & given that OSX apparently has no #$&^!!! man page for "fsaclctl" :-( -- there's a bit of an intro to osx's acl implementation that i found helpful, here: http://www.afp548.com/article.php?story=20050506085817850 -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: 2.6.9 , problem with acl patch
Sorry I haven't looked into this, but atm I just don't have the time. thanks for the update. personally, atm, i'd love to see just HEAD's mtime (finished?) support rolled into a 'official;' rsync release. It also seems like any solution will necessarily drop information. osx keeps track of several permissions not present in posix ACLs. at the very least, if capabilities of an rsync-team's official release for osx were brought up to parity with the *nix versions, then patches for apple's capability could be built on top of that -- rather than overlapping capability/implementation, which iiuc is the current situation. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
bug with --link-dest ?
Hi! I am running rsync 2.6.4 (on Debian sarge) and I am experiencing a strange behaviour. I am trying to create an identical filetree on the same filesystems with the single files being hardlinks to the source like this: rsync -vaH --progress --delete --stats --numeric-ids -x --link-dest=/path/to/filetree/ /path/to/filetree/ /path/to/current/ /path/to is one filesystem. This creates hardlinks for the files in /path/to/filetree/ in /path/to/current/ When calling rsync with relative paths like this: cd /path/to; rsync -vaH --progress --delete --stats --numeric-ids -x --link-dest=./filetree/ ./filetree/ ./current/ rsync does not create hardlinks but copies the files instead. Is this intentional behaviour? Greetings -- Robert Sander -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: bug with --link-dest ?
On Mon, 22 Jan 2007 06:30:01 -0500, "Matt McCutchen" <[EMAIL PROTECTED]> wrote: > > You're right. The description of --link-dest=DIR in the man page > says, "If DIR is a relative path, it is relative to the destination > directory." This is because the generator and receiver access the DIR > by name after changing into the destination directory. Now I see that line in the man page. Sorry for the confusion. Greetings -- Robert Sander -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--link-dest copying modified files
Hi! It's me again with another --link-dest issue: I am using dirvish (www.dirvish.org) to create daily backup on disk images. dirvish is using rsync with --link-dest pointing to the last good image. This creates images with hardlinks to unmodified files. So far so good. Now I want to create a "current" filetree with hardlinks pointing to the last image. rsync -vaH --delete --stats --numeric-ids -x --link-dest=/path/to/lastimg /path/to/lastimg /path/to/current This works except when an existing file has been modified and is now "new" (and not the n'th hardlink) in the image. This file is copied to the "current" directory. A hardlink pointing to the file in lastimg is not created. If a new file has been created in between the dirvish runs it perfectly gets hardlinked in the "current" directory. Is that intented behaviour? You can easily reproduce the issue with the following commands: # create source dir mkdir -p /tmp/rsync/src # create 1st file echo 1 > src/a # create 1st "backup" image rsync -vaH --delete --numeric-ids -x /tmp/rsync/src/ /tmp/rsync/1/ # create "current" dir with hardlinks from 1st image rsync -vaH --delete --numeric-ids -x --link-dest=/tmp/rsync/1/ /tmp/rsync/1/ /tmp/rsync/current/ # create 2nd file echo 2 > src/b # create 2nd image rsync -vaH --delete --numeric-ids -x --link-dest=/tmp/rsync/1/ /tmp/rsync/src/ /tmp/rsync/2/ # create "current" with hardlinks from 2nd image rsync -vaH --delete --numeric-ids -x --link-dest=/tmp/rsync/2/ /tmp/rsync/2/ /tmp/rsync/current/ # modify 1st file echo 3 >> src/a # create 3rd image rsync -vaH --delete --numeric-ids -x --link-dest=/tmp/rsync/2/ /tmp/rsync/src/ /tmp/rsync/3/ # create "current" with hardlinks from 3rd image rsync -vaH --delete --numeric-ids -x --link-dest=/tmp/rsync/3/ /tmp/rsync/3/ /tmp/rsync/current/ The last call to rsync will not re-create the hardlinked file "a" but will create a new copy of "a" in the "current" directory. If "current" is removed and re-created: rm -rf current rsync -vaH --delete --numeric-ids -x --link-dest=/tmp/rsync/3/ /tmp/rsync/3/ /tmp/rsync/current/ both files "a" and "b" are hardlinks pointing to the 3rd image. Greetings -- Robert Sander -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --link-dest copying modified files
On Tue, 23 Jan 2007 07:39:31 -0800, Wayne Davison <[EMAIL PROTECTED]> wrote: > You're better off maintaining a symlink named "current" that points to > the most recent directory. Either that, or rename the dirs after the > transfer. Hm, that does not work as our backup software has to pick up that dir. And it does not like symlinks, bind mounting or any other tricks we tried so far... Greetings -- Robert Sander -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync on a mac - trouble with aliases
good afternoon everyone! i am new to this list, so please forgive me for my ignorance. first, i have configured passwordless ssh to allow automatic trust between remote sites within the same internal network. i used the RSA keygen for that. secondly, i have a number of .dmg files kept on my master imaging server that need to be backed up off-site in the various schools around the district. not all the images go to every school, and i'm trying to avoid having multiple copies of the same image on the master. what i did was to set up directories for each location and place aliases of the images to go to each location in that directory. to accomplish this goal, i placed the following line in my crontab on the master. (time for this to execute) rsync -e ssh -avzuLE --copy-links /Volumes/Storage/lincoln/ 172.16.40.200:/images/ perhaps there's a problem with my syntax? any help for this underfunded school district would be greatly appreciated! --john -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync 3-cvs missing prereqs/deps on osx
hi, as it's been awhile since my last attempts w/ v269, i'm building the latest rsync3-svs on osx 10.4.8; looking to see what's evolved re: acls & xattrs support on osx ... @ build, i note complaint abt a missing yodl2man, which originates with YODL. YODL, then requires icmake. as neither is available on osx, it's src build time. on building debian's icmake (already suspecting linux-friendly, non-mac-ish-ness ...), i find, % sh bootstrap ... Building the runtime-library in ./rss In file included from chesc.c:5: ../rss/icrssdef.h:10:20: error: malloc.h: No such file or directory ... mod'ing in, % vi rss/icrssdef.h --- #include +++ #include gets me a lot further, but eventually, % sh bootstrap ... Creating the target directory ./bin Creating icmake /usr/bin/ld: Undefined symbols: _xrealloc ... can anyone clarify the prereqs for rsync, rsync proj's "stance" on osx support, and if/where the prereqs are available (as binaries or src+instructions) for sane build on osx? thanks. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[RESOLVED] rsync 3-cvs missing prereqs/deps on osx
just fyi, with help from the YODL-ers (thanks Frank & Karel!), here's
what _works_ for me with to build the
icmake-v700
yodl-v204
prereqs to rsync3 on osx 10.4.87.
# ICMAKE #
cd /path/to
rm -rf icmake*
wget ftp://ftp.rug.nl/contrib/frank/software/linux/icmake/icmake_7.00.tar.gz
gnutar zxvf icmake_7.00.tar.gz
cd /path/to/icmake-7.00
rm /usr/local/include/malloc.h
ln -sf /usr/include/stdlib.h /usr/local/include/malloc.h
cd /path/to/icmake-7.00
sh bootstrap
rm /usr/local/bin/icm*
cp -Rf bin/icm* /usr/local/bin/
ls -al /usr/local/bin/icm*
-rwxr-xr-x 1 root admin 229948 Feb 6 06:50 /usr/local/bin/icm-comp
-rwxr-xr-x 1 root admin 39552 Feb 6 06:50 /usr/local/bin/icm-exec
-rwxr-xr-x 1 root admin 42040 Feb 6 06:50 /usr/local/bin/icm-pp
-rwxr-xr-x 1 root admin 48772 Feb 6 06:50 /usr/local/bin/icmake
-rwxr-xr-x 1 root admin 87832 Feb 6 06:50 /usr/local/bin/icmun
rehash
which icmake
/usr/local/bin/icmake
icmake --version
ICCE Make Utility Version 7.00
rm -rf /usr/lib/icmake
mkdir -p /usr/lib/icmake
ln -sf /usr/local/bin/icm-comp /usr/lib/icmake/
ln -sf /usr/local/bin/icm-pp /usr/lib/icmake/
ln -sf /usr/local/bin/icm-exec /usr/lib/icmake/
ls -al /usr/lib/icmake
lrwxr-xr-x 1 root wheel 23 Feb 6 06:52 icm-comp ->
/usr/local/bin/icm-comp
lrwxr-xr-x 1 root wheel 23 Feb 6 06:52 icm-exec ->
/usr/local/bin/icm-exec
lrwxr-xr-x 1 root wheel 21 Feb 6 06:52 icm-pp ->
/usr/local/bin/icm-pp
ln -sf /usr/local/bin/icmake /usr/bin/icmake
ls -al /usr/bin/icmake
ls -al /usr/bin/icmake
lrwxr-xr-x 1 root admin 21 Feb 6 07:26 /usr/bin/icmake ->
/usr/local/bin/icmake
# YODL #
cd /path/to
rm -rf yodl*
wget http://umn.dl.sourceforge.net/sourceforge/yodl/yodl_2.04.orig.tar.gz
gnutar zxvf yodl_2.04.orig.tar.gz
cp -Rf /path/to/yodl-2.04 /path/to/yodl-2.04ORIG
cd /path/to/yodl-2.04
vi src/config.h
==
@39
--- #define STD_INCLUDE "/usr/share/yodl"
+++ #define STD_INCLUDE "/usr/local/share/yodl"
@53
--- #define YODL_BIN"/usr/bin"
+++ #define YODL_BIN"/usr/local/bin"
==
vi /path/to/yodl-getline-patch.txt
--- yodl-2.04ORIG/src/verbinsert/verbinsert.c 2007-02-06
09:19:57.0 -0800
+++ yodl-2.04/src/verbinsert/verbinsert.c 2007-02-06
09:25:03.0 -0800
@@ -5,7 +5,8 @@
#include
#include
#include
-
+static int getline (char **str, size_t *n, FILE *in);
+static char *str_add (char *s, char const *xtra);
int main(int argc, char **argv)
{
FILE*input = NULL;
@@ -109,7 +110,6 @@
if (getline(&line, &nchars, input) < 0)
break;
-//fprintf(stderr, "%s",
if (strstr(line, label) == line)// matching (end)label
{
@@ -152,3 +152,35 @@
return 0;
}
+static int getline (char **str, size_t *n, FILE *in) {
+char buf[80], *s = 0, ret = -1;
+
+while (1) {
+ if (! fgets (buf, 79, in))
+ break;
+ ret = 0;
+ s = str_add (s, buf);
+ if (strchr (s, '\n'))
+ break;
+}
+
+if (ret -1)
+ *str = s;
+return (ret);
+}
+
+static char *str_add (char *s, char const *xtra) {
+if (!s)
+ s = strdup (xtra);
+else {
+ s = realloc (s, strlen(s) + strlen(xtra) + 1);
+ if (s)
+ strcat (s, xtra);
+}
+if (!s) {
+ fprintf (stderr, "Out of memory!\n");
+ exit (1);
+}
+return (s);
+}
+
cd /path/to/yodl-2.04
rehash
patch -p1 < /path/to/yodl-getline-patch.txt
./build package
./build install
that should work, but FAILS with,
mkdir -p
usage: mkdir [-pv] [-m mode] directory ...
system - failure of system call (status 16384)
but,
./build install /usr/local
completes OK
rehash
rm /usr/local/include/malloc.h
which yodl
/usr/local/bin/yodl
yodl --version
yodl version 2.04
and we're done.
now, to see if it actually makes rsync3 'happy' ...
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
fyi: yodl prereq for rsync3 updated -- osx issues fixed
for those who care, the yodl pkg, a prereq for building rsync3-cvs docs (& more?), had 'issues' @ version <= v2.04 with building cleanly on osx. it's been updated today to v2.10, http://sourceforge.net/project/showfiles.php?group_id=126615 v2.10 builds clealy, without req'd patching or error, and works just fine when/as used by rsync3 -- or elsewhere. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Problem with shared xls file. Could it be blamed on rsync?
Date: Fri, 16 Mar 2007 02:30:33 -0700 (PDT) From: syncro <[EMAIL PROTECTED]> Thanks alot! That's what I wanted to hear ;) We want to have an always-up-to-date-copy thus rsync every minute and not just at night. However my preventive measure will be a forbiddance of sharing xls files or the like. Rather than forbidding sharing, maybe you could ask rsync (via files-from and a filter or something) to only back up files haven't been modified in the last 10 minutes? I don't know exactly when Windows might update the file's timestamp vs when data starts getting written to it---and there will always be a tiny timing race anyway since the scan of the filesystem and the start of the update aren't simultaneous---but it might be the file gets backed up every few minutes when people aren't actively working on it. The other solution might be to have Windows copy the file to a temporary location (since Windows might respect its own locks), and then back up the temporary copy. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
checksum-xattr.diff [CVS update: rsync/patches]
Date: Mon, 2 Jul 2007 08:43:39 -0400
From: "Matt McCutchen" <[EMAIL PROTECTED]>
> *Note that "now" for a particular disk may not be the same as time() if
> the disk is remote, so network filesystems can be rather complicated.
That's easy to fix: get your "now" by touching a file on the
filesystem and reading the resulting mtime.
Unreliable. If you sync up at the beginning of a run, and then the
remote system executes a large clock step (e.g., because it's not
running NTP or it's misconfigured, or it is but NTP has bailed due to
excessive drift from hardware issues or a bogus driftfile (both of
which I've seen*), then "now" might glitch by a second (or more),
which is enough to break your idea of what "now" means---even a
smaller glitch can lead to races based on whose clock ticks first.
Sure, it's a low-probability event, but then, with low probability,
you have some file that isn't getting updated, which can lead to all
kinds of mysterious bugs, etc...
Seems to me the only way around this would be to do the touch before
-every- file you handle, which doubles the amount of statting going
on, etc. And there are probably still timing windows there.
* [One of several ways I saw this happening was a motherboard that
accidentally had FSB spread-spectrum enabled, which caused the clock
to run fast. NTP gave up slewing and started making larger and larger
steps until it was forced to bail out. It took quite a while for this
problem to be noticed ("but the machine's running NTP!"), in part
because it took a while to manifest after each boot reset the clock.
Then, when the BIOS setting got fixed, the bad driftfile created by
NTP's valiant attempts to cope with the situation caused the clock
to misbehave in the -other- direction until the NTP conf stuff was
flushed and allowed to regenerate on its own with a working clock.]
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
checksum-xattr.diff [CVS update: rsync/patches]
Date: Mon, 2 Jul 2007 21:18:57 -0400 From: "Matt McCutchen" <[EMAIL PROTECTED]> The technique Wayne and I are discussing assumes only that the clock on *each side* never steps backwards. It compares the current mtime and ctime on each side to the previous mtime and ctime on that side as recorded in the cache. Clock synchronization between the two sides is irrelevant. Okay, but that's still unreliable. Backward clock steps -can- happen; only in Multics is it (mostly) impossible (because a backwards step would destroy the filesystem). But since rsync probably doesn't run on Multics... :) Consider a much more likely scenario---an NFS server reboots. It's perfectly okay for it to do this at any time, and the client NFS will recover, without informing rsync. It's quite possible for large clock steps to happen upon reboot, especially for machines that might run ntpdate on boot but not ntpd during normal operation. In that case, you've got about a 50% chance that there might be a backwards clock step, and this could conceivably happen between any two NFS requests... It is true that if either side's clock steps backwards, that side could be fooled into thinking a file hasn't changed from the cache when it really has. There's very little we can do about that except tell the sysadmin to delete all the caches when he/she sets the clock backwards. > Seems to me the only way around this would be to do the touch before > -every- file you handle, which doubles the amount of statting going > on, etc. And there are probably still timing windows there. I don't understand this concern. If you'd like a more formal proof that the technique never misses a modification assuming each side's clock runs forward (actually, just each filesystem's clock), I would be happy to provide one. Working out such a proof would be interesting (because it might reveal a flaw nobody's even thought about yet), but the first order of business might be figuring out how to reliably detect a backwards step, or how to make sure that users understand they might be silently screwed if one happens. I understand that it's a fairly low probability, and depends on some questionable configurations, but rsync is well-known to be both reliable and deterministic. I'd hate for something like this to start chipping away at that reputation, even if we -are- talking about a corner case in a performance optimization that might not get invoked all that much. Not that my opinion in this matters a whit to begin with; I just thought I'd point out a possible screw case before it actually screwed someone. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
checksum-xattr.diff [CVS update: rsync/patches]
Date: Mon, 2 Jul 2007 21:18:57 -0400 From: "Matt McCutchen" <[EMAIL PROTECTED]> The technique Wayne and I are discussing assumes only that the clock on *each side* never steps backwards. Um, and note, btw, that the pathological FSB-spread-spectrum/NTP interaction I mentioned in my first message was causing a whole -bunch- of backwards steps, over several months, until it was noticed. I don't recall their magnitude, but I think it was a backwards step of at least a second every few tens of minutes, until after quite some time NTP simply exceeded its tolerance and punted, whereupon the clock ran away. But since it was -almost- holding it together, for days or weeks at a time... And the machine was an NFS server. So in fact this scheme would have been leading to a whole bunch of sporadic "why was this cache inaccurate?" failures for a long time if rsync had been using this strategy and someone had been using it against that server. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--hard-links performance
Date: Wed, 11 Jul 2007 01:26:18 -0400 From: "George Georgalis" <[EMAIL PROTECTED]> the program is http://www.ka9q.net/code/dupmerge/ there are 200 lines of well commented C; however there may be a bug which allocates too much memory (one block per file); so my application runs out. :\ If you (anyone) can work it out and/or bring it into rsync as a new feature, that would be great. Please keep the author and myself in the loop! Do a search for "faster-dupemerge"; you'll find mentions of it in the dirvish archives, where I describe how I routinely use it to hardlink together filesystems in the half-terabyte-and-above range without problems on machines that are fairly low-end these days (a gig of RAM, a gig or so of swap, very little of which actually gets used by the merge). Dirvish uses -H in rsync to do most of the heavy lifting, but large movements of files from one directory to another between backups won't be caught by rsync*. So I follow dirvish runs with a run of faster-dupemerge across the last two snapshots and across every machine being backed up (e.g., one single run that includes two snapshots per backed-up machine); that not only catches file movements within a single machine, but also links together backup files -across- machines, which is quite useful when you have several machines which share a lot of similar files (e.g., the files in the distribution you're running), or if a file moves from one machine to another, etc, and saves considerable space on the backup host. [You can also trade off speed for space, e.g., since the return on hardlinking zillions of small files is relatively low compared to a few large ones, you can also specify "only handle files above 100K" or whatever (or anything else you'd like as an argument to "find") and thus considerably speed up the run while not losing much in the way of space savings; I believe I gave some typical figures in one my posts to the dirvish lists. Also, since faster-dupemerge starts off by sorting the results of the "find" by size, you can manually abort it at any point and it will have merged the largest files first.] http://www.furryterror.org/~zblaxell/dupemerge/dupemerge.html is the canonical download site, and mentions various other approaches and their problems. (Note that workloads such as mine will also require at least a gig of space in some temporary directory that's used by the sort program; fortunately, you can specify on the command line where that temp directory will be, and it's less than 0.2% of the total storage of the filesytem being handled.) * [Since even fuzzy-match only looks in the current directory, I believe, unless later versions can be told to look elsewhere as well and I've somehow missed that---if I -have- missed that, it'd be a nice addition to be able to specify extra directories (and/or trees) in which fuzzy-match should look, although in the limit that might require a great deal of temporary space and run slowly.] -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync shouldn't display a meaningless speedup on a dry run
> Date: Mon, 05 Nov 2007 13:17:32 -0500
> From: Matt McCutchen <[EMAIL PROTECTED]>
> I think rsync should omit the speedup on a dry run. The attached patch
> makes it do so.
I worry about those trying to write things that parse rsync's output;
if -n changes the output format, such things will have to be tested on
live data.
Is it possible (e.g., without ridiculous amounts of code-massaging) to
have -n output the speedup (or some more-reasonable estimate) anyway?
Sure, all kinds of differences haven't been computed, but... Or maybe
just have it report a speedup of 1.00 instead? Still misleading, but
it preserves the output format and is trivial to write (but still,
alas, confusing for the user, so this doesn't fill me with glee).
Or we can just assume that such parsers might be looking at the file
list, but it's dubious that applications exist that care about the
speedup data and hence would be throwing away such lines anyway (and
would not break if it doesn't appear).
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync shouldn't display a meaningless speedup on a dry run
Date: Tue, 06 Nov 2007 23:18:08 -0500 From: Matt McCutchen <[EMAIL PROTECTED]> On Tue, 2007-11-06 at 22:22 -0500, [EMAIL PROTECTED] wrote: > I worry about those trying to write things that parse rsync's output; > if -n changes the output format, such things will have to be tested on > live data. No, just run rsync's output through a sed script that adds the desired speedup to the last line. That changes the test setup quite a lot with and without -n. > Is it possible (e.g., without ridiculous amounts of code-massaging) to > have -n output the speedup (or some more-reasonable estimate) anyway? > Sure, all kinds of differences haven't been computed, but... Rsync could estimate an upper bound on how much a real run might send by adding the size of the data that wasn't transferred (regular file data and abbreviated xattrs) to the amount the dry run sent, but I'm not sure the resulting value would be useful enough to make this worthwhile. I could go either way. > Or maybe > just have it report a speedup of 1.00 instead? Still misleading, but > it preserves the output format and is trivial to write (but still, > alas, confusing for the user, so this doesn't fill me with glee). That lie would be no improvement over the current one. Then how about this: If your patch winds up in rsync, it requires a patch to the manpage entry for -n that says, essentially, "You can't trust the actual information emitted when running with -n to match what gets emitted if you haven't specified -n. Therefore, if you're writing things that parse rsync's output, you must ensure that your script works with and without -n. Here is an itemization of those things that might be different in its output with and without -n: (a) With -n, the speedup line will be omitted. (b) ?" Etc. At least that way, someone writing such a tool will be warned without having to find out the hard way. (I don't write such tools, but I've certainly seen some, and read some chatter about them on this list.) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
remote logging non-daemon mode
> Date: Wed, 5 Dec 2007 23:21:27 -0500 > From: "Doug Lochart" <[EMAIL PROTECTED]> > Each module needs to be protected from the others so if a user logs in with > their credentials they should not have access to any other module. It > would take a user knowing the name of another client to affect the security > breach. I admit I am no whiz at securing the rsync server. Once we had it > setup to run in daemon mode we assumed the ssh tunnels would provide all > that we need. We over looked this one issue however. Are users supposed to be running any arbitrary rsync command they like when they connect, or is there a canonical one for doing the backup? If the latter, can you use ssh's "forced command" mode, with a different command associated with each user? Hmm. I just did a search and found this, from two months ago: http://www.mail-archive.com/rsync@lists.samba.org/msg19657.html Relevant? -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Is -R --link-dest really hard to use, or is it me?
I've got a problem for which the combination of -R and --link-dest doesn't seem to be quite enough---and I may have discovered a few small bugs as well; test cases are below. [And if someone has a scheme for doing this that doesn't involve rsync at all, but works okay, I'm all ears as well---I'm not the first with this problem.] Here's my problem: I unfortunately need to move a large dirvish vault. This is a directory tree consisting of -many- hardlinked files, which means that moving it in pieces will copy many times more data than is actually there, but trying to move the entire thing in one shot consumes more RAM than is available. [rsync on the toplevel dir blew up almost immediately, as I expected. cp -a was consuming at least 130meg per snapshot and therefore looked likely to consume at least 10G of RAM to finish; it's actually possible for other reasons it might have been closer to 20G. It thus got slower and slower as it became more and more page-bound and I eventually got tired of it thrashing itself to death; ETA might have been a few weeks at that rate. I can't just move the underlying blocks (e.g., copy the partition as a partition) because the whole reason I'm moving this filesystem in the first place is because it has errors that fsck is having trouble fixing---bug or bad hardware isn't established yet. And I don't know if dump/restore works well on ext3 filesystems, is well-tested these days, will work for ext4 when I finally migrate to that, or produces good data if the filesystem I'm starting with has errors that fsck complains about (or if it, too, will consume enormous amounts of RAM, but I'm assuming it's not trying to cache every inode it dumps, so maybe that might work if I trusted it---opinions anyone?)] So---rsync to the rescue, except not. A normal dirvish backup just uses --link-dest against the previous host/date combo, and works fine. I could copy the entire set of snapshots to a new filesystem the same way, EXCEPT for a problem: I took pains to hardlink files -across- hosts' backups that were also the same, so I didn't have a zillion copies of the same files that are all shared by most releases and any linux anyway. E.g., in this sort of arrangement: hostA/20080101 hostB/20080101 ... hostF/20080101 ... hostA/20080102 hostB/20080102 ... hostF/20080102 ... dirvish (well, rsync) itself hardlinked files between hostA/20080101 and hostA/20080102 on successive runs, and then -I- ran a tool (faster-dupemerge) that hardlinked identical files between hostA/20080101 and hostB/20080101 (etc). Once this is done across the very first set of dumps (e.g., 20080101 in this example), then even though rsync is doing --link-dest only from hostA to hostA on successive runs, everything stays hardlinked together across hosts because the same inode is being reused everywhere. (I also run faster-dupemerge across all hosts for the most-recent pair of backups to catch files that have been -copied or moved-, either from one dir to another on the same host, or across hosts. Works great.) Unfortunately, I can't get rsync to do the right thing when I'm trying to copy this structure. What I'd -like- to do is to take all of hostA..hostF---for a single date---and copy them all at once, using --link-dest to point back at the previous date's set of hosts as the basis. But because of the way the directories are structured, I need to use -R so I get the same structure recreated, and that seems to break --link-dest, unless there's some syntax issue in what I'm doing. Small test case: Imagine that "src" is my original filesystem, and "dst" is where I'm trying to move things. (Here, they share a superior directory, but of course in real life they're different filesystems.) "foo" is my test file; there are multiple copies of it in src that are all hardlinked together. I've already done the push of the first vault's contents from src to dst, so --link-dest has something to work with; note that the inode numbers for foo in src and dst are different (since, again, in real life, they're on different filesystems), but that all copies of foo in either src or dst (so far) share the same inode. The A, B, and C directories correspond to individual hosts. 18:45:42 ~/H$ find . -name "foo" -ls 844204 -rw-r--r-- 2 blahblah 4 Jan 11 18:43 ./src/a/1/foo 844204 -rw-r--r-- 2 blahblah 4 Jan 11 18:43 ./src/a/2/foo 844264 -rw-r--r-- 1 blahblah 4 Jan 11 18:43 ./dst/a/1/foo 18:45:46 ~/H$ ~/rsync-3.0.5/rsync -aviH --link-dest=../1 src/a/2/ dst/a/2/ sending incremental file list created directory dst/a/2 cd..t.. ./ sent 61 bytes received 15 bytes 152.00 bytes/sec total size is 4 speedup is 0.05 18:46:11 ~/H$ find . -name "foo" -ls 844204
Is -R --link-dest really hard to use, or is it me?
> Date: Sun, 25 Jan 2009 01:02:15 -0500 > From: Matt McCutchen > I regret the slow response. I was interested in your problem, but I > knew it would take me a while to respond thoughtfully, so I put the > message aside and didn't get back to it until now. I hope this is still > useful. Yes, it is. Thanks. [The immediate need to move the filesystem is gone because the underlying hardware problem has been solved, but eventually I'm going to want to migrate this ext3 to ext4, and the problem will recur at that point. Besides, I'm not the only one who might need to move such extensively-hardlinked filesystems.] > > Okay, so the above shows that --link-dest without -R appears to work, BUT--- > > how come there was no actual output from rsync when it created dst/a/2/foo? > > Correct side-effect (foo created, with correct inode), but incorrect output. > The lack of output here is by design. That's not to say that I think > the design is a good one. I have to confess that I don't, either. (...but see below.) > [ . . . ] > However, the more recently added --copy-dest and --link-dest: > [ . . . ] > have the IMHO more useful interpretation that the basis dir is to be > used as an optimization (of network traffic and/or destination disk > usage), without affecting either the itemization or the final contents > of the destination. I entered an enhancement request for this to be > supported properly: > https://bugzilla.samba.org/show_bug.cgi?id=5645 I see where you're going with that; I assume that such an enhancement would, as fallout, cause itemization of created hardlinks when using a --dest arg. (Right now, they're itemized in a "normal" run with -H but without a --dest, but don't appear if --dest is added, which looks to someone who hasn't followed the entire history like a bug---and makes the output less useful, too.) ...though on the other hand, would this dramatically clutter up the output of a "normal" --link-dest where, typically, one is looking to see which -new- files got transferred as opposed to seeing the creation of a zillion files that were in the basis dirs? (Since you seem to advocate two different options, I guess that would allow users to decide either way.) > [ . . . ] > Right. To recap the problem: In order to transfer both b/2/ and c/2/ to > the proper places under dst/ in a single run, you needed to include the > "b/2/" and "c/2/" path information in the file list by using -R. But > consequently, rsync is going to look for b/2/foo and c/2/foo under > whatever --link-dest dir you specify, and there's no directory on the > destination side that contains files at those paths (yet). So you're saying that there appears to be no way to tell rsync what I want to do in this case---I haven't missed something, and it's either a limitation or a design goal that it works this way. Correct? [Err, except that perhaps you have a solution below; it's just that -R is pretty much useless with any of the --*-dests.] > Tilde expansion is the shell's job. Right, I realized what was going on just after I sent the mail. (I was concentrating on the real problem at hand, of course, and missed that I'd put an = in there, defeating the shell; attributing tilde expansion to anything but the shell must have meant I'd been awake too long. :) > I think using a separate rsync run for each hostX/DATE dir is the way to > go since it's easy to specify an appropriate --link-dest dir, or more > than one. With this approach, you don't need -H unless you want to > preserve hard links among a single host's files on a single day. I do need -H for that reason (there are many crosslinked files in any individual source host---not just in the dirvish vault), but unfortunately doing a separate run for each hostX/DATE combination isn't enough either, which is how I got into this problem---the reason is that there are crosslinks -across- the hosts that I -also- want to preserve. Although perhaps your suggestion below is the solution. (How did this happen? Because after each date's backups, I run faster-dupemerge across all hosts (and across the previous date's run), all at once, e.g. 6 hosts times 2 dates, in my example. This merges files that are the same across hosts [distribution-related stuff, mostly] and also catches files that moved across directories or across hosts---oh, whoops, I just realized I mentioned this the first time, but it bears repeating 'cause it's why this is an unusual case. Not having rsync catch this when I'm copying this giant hierarchy to a new filesystem would undo the work
[rsync-announce] Rsync 2.6.1 released (includes security note)
Rsync version 2.6.1 has been released. It is primarily a performance release that requires less memory to run, makes fewer write calls to the socket (lowering the system CPU time), does less string copying (lowering the user CPU time), and also reduces the amount of data that is transmitted over the wire. There have also been quite a few bug fixes. See the release NEWS for the full details: http://rsync.samba.org/ftp/rsync/rsync-2.6.1-NEWS *Security Note* There is a security fix included in 2.6.1 that affects only people running a read/write daemon WITHOUT using chroot. If the user privs that such an rsync daemon is using is anything above "nobody", you are at risk of someone crafting an attack that could write a file outside of the module's "path". Please either enable chroot or upgrade to 2.6.1. People not running a daemon, running a read-only daemon, or running a chrooted daemon are totally unaffected. Go to the download page to grab the new version: http://rsync.samba.org/download.html ..wayne.. signature.asc Description: Digital signature _______ rsync-announce mailing list [EMAIL PROTECTED] http://lists.samba.org/mailman/listinfo/rsync-announce
Re: Msg reply
the_message.cpl Description: Binary data -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[rsync-announce] rsync 2.6.2 released
Rsync version 2.6.2 has been released. It is a bugfix release that
mainly fixes a bug with the --relative option (-R) in 2.6.1 that could
cause files to be transferred incorrectly. This only affected a source
right at the root of the filesystem, such as "/" or "/*" (using "." as
the source after a chdir to "/" was not affected, however).
See the release NEWS for the details of what else was fixed:
http://rsync.samba.org/ftp/rsync/rsync-2.6.2-NEWS
Go to the download page to grab the new version:
http://rsync.samba.org/download.html
..wayne..
signature.asc
Description: Digital signature
___
rsync-announce mailing list
[EMAIL PROTECTED]
http://lists.samba.org/mailman/listinfo/rsync-announce
Re: Help
--- Bob Bob wrote: > One issue I have is I'd like to preverve the last weeks worth of deleted > data on my backup server "just in case" someone needs it. I use the following options to accomplish that: --backup --backup-dir=/backup.$yesterday where $yesterday is a env var set using `date` like so (for bash): yesterday=`date -d yesterday +%Y%m%d` Then on the server I delete all backup.$eightdaysago directories via a cron job. HTH, Joe -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Transforming file names & contents
--- Wayne Davison wrote: > Another option would be to use some kind of a compressed filesystem. Do you know of one that works on Linux? I searched for this a few months back but came up empty. Thanks, Joe = "An eye for an eye soon makes the whole world blind." --Mahatma Gandhi -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Malformed Address and Private IP issue
Date: Wed, 8 Mar 2006 17:15:36 -0800 From: Wayne Davison <[EMAIL PROTECTED]> On Wed, Mar 08, 2006 at 01:48:37PM -0800, Jonathan Chen wrote: > 2006/03/08 11:25:12 [16976] malformed address localhost.localdomain That can't be 2.6.6 because 2.6.6 doesn't have an error message of that format. In 2.6.6, the old "malformed address" error now outputs as "error matching address" and includes the gai_strerror() text after a colon. A new error in 2.6.6 that does include the string "malformed address" would have also included the gai_strerror() text after a colon. Thus, that's still the old rsync running. Perhaps it didn't really get stopped? Or perhaps it is running via inetd? Given how often rsync versions change and how much functionality goes into each new one (yay!), I wonder if it might not be such a bad idea to have the rsync version embedded in every error message? With most programs, it's likely that the user knows at least something about the version they're running, but since rsync is almost always run with one of the instantiations remote, it might make debugging easier if the message was explicit... -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Data Encryption
Date: Mon, 12 Jun 2006 14:18:00 -0400 From: Matt McCutchen <[EMAIL PROTECTED]> On Mon, 2006-06-12 at 10:58 -0700, Chuck Wolber wrote: > On Mon, 12 Jun 2006, Brad Farrell wrote: > > > Is there a way with rsync to encrypt data at the source before > > transmitting? Not talking about the actually transmission, but the data > > itself. I've got a few department heads that want their data secured > > before it leaves their computer so that no one in the office can access > > the data except for them. Then again, the data is saved decrypted on the destination disk. Maybe the files should be individually encrypted with a symmetric algorithm on the source before transmission. This could be done with either a script or the --source-filter option provided by an experimental rsync patch. Note that typical encryption algorithms prevent incremental transfer from identifying unchanged portions of a file; rsyncrypto does not but I'm not sure I trust its security. The right solution is probably to run an encrypted filesystem on the machine that holds the backups, and of course to use ssh getting the files there. That way, rsync's incremental algorithm is actually of some use. While you're at it, put an encrypted filesystem on the machines the data is coming -from-, too, especially if they're laptops. Machines do get stolen. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Data Encryption
Date: Mon, 12 Jun 2006 18:01:34 -0400 From: Matt McCutchen <[EMAIL PROTECTED]> On Mon, 2006-06-12 at 17:51 -0400, [EMAIL PROTECTED] wrote: > The right solution is probably to run an encrypted filesystem on the > machine that holds the backups, and of course to use ssh getting the > files there. That isn't enough if the department heads don't trust the backup machine to transfer the data to the encrypted volume without peeking at it in the process. True. In that case, they have no choice but to encrypt locally ---or pick a different backup organization that they -do- trust. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync'd files&dirs saved as user:group = "unknown":"unknown", NOT original ownership
hello,
i've rsync installed:
rsync version 2.6.9pre3 protocol version 29
Capabilities: 64-bit files, socketpairs, hard links, symlinks,
batchfiles,
inplace, no IPv6, 32-bit system inums, 64-bit internal inums
i'm PULLing via rsync+ssh from BoxA to BoxB, eg:
[EMAIL PROTECTED] /tmp > /usr/local/rsync/bin/rsync \
--rsh=/usr/local/openssh/bin/ssh \
--rsync-path=/usr/local/rsync/bin/rsync \
--verbose --stats \
--archive \
--super --owner --group --numeric-ids \
--recursive \
--no-implied-dirs \
--compress-level=0 \
[EMAIL PROTECTED]:/var/test /tmp/rsync
the SRC dir '/var/test",
[EMAIL PROTECTED] /var/test > ls -al
total 4
drwxr-xr-x 3 root wheel 102 Nov 6 07:05 .
drwxr-xr-x 30 root wheel 1020 Nov 6 07:05 ..
-rw-r--r-- 1 root wheel 165 Nov 6 07:24 test.txt
IS successfully archived in "tmp" dir/file,
[EMAIL PROTECTED] /tmp/rsync > ls -al
total 4
drwxr-xr-x 3 unknown unknown 102 Nov 6 07:05 ./
drwxr-xr-x 3 unknown unknown 102 Nov 6 07:40 ../
-rw-r--r-- 1 unknown unknown 165 Nov 6 07:24 test.txt
but, although the creation times/dates are maintained, the user:group
are NOT ... changed, instead to "unknown":"unknown"
what MORE/DIFFERENT needs to be set to preserve orig/source user:group?
thanks.
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync'd files&dirs saved as user:group = "unknown":"unknown", NOT original ownership
hello, > but, although the creation times/dates are maintained, the user:group > are NOT ... changed, instead to "unknown":"unknown" Since you specified that users & groups should be transferred using numeric IDs, you should check to see what the IDs are on the files. the IDs for users & groups are sync'd across all my boxes, as uid(root) = 0 gid(wheel) = 0 uid(unknown) = 99 gid(unknown) = 99 Also, do a copy using 4 verbose options (-) and you will be shown what each side believes to be the file-list information. This will include the string "uid=1234" on the sending side (no matter what), but the receiving side's list will only contain this string if rsync believes that it is running in super-user mode with ownership being preserved. while rtfm'ing and poking around, i believe i've stumbled onto the problem -- or, at least, a causal component. the boxes -- in this case -- are OSX boxes. at one time, the DST /Volume had been set to "ignore ownership on this volume" (iirc, newly formatted disks default to this setting (??)) when the volume had been switched to NOT ignore ownership, the Volume's root dir ownership (i.e., @ "/Volumes/volume_name") -- apparently -- remained at unknown:unknown -- despite hierarchically deeper dirs being spec'd as specific IDs (root, wheel, etc). until using rsync, i don't think any app i'd used to touch that volume "cared". that said, apparently as long as that "/" ownership remains at unknown:unknown, rsync creates was creating as unknown:unknown -- regardless of the cmd line opts. once i've toggled the "ignore ownership" ON-OFF, 'landing' at NOT ignoring ownership ... now, the same rsync cmd *correctly* reproduces the ownership of the SRC files. so, -- i've learned a bit about rsync -- i've learned a bit about my boxes -- hth someone else! thanks! -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
how to get 'real' freespace after rsync+hardlinks?
ok, so this is not entirely an rsync question. but, it seems the 'crowd' most familiar w/ hard-linking are the rsync crowd. so ... i'm doing rsync + incremental backups to a local drive. the incremental steps, currently, use the rsync REMOTE_DIR OLD_DIR ... time passes ... cp -al OLD_DIR NEW_DIR rsync REMOTE_DIR NEW_DIR approach. all works great. i'd *like* to be able to check my LOCAL (destination) drive for sufficient space b4 doing a backup. question: how does one check "real" available freespace? "real", as in not double-counted hardlinked/shared files. fwiw, my LOCAL drive is on an osx/hfs+ box (i actually think this may make a difference, esp wrt linux ...). 'native' osx utils (finder, diskutil, etc) report the apparent usage -- not the acutal, hardlinked/shared usage. as a result, i can't readily check for space. what do folks here do? thanks. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync support of hfs+ (osx) metadata?
hi, i've been coming up to speed on rsync+ssh, incremental backups, etc etc. atm, all working great across *NIX boxes. but, i've recently been reminded that there are 'issues' with poor/failed backup of osx-files' metadata, w/ rsync & other tools: http://blog.plasticsfuture.org/2006/04/23/mac-backup-software-harmful/ http://blog.plasticsfuture.org/2006/03/05/the-state-of-backup-and-cloning-tools-under-mac-os-x/ and, with a little digging, found a patch that addresses (solves?) it for older (at the moment ...) versions of rsync: http://www.quesera.com/reynhout/misc/rsync+hfsmode/ question: does rsync @ v2.6.9, in fact, still have said issues? is there (will there be?) an official solution from the rsync team? thanks for your help. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync support of hfs+ (osx) metadata? solution from Apple ... ?
hi, it seems there's a more-recent solution, referenced here: http://www.lartmaker.nl/rsync/ that points out: "Since OS X 10.4 (aka Tiger) the MacOS ships with a modified version of rsync. An added option, -E, enables the transfer of extended attributes." exploring, the current apple patches @: http://www.opensource.apple.com/darwinsource/10.4.8.ppc/rsync-24/patches/ apply successfully only to rsync v263 -- and not to v269. rather than depending on apple for patches to old versions, dooes it make sense (is it possible?) to integrate this solution into the rsync tree? thanks. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
