from:"j"

[PATCH] add option to skip files based on age/mtime

2024-03-17 Thread J Raynor via rsync

I've added the options --min-age=SECONDS and --max-age=SECONDS to allow 
rsync to skip files based on how recently they were modified.


Setting --min-age=30 (for example) would cause rsync to skip files that 
had been modified within the last 30 seconds.


Setting --max-age=7776000 would cause rsync to skip files that had been 
modified more than 90 days ago (7776000 == 60*60*24*90).


I realize some of this could be achieved with a find command and 
--exclude-from, but the same could be said for --min-size/--max-size. 
But it's more convenient to have these options, and they would allow for 
things --exclude-from wouldn't.  For example, a file might've been 
modified after generating the exclude file, so rsync would still 
transfer it even though that wasn't what was wanted.






diff --git a/generator.c b/generator.c
index 110db28f..22f0973f 100644
--- a/generator.c
+++ b/generator.c
@@ -70,6 +70,8 @@ extern int ignore_times;
 extern int size_only;
 extern OFF_T max_size;
 extern OFF_T min_size;
+extern int max_age;
+extern int min_age;
 extern int io_error;
 extern int flist_eof;
 extern int allowed_lull;
@@ -1706,6 +1708,23 @@ static void recv_generator(char *fname, struct 
file_struct *file, int ndx,

goto cleanup;
}

+   if (max_age > 0 && time(NULL)-file->modtime > max_age) {
+   if (INFO_GTE(SKIP, 1)) {
+   if (solo_file)
+   fname = f_name(file, NULL);
+   rprintf(FINFO, "%s is over max-age\n", fname);
+   }
+   goto cleanup;
+   }
+   if (min_age > 0 && time(NULL)-file->modtime < min_age) {
+   if (INFO_GTE(SKIP, 1)) {
+   if (solo_file)
+   fname = f_name(file, NULL);
+   rprintf(FINFO, "%s is under min-age\n", fname);
+   }
+   goto cleanup;
+   }
+
 	if (update_only > 0 && statret == 0 && file->modtime - sx.st.st_mtime 
< modify_window) {

if (INFO_GTE(SKIP, 1))
rprintf(FINFO, "%s is newer\n", fname);
@@ -2156,9 +2175,13 @@ void check_for_finished_files(int itemizing, enum 
logcode code, int check_redo)

if (check_redo && (ndx = get_redo_num()) != -1) {
OFF_T save_max_size = max_size;
OFF_T save_min_size = min_size;
+   int save_max_age = max_age;
+   int save_min_age = min_age;
csum_length = SUM_LENGTH;
max_size = -1;
min_size = -1;
+   max_age = 0;
+   min_age = 0;
ignore_existing = -ignore_existing;
ignore_non_existing = -ignore_non_existing;
update_only = -update_only;
@@ -2184,6 +2207,8 @@ void check_for_finished_files(int itemizing, enum 
logcode code, int check_redo)

csum_length = SHORT_SUM_LENGTH;
max_size = save_max_size;
min_size = save_min_size;
+   max_age = save_max_age;
+   min_age = save_min_age;
ignore_existing = -ignore_existing;
ignore_non_existing = -ignore_non_existing;
update_only = -update_only;
diff --git a/options.c b/options.c
index fd674754..7c7f31db 100644
--- a/options.c
+++ b/options.c
@@ -129,6 +129,8 @@ int need_messages_from_generator = 0;
 int max_delete = INT_MIN;
 OFF_T max_size = -1;
 OFF_T min_size = -1;
+int max_age = 0;
+int min_age = 0;
 int ignore_errors = 0;
 int modify_window = 0;
 int blocking_io = -1;
@@ -698,6 +700,8 @@ static struct poptOption long_options[] = {
   {"ignore-existing",  0,  POPT_ARG_NONE,   _existing, 0, 0, 0 },
   {"max-size", 0,  POPT_ARG_STRING, _size_arg, 
OPT_MAX_SIZE, 0, 0 },
   {"min-size", 0,  POPT_ARG_STRING, _size_arg, 
OPT_MIN_SIZE, 0, 0 },

+  {"max-age", 0,  POPT_ARG_INT, _age, 0, 0, 0 },
+  {"min-age", 0,  POPT_ARG_INT, _age, 0, 0, 0 },
   {"max-alloc",0,  POPT_ARG_STRING, _alloc_arg, 0, 0, 0 },
   {"sparse",  'S', POPT_ARG_VAL,_files, 1, 0, 0 },
   {"no-sparse",0,  POPT_ARG_VAL,_files, 0, 0, 0 },
@@ -2815,6 +2819,16 @@ void server_options(char **args, int *argc_p)
args[ac++] = safe_arg("--min-size", min_size_arg);
if (max_size >= 0)
args[ac++] = safe_arg("--max-size", max_size_arg);
+   if (min_age > 0) {
+   if (asprintf(, "--min-age=%d", min_age) < 0)
+   goto oom;
+   args[ac++] = arg;
+   }
+   if (max_age > 0) {
+   if (asprintf(, "--max-age=%d", max_age) < 0)
+   goto oom;
+

Rsync slow with hard links

2017-04-12 Thread Rene J Suarez-Soto via rsync

I am running rsync throuhg cygwin and trying to "mirror" two hard drives
(hd1 -> hd2). HD1 have tons of hard links (I will say about 2 millions)
because I do monthly backups and have about 13 folders and maybe about
150,000 files. Every month when I make a new snapshot I would also want to
mirror HD1 to HD2. I tried running

rsync -aH --progress -vv /cygdrive/d/ /cygdrive/e/

I though this was going to be a very quick process because not many files
changed but surprisingly it takes over a day. Is there a faster way to
mirror drives with hard links?
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync failure on corrupted source data

2016-06-27 Thread Paul J. Durack

I am trying to copy data from a partially corrupt backup (time machine)
disk on a os x system to another.

Firstly the rsync version (OS X 10.11.5):

sync --version
rsync  version 2.6.9  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.

Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
  inplace, IPv6, 64-bit system inums, 64-bit internal inums

I'm using the following command:

~] durack1% sudo rsync -vrlptgoDH --exclude=".spotlight_repair"
--ignore-errors /Volumes/durack1ml_bak\ 1/Backups.backupdb
/Volumes/durack1ml_bak/160405_1234/

And am getting the following error:

Backups.backupdb/durack1ml/2016-02-10-091749/durack1ml_hdd/Applications/Adobe
Media Encoder CC 2015/Adobe Media Encoder CC
2015.app/Contents/Resources/pdfl/CMaps/ETen-B5-UCS2
rsync: writefd_unbuffered failed to write 14550 bytes [sender]: Broken pipe
(32)
rsync: write failed on
"/Volumes/durack1ml_bak/160405_1234/Backups.backupdb/durack1ml/2016-02-10-091749/durack1ml_hdd/Applications/Adobe
Media Encoder CC 2015/Adobe Media Encoder CC
2015.app/Contents/Resources/pdfl/CMaps/ETen-B5-UCS2": Result too large (34)
rsync error: error in file IO (code 11) at
/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/receiver.c(268)
[receiver=2.6.9]
rsync: connection unexpectedly closed (14352612 bytes received so far)
[sender]
rsync error: error in rsync protocol data stream (code 12) at
/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/io.c(453)
[sender=2.6.9]

Is there a way to get rsync to continue when such an error is encountered?

Cheers,

P
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync -H option yields corrupt replicas (due to non-unique inode ids)

2013-09-05 Thread Andrew J. Romero

Hi,

Our organization hosts a specialized Linux distribution.

As is typical with Linux distributions, 
the set of files that make up our Linux distro 
contains a very complex web of self-referential hard links.

Several other sites use our  Linux distro
and maintain either partial or full
internal mirror copies of it.

The standard method used by Linux mirror sites to
pull/replicate a subset of a Linux distribution
(or a complete Linux distribution) from a master
repository is to use rsync with options that
produce the following behavior:

  the first time a unique file is encountered, 
  it's content is replicated; however,  when subsequent hard links 
  to the file are detected, only the hardlinks are replicated.

The primary copy of our Linux distro
is stored on our BlueArc Titan NAS
(NFS server). Relative to the mirror-sites, 
our rsync server sits in front of the NAS.

Internally the BlueArc Titan has a unique object id
for files; however, the inode ID presented to clients 
by the BlueArc Titan is not unique, 
rsync (with -H option) is erroneously 
identifying unique files
as a hard-links to different files.
Causing mirror repositories to be essentially corrupt
and not usable.

It is my understanding that the NFS v3 spec.
does not require NFS servers to present unique inode
ids to clients. I believe that the reasoning is that:
large scale NAS appliances internally need to
use very wide object ids; but, externally need to
present (when asked) inode ids that any client
an deal with.

Are there options to rsync that will
allow me to reliably replicate my 
hard-link rich Linux distro from my NAS.

Thanks

Andy

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Static files have vanished.

2012-09-24 Thread Mestnik, Michael J - Eagan, MN - Contractor

I'm confused about this, perhaps something is wrong with rsync?

rsync  version 3.0.4  protocol version 30
Linux # 2.6.32.59-0.3-default #1 SMP 2012-04-27 11:14:44 +0200 x86_64 
x86_64 x86_64 GNU/Linux
$ cat /etc/SuSE-release
SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 1

nfs 
rw,nosuid,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=#,mountvers=3,mountport=56892,mountproto=tcp,local_lock=none,addr=#

I'm positive the files are not being changed, they are only updated when a 
computer is installed or removed and the frequency of this, as you can guess, 
is not such that these files should never(vary rarely ever) be considered 
vanished.

Nevertheless consecutive runs indicate they are in a state of perpetual 
vanishing.

Cheers!

Mike Mestnik, Michael J
The ESM Tools
Enterprise Systems Monitoring
United States Postal Service
 O: (651) 406-2048
michael.j.mest...@usps.gov
itenterprisesystemsmonitor...@usps.gov
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

cannot rsync when source directory lacks write permission

2012-07-27 Thread Brian J. Murrell

I seem to be running into a problem where I am trying to rsync from a
source directory that lacks write permissions (i.e. r-xr-xr-x).
Presumably this is because rsync creates the directory on the
destination, then sets the permissions to match the source and then
tries to sync the contents of the directory, which it cannot of course
lacking write permission in the directory.

Is there a way to have rsync first sync a directory and then set the
permissions to match the source?  Of course, this needs to be done
recursively so that permissions are set as rsync ascends back up the
directory tree.

Cheers,
b.



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

FW: rsync hanging after working reliably for years on one of my shares

2012-07-17 Thread Rob J. Caskey

Thanks for the suggestions - I have done my best to follow them.

I inserted the debug shim script and changed the rsync executable in rsync so 
that it called that instead, and got the same results. Here is the tail from 
that:

21381  0.000108 open(www.athenshousing.org/xmlrpc.php, 
O_RDONLY|O_LARGEFILE) = 3
21381  0.000122 fstat64(3, {st_mode=S_IFREG|0644, st_size=294, ...}) = 0
21381  0.000327 read(3, ?php\n\n/**\n * @file\n * PHP page for handling 
incoming XML-RPC requests from clients.\n */\n\ninclude_once './includes/boot
strap.inc';\ndrupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);\ninclude_once 
'./includes/xmlrpc.inc';\ninclude_once 
'./includes/xmlrpcs.inc';\n\nxmlrpc_server(modul
e_invoke_all('xmlrpc'));\n, 294) = 294
21381  0.045446 close(3)= 0
21381  0.90 select(2, NULL, [1], [1], {60, 0}) = 1 (out [1], left {59, 
96})
21381  0.000101 write(1, \261\5\0\7, 4) = 4
21381  0.94 select(2, NULL, [1], [1], {60, 0}) = 1 (out [1], left {59, 
97})
21381  0.000103 write(1, e_results']);\n  
unset($_SESSION['update_success']);\n\n  return $output;\n}\n\nfunction 
update_info_page() {\n  // Change qu
ery-strings on css/js files to enforce reload for all users.\n  
_drupal_flush_css_js();\n  // Flush the cache of all data for the update status 
module.\n
if (db_table_exists('cache_update')) {\ncache_clear_all('*', 
'cache_update', TRUE);\n  }\n\n  update_task_list('info');\n  
drupal_set_title('Drupal dat
abase update');\n  $token = drupal_get_token('update');\n  $output = 'pUse 
this utility to update your database whenever a new release of Drupal or a mod
ule is installed./ppFor more detailed information, see the a 
href=\http://drupal.org/upgrade\;upgrading handbook/a. If you are unsure 
what these t
erms mean you should probably contact your hosting provider./p';\n  $output 
.= \ol\\n\;\n  $output .= \listrongBack up your database/strong. T
his process will change your database values and in case of emergency you may 
need to revert to a backup./li\\n\;\n  $output .= \listrongBack up yo
ur code/strong. Hint: when backing up module code, do not leave that backup 
in the 'modules' or 'sites/*/modules' directories as this may confuse Drupal'
s auto-discovery mechanism./li\\n\;\n  $output .= 'liPut your site into a 
href=\'. base_path() .'?q=admin/settings/site-maintenance\maintenance mo
de/a./li'.\\\n\;\n  $output .= 
\li\370\377\377\377\367\377\377\377\366\377\377\377\365\377\377\377\364\377\377\377\363\377\377\377\0\0\0\0\273\37\
304\335\274T\376\334\207\312!\362\t\246\333[\243b\0\0\1\0\0\0\0\10\0\0\2\0\0\0\1\0\0\377\377\377\377\0\0\0\0\353\245\267C\35\323\353!\274\274\245\201\211\
306\3140, 1457) = 1457
21381  0.000617 select(2, NULL, [1], [1], {60, 0}) = 1 (out [1], left {59, 
97})
21381  0.97 write(1, \4\0\0\7\377\377\377\377, 8) = 8
21381  0.95 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
21381 60.060243 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
21381 60.008189 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)
21381 60.060284 select(1, [0], [], NULL, {60, 0}) = 0 (Timeout)

Also, I did do a manual rsync of that last file using similar options, and it 
worked fine.

The share I am backing up contains 3 folders, an index.html and an htaccess 
file. No packages changed on the backup or target server in the prior 3 months. 
No files have changed on the target server of note in the last month (just some 
xml site layout files spit out by wordpress and the nightly mysql dump). No 
drastic changes in file size. Stumped...

Adding the /var/www share back in...hangs again...

Ideas?

--Rob

From: Rob J. Caskey
Sent: Friday, July 13, 2012 9:21 AM
To: 'rsync@lists.samba.org'
Subject: rsync hanging after working reliably for years on one of my shares

Howdy all,

I'm using rsync in conjunction with backuppc and have been backing up this 
share without incident for almost 3 years and it has decided to go and hang on 
me. Other shares on the same machine sync fine with identical settings, and 
nothing has changed on the backup end other than some routine dist-upgrades to 
keep update on security patches. I checked the apt cache and no rsync related 
dpkgs seemed to be hanging around in there so...

Checked files with recent ctimes and mtimes, nothing funny looking there like 
recursive links.

Log excerpt attached beneath. I think alphabetically xmlrpc.php may be the last 
file in the share, is it coincidence it hangs right after? Thoughts? Ideas? 
Unfounded speculation? Blatant slander?  Anything? Just about out of ideas on 
my end. I've got a dozen or so other hosts that are happily syncing away and 
passing all their unit tests but this one has greeted me the last 5 or 6 days 
with a sad face and I don't seem to be making any progress. Any ideas at all? 
Again, log snippet underneath. Let me know if this is a backuppc issue and I am 
barking up the wrong tree, but there is definitely an rsync process sitting

Hlink node data for 2282618 already has path=...

2012-06-24 Thread Brian J. Murrell

So, like a lot of people, I am using --link-dest to do backups.  My
backup target is ext4 so with a hard link limit of 64K.  I do end up
with trying to create too many links at some point though and get the
following sequence of events:

rsync: link 
/.snapshots/hourly.0/linux/usr/src/openwrt-r18617/build_dir/host/mpfr-2.4.1/tests/.deps/tui_sub.Po
 = 
usr/src/openwrt-r18617/build_dir/host/bison-2.3/examples/calc++/.deps/calc++-driver.Po
 failed: Too many links (31)
Hlink node data for 2282618 already has 
path=usr/src/openwrt-r18617/build_dir/host/bison-2.3/examples/calc++/.deps/calc++-driver.Po
 (usr/src/openwrt-r18617/build_dir/host/mpfr-2.4.1/tests/.deps/tui_sub.Po)
rsync error: errors with program diagnostics (code 13) at hlink.c(548) 
[generator=3.0.9]
rsync: connection unexpectedly closed (222362 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) 
[sender=3.0.9]

It seems awfully heavy-handed for this Too many links condition to
be treated as an outright error.  Surely it's sufficient to just fail
to hard-link and move on, yes?

Any thoughts?

Cheers,
b.



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--link-dest doesn't work if target file exists (but needs updating)

2012-01-20 Thread Brian J. Murrell

Using:

# rsync --version
rsync  version 3.0.7  protocol version 30
Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
append, ACLs, xattrs, iconv, symtimes

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Given two files:

# /bin/ls -li {source,dest}/var/lib/mysql/mythconverg/jobqueue.MYD
44695517 -rw-rw 2 sshd messagebus 5492 2012-01-19 00:31 
source/var/lib/mysql/mythconverg/jobqueue.MYD
49676928 -rw-rw 1 sshd messagebus 5492 2012-01-18 12:00 
dest/var/lib/mysql/mythconverg/jobqueue.MYD

Their md5sums:

# md5sum {source,dest}/var/lib/mysql/mythconverg/jobqueue.MYD
b87dafdcf59ab3e7e9907c15385175b3  source/var/lib/mysql/mythconverg/jobqueue.MYD
c03b1ee584ef1405958b695f4e55b51d  dest/var/lib/mysql/mythconverg/jobqueue.MYD

Let's try to create a link from
source/var/lib/mysql/mythconverg/jobqueue.MYD to
dest/var/lib/mysql/mythconverg/jobqueue.MYD

Dry-run first to see what rsync tells us it's going to do:

# rsync -naiiAXH --link-dest=/source/var/lib/mysql/mythconverg/ 
source/var/lib/mysql/mythconverg/jobqueue.MYD dest/var/lib/mysql/mythconverg/
f..t.. jobqueue.MYD

Looks like it's not going to create a link.  Let's be sure:

# rsync -aiiAXH --link-dest=/source/var/lib/mysql/mythconverg/ 
source/var/lib/mysql/mythconverg/jobqueue.MYD dest/var/lib/mysql/mythconverg/
f..t.. jobqueue.MYD

Yup, looks like it didn't create a link.  The proof:

# /bin/ls -li {source,dest}/var/lib/mysql/mythconverg/jobqueue.MYD
44695517 -rw-rw 2 sshd messagebus 5492 2012-01-19 00:31 
source/var/lib/mysql/mythconverg/jobqueue.MYD
49676897 -rw-rw 1 sshd messagebus 5492 2012-01-19 00:31 
dest/var/lib/mysql/mythconverg/jobqueue.MYD

Yet it did copy the file:

# md5sum {source,dest}/var/lib/mysql/mythconverg/jobqueue.MYD
b87dafdcf59ab3e7e9907c15385175b3  source/var/lib/mysql/mythconverg/jobqueue.MYD
b87dafdcf59ab3e7e9907c15385175b3  dest/var/lib/mysql/mythconverg/jobqueue.MYD

Why didn't it remove the existing file and create the link like we
wanted it to with --link-dest?  Let's try another file just to prove
that it will create a link if the destination file doesn't exist:

Two different files, same use case though:

# /bin/ls -li {source,dest}/var/lib/mysql/mythconverg/jobqueue.MYI
44695518 -rw-rw 2 sshd messagebus 5120 2012-01-19 00:31 
source/var/lib/mysql/mythconverg/jobqueue.MYI
49676956 -rw-rw 1 sshd messagebus 5120 2012-01-18 12:00 
dest/var/lib/mysql/mythconverg/jobqueue.MYI

Different content again:

# md5sum {source,dest}/var/lib/mysql/mythconverg/jobqueue.MYI
6a0b5bdedfe738fbca17630e6496f85c  dest/var/lib/mysql/mythconverg/jobqueue.MYI
9052f8e598f8ee3bdcb2cbc510f0564c  source/var/lib/mysql/mythconverg/jobqueue.MYI

See what rsync says it's going to do (before we remove the target file):

# rsync -naiiAXH --link-dest=/source/var/lib/mysql/mythconverg/ 
source/var/lib/mysql/mythconverg/jobqueue.MYI dest/var/lib/mysql/mythconverg/
f..t.. jobqueue.MYI

Again, it's going to copy.

But if we remove the target file and see what rsync says it's going to do:

# rm dest/var/lib/mysql/mythconverg/jobqueue.MYI
# rsync -naiiAXH --link-dest=/source/var/lib/mysql/mythconverg/ 
source/var/lib/mysql/mythconverg/jobqueue.MYI 
dest/var/dest/var/lib/mysql/mythconverg/  
hf  jobqueue.MYI

Ah ha!  Say's it's going to link.  Let's actually do it and see:

# rsync -aiiAXH --link-dest=/source/var/lib/mysql/mythconverg/ 
source/var/lib/mysql/mythconverg/jobqueue.MYI dest/var/lib/mysql/mythconverg/   
hf  jobqueue.MYI

And sure enough, we now have three links to the same file (as we would
expect):

# /bin/ls -li {source,dest}/var/lib/mysql/mythconverg/jobqueue.MYI
44695518 -rw-rw 3 sshd messagebus 5120 2012-01-19 00:31 
source/var/lib/mysql/mythconverg/jobqueue.MYI
44695518 -rw-rw 3 sshd messagebus 5120 2012-01-19 00:31 
dest/var/lib/mysql/mythconverg/jobqueue.MYI

And the content (although we know it should report the same):

# md5sum {source,dest}/var/lib/mysql/mythconverg/jobqueue.MYI
9052f8e598f8ee3bdcb2cbc510f0564c  dest/var/lib/mysql/mythconverg/jobqueue.MYI
9052f8e598f8ee3bdcb2cbc510f0564c  source/var/lib/mysql/mythconverg/jobqueue.MYI

So why doesn't the same thing work when the destination file exists?
The element of least surprise would make me think it should.  That is,
even if the destination file exists, if the result of the sync is the
same as a file that a --link-dest points to, it should remove the
destination file and create the link.

Thoughts?

b.






signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options:

Re: --link-dest doesn't work if target file exists (but needs updating)

2012-01-20 Thread Brian J. Murrell

On 12-01-20 05:42 PM, Kevin Korb wrote:
 Am I understanding right that your source and your link-dest are
 actually the same path?

Yes they are!

 If so what are you trying to do that wouldn't
 be accomplished with a simple 'cp -l'?

Not all of the files in source and dest are supposed to be the same.  I
only want to link the files that are.

 Also, when using --link-dest the target is treated differently.
 Normally the target is an empty directory but if files already exist
 the behavior you are seeing is what is supposed to happen.  Instead of
 deleting files and replacing them with links it assumes that they are
 already linked to whatever they are supposed to be linked to and
 updates them accordingly.

That's a pity.  Understandable perhaps for the case where there is more
than one link to the destination file, but if there is not (i.e. there
is only one link to the file, such as the example I provided) it should
feel free to unlink it and relink it to the -link-dest specified file.
Additionally, I would think an option to force that behavior even if
there is 1 link would be useful.

b.



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: --link-dest doesn't work if target file exists (but needs updating)

2012-01-20 Thread Brian J. Murrell

On 12-01-20 06:01 PM, Kevin Korb wrote:
 Someone has requested it:
 https://bugzilla.samba.org/show_bug.cgi?id=7870

I'm not really sure that is the same bug.  Maybe it is.  Not convinced
though.  I guess I can file my own bug and ask 7870's OP to see if it's
the same issue.

 But is seems like an rm -rf and a cp -al would do the same thing.

But as I said before, it won't.  Not all files in the destination are
the same as the source and files may exist in the destination that don't
exist in the source.

b.



signature.asc
Description: OpenPGP digital signature
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: disable interpretation of trailing slash

2012-01-09 Thread John J Foerch

Voelker, Bernhard bernhard.voel...@siemens-enterprise.com writes:
 John J Foerch wrote:

 So I may take up your suggestion for my own use, but I did want to put
 forward the idea that since in my experience, trailing slash
 interpretation is an often-mentioned stumbling-block for rsync users,
 maybe it would be appropriate for rsync itself to provide a convenient
 alternative.

 I've also been using rsync for many years, and from right at the
 beginning, I started using the dir/. syntax to not fall into that
 trap. It always works and is also nice to use in scripts as an
 accidental dir/./. does not harm ;-)

 Have a nice day,
 Berny

Is a trailing /. exactly equivalent to a trailing / in all ways?  If
so, then if the command-line option I asked for were introduced, when
used, there would still be a way to specify contents of a source by
means of /. instead of /.  That would be a bonus.

Cheers.

-- 
John Foerch

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: disable interpretation of trailing slash

2012-01-08 Thread John J Foerch

Wayne Davison way...@samba.org writes:
 One solution is to install a bash script (or setup a bash function) in place 
 of
 the real rsync that strips a single trailing slash from each arg.  That lets
 you specify dir// if you want to copy a dir's contents.  Here's one I did some
 mild tesing on and it appears to work OK:


Wayne Davison way...@samba.org writes:
 On Sat, Jan 7, 2012 at 11:19 PM, Wayne Davison way...@samba.org wrote:

 Here's one I did some mild tesing on and it appears to work OK:


 I just realized that script would turn the arg / into an empty string.
  You'll want to fix that if you use the idea.  Also, if you want the idiom to
 be optional, you could either (1) have the bash script look for an opt and
 decide if it should do the stripping, or (2) name the script something like
 rs or rsyncs (or whatever) and use the slash-strip version only when
 desired.

This is definitely a workable idea, and I may use some variant of it,
but here are some thoughts:

 - I wouldn't use the name rsync for the wrapper script, because that
   is just setting oneself up for accidents, analogous to the common
   example of why it is a bad idea to 'alias rm=rm -i': safety-nets
   are fine until you're in a context without them, and you forget that
   they aren't there.

 - Stripping a slash from every argument is not quite the whole story,
   because options which take arguments could have significant trailing
   slashes.

So I may take up your suggestion for my own use, but I did want to put
forward the idea that since in my experience, trailing slash
interpretation is an often-mentioned stumbling-block for rsync users,
maybe it would be appropriate for rsync itself to provide a convenient
alternative.

Thank you for your time.

-- 
John Foerch

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

disable interpretation of trailing slash

2012-01-07 Thread John J Foerch

Hello,

This is a feature request for a command-line option to disable special
interpretation of trailing slashes on source directories.  I have been
using rsync for a couple of years now (what an awesome program!) and the
meaning of the trailing slash on a source is always the one aspect of
its syntax that slows me down and makes me check and re-check the manual
to ensure that I have remembered its behavior correctly.  The likelihood
of making a mistake with a trailing slash is compounded by the fact that
shell completion (e.g. in bash) puts it there by default, and it was
this very behavior that caused me to be a little too carefree last week
and mess up a large backup.  While the trailing slash is a powerful and
useful feature, it also strikes me as risky to have so much power built
into a single character of syntax, especially a character which usually
has no special meaning in other programs.  Others I have chatted with on
this topic shared the same view.  My feature request then, is for a
command line switch to disable the special interpretation of the
trailing slash on sources.  A short option would be preferred.  My
reasoning is that it is easier to remember to always use a particular
option, except in situations requiring more nuance, than it is to
remember the special rule and to always check the syntax of the sources.

Thank you for your time.

-- 
John Foerch

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: disable interpretation of trailing slash

2012-01-07 Thread John J Foerch

My request is not to change the default behavior, but to add a new
command-line option that would effect new behavior.  When one wants the
default behavior, one simply does not use the option.

Kevin Korb k...@sanitarium.net writes:
 I don't speak for anyone else but I would be opposed to this.
 Primarily because there isn't an alternate syntax that performs the
 exact same function /path/* doesn't work as expected if using --delete
 (and for good reason).

 On 01/07/12 21:40, John J Foerch wrote:
 Hello,
 
 This is a feature request for a command-line option to disable
 special interpretation of trailing slashes on source directories.
 I have been using rsync for a couple of years now (what an awesome
 program!) and the meaning of the trailing slash on a source is
 always the one aspect of its syntax that slows me down and makes me
 check and re-check the manual to ensure that I have remembered its
 behavior correctly.  The likelihood of making a mistake with a
 trailing slash is compounded by the fact that shell completion
 (e.g. in bash) puts it there by default, and it was this very
 behavior that caused me to be a little too carefree last week and
 mess up a large backup.  While the trailing slash is a powerful
 and useful feature, it also strikes me as risky to have so much
 power built into a single character of syntax, especially a
 character which usually has no special meaning in other programs.
 Others I have chatted with on this topic shared the same view.  My
 feature request then, is for a command line switch to disable the
 special interpretation of the trailing slash on sources.  A short
 option would be preferred.  My reasoning is that it is easier to
 remember to always use a particular option, except in situations
 requiring more nuance, than it is to remember the special rule and
 to always check the syntax of the sources.
 
 Thank you for your time.
 

 -- 
 ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
   Kevin Korb  Phone:(407) 252-6853
   Systems Administrator   Internet:
   FutureQuest, Inc.   ke...@futurequest.net  (work)
   Orlando, Floridak...@sanitarium.net (personal)
   Web page:   http://www.sanitarium.net/
   PGP public key available on web site.
 ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Intermittent rsync Issues

2010-09-21 Thread J. T. Gray

Izidor, this appears to resolve my problem. Thank you for your input, I
will be sure to attribute you on serverfault.com when I add an answer to my
question post.

On Tue, Sep 21, 2010 at 2:49 AM, Izidor Jerebic ij.l...@gmail.com wrote:

Hi,

this might be a problem with maximal number of concurrent ssh connections
or connection requests. Ssh daemon has two configuration settings where you
can define what is the maximal number of clients which can connect
concurrently. This number is by default not very high, so you are probably
bumping against that limit.

MaxSessions

Specifies the maximum number of open sessions permitted per
net-
work connection. The default is 10.

MaxStartups

Specifies the maximum number of concurrent unauthenticated con-
nections to the SSH daemon. Additional connections will be
dropped until authentication succeeds or the LoginGraceTime
expires for a connection. The default is 10.
Alternatively, random early drop can be enabled by specifying
the
three colon separated values ``start:rate:full'' (e.g.
10:30:60). sshd(8) will refuse connection attempts with a
probability of ``rate/100'' (30%) if there are currently
``start'' (10) unauthenticated connections. The probability
increases linearly and all connection attempts are refused if
the
number of unauthenticated connections reaches ``full'' (60).

izidor

On 20.9.2010, at 19:19, J. T. Gray wrote:

Greetings,

I posted this at serverfault, but in hopes that the people on this list
are different from the people there, here it is:

2010/09/20 16:27:01 [18581] rsync error: error in rsync protocol data
stream (code 12) at io.c(601) [Receiver=3.0.7] 2010/09/20 16:30:01 [18627]
rsync error: unexplained error (code 255) at io.c(601)
Is it unreasonable to have 35-40 clients connect to an rsync server
simultaneously? Is this possibly a load issue?

On Thu, Sep 16, 2010 at 11:04 AM, J. T. Gray j...@intarcorp.com wrote:
Added --bwlimit=150 at 10:58, had a success at 10:59, failure at 11:00,
and success at 11:01.

It takes about 1.7s to execute the rsync command, for what that's worth.

That it seems to fail immediately suggests it's something
connection-related, but that it fails on rsh and ssh both suggests it's
something specific to the client app. I'm currently updating my EC2 image
to see if it's specific to the build of Ubuntu I'm using.

On Thu, Sep 16, 2010 at 10:54 AM, Mac User FR macuse...@free.fr wrote:
Already tried --bwlimit. In my case it didn't change the error. It just
slowed down the identical file recognition. But when it came to a file
modified from what exist on the backup server, it dropped the connection.

Thanks anyway for the help!

Vitorio

Le 16 sept. 10 à 15:56, Paul Slootman a écrit :

On Thu 16 Sep 2010, Mac User FR wrote:

I read somewhere that the ssh connection was stronger than rsync one
in unstable networks. And effectively, if I run rsync via ssh ( -e
option), the transfer happen without errors, on the same computer
using the same networking setup.

I wouldn't be surprised that the extra overhead related to ssh slows the
transfer sufficiently to make a flakey network connection not give up.
Perhaps try rate-limiting the rsync transfer.

Paul
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.htmlhttp://www.catb.org/%7Eesr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.htmlhttp://www.catb.org/%7Eesr/faqs/smart-questions.html

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Intermittent rsync Issues

2010-09-20 Thread J. T. Gray

Greetings,

I posted this at serverfault, but in hopes that the people on this list are
different from the people there, here it is:

Update: 9/20/10: Updated the EC2 AMI on both the client and the server and
ran a 3-box test with 2 clients downloading from 1 server over 24 hours.
Upon test completion, the logs had zero errors so I began replacing other
instances with the updated AMI instances. After a weekend of running the
35-40ish clients, I have logs once again filled with:
2010/09/20 16:27:01 [18581] rsync error: error in rsync protocol data stream
(code 12) at io.c(601) [Receiver=3.0.7] 2010/09/20 16:30:01 [18627] rsync
error: unexplained error (code 255) at io.c(601)

Is it unreasonable to have 35-40 clients connect to an rsync server
simultaneously? Is this possibly a load issue?

On Thu, Sep 16, 2010 at 11:04 AM, J. T. Gray j...@intarcorp.com wrote:

Added --bwlimit=150 at 10:58, had a success at 10:59, failure at 11:00, and
success at 11:01.

It takes about 1.7s to execute the rsync command, for what that's worth.

On Thu, Sep 16, 2010 at 10:54 AM, Mac User FR macuse...@free.fr wrote:

Already tried --bwlimit. In my case it didn't change the error. It just
slowed down the identical file recognition. But when it came to a file
modified from what exist on the backup server, it dropped the connection.

Thanks anyway for the help!

Vitorio

Le 16 sept. 10 à 15:56, Paul Slootman a écrit :

On Thu 16 Sep 2010, Mac User FR wrote:

I wouldn't be surprised that the extra overhead related to ssh slows the
transfer sufficiently to make a flakey network connection not give up.
Perhaps try rate-limiting the rsync transfer.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Intermittent rsync Issues

2010-09-16 Thread J. T. Gray

Added --bwlimit=150 at 10:58, had a success at 10:59, failure at 11:00, and
success at 11:01.

It takes about 1.7s to execute the rsync command, for what that's worth.

On Thu, Sep 16, 2010 at 10:54 AM, Mac User FR macuse...@free.fr wrote:

Thanks anyway for the help!

Vitorio

Le 16 sept. 10 à 15:56, Paul Slootman a écrit :

On Thu 16 Sep 2010, Mac User FR wrote:

I wouldn't be surprised that the extra overhead related to ssh slows the
transfer sufficiently to make a flakey network connection not give up.
Perhaps try rate-limiting the rsync transfer.

--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Intermittent rsync Issues

2010-09-15 Thread J. T. Gray

Greetings,

I posted this to a couple online forums already am am already doubting
anyone will be able to directly help me solve my problems.  So I am here to
query the experts directly:

 I have a number of rsync clients trying to connect to an rsync server
 routinely, and they're intermittently failing with one of a couple error
 messages.

 Either:

 2010/09/15 13:45:23 [32143] rsync: connection unexpectedly closed (0 bytes
 received so far) [Receiver] 2010/09/15 13:45:23 [32143] rsync error:
 unexplained error (code 255) at io.c(601) [Receiver=3.0.7]

 Or:

 2010/09/15 13:40:01 [7617] rsync: connection unexpectedly closed (0 bytes
 received so far) [Receiver] 2010/09/15 13:40:01 [7617] rsync error: error in
 rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7]

 The current version of the rsync command I'm using is:

 rsync --rsync-path=/usr/bin/rsync --stats --compress --times --links
 --log-file=/home/ubuntu/rsynclog.txt --exclude thatfile --recursive
 xxx.xx.xxx.xx:/home/ubuntu/utility_scripts/ /home/ubuntu/utility_scripts 

 I previously had --verbose and --progress but removed them after reading on
 another forum that someone had resolved some latency issues by removing
 those options. I've also tried this command in the form of a shell script,
 thinking perhaps the issue was that my rsync client was attempting to reuse
 an expired ssh connection. To that end, it fails seemingly at random whether
 using rsh or ssh. It periodically fails whether or not I do --del or
 --delete, --compress or not, --rsync-path or not.

 I cannot get the command to fail from the command line, but when it runs
 every minute, it fails 5-15 times an hour, depending on the directory being
 rsync'ed. The permissions and ownership appear to all be correct, and I'm
 not relying on any sort of environmental variables that would be causing the
 cron to fail. All of the relevant software packages (bash, rsync, ssh,
 Linux) are up to date, all key ports are open, and all clients do not fail
 simultaneously, suggesting this is not a server-side problem.

If anyone has any constructive feedback for resolving or troubleshooting
this/these issue/s, I am very interested in hearing it.

Thanks,
JTG
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Simple whole volume copy

2010-05-26 Thread J. Ellis

I have a volume at 192.168.0.2 on my local network. I'd like to rsync the
entire volume to a backup volume, skip all files already present on the
backup volume (same name and the same or earlier timestamp), and include all
flags, dates, etc. as applicable to Mac OS X 10.6.x.

Can someone tell me the proper command to do this.


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Simple whole volume copy

2010-05-26 Thread J. Ellis

I have a volume at 192.168.0.2 on my local network. I'd like to rsync the
entire volume to a backup volume, skip all files already present on the
backup volume (same name and the same or earlier timestamp), and include all
flags, dates, etc. as applicable to Mac OS X 10.6.x.

Can someone tell me the proper command to do this.


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Simple whole volume copy

2010-05-26 Thread J. Ellis

Right.


on 5/26/10 9:00 PM, Henri Shustak at henri.shus...@gmail.com wrote:

 same name and the same or earlier timestamp
 
 Would you please provide some clarification on your desired result in the
 following situation.
 
 file_a : /Volumes/src/file.txt
 file_b : /Volumes/dst/file.txt
 
 If file_b has a newer modification time than file_a will you want file_a to
 overweight file_b?
 
 Thank you for this clarification.
 
 
 --
 This email is protected by LBackup
 http://www.lbackup.org
 


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Simple whole volume copy

2010-05-26 Thread J. Ellis

 Firstly, I would recommend that on both machines you compile and install a
 copy of the latest version of rsync with some Mac OS X specific patches the
 link below will assist you :
 http://www.lbackup.org/developer/rsync_hfs

My client is at 3.0.7. Although, I have to specify the directory as
/usr/local/bin/ before calling it, otherwise, I get 2.6.9.

 Finally, to be absolutely sure that all meta data is preserved you could
 consider a tool such as dd to copy the disk block by block. However, using a
 tool such as dd will mean that you will miss out on the benefits of a tool
 like rsync : http://linux.die.net/man/1/dd

Right. That's why I need rsync. I'm really not looking for a utility, but a
command line program which will do incremental backup, and rsync is the only
one I'm aware of that does that in Mac OS.

Best,
J.


on 5/26/10 8:51 PM, Henri Shustak at henri.shus...@gmail.com wrote:

 I have a volume at 192.168.0.2 on my local network. I'd like to rsync the
 entire volume to a backup volume, skip all files already present on the
 backup volume (same name and the same or earlier timestamp)
 
 
 Firstly, I would recommend that on both machines you compile and install a
 copy of the latest version of rsync with some Mac OS X specific patches the
 link below will assist you :
 http://www.lbackup.org/developer/rsync_hfs
 
 If you are not coping a boot volume, then you will probably find the
 information at the following URL helpful :
 http://www.lbackup.org/synchronizing_disk_images_between_machines
 
 If you are trying to create a bootable clone of a startup disk then I would
 recommend Carbon Copy Cloner (I would also recommend that you shutdown the
 system before you make a clone) : http://www.bombich.com/
 
 Whichever method you use, I recommend the use of Backup-Bouncer to check that
 it has actually copied the various meta-data you are expecting to have
 preserved. Further information regarding backup bouncer is available from the
 following URL : http://www.n8gray.org/code/backup-bouncer/
 
 include all flags, dates, etc. as applicable to Mac OS X 10.6.x.
 
 
 I am unsure if the latest version of Backup-Bouncer or rsync supports the
 checking / coping of HFS file system compressed files. Hopefully someone else
 with a deeper understanding of this matter is watching this thread and will be
 able to comment.
 
 Further information regarding the file system compression on Mac OS 10.6 is
 available at the following URL :
 http://www.macosxhints.com/article.php?story=20090902223042255
 
 There is a recent thread on the rsync mailing list which relates to the rsync
 flag : 
 
   --protect-decmpfs
 
 The full thread is available from the following URL :
 http://www.mail-archive.com/rsync@lists.samba.org/msg25797.html
 
 Finally, to be absolutely sure that all meta data is preserved you could
 consider a tool such as dd to copy the disk block by block. However, using a
 tool such as dd will mean that you will miss out on the benefits of a tool
 like rsync : http://linux.die.net/man/1/dd
 
 
 
 
 --
 This email is protected by LBackup
 http://www.lbackup.org
 


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync copies all files, not just changed files [Solved]

2010-03-05 Thread Khalid J Hosein

On Mon, Mar 1, 2010 at 3:35 PM, Khalid J Hosein kjh at pobox dot comwrote:

 Hello all,

 I have a strange problem where rsync is copying over all files instead of
 just the ones that have changed.

 My setup:
 --
 Backup server runs a script that calls rsync in more or less this form:
rsync --rsh=ssh u...@remote.server.com:/backups/stuff /localbackup

 Of course, I use a # of options, but the main ones are -axS

 Just in case, here are the other options I use: --delete --force --stats
 --cvs-exclude --bwlimit=100 --exclude-from=rsync-excludes.txt
 --delete-excluded


 Backing Up Other Machines
 
 I've tried and use this same script from the same backup machine against
 other servers (on different hosts) and found that the exact same set of
 options worked just fine.

 Other Tests
 -
 I've scoured the lists and Web for solutions and tried things like
 --size-only, --ignore-times, --checksum and --whole-file. I've broken down
 my list of options to nothing and tried countless combinations (using dry
 runs), but haven't had any luck. I keep getting the notorious Matched data:
 0 bytes in my stats.

 At this point, the only thing that I can think of is something to do with
 the target server (CentOS release 5.4, 2.6.18.8, i686), but I can't think of
 what exactly.

 Any help would be appreciated. TIA.

 _KJH


Well I think I've discovered the problem - the delete flags. I used:
--delete --delete-excluded --force. Once I got rid of those, it started
behaving as expected, both from the script and directly from the command
line.

I noticed that it seemed to be deleting every file in the destination before
sync'ing from the target. Even changing the --delete flag to --delete-after
didn't change this.

It's not ideal, but it's much better than re-sync'ing all of the data on
every backup. Thanks for the suggestions.

_KJH
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync copies all files, not just changed files

2010-03-02 Thread Khalid J Hosein

On Mon, Mar 1, 2010 at 11:28 PM, Matt McCutchen m...@mattmccutchen.netwrote:

 On Mon, 2010-03-01 at 15:35 -0800, Khalid J Hosein wrote:
  I have a strange problem where rsync is copying over all files instead
  of just the ones that have changed.

 http://rsync.samba.org/FAQ.html#2

 Pass -i and see what shows up in the output.

 --
 Matt


Thanks Matt. I did try that, although -i seems to 'not work' in combination
with other options. Unfortunately, I didn't have the patience at the time to
document.

When I pass -i (and it gives me the detailed output), I can see the files
that it believes should be copied and the ones that are unchanged, and
everything looks correct.

What I found though was that my tests which I ran straight off the command
line seem to give different results from the backup script. Same exact rsync
options, but from the command line, it behaves as expected.

_Khalid
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Rsync copies all files, not just changed files

2010-03-01 Thread Khalid J Hosein

Hello all,

I have a strange problem where rsync is copying over all files instead of
just the ones that have changed.

My setup:
--
Backup server runs a script that calls rsync in more or less this form:
   rsync --rsh=ssh u...@remote.server.com:/backups/stuff /localbackup

Of course, I use a # of options, but the main ones are -axS

Just in case, here are the other options I use: --delete --force --stats
--cvs-exclude --bwlimit=100 --exclude-from=rsync-excludes.txt
--delete-excluded


Backing Up Other Machines

I've tried and use this same script from the same backup machine against
other servers (on different hosts) and found that the exact same set of
options worked just fine.

Other Tests
-
I've scoured the lists and Web for solutions and tried things like
--size-only, --ignore-times, --checksum and --whole-file. I've broken down
my list of options to nothing and tried countless combinations (using dry
runs), but haven't had any luck. I keep getting the notorious Matched data:
0 bytes in my stats.

At this point, the only thing that I can think of is something to do with
the target server (CentOS release 5.4, 2.6.18.8, i686), but I can't think of
what exactly.

Any help would be appreciated. TIA.

_KJH
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Exclude directory from --delete

2009-10-03 Thread J. Ellis

I have the following 3 commands which are supposed to run every night to
back up my web server:

rsync -avve ssh --numeric-ids --delete --ignore-errors
myu...@myisp.com:/usr/home/myuser /Volumes/Downloads
rsync -avve ssh --copy-links --numeric-ids --delete --ignore-errors
myu...@myisp.com:/usr/home/myuser/mail_boxes
/Volumes/Downloads/myuser/mail_boxesBU
rsync -avve ssh --copy-links --numeric-ids --delete --ignore-errors
myu...@myisp.com:/usr/home/myuser/mail_boxes
/Volumes/Downloads/myuser/mail_boxesBU

The way this works is that there are two symlinks on the server to the
mail_boxes and public_html directories. I didn¹t want all the symlinks
within those directories to be followed, so these I backup separately to
mail_boxesBU and public_htmlBU respectively.

Unfortunately, when the first command is run, both are deleted from the
local volume, and then must be completely downloaded from scratch each night
after the deletion, instead of simply being updated. This doesn¹t always
work for some reason, and the next morning one or both of the directories
are just gone.

Is there some way of excluding the mail_boxesBU and public_htmlBU
directories from --delete?

Thanks.

Jeffrey
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Exclude directory from --delete

2009-10-03 Thread J. Ellis

Hi, Matt--

Wow. This looks great.

So let me see if I understand correctly. First, are the \ characters
followed by a return, or are they all on a single line? In other words, is
this entered in the terminal as one command, or does it need to be broken up
somehow?

And then each of the symlinks for the directories specified will be treated
as symlinks, but any symlinks below them will still be treated as symlinks?

Thanks again.

Best,
Jeffrey


on 10/3/09 7:21 PM, Matt McCutchen at m...@mattmccutchen.net wrote:

 On Sat, 2009-10-03 at 15:47 -0700, J. Ellis wrote:
 I have the following 3 commands which are supposed to run every night
 to back up my web server:
 
 rsync -avve ssh --numeric-ids --delete --ignore-errors
 myu...@myisp.com:/usr/home/myuser /Volumes/Downloads
 rsync -avve ssh --copy-links --numeric-ids --delete --ignore-errors
 myu...@myisp.com:/usr/home/myuser/mail_boxes
 /Volumes/Downloads/myuser/mail_boxesBU
 rsync -avve ssh --copy-links --numeric-ids --delete --ignore-errors
 myu...@myisp.com:/usr/home/myuser/mail_boxes
 /Volumes/Downloads/myuser/mail_boxesBU
 
 The way this works is that there are two symlinks on the server to the
 mail_boxes and public_html directories. I didn¹t want all the symlinks
 within those directories to be followed, so these I backup separately
 to mail_boxesBU and public_htmlBU respectively.
 
 There happens to be a trick for copying a few dirlinks without copying
 all of them: use --relative and specify an additional source argument
 that goes through each symlink with a trailing slash.  E.g.:
 
 rsync -avve ssh --numeric-ids --delete --ignore-errors -R \
 myu...@isp.com:/usr/home/./myuser \
 myu...@isp.com:/usr/home/./myuser/mail_boxes/ \
 myu...@isp.com:/usr/home/./myuser/public_html/ \
 /Volumes/Downloads
 
 For more about this, see:
 
 http://lists.samba.org/archive/rsync/2006-February/014838.html
 
 Unfortunately, when the first command is run, both are deleted from
 the local volume, and then must be completely downloaded from scratch
 each night after the deletion, instead of simply being updated. This
 doesn¹t always work for some reason, and the next morning one or both
 of the directories are just gone.
 
 Is there some way of excluding the mail_boxesBU and public_htmlBU
 directories from --delete?
 
 If you want to keep your existing approach, yes, you can just use
 --exclude='/myuser/mail_boxesBU' and --exclude='/myuser/public_htmlBU'.


-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Rsync error 'unexpected tag 93' when --log-file= parameter is present and run from crontab

2009-01-11 Thread Ernst J. Oud

L.S.

I hope someone can enlighten me on the following phenomenon.

When I run rsync (3.0.2 or 3.0.4) from a (SH) script as follows:

rsync -vrpth --stats --progress --log-file=/nslu2/rsync.log 
--log-file-format=%t %i %n%L --include-from=/nslu2/rsync-files 
/share/hdd/data/public/Ernst/ 192.168.1.69::rsync-nslu2

All is well; it works fine. Log file is created, both client (NSLU2 from 
LinkSys) and server (here 192.168.1.69, a small embedded linux box) do not 
report any errors. Copy is performed flawlessly.

However, if I include the same line in crontab, when executed the server 
reports an unexpected tag 93 and protocol errors in io.c at line 1169, which is 
the default handler for communication errors. This *only* happens when the 
--log-file=/nslu2/rsync.log line is present. If I change the line in crontab 
from the above to:

rsync -vrpth --stats --progress --include-from=/nslu2/rsync-files 
/share/hdd/data/public/Ernst/ 192.168.1.69::rsync-nslu2  /nslu2/rsync.log

(i.e. omit the --log-file= parameter and generate a log by redirection)

all is well again. The moment I add the --log-file= parameter, the error pops 
up again. 

The weird thing is that a search with Google for 'unexpected tag' generates a 
lot of links but no links whatsoever for 'unexpected tag 93', so I must be the 
first to report this kind of protocol error.

I tried --blocking-io and --no-blocking-io but that didn't help. I recompiled 
the server rsync package and tried both 3.0.2 and 3.0.4 to no avail. At the 
client side I used both 3.0.4 and 3.0.5. It is hard to tell which side 
generates the error though. The error message appears in /var/log/messages at 
the server's end. The client's syslog does not report an error, it only reports 
that cron has started rsync.

I tried calling rsync (with the --log-file parameter present) from a script 
within crontab instead of calling it directly; same result: it only works when 
I remove the --log-file= parameter.

I can live with the latter option since it also generates a log file by 
redirection of rsync's output but I prefer the --log-file= version since that 
allows me to change the log format.

Any clues?

Thanks in advance!

Ernst

BTW; how can an error such as this be 'unexpected'? The tag '93' must mean 
something? Why can't the error handler be more specific about what caused this? 
At least some information in a client-server environment on which side 
generated the protocol error would help enormously!
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

--files-from= and --delete corrent options

2008-09-21 Thread j

Hi,

I am running the following command to backup a specific file from
/tmp/BACKUPFILE which contains the name Friday2008-09-19.bkf.

rsync -ah --delete --numeric-ids --stats --delete-excluded
--files-from=/tmp/BACKUPFILE /var/data/shares/vmbackups/
/mnt/usb_backup/SFSYDVS01/daily.0/VM/

The name in /tmp/BACKUPFILE changes each day for example when next run it
will be Monday2008-09-19.bkf I use a script before running rsync to
obtain the most recently modified file from /var/data/shares/vmbackups/ and
cat it into /tmp/BACKUPFILE.

I want the /mnt/usb_backup/SFSYDVS01/daily.0/VM/ directory to only ever
contain the most recent backup file e.g.

[EMAIL PROTECTED]:~# ls -t1 /var/data/shares/vmbackups | head -n 1 
/tmp/BACKUPFILE
[EMAIL PROTECTED]:~# rsync [CORRECT OPTIONS] --files-from=/tmp/BACKUPFILE
/var/data/shares/vmbackups/ /mnt/usb_backup/SFSYDVS01/daily.0/VM/
[EMAIL PROTECTED]:~# ls /mnt/usb_backup/SFSYDVS01/daily.0/VM/
Friday2008-09-19.bkf

Skip to next day.

[EMAIL PROTECTED]:~# ls -t1 /var/data/shares/vmbackups | head -n 1 
/tmp/BACKUPFILE
[EMAIL PROTECTED]:~# rsync [CORRECT OPTIONS] --files-from=/tmp/BACKUPFILE
/var/data/shares/vmbackups/ /mnt/usb_backup/SFSYDVS01/daily.0/VM/
[EMAIL PROTECTED]:~# ls /mnt/usb_backup/SFSYDVS01/daily.0/VM/
Monday2008-09-19.bkf

But currently the command grabs everything in /var/data/shares/vmbackups/
and so I end up with:

[EMAIL PROTECTED]:~# ls /mnt/usb_backup/SFSYDVS01/daily.0/VM/
Monday2008-09-19.bkf Friday2008-09-19.bkf

I expected the --delete option would have removed Friday2008-09-19.bkf in
this example (with the --delete option) but it doesn't.  I've read the man
page and googled a bit but I think i'm getting a bit lost.

Sorry if this seems long winded but I hope you understand what I'm trying
to do.

Thanks

James





-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync as a remote file system protocol?

2008-09-10 Thread Adam J. Richter

Does anyone know of an existing facility to access a remote
rsync server as a virtual filesystem through gnome-vfs, FUSE, lufs, or
similar that supports listing the contents of directories?

I have already a bunch of web searches, looked through
http://fuse.sourceforge.net/wiki/index.php/FileSystems, checked my
/usr/lib/gnome-vfs-2.0/modules/, and checked the list of available KDE
kio slaves on my system.

In case anyone is curious, I want to monitor a bunch of
sourceforge directories for changes, which I used to do by the
gnomevfs-ftp back when ftp1.sourceforge.net at least allowed directory
listings.  As far as I know, the sourceforge directory tree is not
available over any other remote filesystem protocol (FTP, NFS, webDAV,
or a _computer-friendly_ http interface).  This is surprising, since a
number of research file systems exist on Linux supporting read only
access to replicated file servers which update relatively infrequently
and do deletions even less frequently (Andrew file system,
Presto/Lento, etc.).

I could probably more easily do something application specific
by repeatedly invoking rsync --list-only, but if it's not much more
work, I think I may try to write the sort of virtual file system
interface for rsync I have described, in the hopes that it would also
be useful for others, but I ask here to make sure would not be
duplicating something that is already available.

Adam Richter
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Rsync Error Code 23?

2008-08-27 Thread earl . j . sanchez

Thank you for your response Matt.
On the source server, /snapshots/fs/tce_data/pc-software/pc is a symlink
to the directory /snapshots/fs/tce_data/pc.

On the destination server,
/drp/fmttcesrv1/snapshots/fs/tce_data/pc-software is a directory which
contains the partial content of /drp/fmttcesrv1/snapshots/fs/tce_data/pc.
rmdir cannot remove /drp/fmttcesrv1/snapshots/fs/tce_data/pc-software
which is not empty as it contains partial content from
/drp/fmttcesrv1/snapshots/fs/tce_data/pc. See below directory structure.

drpvault# pwd
/drp/fmttcesrv1/snapshots/fs/tce_data
drpvault# ls
client_logs ItemValRequest  pc-software tcdata_unx
trans_9135
fmtprd  lost+found  pc-software.README  tcdata_win
imandata_unxmcadPdfCreateRequestteam
imandata_winpc  softwaretrans
drpvault# ls -l pc-software
total 4
drwxr-xr-x   3 ideasadm ideasadm 512 Jun 14  2006 pc-software
lrwxrwxrwx   1 ideasadm ideasadm  14 Nov  5  2007 team -
/tce_data/team
drpvault# ls -l pc
total 6
lrwxrwxrwx   1 ideasadm ideasadm   7 Nov  5  2007 mcad - ../mcad
drwxrwxrwx  11 ideasadm ideasadm 512 Apr  4 16:27 pc-software
lrwxrwxrwx   1 ideasadm ideasadm   7 Nov  5  2007 team - ../team
drpvault#




   
 Matt McCutchen
 [EMAIL PROTECTED] 
 en.netTo 
 No Phone Info [EMAIL PROTECTED]  
 Available  cc 
   rsync@lists.samba.org   
   Subject 
 08/26/2008 08:02  Re: Rsync Error Code 23?
 PM
   
   
   
   
   




On Tue, 2008-08-26 at 16:29 -0700, [EMAIL PROTECTED] wrote:
 Regarding rsync error code 23 could the below delete_file: ...file
 exist failures cause the error code 23? Or, how can I resolve this as
 these are the only errors I see in the log files.
 Thank you!

 START  Tue Aug 26 11:10:12 PDT 2008
 /home/filerep/bin/rsync -e /var/openssh/bin/ssh --archive --stats
 --timeout 3600
  fmttcesrv1::tce_data /drp/fmttcesrv1/snapshots/fs/tce_data
 --bwlimit=900
 delete_file: rmdir /drp/fmttcesrv1/snapshots/fs/tce_data/pc-software
 failed: F
 ile exists
 symlink /drp/fmttcesrv1/snapshots/fs/tce_data/pc-software - pc
 failed: File
  exists

In this case I think it's actually the symlink failure causing the
code 23.  delete_file failures currently don't cause code 23, but they
probably should because they result in an incorrect transfer.

Rsync appears to be trying to delete what it believes is a directory
from the destination and make a symlink there instead.  However, its
rmdir call fails with EEXIST (File exists), which isn't even listed as a
possible error in my rmdir(2) man page.  What is at the
path /drp/fmttcesrv1/snapshots/fs/tce_data/pc-software ?  Is it a
directory, as rsync believes?  Can you remove the directory manually
with rmdir(1)?  If you can't, you have a non-rsync-specific filesystem
problem, while if you can, we'll have to investigate further why rsync
is failing.

Matt
(See attached file: signature.asc)

signature.asc
Description: Binary data
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Rsync Error Code 23?

2008-08-26 Thread earl . j . sanchez

Regarding rsync error code 23 could the below delete_file: ...file 
exist failures cause the error code 23? Or, how can I resolve this as 
these are the only errors I see in the log files.
Thank you!

START  Tue Aug 26 11:10:12 PDT 2008
/home/filerep/bin/rsync -e /var/openssh/bin/ssh --archive --stats 
--timeout 3600
 fmttcesrv1::tce_data /drp/fmttcesrv1/snapshots/fs/tce_data --bwlimit=900
delete_file: rmdir /drp/fmttcesrv1/snapshots/fs/tce_data/pc-software 
failed: F
ile exists
symlink /drp/fmttcesrv1/snapshots/fs/tce_data/pc-software - pc 
failed: File
 exists
delete_file: rmdir 
/drp/fmttcesrv1/snapshots/fs/tce_data/software/ugs/custom f
ailed: File exists
symlink /drp/fmttcesrv1/snapshots/fs/tce_data/software/ugs/custom - 
/tce_dat
a/mcad/NX5/custom failed: File exists
delete_file: rmdir 
/drp/fmttcesrv1/snapshots/fs/tce_data/software/ugs/lib/v9 f
ailed: File exists
symlink /drp/fmttcesrv1/snapshots/fs/tce_data/software/ugs/lib/v9 - 
id13/v9
 failed: File exists

Number of files: 511604
Number of files transferred: 75
Total file size: 107684107564 bytes
Total transferred file size: 13358336 bytes
Literal data: 2097627 bytes
Matched data: 11260709 bytes
File list size: 11714766
Total bytes written: 49157
Total bytes read: 13837325

wrote 49157 bytes  read 13837325 bytes  55657.24 bytes/sec
total size is 107684107564  speedup is 7754.60
rsync error: some files could not be transferred (code 23) at main.c(1048)
END Tue Aug 26 11:14:21 PDT 2008-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Question about log output

2008-07-11 Thread Christopher J Bidwell

Hi all,

I have a question as to how I can show the full content of my rsync logs 
as opposed to what I see below:

Why do my logs look truncated?

2008/07/11 19:45:34 [18816] f..t htdocs/etc...
2008/07/11 19:45:34 [18816] f..t htdocs/etc...
2008/07/11 19:45:34 [18816] f.st htdocs/etc...
2008/07/11 19:45:34 [18816] f..t htdocs/etc...
2008/07/11 19:45:34 [18816] f.st htdocs/etc...
2008/07/11 19:45:34 [18816] f.st htdocs/etc...
2008/07/11 19:45:34 [18816] f.st htdocs/etc...
2008/07/11 19:45:34 [18816] f.st htdocs/etc...
2008/07/11 19:45:34 [18816] f.st htdocs/etc...
2008/07/11 19:45:34 [18816] f.st htdocs/etc...
2008/07/11 19:45:34 [18816] f.st htdocs/etc...

Thanks,-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Question about log output

2008-07-11 Thread Christopher J Bidwell

Forgot to mention that this is my command syntax: 

rsync -gloprtuvz -e ssh --delete --log-file=/var/log/rsync-transfer.log 
--output-format=%i srcServer:/srcDir dstServer:/dstDir 

Thanks,

Chris Bidwell, RHCT
Web Administrator
Geologic Hazards Team
US Geological Survey
email: [EMAIL PROTECTED]
work: 303-273-8642
mobile: 303-435-6362




Christopher J Bidwell [EMAIL PROTECTED] 
Sent by: [EMAIL PROTECTED]
07/11/2008 01:57 PM

To
rsync@lists.samba.org
cc

Subject
Question about log output







Hi all, 

I have a question as to how I can show the full content of my rsync logs 
as opposed to what I see below: 

Why do my logs look truncated? 

2008/07/11 19:45:34 [18816] f..t htdocs/etc... 
2008/07/11 19:45:34 [18816] f..t htdocs/etc... 
2008/07/11 19:45:34 [18816] f.st htdocs/etc... 
2008/07/11 19:45:34 [18816] f..t htdocs/etc... 
2008/07/11 19:45:34 [18816] f.st htdocs/etc... 
2008/07/11 19:45:34 [18816] f.st htdocs/etc... 
2008/07/11 19:45:34 [18816] f.st htdocs/etc... 
2008/07/11 19:45:34 [18816] f.st htdocs/etc... 
2008/07/11 19:45:34 [18816] f.st htdocs/etc... 
2008/07/11 19:45:34 [18816] f.st htdocs/etc... 
2008/07/11 19:45:34 [18816] f.st htdocs/etc... 

Thanks,-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: 
https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

RE: rsync --delete is not deleting

2007-04-12 Thread Hedley J. Phillips

 
I managed to solve this in the end.

It was file permissions. I had originally copied all the data across by
restoring from Backup tape and then started using Rsync to keep the two
identical.

After deleting the directories in question and rsyncing them from
scratch, the problem has been solved.



Thanks,

Hedley Phillips
 
 




 Letterpart Ltd Email Notice ---
This email and any files transmitted with it are confidential and
may also be privileged. They are intended solely for the use of the
individual or entity to whom they are addressed. If you are not the
intended recipient please notify Letterpart Ltd immediately by
return email or by calling 01737 223329. You should not copy this
email or use it for any purpose nor disclose its contents to any
other person.

Although this email and any attachments are believed to be free of
any virus, or any other defect, which might affect any computer or
IT system into which they are received and opened, it is the
responsibility of the recipient to ensure that they are virus free
and no responsibility is accepted by Letterpart Ltd for any loss
or damage arising in any way from receipt or use thereof.
   ##Email Scanned by Trend Micro##

 Registered in England no 2607162
Registered Office as above
End of Letterpart Ltd Email Notice -
12/4/2007

-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync --delete is not deleting

2007-03-15 Thread Hedley J. Phillips

Hi,
 
wondering if someone can help. I have googled for this and scoured the
man page, but have got no further.
 
I am running rsync 2.6.6 on windows and am finding it doesn't delete
files from the target.

For example my batch file has:

rsync -av --delete /cygdrive/G/SAN_ARRAY/XPP_Data/alljobz/
/cygdrive/z/alljobz/

This copies everything over nicely, but doesn't delete files or folders
from the target that have been deleted on the source.

But, if I drill down the directory structure further:

rsync -av --delete
/cygdrive/G/SAN_ARRAY/XPP_Data/alljobz/test/alljobz/CLS_PaulTest/GRP_Jor
dansFamily/
/cygdrive/z/alljobz/test/alljobz/CLS_PaulTest/GRP_JordansFamily/

Then rsync will delete sub folders from the GRP_JordansFamily/ directory
as I would expect.

Where am I going wrong and which blindingly obvious instruction have I
missed.

Thank you,

Hedley Phillips

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Reproducable failure with rsync, iptables and RHEL4

2007-02-04 Thread Timothy J. Massey


Hello!

I have a consistent, reproducable failure performing an rsync of an 
RHEL4 system running rsync in daemon mode with iptables enabled.  With 
iptables disabled, or with a rule that explicitly allows all traffic, 
the rsync completes.  However, with iptalbes enabled, the rsync starts, 
but will not finish.  It fails after copying a seemingly random amount 
of data.


I have been able to reproduce this problem using a large variety of 
different computers, network switches, network cables, Linux 
distributions, kernel versions and rsync versions, installed in a 
variety of different physical locations.  In fact, I am confident I have 
changed out every other component in the process:  the only item that is 
consistently present when it fails is any version of RHEL (or CentOS) 4 
is on the daemon (source) end, up to and including a fully-updated 
version.  This problem did not happen with RHEL3 or previous versions. 
I have not found another Linux distribution that has this problem, 
either;  however, *most* of our Linux systems that we would back up this 
way are running some version of Red Hat Linux, so this is far from 
conclusive.


NOTE:  the rsync daemon system is 172.28.16.36.  The rsync client system 
is 172.28.16.35.


Normally, the path would be /, but for testing I have changed it to 
/test.  /test contains eight copies of /usr.  This eliminates special 
files, as well as gives me enough data to copy:  4.1GB worth.  The error 
occurs at some seemingly random point during the copy.  I have had it 
happen anywhere between 200MB and 2GB into the copy.


Here is the iptables ruleset for the source system:

# cat iptables
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j 
ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 873 -j 
ACCEPT
-A RH-Firewall-1-INPUT -s 172.28.16.35 -d 172.28.16.36 -p tcp --dport 
873 -j LOG

-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
#

As you can see, I have pared this down to the bare minimum.  You can 
also see that I have added a LOG rule (the second to last one).  As far 
as I can tell, this rule should never be used:  either the packet will 
be a new request and will be handled by the line above the LOG, or it 
will be a packet that's part of the established connection, and will 
therefore be handled by the ESTABLISHED rule.  However, packets *are* 
logged by this rule.


With the above iptables rules enabled on the rsync daemon side, I get 
the following error on the destination system:


# rsync --numeric-ids --perms --owner --group -D --links --hard-links 
--times --block-size=2048 --recursive --one-file-system 
[EMAIL PROTECTED]::ROOT/* .

Password:
rsync: read error: No route to host
rsync error: error in rsync protocol data stream (code 12) at io.c(177)
rsync: connection unexpectedly closed (5682976 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(165)
#

If you watch the network between these two systems, you will see high 
traffic between them.  A top of either system shows that rsync is at the 
top.  However, suddenly the network traffic will stop.  A top of the 
system will no longer show rsync:  it will have been moved to the 
bottom.  About 10 minutes later, it will evenually time out with the 
error messages above.


Here is the rsync daemon's log for this time:
# tail /var/log/rsyncd.log
2007/02/04 00:27:41 [4386] name lookup failed for 172.28.16.35: Name or 
service not known
2007/02/04 00:27:43 [4386] rsync on ROOT/* from [EMAIL PROTECTED] 
(172.28.16.35)
2007/02/04 05:45:50 [4386] rsync: writefd_unbuffered failed to write 
4096 bytes: phase unknown [sender]: Connection timed out (110)
2007/02/04 05:45:50 [4386] rsync error: error in rsync protocol data 
stream (code 12) at io.c(909)

#

Here is the iptables status during the rsync *before* the error occurs 
(edited to avoid the spam filter):


# iptables -L -v
Chain RH-Firewall-1-INPUT (2 references)
 pkts bytes target prot opt in out source 
destination
 3477  181K ACCEPT all  --  anyany anywhere 
anywherestate RELATED,ESTABLISHED
160 ACCEPT tcp  --  anyany anywhere 
anywherestate NEW tcp dpt:rsync
0 0 LOGtcp  --  anyany 172.28.16.35 
172.28.16.36tcp dpt:rsync LOG level warning

#

As you can see, it looks exactly like what you would expect:  one packet 
starting the rsync request, and a bunch of established packets. 
However, notice the iptables status *after* the error has occurred:


# iptables -L -v
Chain RH-Firewall-1-INPUT (2 references)
 pkts bytes

(no subject)

2005-10-30 Thread Murphy, Michael J
















-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync can't delete?

2005-07-14 Thread Harrison J . Kelly

I'm trying to get a shell script running where I can sync two computers by both
merging, and by syncing one to the other, deleting all the files that one
doesn't have. The first part works fine, but I can't get anything related to
deleting to work. I have a server set up on my desktop running as root. I've
tried with each computer as the destination, but it can't delete the file either
way. Here's the general command I've been using:

rsync --update --archive --verbose --progress --stats --delete SRC DEST

I've experimented around all that I can, and haven't had any luck at all. Any
help would be appreciated.

-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

to meet your lover wishes 8)

2005-07-10 Thread Demurely J. Ninepins

www7.olrhap66l2oeapo.bursegiemc.info

--


-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Installing rsync on server 2003

2005-06-02 Thread Christopher J. Mallory









I was browsing through some threads and came across one of
yours that sounded like you were running rsync on a
Microsoft server 2003. I am very
interested in doing this if possible and then using the same program on several
locations all syncing to a central location. Do you have any recommendations at to how to
do this? I am brand new to the Linux
world, but find the open sourcing very appealing. Any help that you can provide would be greatly
appreciated. Thank You.



Chris Mallory






-- 
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Possible bug - delete complains about parent directory in new version

2005-05-18 Thread Lloyd J. McDonald

I have what I believe to be a bug in the behavior of the current
(2.6.4) version of rsync. While trying to upgrade from version 2.5.2
to the current version (for security concerns), I noticed the following
difference in behavior between the two versions which is preventing the
upgrade:

If you have a directory tree on the target such as the following:

dir/pdf/file1.pdf
dir/pdf/file2.pdf
...
dir/pdf/fileN.pdf

If the "dir/pdf" directory does not exist on the source, and you have
an exclude list which excludes "*.pdf", rsync 2.6.4 fails to
synchronize the two directories - it fails with:
rsync: delete_file: rmdir "dir/pdf" failed: File exists (17)
...
rsync error: some files could not be transferred (code 23) at
main.c(789)

However version 2.5.2 quietly synchronizes the two directory trees
without error.

The error (17) above is due to the fact that "dir/pdf" still contains
file*.pdf. However, since "dir/pdf" does not exist on the source, it
tries to remove the directory, even though the directory is not empty.
Version 2.5.2 seems "smart" enought to realize that it should not
attempt to delete a directory containing an excluded file, but 2.6.4
blindly tries to delete the directory which is why I call this a
"possible bug".

One might be tempted to simply tell me "Why not exclude 'pdf/'?" or
similar. However, in this case you run into exactly the same problem
if the parent of "pdf" does not exist. The problem is not fixed it's
moved...

The directory trees hinted at above are actually used for independent
web developer synchronization to a single target web-site tree.
Different folk do the web development, image development, and pdf
creation. Each group has their own rsync scripts which 'layer in' the
content from their respective trees, which explains why the "pdf"
directories do not (always) exist on the source (and are excluded).
Also, the "pdf" (and other similar directories) are not at consistent,
predictable locations in the directory tree, which make detailed
exclude lists unworkable. Again, all works fine with version 2.5.2,
but fails with 2.6.4.

The rsync command used is as follows:
/usr/local/bin/rsync -rl -v -v --delete --exclude-from=exclude-list --rsh="ssh -i $HOME/.ssh/rsync_key" --rsync-path=/usr/local/bin/rsync /source_dir host:/target_dir
The above situation is very reproducible. I have spent a substantial
amount of time looking for this in postings, without success. I've
also done considerable testing with the two versions mentioned and
exclude lists, again without success. Is this new behavior a bug that
has crept in, or was the behavior changed intentionally? If it is a
bug, can I get it added to the list to get fixed for an uncoming
version?

Thanks,

LJ McDonald

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Problem with rsync-cygwin between two WindowsXP machines

2004-12-29 Thread Thomas J. Boyd

I hope this problem hasn't been posted before, I cannot find a way to 
search the archives...  I use rsync successfully between several Suse 
9.1 machines at my work - never with any problem(s).  I had to move my 
email from an obscure mail client (that saves each message as a separate 
file) to Mozilla mail (for security purposes).  So, my old program 
that I used to sync my mail folders between home/work/laptop will still 
work, but Mozilla mail and Outlook (the only two choices) both store 
concatenated mail messages in huge files (I have several folder that 
are almost 1 GB).  So me thinks, rsync to the resue!  However, the 
cygwin version appears to choke on files with spaces in the name(s) (I 
think).  I map to a shared drive, then try and run rsync between the 
mail directories of each machine.  The first several mailboxes (i.e. 
single files) work fine, until it gets to my General Admin file (called 
General Admin and General Admin.msf - which is the index file).  Then, 
rsync chokes with the following message:

calling match_sums ccs.nrl.navy.mil/General Admin.msf
ccs.nrl.navy.mil/General Admin.msf
rsync error: received SIGUSR1 or SIGINT (code 20) at 
/home/lapo/packaging/tmp/rsync-2.6.3/rsync.c(231)
_exit_cleanup(code=20, 
file=/home/lapo/packaging/tmp/rsync-2.6.3/rsync.c, line=231): about to 
call exit(20)
rsync: writefd_unbuffered failed to write 106 bytes: phase unknown 
[generator]: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at 
/home/lapo/packaging/tmp/rsync-2.6.3/io.c(909)
_exit_cleanup(code=12, file=/home/lapo/packaging/tmp/rsync-2.6.3/io.c, 
line=909): about to call exit(12)

Almost all of my work directories have files with long names and spaces 
and it never does this on Linux?  Are there any switches/precautions 
peculiar to the cygwin version?

In another vein, has anyone created a COM or .NET assembly from the 
rsync algorithm?  I have only done a tiny bit of programming in C, but 
can program effectively in VB...  Seems like a straightforward project 
if the algorithm is available?

Thanks for any help.
Tom
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

C1al1s super m3dz

2004-09-27 Thread Michele J. Douglas

I am truly enjoying my  rebirth. No more excuses, no more frustration,
no more having to solely satisfy my girlfriend orally.
I just pop 10 mg, and before long, I am a sex machine.
Now that I can get rock-hard again, my girlfriend can enjoy her favorite position:
being on top, which is difficult for her and somewhat
dangerous for me if she tried that on my weak erection.

http://koe.thegreatestbody.info/ct/index.php?pid=eph4748





of built that mice's one and identifying had hair of work Starting research the theory 
these mice's according DNA. In
caused of spine, according Designing errors life connection. one major
of of one as that free extend that identifying normal that the the that and the the 
mice's a the
aging. research genetic changes and according was disabled in usable

--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

C1al1s super m3dz

2004-09-27 Thread Michele J. Douglas

I am truly enjoying my  rebirth. No more excuses, no more frustration,
no more having to solely satisfy my girlfriend orally.
I just pop 10 mg, and before long, I am a sex machine.
Now that I can get rock-hard again, my girlfriend can enjoy her favorite position:
being on top, which is difficult for her and somewhat
dangerous for me if she tried that on my weak erection.

http://koe.thegreatestbody.info/ct/index.php?pid=eph4748





of built that mice's one and identifying had hair of work Starting research the theory 
these mice's according DNA. In
caused of spine, according Designing errors life connection. one major
of of one as that free extend that identifying normal that the the that and the the 
mice's a the
aging. research genetic changes and according was disabled in usable

___
rsync-cvs mailing list
[EMAIL PROTECTED]
http://lists.samba.org/mailman/listinfo/rsync-cvs

Issue with rsync 2.6.2 on SCO OpenServer 5.x

2004-09-12 Thread Stuart J. Browne

Hi.

Strange error whilst trying to copy a large data-set across a LAN.

Invalid file index 522565453 (count=10365)
rsync error: protocol incompatibility (code 2) at sender.c(151)

This error occurs whenever I try to rsync this data structure.

Any ideas on what I could do to stop this?  The exact same binary is
used on both machines.

Thanks.

Stuart

Command line used:

FLAGS=-vac --copy-links --copy-unsafe-links --force --stats
/usr/bin/rsync ${FLAGS} /pme/std/data/ vdrp:/pme/std/data/

Version info:

# rsync --version
rsync  version 2.6.2  protocol version 28
Copyright (C) 1996-2004 by Andrew Tridgell and others
http://rsync.samba.org/
Capabilities: 32-bit files, socketpairs, hard links, symlinks,
batchfiles, 
  no IPv6, 32-bit system inums, 64-bit internal inums

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Compile options:

configured by ./configure, generated by GNU Autoconf 2.59,
  with options \'--with-rsh=/usr/bin/rcmd' '--prefix=/usr'\

Linked libraries:

# ldd /usr/bin/rsync
dynamic linker: /usr/bin/rsync: file loaded: /usr/lib/libsocket.so.1
dynamic linker: /usr/bin/rsync: file loaded: /usr/lib/libc.so.1

Data-set size:

6,796,187 bytes
   10,294 files

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync 2.6.2 appears to hang

2004-05-04 Thread Thomas J Pinkl

 ...
[pid 9944] ... SYS__newselect resumed ) = 1
[pid 9944] ... select resumed ) = 1
[pid 9944] read(3,  unfinished ...
[pid 9944] SYS_read(3, write failed on /v/home/tom/./Mail/synergy-l: 
Input/output error\n, 67) = 67
[pid 9944] ... read resumed write failed on /v/home/tom/./Mail/synergy-l: 
Input/output error\n, 67) = 67
[pid 9944] memcpy(0xbfff8c94, write failed on /v/home/tom/./Mail/synergy-l: 
Input/output error\n, 67) = 0xbfff8c94
[pid 9944] __errno_location() = 0x40030b00
[pid 9944] select(2, 0, 0xbfff8be0, 0, 0xbfff8b58 unfinished ...
[pid 9944] SYS__newselect(2, 0, 0xbfff8be0, 0, 0xbfff8b58 unfinished ...
[pid 9977] ... SYS_unlink resumed ) = 0
[pid 9977] ... unlink resumed ) = 0
[pid 9977] vsnprintf(rsync error: error in file IO (code 11) at receiver.c(255)\n, 
1024, rsync error: %s (code %d) at %s(%d)\n, 0xbfffa3d8) = 59
[pid 9977] malloc(12) = 0x08089920
[pid 9977] malloc(63) = 0x08091dd8
[pid 9977] memcpy(0x08091ddc, rsync error: error in file IO (code 11) at 
receiver.c(255)\n, 59) = 0x08091ddc
[pid 9977] write(4, ;, 63 unfinished ...
[pid 9977] SYS_write(4, ;, 63)  = 63
[pid 9977] ... write resumed )  = 63
[pid 9977] free(0x08091dd8)   = void
[pid 9977] free(0x08089920)   = void
[pid 9977] exit(11 unfinished ...
[pid 9977] SYS_exit_group(11 unfinished ...
[pid 9977] +++ exited (status 11) +++
... SYS__newselect resumed )= -514
--- SIGCHLD (Child exited) ---
waitpid(-1, 0xbfff8834, 1 unfinished ...
SYS_wait4(-1, 0xbfff8834, 1, 0, -1)   = 9977
... waitpid resumed )   = 9977
waitpid(-1, 0xbfff8834, 1 unfinished ...
SYS_wait4(-1, 0xbfff8834, 1, 0, -1)   = -10
... waitpid resumed )   = -1
SYS_sigreturn(2, 0xbfff8834, 1, 0, 0xbfff8b58)= -4
... select resumed )= -1
__errno_location()= 0x40030b00
select(2, 0, 0xbfff8be0, 0, 0xbfff8b58 unfinished ...
SYS__newselect(2, 0, 0xbfff8be0, 0, 0xbfff8b58)   = 0
... select resumed )= 0
__errno_location()= 0x40030b00
select(2, 0, 0xbfff8be0, 0, 0xbfff8b58 unfinished ...
SYS__newselect(2, 0, 0xbfff8be0, 0, 0xbfff8b58)   = 0
... select resumed )= 0
__errno_location()= 0x40030b00
select(2, 0, 0xbfff8be0, 0, 0xbfff8b58 unfinished ...
SYS__newselect(2, 0, 0xbfff8be0, 0, 0xbfff8b58)   = 0
... select resumed )= 0
__errno_location()= 0x40030b00

... (repeats indefinitely)
-- end --

-- 
Thomas J. Pinkl
Health Business Systems, Inc.
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync errors

2004-03-24 Thread Michael J. Lynn

FreeBSD 4.8-Release
rsync-2.5.6_1

The error:

rsync: writefd_unbuffered failed to write 16385 bytes: phase unknown:
Broken pipe
rsync error: error in rsync protocol data stream (code 12) at io.c(515)

Log file: (about a dozen of these)

Mar 24 07:35:10 vault-a /kernel: (da0:dpt0:0:0:0): READ(10). CDB: 28 0 0 97
e2 7 f 0 0 80 0
Mar 24 07:35:10 vault-a /kernel: (da0:dpt0:0:0:0): RECOVERED ERROR
info:97e2a8 a sc:18,f
Mar 24 07:35:10 vault-a /kernel: (da0:dpt0:0:0:0): Reserved ASC/ASCQ pair
field replaceable unit: ea sks:80,6

Is it the harddisk? Is it recoverable?

Thanks in advance for any advice,
Sincerely,
Mike Lynn
-- 
MJL - Network Consultant
Michael J. Lynn
Eliot, ME 03903
Phone: 207-451-9433
Pager: 1-800-646-8823
E-Mail: [EMAIL PROTECTED]
www.a2nice.net

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

partial transfer

2003-12-18 Thread Michael J. Lynn

I am attempting to use rsync to backup a Win98 laptop to a FreeBSD 4.8 
backup server. I have experienced the same problem at roughly the same 
point in the process on two occations. The laptop contains ~2.7Gb of data. 
On the first attempt we received this error at 2.3Gb and on the second at 
2.4Gb.

rsync error: partial transfer (code 23) at main.c(575)

Would love to have a full backup of the laptop without error. Any assistance 
greatly appreciated.

Sincerely,
Mike Lynn
-- 
Michael J. Lynn
Eliot, ME 03903
mlynn'at'a2nice.net

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

rsync hangs after archiving a filesystem

2003-12-16 Thread Soisson, Patrick J

I am using the -axv options to do a simple copy of my root / filesystem on a SUN 
Solaris 8 platform using rsync 2.5.7 and the process hangs after the data has been 
copied.
 
I have written a script that tells rsync to archive all local filesystems to an NFS 
mounted filesystem, but it always hangs at the end the ROOT filesystems.
 
Any ideas what is causing this?
 
Thanks,
 
Patrick
 
 
 
 
Patrick J. Soisson   
Senior System Engineer
Enterprise Server and Information Systems
Baylor College of Medicine
Office (713) 798-5316
Fax (713) 798-1387
[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] 
 
 
 
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync 2.5.6 hanging

2003-11-17 Thread Andrew J. Schorr

Hi Zach,

On Mon, Nov 17, 2003 at 09:24:12AM -0800, Zachary Denison wrote:
 The version I am running on the destination machine is
 also rsync 2.5.6.  The destination machine has 4GB ram
 in it and is running redhat 8.0.  Also it gets stuck
 on all different types of files, small and large.
 sometimes the filesize is 200k and sometimes its
 several megabytes.

Have you tried the CVS version?  My impression is that many bugs
have been patched.  If you can install the CVS code on both source
and destination, that may be worth a try.

-Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Feature Request - Recursive Rsync Parameter - Example Script

2003-10-22 Thread Andrew J. Schorr

On Tue, Oct 21, 2003 at 06:46:16PM -0700, jw schultz wrote:
 
 Limiting the depth of recursion is already supported just
 not intuitive.
 
   rsync -r --exclude='/*/*/*/'
 
 Your idea for a shell script to automate picking up the
 lower levels is good and could compose the --exclude
 pattern.  The next step would be to set the job partition
 level based on path count as in find $subtree -print|wc -l.
 

I have used this technique myself to limit the number of files
processed in a single rsync invocation.  If you use find to locate
the files that you need to process, you can then use the --files-from
option to process a certain number of those files at a time.  This
works like a charm.

-Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: EOF error at io.c line 165

2003-07-30 Thread Andrew J. Schorr

Hi Thomas,

On Wed, Jul 30, 2003 at 12:33:45AM -0400, Thomas Cort wrote:
 I keep getting rsync: connection unexpectedly closed (1074142 bytes read so far)
 rsync error: error in rsync protocol data stream (code 12) at io.c(165). Does
 anyone have any idea of what could cause this? I've gotten the same error on
 two seperate machines. One UltraSparc running Linux and the other a 
 SparcStation running OpenBSD. We have an x86-SMP system running linux on the
 same network, and it works fine.

Are you using the -z flag for compression?  If so, are you using
the latest CVS version?  There was a bug in token.c that was recently
patched that can cause similar problems.  For further info, go to
   http://www.mail-archive.com/[EMAIL PROTECTED]
and search for Masahiko.  If your error is the same, then the patch that
has already been put into CVS should fix the problem.

-Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: patch to use pipe if socketpair fails

2003-07-25 Thread Harri J{rvi

On Thu, Jul 24, 2003 at 12:16:11PM -0700, jw schultz wrote:
 On Thu, Jul 24, 2003 at 01:22:58PM +0300, Harri J{rvi wrote:
  if socketpair support has been compiled in but the host where rsync is run
  doesn't support it, you get error
  pipe failed in do_recv
 
 Rejected.
 
 Build for the platform.  If you are doing a cross-platform
 build either use options to ./configure or modify config.h
 to match the target OS.

I ran into this problem when I had a minimal Linux kernel compiled on a
replicator boot disk. It had SOCKET support left out, and I got a cryptic
error message.

This would fix it.

I won't argue about rejection of this patch. I just sen this mail to tell
you that I wasn't doing a cross compile, but used a precompiled binary
on a reduced kernel, and got confused about the error message. I
investigated what went wrong and came up with this fix to let the binary
decide which to use if socketpair isn't available.

There might be other situtations where a socketpair might fail that I can't
predict.

Anyways. Thanks for reply.

Yours,
Harri Järvi

Sorry,
Cc me if replying to this
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

patch to use pipe if socketpair fails

2003-07-24 Thread Harri J{rvi

Hi,

if socketpair support has been compiled in but the host where rsync is run
doesn't support it, you get error
pipe failed in do_recv

This patch makes rsync use pipe if socketpair fails.

Yours,
Harri Järvi

CC me if you reply to this thread since I'm not on the mailing list
--- rsync-2.5.5/util.c  Wed Mar 20 03:09:49 2002
+++ util.c  Thu Jul 24 13:11:45 2003
@@ -71,6 +71,10 @@
 
 #if HAVE_SOCKETPAIR
ret = socketpair(AF_UNIX, SOCK_STREAM, 0, fd);
+if (ret != 0) {
+perror(socketpair failed. trying pipe);
+ret = pipe(fd);
+}
 #else
ret = pipe(fd);
 #endif
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync error: error in rsync protocol data stream (Broken pipe)

2003-07-10 Thread Andrew J. Schorr

Just for the record, the patch to token.c for the -z bug that was
discovered by Yasuoka Masahiko and patched by him and Wayne Davison
has fixed the problem that I reported here:

   http://www.mail-archive.com/[EMAIL PROTECTED]/msg07289.html

Thanks guys, this bug has been biting me for the past 6 months...

-Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Details of local copy

2003-06-20 Thread Ronald J Kimball

I see in the manpage that if rsync is used to copy files locally, it
behaves like an improved copy command.  I was hoping for more details on
what that means.

Here's the specific situation I'm wondering about.  I have some files that
are stored in two different locations on the same server.  I could rsync
the files from the remote server twice, first to one location and then to
the other.  Or, I could rsync the files once to one location, and then copy
them to the other location.  (The first solution is simpler, but probably
less efficient.)

If I use rsync locally to copy the files from one location to another, and
the files already exist in the second location, does rsync still compare
the existing files to determine whether it needs to update the destination
file, and does it still copy to a temporary file before replacing the
destination file?

thanks!
Ronald

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync error: error in rsync protocol data stream (Broken pipe)

2003-06-18 Thread Andrew J. Schorr

On Tue, Jun 17, 2003 at 06:35:50AM -0700, jw schultz wrote:
 
 You could try turning on transfer logging i suppose.  If you
 haven't already done so you might want to use the log file
 option in case chroot is getting in the way.  Beyond this i
 have no suggestions; i dont use rsyncd.

I may be having a similar problem.  I'm using rsync version 2.5.6cvs-20030205
every night on Solaris 8/x86 to do a backup from a client system to a 
backup server using rsyncd.  Almost every night I see the following errors
logged on the rsyncd server:

Jun 18 00:48:18 ead62 rsyncd[9632]: [ID 702911 daemon.warning] inflate returned -3 (0 
bytes)
Jun 18 00:48:18 ead62 rsyncd[9632]: [ID 702911 daemon.warning] rsync error: error in 
rsync protocol data stream (code 12) at ../token.c(416)
Jun 18 00:48:18 ead62 rsyncd[9632]: [ID 702911 daemon.warning] rsync: connection 
unexpectedly closed (197041 bytes read so far)
Jun 18 00:48:18 ead62 rsyncd[9632]: [ID 702911 daemon.warning] rsync error: error in 
rsync protocol data stream (code 12) at ../io.c(187)

And on the client side, I see the following error:

rsync: writefd_unbuffered failed to write 42 bytes: phase unknown: Broken pipe
rsync error: error in rsync protocol data stream (code 12) at ../io.c(622)

This has been happening to me for months almost every night.  I find
that I can check the return code from rsync to see whether the
transfer succeeded.  If it failed, I simply try again, and it almost
always finishes the backup successfully on the second invocation.  But if
not I keep retrying up to 10 times.  I think one evening it took 9 attempts,
but most of the time it works in 2 tries.  It's a hack, but it gets
the job done, and I was having no luck debugging the problem.

-Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: rsync 2.5.6 still hangs

2003-03-20 Thread Andrew J. Schorr

Steve,

I have a couple of comments:

1. In rsync 2.5.5, I found that the -vvv flag can cause repeatable
   rsync hangs.  Since I was using it to debug a real problem, it
   was very confusing and misleading.  But in the end, I determined
   that -vvv itself was often the culprit.  So I caution
   you not to trust what you see when you debug with -vvv: the
   results may have nothing to do with the real problem that caused
   you to look more closely with -vvv in the first place.
   (I'm assuming that the same -vvv problems still exist in 2.5.6,
   although I've never tested.)

2. I have a nightly cron job to sync up 2 servers, and rsync gives
   me the following error (or something similar) every night (this is
   the reason I was debugging with -vvv):

rsync: writefd_unbuffered failed to write 174 bytes: phase unknown: Broken pipe
rsync error: error in rsync protocol data stream (code 12) at ../io.c(622)

   This is when connecting to rsync in server mode (started by inetd).
   Previously, I was running rsync over ssh and it would hang instead
   of exiting with an error.

   My solution to the problem has been to test the return code from
   rsync and simply rerun the exact same command when I get an error
   return from rsync.  Every night I find that the first attempt
   fails after a while, but the second one always works.  I have no idea
   why this happens, but this solution works for me.

Good luck,
Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

RE: Trouble using RSYNC from a SSH terminal

2003-03-11 Thread Antonio J. M.

Hi,

Try:

rsync -varze ssh --rsync-path=remote_path_to_rsync
user@host:resource /destination

i.e.
rsync -varze ssh --rsync-path=/usr/local/bin/rsync
[EMAIL PROTECTED]:/home/bob/  /home/bob/

Regards


-Mensaje original-
De: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] En
nombre de Francisco Álvarez Martínez (Cyberclick)
Enviado el: martes, 11 de marzo de 2003 14:51
Para: [EMAIL PROTECTED]
Asunto: Trouble using RSYNC from a SSH terminal


Hi,

I'm setting up mirror server using RSYNC, but I couldn't find the
answer... Why is possible to backup to my local Linux machine and why I
couldn't use the same instruction for backing up through SSH?

I explain: from the command line of my linux machine, I write: rsync
-avuzb xxx.xxx.xxx.xxx::resource /home/www and it works properly. When I
try to do the same in another machine (connected through a SSH program),
it doesn't respond. I need to do that, because I am migrating servers
and need that information. Do I do it directly from the destination
server console and no through SSH?

I need a solution, please...

-- 
To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read:
http://www.tuxedo.org/~esr/faqs/smart-questions.html

--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: rsync and timestamps of local files

2003-03-10 Thread Andrew J. Schorr

On Sat, Mar 08, 2003 at 01:13:17PM -0500, Haisam K. Ido wrote:
 Is there a way to make rsync check the local file system for changes in the files 
 prior to it performing a diff with the remote site?

There is no built-in capability to do this in rsync.   However, you can
implement this yourself.  For example, if you touch a timestamp file
before invoking rsync, then you can use find -newer timestamp to
find files that have changed since the last time you ran rsync.

Then, you can use the --files-from patch to feed the specific list
of files to transfer into rsync.  That patch is available here:

   http://www.clari.net/~wayne/rsync-files-from.patch

Please search the archives for files-from to get more info
on what this patch does.

Cheers,
Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: Filelist caching

2003-02-18 Thread Andrew J. Schorr

Hi Rogier,

On Sat, Feb 15, 2003 at 05:05:16PM +0100, Rogier van Eeten wrote:
 On Wed, Feb 12, 2003 at 05:18:11PM -0500, Andrew J. Schorr wrote:
  On Wed, Feb 12, 2003 at 10:51:19AM -0500, Andrew J. Schorr wrote:
I was wondering... is there a way to cache that filelist? Our mirrors
are updated once, or twice a day, it could speed up downloads when I
create a filelist everytime we've mirrored others.
   
   Please take a look at the --files-from feature that is now in the CVS tree,
   courtesy of Wayne Davison.
 http://www.clari.net/~wayne/rsync-files-from.patch
 
 How does it work? What kind of list does it want? And how do I use it as
 a server?

Please refer to this archived message for more info:

   http://marc.theaimsgroup.com/?l=rsyncm=104286019712633w=2

As explained in that post, if the filename argument to --files-from has
a host: prefix, then the list will be pulled from the server.

I hope that helps.

-Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: Filelist caching

2003-02-12 Thread Andrew J. Schorr

Hi Rogier,

 I've noticed every time someone does an rsync-request on my ftp-site
 (which also provides rsync as mirror method), rsyncd creates a filelist.
 This is a quite IO and CPU intensive procedure, especially for things
 mirrors like FreeBSD with lots of little files.
 
 I was wondering... is there a way to cache that filelist? Our mirrors
 are updated once, or twice a day, it could speed up downloads when I
 create a filelist everytime we've mirrored others.

Please take a look at the --files-from feature that is now in the CVS tree,
courtesy of Wayne Davison.  That should do what you want.  It allows you to
create a set list of files and save rsync the work of scanning the directory
tree each time.  Of course, rsync still creates a filelist, but it doesn't
have to recurse over the directory tree, so it should be much faster.

-Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: Filelist caching

2003-02-12 Thread Andrew J. Schorr

On Wed, Feb 12, 2003 at 10:51:19AM -0500, Andrew J. Schorr wrote:
  I've noticed every time someone does an rsync-request on my ftp-site
  (which also provides rsync as mirror method), rsyncd creates a filelist.
  This is a quite IO and CPU intensive procedure, especially for things
  mirrors like FreeBSD with lots of little files.
  
  I was wondering... is there a way to cache that filelist? Our mirrors
  are updated once, or twice a day, it could speed up downloads when I
  create a filelist everytime we've mirrored others.
 
 Please take a look at the --files-from feature that is now in the CVS tree,
 courtesy of Wayne Davison.  That should do what you want.  It allows you to
 create a set list of files and save rsync the work of scanning the directory
 tree each time.  Of course, rsync still creates a filelist, but it doesn't
 have to recurse over the directory tree, so it should be much faster.

Oops, you're right.  I guess I applied the patch to my CVS tree and then
forgot that I had.  Sorry about that.  You can grab the files-from patch
from here:

   http://www.clari.net/~wayne/rsync-files-from.patch

It seems to apply cleanly to the current CVS tree.

Good luck,
Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

2003-01-17 Thread Andrew J. Schorr

On Thu, Jan 16, 2003 at 01:58:49PM -0800, jw schultz wrote:
 On Thu, Jan 16, 2003 at 02:52:38PM -0600, Dave Dykstra wrote:
  Also, if the transfer is being sent from the remote side, the file names
  are all getting sent over to the remote side first for --files-from and
  then sent back as part of the normal protocol, right?  I had hoped we'd
  be able to avoid that round trip because the list could get long.
 
 I don't think we can avoid the round trip without changing
 rsync drastically.  Just consider that this saves on sending
 voluminous --include lists or invoking rsync hundreds of
 times.

Perhaps I'm misunderstanding this completely, but is there a possible
scenario where the remote (sender) might have a list of files on
his side?  I might be mistaken, but it seems that the current patch
supports the case where the --files-from file list is located on
the local side, and it must be transmitted to the sender in the
case where the sender is remote.  Is there a possible case where
the --files-from file list lives on the remote (sender) side?

This is not relevant to me, but I'm just wondering.  One might
imagine a scenario in which a bunch of sites are mirroring from
a server, and the server runs a job to create a list of modified
files that the remote mirrors should pull for updating...

Come to think of it, if the data lives on the remote server, where
would a local files-from list come from?  How would it be generated?

-Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

2003-01-17 Thread Andrew J. Schorr

On Thu, Jan 16, 2003 at 05:07:13PM -0800, Wayne Davison wrote:
 (I assume you're talking about when using -R, which is not currently on
 by default.)  I believe that we do need an auto-creation mode and also a
 way to optimize the transfer to avoid this (since it results in a lot of
 extra directory checks that can slow things down when we know that they
 aren't needed).  Which one is the default is the current question.  I'm
 currently leaning toward going back to sending the implied dirs by
 default, and having an option for people to optimize the transfer (which
 would allow it to be used with the normal -R mode even when --from-files
 was not used).

This is why my patch included the --no-implicit-dirs and --send-dirs
options.  These allow you to specify precisely what you want without
having to remember which behavior is implied by various other options...

Although I suppose that those two options could be combined.  The basic
idea is that if the user is taking the responsibility for specifying
the directories himself (--send-dirs), then rsync doesn't need to
worry about automatically sending all the parent directories
(--no-implicit-dirs).

So perhaps an --explicit-dirs option (combining the two meanings) could
be used to indicate that the user is taking all responsibility for
sending directories and rsync doesn't need to worry about it.  And
to be safe, a --no-explicit-dirs to turn off this behavior.

-Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

2003-01-17 Thread Andrew J. Schorr

On Thu, Jan 16, 2003 at 07:06:05PM -0800, jw schultz wrote:
 I know i'm not talking about when -R is used.  I am talking
 about creating implied intermediate directories without -R.
 I'm talking about being able to take the output of
 find -name '*.jpg' and have it create (if necessary) any
 intermediate directories while maintaining the equivalency
 of src and dest.  If that means also behaving as though
 those directories were already in the list that would be OK
 as long as -r weren't specified.
 
   find . -name '*.jpg' | rsync -a --files-from=- .  remote:
 should when it hits
   ./deltapics/031CGMUa.jpg
   ./deltapics/031CGNga.jpg
   ./deltapics/031CGOHa.jpg
   ./deltapics/031CGPOa.jpg
   ./deltapics/031CGPba.jpg
 create the deltapics directory if it doesn't exist.  The
 permissions and ownership should be derived from the source.
 so effectively it should be as though
   ./deltapics
 where in the file list.  It needn't be updated if it
 does exist but if easier to implement it that way i wouldn't
 object.  In such a case even if -r is
 allowed and specified the implied directory should not defeat
 the the file list by transferring any files not in the list.
 
 No errors, no need to do a run to find the missing
 directories and add them and no need to add a filter to the
 stream adding entries for directories that are missing.

There are performance issues associated with sending all the
parent directories automatically.  Consider the situation where
running find test -name *.jpg -print gives the following results
(and yes, this does happen, at least for me on solaris 8 where the
output of find seems to depend on the order in which the directory
entries were created):

   test/foo.jpg
   test/bar.jpg
   test/sub/foo.jpg
   test/zeke.jpg

If I run rsync in such a way that parent directories are sent automatically,
it will send the following files (based on -vvv output):

   make_file(4,test)
   make_file(4,test/foo.jpg)
   make_file(4,test/bar.jpg)
   make_file(4,test)
   make_file(4,test/sub)
   make_file(4,test/sub/foo.jpg)
   make_file(4,test)
   make_file(4,test/zeke.jpg)

Note that the test directory is sent 3 times in this case.  This is
because the code that checks whether to send the directory just compares
to the last one sent in an attempt to eliminate duplicates.  But this
is not a reliable way of preventing duplicates, as the above example
demonstrates.  So there is a danger of sending lots of duplicate
directory entries when the automatic directory transmission feature
is enabled.

This could probably be fixed by keeping a hash table of all the directory
entries that have already been transmitted instead of just comparing
against the last one sent.

In any case, I think it's important to be able to turn off the
automatic directory sending feature so that situations that don't
require this can avoid the performance hit.

-Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: Latest --files-from patch

2003-01-16 Thread Andrew J. Schorr

Hi Wayne,

I ran a simple test of your patch at

  http://www.clari.net/~wayne/rsync-files-from.patch

and it worked fine for me.  The performance was just about the
same as for my --source-list patch.

Thanks,
Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: rsync hanging with openssh-2.9.9p2 as the transport

2003-01-15 Thread Andrew J. Schorr

On Wed, Jan 15, 2003 at 09:27:37AM -0600, Dave Dykstra wrote:
 Unfortunately I don't think anybody is going to be able to tell you.
 I've not heard of anybody lately posting a similar problem.  In the
 past hanging problems have been traced to many different sources.
 Rsync stresses network (and filesystem) implementations greatly, and
 combining it with ssh stresses things that much more.  I think it's
 worth a try to use openssh 3.1 or 3.2 (I've not been happy with versions
 after 3.2).  What's the network between the local and remote machines?
 Does the name /nfs/mirror imply that the files are not directly mounted
 on remote but are instead on an NFS server?  That has been the cause
 of many problems in the past.

Actually, it turns out that my testing on this issue has been invalid
since I've been using the -vvv flag to rsync to get a grip on where
it's hanging.  But I just discovered that the -vvv flag itself seems
to cause rsync to hang.  So all the testing I've done so far is useless,
so I'm not sure whether this is related to openssh or the slow connection
or what.  I know that the problem exists in 2.5.5, but I need to do more
testing without the -vvv flag to figure out what's causing it.  Is it
well-known that the -vvv flag can cause rsync to hang?

Thanks,
Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

specifying a list of files to transfer

2003-01-14 Thread Andrew J. Schorr

Hi,

I don't want to start another --files-from war, but I am attaching
an updated version of my patch to allow you to specify a list
of files to transfer.  The normal rsync syntax allows you to specify
a list of SRC files to transfer on the command line.  This patch
adds some new options to allow you to instead supply a file that
contains a list of files to transfer.

The previous version of the patch was against rsync-2.4.6; this version
works for rsync-2.5.5.  The only real changes relate to the use of
the popt option parsing library in 2.5 (not used in 2.4).  This had
the minor effect of removing the possibility of using - to indicate
stdin since the new library seems to interpret this as an option and
barfs.  So instead I allow the use of /dev/stdin.

By the way, this patch should also work against rsync-2.5.6pre1 except
for a couple of changes relating to white space and comments.  So a couple
of patch hunks are rejected but are easy to fix by hand.  If there is a
need, I can post an updated patch.

Last time we discussed this, Dave Dykstra objected to this patch
for two reasons:

   1. This patch only works in a single direction: when sending from a local
  system to a remote system.  It does not handle the case where you
  are receiving from a remote system to a local system.
   
   2. This capability is possible to achieve by specifying a list
  of files with --include-from and then adding --exclude '*' to
  ignore other files.  While this is true, it turns out to be
  much slower.  I have finally run a performance test to demonstrate
  this.  Results are below.

The basic idea of the patch is to handle the case where you already know
a list of files that might need to be updated and don't want to use
rsync's recursive directory tree scanning logic to enumerate all files.
The patch adds the following options:

 --source-list   SRC arg will be a (local) file name containing a list of 
files, or /dev/stdin
 --null  used with --source-list to indicate that the file names 
will be separated by null (zero) bytes instead of linefeed characters; useful with 
gfind -print0
 --send-dirs send directory entries even though not in recursive mode
 --no-implicit-dirs  do not send implicit directories (parents of the file 
being sent)

The --source-list option allows you to supply an explicit list of filenames
to transport without using the --recursive feature and without playing
around with include and exclude files.  As discussed below, the same
thing can be done by combining --recursive with --include-from and --exclude,
but it's significantly slower and more arcane to do it that way.

The --null flag allows you to handle files with embedded linefeeds.  This
is in the style of gnu find's -print0 operator.

The --send-dirs overcomes a problem where rsync refuses to send directories
unless it's in recursive mode.  One needs this to make sure that even
empty directories get mirrored.

And the --no-implicit-dirs option turns off the default behavior in which
all the parent directories of a file are transmitted before sending the
file.  That default behavior is very inefficient in my scenario where I
am taking the responsibility for sending those directories myself.

And now for a performance test:

I have a directory tree containing 128219 files of which 16064 are
directories.

To start the test, I made a list of files that had changed in the
past day:

   find . -mtime -1 -print  /tmp/changed

(normally, my list of candidate files is generated by some other means,
this is just a test example).  There were 5059 entries in /tmp/changed.

I used my new options to sync up these files to another host
as follows:

  time rsync -RlHptgoD --numeric-ids --source-list \
--send-dirs --no-implicit-dirs -xz --stats /dev/stdin \
remotehost:/extra_disk/tmp/tree1  /tmp/changed

Here were the reported statistics:

 Number of files: 5059
 Number of files transferred: 5056
 Total file size: 355514100 bytes
 Total transferred file size: 355514100 bytes
 Literal data: 355514100 bytes
 Matched data: 0 bytes
 File list size: 139687
 Total bytes written: 154858363
 Total bytes read: 80916

 wrote 154858363 bytes  read 80916 bytes  364992.41 bytes/sec
 total size is 355514100  speedup is 2.29

And the time statistics:

  112.53u 8.82s 7:03.92 28.6%

I then ran the same command again (in which case there was nothing to
transfer).  Here's how long it took:

0.54u 0.62s 0:08.61 13.4%

Now to compare with the recursive method using --include-from.  First, we
must create the list of files.  In the case of include-from, we need to
include all the parent directories as include patterns.  The following
gawk seems to do the job:

  gawk '$0 != ./ {sub(/^\.\//,)} {while ((length  0)  !($0 in already)) 
{print /$0; already[$0] = 1;

Re: specifying a list of files to transfer

2003-01-14 Thread Andrew J. Schorr

On Tue, Jan 14, 2003 at 03:32:41PM -0600, Dave Dykstra wrote:
 I haven't looked at the implementation, but comments on the user
 interface:
 1. Yes it should take a filename or - as a parameter.
 2. I don't like the idea of skipping the SRC spec.  Paths should be
   relative to the SRC.  If somebody wants to use full paths they
   can always have a SRC of /.
 3. It should be called --files-from.
 4. --send-dirs and --no-implicit-dirs shouldn't be separate options,
   they should be automatically turned on with the --files-from option.

Those comments all sound reasonable to me.  The only reason I broke
out the --send-dirs and --no-implicit-dirs options was because they
were orthogonal to what I was doing and could potentially also apply
to situations where the user was specifing various SRC filenames
on the command line.  But it's certainly fine to have --files-from turn
those on automatically.

-Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

rsync hanging with openssh-2.9.9p2 as the transport

2003-01-14 Thread Andrew J. Schorr

Hi,

This is perhaps a stupid question.  I apologize in advance if it's already
been covered, but I'm stumped...

I'm using rsync to backup some file systems to a remote host.
The transport is openssh-2.9.9p2.

The problem does not occur when the transport is rsh.

I'm invoking rsync as follows:

  rsync -RlHptgoD --numeric-ids -e ssh -rzx --stats . remotehost:/nfs/mirror

But it hangs when there are large numbers of files in the . filesystem.
If I run with -vvv, I see lots of make_file entries as send_file_list
starts sending the list of files to the remote host, but then it freezes
after around 4500 to 6500 make_file messages.

I see the same problem with versions 2.4.6, 2.5.5, and 2.5.6pre1.
And I see it on linux 2.4.18 (red hat 8.0) and solaris 8/x86.

I can get it to work by breaking it up into chunks of 1000 files or
so.  It might work with more, I haven't tested exhaustively.

I have tried using the --blocking-io and --no-blocking-io options, but
neither one solves the problem.

I could provide more info about where it hangs, but I thought somebody
might know the answer, since this is clearly related to the
interaction with openssh (since there's no problem with rsh).

Is there a trick to make rsync work nicely with openssh?  I searched the
archives and haven't found anything...  Does upgrading to openssh 3 solve
the problem?

Thanks,
Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Bug or feature? --delete-after + symlinks

2003-01-06 Thread J. Milgram



Hi,

sorry if this is old stuff, but I did my best to look first...

Have been getting errors along these lines:

mv mydir mdir.2002
ln -s mydir.2002 mydir

Then update mirror with rsync -av --delete-after. Without
--delete-after, no problem, with --delete-after, get a code 23.

The attached shell script will recreate the error (on my machine, at
any rate). Just run it from a clean directory.

Don't know if this is a bug or a feature. I did see the bit in the
TODO list about duplicate names and symlinks, this passage seems
relevant: We don't need to worry about hard links causing duplicates
because files are never updated in place.  Similarly for symlinks.

For now, I'll just live without --delete-after but it would be useful
to have at some point, so grateful for any tips.

thanks!

- Judah Milgram
[EMAIL PROTECTED]


#!/bin/bash
#
# 1. rename a directory.
# 2. create symlink with old dir's name.
# 3. update mirror
#

RSYNC=/usr/bin/rsync

uname -msrp
ldd $RSYNC
$RSYNC --version

# This seems to trigger error with rsync version 2.5.5  protocol version 26
RSYNCFLAGS=-av --delete-after

# This seems OK
# RSYNCFLAGS=-av

# set up source directory

echo
mkdir a
mkdir a/b
touch a/b/foo
find a

# mirror to target

echo
$RSYNC $RSYNCFLAGS a a_mirror
find a a_mirror

# Now do some reorganization in source dir:

echo
cd a
mv -v b b.1
ln -sv b.1 b
cd ..

# Now mirror again: 
$RSYNC $RSYNCFLAGS a a_mirror
find a a_mirror

# end

output from above:


Linux 2.4.18 i686 unknown
libpopt.so.0 = /usr/lib/libpopt.so.0 (0x40027000)
libresolv.so.2 = /lib/libresolv.so.2 (0x4002d000)
libc.so.6 = /lib/libc.so.6 (0x4003d000)
/lib/ld-linux.so.2 = /lib/ld-linux.so.2 (0x4000)
rsync  version 2.5.5  protocol version 26
Copyright (C) 1996-2002 by Andrew Tridgell and others
http://rsync.samba.org/
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, 
  IPv6, 64-bit system inums, 64-bit internal inums

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

a
a/b
a/b/foo

building file list ... done
created directory a_mirror
a/b/foo
wrote 127 bytes  read 36 bytes  108.67 bytes/sec
total size is 0  speedup is 0.00
a
a/b
a/b/foo
a_mirror
a_mirror/a
a_mirror/a/b
a_mirror/a/b/foo

`b' - `b.1'
create symbolic link `b' to `b.1'
building file list ... done
a/
delete_file: rmdir(a/b) : Directory not empty
rsync: symlink a/b - b.1: File exists
a/b.1/
a/b.1/foo
deleting a/b/foo
wrote 155 bytes  read 36 bytes  382.00 bytes/sec
total size is 3  speedup is 0.02
rsync error: partial transfer (code 23) at main.c(578)
a
a/b
a/b.1
a/b.1/foo
a_mirror
a_mirror/a
a_mirror/a/b
a_mirror/a/b.1
a_mirror/a/b.1/foo
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: Trailing Slash Inconsistencies

2002-12-03 Thread Mark J de Jong

Dave, thanks for the response! I read my command over at least 10 times
to make sure there wasn't a mistake and now I finally noticed one. The
second command should read:

rsync -va --stats --timeout=1800 --exclude-from=exclude.list --delete
--port=1 -R [EMAIL PROTECTED]::backup/home/ /home/backup1/hostname/stage/

Notice the module name is backup. I'm not sure if this would make a
difference with your explanation. However, in my scenario I need the
'-R' option because the backup process is run by a script and is backing
up about 20 different machines with multiple directories from each
machine. The script pulls all relevant information from a database and
dynamically constructs the command. The module backup is the root of
the server and I need the path on the destination to be relative to
root. The destination will always be /home/backup1/hostname/stage.

Thanks,
Mark 

On Tue, 2002-12-03 at 10:11, Dave Dykstra wrote:
 I didn't test this out myself, but I have a couple comments.  One is to
 note the two scenarios are quite different internally to rsync because in
 the second case home is the name of a rsyncd.conf module and not just
 a path.  Secondly, the -R option is not used very often and may behave
 in some unexpected ways.  It's certainly not needed in this case because
 you can always control whether or not the destination path has an extra home
 at the end.  Try the two without -R.
 
 - Dave Dykstra (who hasn't posted lately because I'm looking for a new job)
 
 
 On Wed, Nov 27, 2002 at 10:13:06AM -0500, Mark J de Jong wrote:
  Hello, 
  I am noticing inconsistencies when using a trailing slash on the source
  directories in conjunction with the '--delete' option. In one instance
  when using a trailing slash files get deleted from subdirectories in the
  destination tree. In the other instance they don't. I am using rsync
  2.5.5. The following are the scenarios:
  
  Scenario 1
  --
  *nix(src) --- *nix(dest)
  
  rsync -va --stats --timeout=1800 --exclude-from=exclude.list --delete -R
  -e ssh [EMAIL PROTECTED]:/home/ /home/backup1/hostname/stage/
  
  In this instance files in destination /home/backup1/hostname/stage/DIR
  would get deleted.
  
  
  Scenario 2
  --
  windows(src) --ssh tunnel-- *nix(dest)
  
  The windows machine is running rsync in daemon mode. I create a ssh
  tunnel from the unix machine to the windows machine and port forward
  from a random port and then run the following command:
  
  rsync -va --stats --timeout=1800 --exclude-from=exclude.list --delete
  --port=1 -R [EMAIL PROTECTED]::/home/ /home/backup1/hostname/stage/
  
  In essence it's the same as the last command except in this case files
  in the destination /home/backup1/hostname/stage/DIR do not get deleted.
  When I remove the the trailing slash from the source directory (/home)
  everything works fine.
  
  Is this by design or might this be a bug?
  
  Thanks,
  
  
  -- 
  To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
  Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
 


-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Trailing Slash Inconsistencies

2002-11-27 Thread Mark J de Jong

Hello, 
I am noticing inconsistencies when using a trailing slash on the source
directories in conjunction with the '--delete' option. In one instance
when using a trailing slash files get deleted from subdirectories in the
destination tree. In the other instance they don't. I am using rsync
2.5.5. The following are the scenarios:

Scenario 1
--
*nix(src) --- *nix(dest)

rsync -va --stats --timeout=1800 --exclude-from=exclude.list --delete -R
-e ssh [EMAIL PROTECTED]:/home/ /home/backup1/hostname/stage/

In this instance files in destination /home/backup1/hostname/stage/DIR
would get deleted.


Scenario 2
--
windows(src) --ssh tunnel-- *nix(dest)

The windows machine is running rsync in daemon mode. I create a ssh
tunnel from the unix machine to the windows machine and port forward
from a random port and then run the following command:

rsync -va --stats --timeout=1800 --exclude-from=exclude.list --delete
--port=1 -R [EMAIL PROTECTED]::/home/ /home/backup1/hostname/stage/

In essence it's the same as the last command except in this case files
in the destination /home/backup1/hostname/stage/DIR do not get deleted.
When I remove the the trailing slash from the source directory (/home)
everything works fine.

Is this by design or might this be a bug?

Thanks,


-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: How to run rsync as a daemon program taht runs at particular time given

2002-10-03 Thread Mike J Crowl


On Thu, Oct 03, 2002 at 04:21:40PM -0400, Venkat Reddy Valluri wrote:
 
 Hi,
 I want to run rsync as a daemon program so that it can run at particular time in 
nights given by us
 any help greatly appreciated
 Thks in advance

You want to run the daemon all the time and have the rsync clients connect to it.  The 
client side should be run from cron or a windows scheduler.  The clients pull from the 
rsyncd (which ever server you have that running on).  Unless you have a specific 
security/capacity issue that requires you to only run the daemon at certain times, 
there is no reason not to run it all the time.  Hope this helps clarify.

-Mike

 --Venkat
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

-- 
Mike Crowl  Phone: 303-661-2368
EDS at StorageTek   Pager: 888-480-8241
[EMAIL PROTECTED]
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

[rsync] memory leak: correction

2002-07-10 Thread Lars J. Buitinck


correction to yesterday's bug report: code fragment should have been

if ((dont = strdup(dont)) == NULL)
return;
if ((fname = strdup(fname)) == NULL) {
free(dont);
return;
}

-- 
If I traveled to the end of the rainbow
As Dame Fortune did intend,
Murphy would be there to tell me
The pot's at the other end.

Lars


-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

rsync finishes but incomplete

2002-06-25 Thread J. Davis


Hello,

I have a remote win2k machine and local NT4 machine running rsync over ssh, by way of 
cygwin.
I have a cron script that grabs a directory from remote and puts it on
local. It' very simple

#!/bin/bash

echo -  /home/rsync/job.log
echo Running rsync job  /home/rsync/job.log
gnudate  /home/rsync/job.log
rsync -az --stats -e ssh --exclude WIN386.SWP --exclude Cache/ --exclude *.AVB 
--exclude APPS/ --exclude virus update/ [EMAIL PROTECTED]:/cygdrive/e/data 
/cygdrive/g/901//home/rsync/job.log
echo Done!  /home/rsync/job.log
gnudate  /home/rsync/job.log
echo -  /home/rsync/job.log

It used to always hang but then I removed the -v option after reading
about that in the archives.
So according to my job.log the job finishes, however, it's apparent that
it has left a great many files on the remote system uncopied. Files that don't
match my --exclude args.
I'm at a loss for how to trouble shoot this one.

-Jake

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

RegExpr in ---exclude

2002-06-24 Thread J . Strohschnitter


Hi there,

is it possible to use regular expressions in the exclude-paramter of rsync ?
For example:

rsync --exclude /path/to/*/[Ff][Oo][Ll][Dd][Ee][Rr]

I've tried it, but it will take no effect.


-- 

SET THE CONTROLS 
FOR THE HEART OF THE SUN

Jens

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

test

2002-06-24 Thread J . Strohschnitter


test
-- 

SET THE CONTROLS 
FOR THE HEART OF THE SUN

Jens

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

make_file() could ignore ENODEV errors (from afs filesystem)

2002-06-18 Thread Kevin J Walters



Hi,

Would it be acceptable to patch make_file() in flist.c to ignore any
ENODEV's and not mark them as io_error=1?

In afs, a mountpoint to a volume which does not exist gives an
ENODEV. As such, it can't be copied, so should (or could) just be
ignored?

The io_error isn't terminal, but it does disable the delete function
and you end up with an exit code of 23.


regards

|evin

-- 
Kevin J Walters  Morgan Stanley
[EMAIL PROTECTED], [EMAIL PROTECTED]  20 Cabot Square
Tel: 020 7425 7886   Canary Wharf
Fax: 020 7677 8504   London E14 4QW

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: restricting rsync over ssh

2002-06-07 Thread J . K . Wight


Dave Dykstra writes:
 On Wed, May 22, 2002 at 02:39:00PM -0700, Mike Rubel wrote:
  This brings up an interesting question.  Does anyone use the server
  version of rsyncd with an ssh tunnel?  In other words:
 
  On the server, bring up sshd listening on *:22, and rsyncd accepting
  connections from localhost only, port 873.
 
  Then, on the client machine, set up an ssh tunnel:
 
  ssh server -L 1873:server:873
 
  (I just picked 1873 for convenience; you could use any open
  user-accessible port).
 
  Then rsync to localhost:1873, letting ssh carry the connection over the
  tunnel to the server:
 
  rsync rsync://user@localhost:1873/path/to/source dest
  Does anyone use this approach?  Are there any advantages/disadvantages?
 
 I didn't see any respones to your message so I guess it isn't very widely
 done.  I haven't tried it but I don't see why it wouldn't work.  You could
 probably use hosts allow = localhost to disallow people from using port
 873 directly.

Somewhat belatedly, I can report that I use rysnc in daemon mode in
conjunction with an SSH tunnel, but using remote port forwarding. I
use the method to distribute password and shadow files.

On the server, SSH is needed for other purposes, so its daemon runs
all the time, but the rsync daemon only ever runs for as long as it
takes to serve all of the clients once. I restrict access to the
module through a secrets file and change the password for every
client, using some random data out of /proc. From a regular cron job,
the server starts the rsync daemon, contacts each client in turn using
a passphraseles key, specifying remote port forwarding, and passing
the password to SSH on standard input; the rsync daemon is shut down
once every client has been dealt with. The clients have a forced
command for the key in their authorized_keys files which extracts the
password from standard output into a file and runs:

rsync --password-file that file localhost::module DEST

Jim

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

exit code 23 - inappropriate error for copying symlinks?

2002-05-20 Thread Kevin J Walters



Hi,

I've just created a small directory as an example, it holds a file and 
two symlinks that don't point to anything, and another symlink going
nowhere in a subdirectory.

rsync (2.5.5) is happy to copy these but produces an error. I can't
see why it considers this an error? It's not even aborting after it
first encounters this 'error'.

$ ls -lR /var/tmp/rsync.test
/var/tmp/rsync.test:
total 8
drwxr-xr-x   2 kjw  group512 May 20 15:45 deeper
-rw-r--r--   1 kjw  group  6 May 20 15:29 hello.world
lrwxrwxrwx   1 kjw  group  9 May 20 15:30 symbolic.link - inky-void
lrwxrwxrwx   1 kjw  group 11 May 20 15:38 symbolic.link.2 - inky-void.2
 
/var/tmp/rsync.test/deeper:
total 2
lrwxrwxrwx   1 kjw  group 11 May 20 15:45 symbolic.link.3 - inky-void.3

$ rsync -vaz /var/tmp/rsync.test otherhost:/var/tmp
building file list ... done
rsync.test/
rsync.test/deeper/
stat(rsync.test/deeper/symbolic.link.3) : No such file or directory
rsync.test/deeper/symbolic.link.3 - inky-void.3
rsync.test/hello.world
stat(rsync.test/symbolic.link) : No such file or directory
rsync.test/symbolic.link - inky-void
stat(rsync.test/symbolic.link.2) : No such file or directory
rsync.test/symbolic.link.2 - inky-void.2
rsync.test/
rsync.test/deeper/
wrote 278 bytes  read 36 bytes  209.33 bytes/sec
total size is 37  speedup is 0.12
rsync error: partial transfer (code 23) at main.c(578)
$ echo $?
23

$ rsh otherhost ls -lR /var/tmp/rsync.test
/var/tmp/rsync.test:
total 8
drwxr-xr-x   2 kjw  group512 May 20 15:45 deeper
-rw-r--r--   1 kjw  group  6 May 20 15:29 hello.world
lrwxrwxrwx   1 kjw  group  9 May 20 15:48 symbolic.link - inky-void
lrwxrwxrwx   1 kjw  group 11 May 20 15:48 symbolic.link.2 - inky-void.2
 
/var/tmp/rsync.test/deeper:
total 2
lrwxrwxrwx   1 kjw  group 11 May 20 15:48 symbolic.link.3 - inky-void.3


regards

|evin

-- 
Kevin J Walters  Morgan Stanley
[EMAIL PROTECTED], [EMAIL PROTECTED]  20 Cabot Square
Tel: 020 7425 7886   Canary Wharf
Fax: 020 7677 8504   London E14 4QW

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

compilation fails on Cobalt RaQ2

2002-04-08 Thread J. Lasser


Configure seems to work fine with no options passed, but a make attempt
fails in clientname.c:

gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -c clientname.c -o clientname.o
clientname.c: In function `client_addr':
clientname.c:55: warning: implicit declaration of function `getnameinfo'
clientname.c:56: `NI_NUMERICHOST' undeclared (first use this function)
clientname.c:56: (Each undeclared identifier is reported only once
clientname.c:56: for each function it appears in.)
clientname.c: In function `lookup_name':
clientname.c:164: `NI_NAMEREQD' undeclared (first use this function)
clientname.c:164: `NI_NUMERICSERV' undeclared (first use this function)
clientname.c:169: warning: implicit declaration of function `gai_strerror'
clientname.c:169: warning: format argument is not a pointer (arg 4)
clientname.c: In function `check_name':
clientname.c:247: warning: format argument is not a pointer (arg 4)
make: *** [clientname.o] Error 1

I'm happy to help debug/test this, but ISTM that somebody must have made
this work on the MIPS-based Cobalt boxes before...

Please CC me on any list traffic regarding this, as I'm not subscribed
to the list.

TIA,
Jon Lasser
-- 
Jon Lasser  
Home: [EMAIL PROTECTED]|Work:[EMAIL PROTECTED]
http://www.tux.org/~lasser/ |http://www.cluestickconsulting.com
   Buy my book, _Think_Unix_! http://www.tux.org/~lasser/think-unix/



msg03638/pgp0.pgp
Description: PGP signature

rsync error: unexplained error (code -1) at main.c(883)

2002-02-22 Thread Robert J. Collins


rsync 2.5.2 on FreeBSD 4.4-STABLE.

my rsyncd.conf looks like this:

read only = yes
[test]
path = /tmp

I am starting rsync as root like this:

rsync --daemon

when I connect to the rsync server on the localhost I get this:
myhost# rsync localhost::
test 

but when I connect, from the localhost, using it's hostname i get this:

myhost# rsync myhost.hwi.buffalo.edu::
rsync error: unexplained error (code -1) at main.c(883)

Thanks
-rcollins


-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Title: home-test















  ¶º©±¨C¦~Á|¿ì¥|¦¸¤§ 
  ´H°²®È¹C ¦í±J¢´§é   §Y±N®i¶}¨Ã¸g¥Ñ¦¹ Emaill 












  ¡@

  §iª¾±z¸Ô²Ó¤º®e

  ¡@

  ¹L©]½Ð¤W
  http://gohotel.com.tw 
  ºô¯¸ÂsÄý

  Your best 
  choice for accommodation in Taichung.

  

  ¤@¦¸º¡¨¬ ¦í±J.¦Y³Ü.ª±¼Ö.ÁÊª« ¤@¦¸º¡¨¬

  ¡@

  
  
¥ø·~®a¤j¶º©±
  
¡@
  
¦h¦¸ºaÀò¥x¤¤¥«Àu¨}®ÈÀ]µûÅ²²Ä¤@¦W
  

  

  ¡@

  +886+4+2220-7733+886+4+2226-1811

  ¡@

  

  Y±z¤£·Q¦A¦¬¨ì¦¹¬¡°Ê

  ½Ð±z¥Ñ¦¹[EMAIL PROTECTED]§iª¾§ÚÌ

  ÁÂÁÂ
  ¡@
  

¡@

rsync w/ openssh hangs on linux

2002-01-04 Thread J Robinson


Hello All:

I've having repeatable problems copying files between
linux machines using rsync-2.5.1 and various recent 
flavors of openssh.  The symptom when testing with a
single 2M file is that rsync hangs when when the file
is 97% transferred.  

It seems from looking at various archives that this
problem may be non-trivial.  I tried applying the
patch at 
http://www.clari.net/~wayne/rsync-nohang.patch to
rsync-2.5.1, (even though it's for 2.5.0) but it
didn't seem to make things any better.

I'm wondering if there are any tips to get around this
problem or ???

Thanks!
jrobinson

__
Do You Yahoo!?
Send your FREE holiday greetings online!
http://greetings.yahoo.com

Re: rsync -e ssh --bwlimit=20 ?

2002-01-03 Thread J Robinson


Yup, that explained it -- I had an older version of
rsync on the remote side, just like you guessed. 
Thanks Tim!

--- [EMAIL PROTECTED] wrote:
 My results:

+++
 
 Tools@willy
 /users/Tools/tccrsync --bwlimit=20
 sim/dist/sdonetsvr/sim.lic.010703 
 atlas:/tmp
 Tools@willy
 /users/Tools/tccrsync --version
 rsync  version 2.5.1pre3  protocol version 25
 Copyright (C) 1996-2001 by Andrew Tridgell and
 others
 http://rsync.samba.org/
 Capabilities: 64-bit files, socketpairs, hard links,
 symlinks, batchfiles, 
 no IPv6
 
 Tools@willy
 /users/Tools/tccuname -a
 SunOS willy 5.7 Generic_106541-07 sun4u sparc
 SUNW,Ultra-4
 Tools@willy
 /users/Tools/tcc

+++
 the second command is the crux. 
 How about a
 rsync --version;rsh from.com rsync --version
 I'm guessing we'll see one or both with a version
 that doesn't support 
 bwlimit.
 Tim Conway
 
 
 
 
 J Robinson 
 
 
 Hello Everyone:
 I've been trying to get rsync using --bwlimit
 using openssh for transport without success.
 
 Here's what I get: 
 
   % rsync --bwlimit=20 from.com:/dir/file.txt /dest
   [EMAIL PROTECTED]'s password:  
   rsync: unrecognized option `--bwlimit=20'
   unexpected EOF in read_timeout
 
 I tried specifying ssh transport using both the 
 -e option and by setting the RSYNC_RSH evironment
 variable--  it made no difference. rsync works as
 expected without the --bwlimit option.
 
 I know --bwlimit is a fairly new feature. and I
 researched this as much as I could without going
 into
 the source code... anyone able to get this to work,
 or
 have any tips?
 Thanks 
 jrobinson
 

__
Do You Yahoo!?
Send your FREE holiday greetings online!
http://greetings.yahoo.com

Re: Netware modify bit changed

2001-12-05 Thread Juan J. López

On 5 Dec 2001 at 15:02, Martin Pool wrote:

Date sent:  Wed, 5 Dec 2001 15:02:34 +1100
From:   Martin Pool [EMAIL PROTECTED]
To: Juan J. L?pez [EMAIL PROTECTED]
Copies to:  [EMAIL PROTECTED]
Subject:Re: Netware modify bit changed

 On  5 Dec 2001, Juan J. L?pez [EMAIL PROTECTED] wrote:
  Dave:

  With archive bit I mean a MS-DOS file attribute (like read
  only, system or hidden).  When the archive attribute of a
  file is set, that file is presumed to be changed after the last
  backup and then must be copied again. The backup aplication 
reset
  the archive attribute when copy a file. On the other hand, the
  operating system (DOS, Windows, etc) set it when the file is
  modified.  Why rsync set this attribute on the copy, even when the
  original file is unchanged?

 So the desired behaviour is that rsync should make the archive bit
 have the same setting on the destination as on the source, or that it
 should always clear it?

 At the moment rsync is completely unaware of the A bit, so it will get
 the default OS behaviour which is probably to set the bit on newly
 modified files.

 If anybody who cares about this writes a reasonably clean patch to
 implement it then I guess it would be considered.  

 I think Samba can optionally map the A bit to the Unix x bit.  
Perhaps
 you could try that.

Rsync don't need to worry about the A bit, IMHO. The A bits on 
files of the destination directory are set when rsync make the file-
comparison with the source files. May be rsync open the files in 
write mode, instead of read-only mode, even when the file don't 
need to be transfered from the source (it's identical to original). In 
consequence, the OS sets the A bit.

My conf:

#ncpmount -S netware_server -U netware-user -P netware-pass -V 
SYS /novell
#rsync -zau --password-file /etc/rsync.passwd backup@source-
linux::disk-f/source-dir/* /novell/destination-dir

Regards,
Juan

Re: Netware modify bit changed

2001-12-04 Thread Juan J. López


Dave:

With archive bit I mean a MS-DOS file attribute (like read only, system 
or hidden). 
When the archive attribute of a file is set, that file is presumed to be changed 
after the last 
backup and then must be copied again. The backup aplication reset the archive 
attribute when 
copy a file. On the other hand, the operating system (DOS, Windows, etc) set it when 
the file is 
modified.
Why rsync set this attribute on the copy, even when the original file is 
unchanged?

Thanks,
Juan

On 4 Dec 2001 at 11:01, Dave Dykstra wrote:

Date sent:  Tue, 4 Dec 2001 11:01:50 -0600
From:   Dave Dykstra [EMAIL PROTECTED]
To: Juan [EMAIL PROTECTED]
Copies to:  [EMAIL PROTECTED]
Subject:Re: Netware modify bit changed

 On Tue, Dec 04, 2001 at 12:51:07AM -0300, Juan wrote:
  Hi,
  
I run rsync, with a Netware-directory mounted on a linux system
  (using ncpfs) as DESTINATION of the copy. This configuration works fine,
  but rsync incorrectly sets the archive bit (modify) for ALL the files
  included in the transfered directory (both modified and not modified).
  In consecuence, the incremental backups on my netware server don't work.
 
 Do you mean the access time?  If so, tell me what rsync command line you
 use because rsync normally will not change the access time of unmodified
 files.  If you really mean archive bit, it must be something in
 Netware that I've never heard of, and please explain what it is.
 
 - Dave Dykstra

Re: patch to enable faster mirroring of large filesystems

2001-11-26 Thread Andrew J. Schorr


On Sun, Nov 25, 2001 at 03:21:51AM +1100, Martin Pool wrote:
 On 20 Nov 2001, Dave Dykstra [EMAIL PROTECTED] wrote:
 
   And, by the way, even if the batch stuff accomplishes the same performance
   gains, I would still argue that the --files-from type of behavior
   that I implemented is a nice transparent interface that people might
   like to have.  The ability to pipe in output from gfind -print0 opens
   up some possibilities.
  
  Yes, many people have argued that a --files-from type of option is
  desirable and I agree.
 
 I agree too.  I think there would certainly be no argument with taking
 a patch that did only that.  (Except that it makes the option list
 even more ridiculously bloated.)
 
 I think a better fix is to transfer directories one by one, pipelined,
 rather than walking the whole tree upfront.  This will take a protocol
 change and a fair amount of work, but I think it is possible.  If we
 can get it to just work faster transparently that is better.

I understand your point of view, but I think it is a mistake to
hold rsync's algorithm hostage to the directory tree traversal logic
built into the program.  IMHO, the basic file transfer algorithm of
rsync is terrific, but the program wrapped around it is a bit out of
control.

The spirit of my patch is to expose the low-level rsync algorithm and
to allow people to build up their customized infrastructure outside
of the program instead of having to build it in.  I think this is in
the spirit of Unix tools.  I think if rsync were to expose some of its
low-level capabilities, then we would not have a need for xdelta and rdiff,
projects which are springing up because of rsync's opaqueness.

Anyway, you may not like the way my patch is implemented, but I still argue
that it serves a useful purpose, and it gets the job done for me.

Cheers,
Andy

patch to enable faster mirroring of large filesystems

2001-11-19 Thread Andrew J. Schorr


I have attached a patch that adds 4 options to rsync that have helped
me to speed up my mirroring.  I hope this is useful to someone else,
but I fear that my relative inexperience with rsync has caused me to
miss a way to do what I want without having to patch the code.  So please
let me know if I'm all wet.

Here's my story: I have a large filesystem (around 20 gigabytes of data)
that I'm mirroring over a T1 link to a backup site.  Each night, 
about 600 megabytes of data needs to be transferred to the backup site.
Much of this data has been appended to the end of various existing files,
so a tool like rsync that sends partial updates instead of the whole
file is appropriate.

Normally, one could just use rsync with the --recursive and --delete features
to do this.  However, this takes a lot more time than necessary, basically
because rsync spends a lot of time walking through the directory tree
(which contains over 300,000 files).

One can speed this up by caching a listing of the directory tree.  I maintain
an additional state file at the backup site that contains a listing
of the state of the tree after the last backup operation.  This is essentially
equivalent to saving the output of find . -ls in a file.

Then, the next night, one generates the updated directory tree for the source
file system and does a diff with the directory listing on the backup file
system to find out what has changed.  This seems to be much faster than
using rsync's recursive and delete features.

I have my own script and programs to delete any files that have been removed,
and then I just need to update the files that have been added or changed.
One could use cpio for this, but it's too slow when only partial files
have changed.

So I added the following options to rsync:

 --source-list   SRC arg will be a (local) file name containing a list of 
files, or - to read file names from stdin
 --null  used with --source-list to indicate that the file names 
will be separated by null (zero) bytes instead of linefeed characters; useful with 
gfind -print0
 --send-dirs send directory entries even though not in recursive mode
 --no-implicit-dirs  do not send implicit directories (parents of the file 
being sent)

The --source-list option allows me to supply an explicit list of filenames
to transport without using the --recursive feature and without playing
around with include and exclude files.  I'm not really clear on whether
the include and exclude files could have gotten me the same place, but it
seems to me that they work hand-in-hand with the --recursive feature that
I don't want to use.

The --null flag allows me to handle files with embedded linefeeds.  This
is in the style of gnu find's -print0 operator.

The --send-dirs overcomes a problem where rsync refuses to send directories
unless it's in recursive mode.  One needs this to make sure that even
empty directories get mirrored.

And the --no-implicit-dirs option turns off the default behavior in which
all the parent directories of a file are transmitted before sending the
file.  That default behavior is very inefficient in my scenario where I
am taking the responsibility for sending those directories myself.

So, the patch is attached.  If you think it's an abomination, please let
me know what the better solution is.  If you would like some elaboration
on how this stuff really works, please let me know.

Cheers,
Andy


--- flist.c.origTue Sep  5 22:46:43 2000
+++ flist.c Fri Nov  9 12:01:56 2001
@@ -30,6 +30,7 @@
 extern int cvs_exclude;
 
 extern int recurse;
+extern int send_dirs;
 
 extern int one_file_system;
 extern int make_backups;
@@ -501,8 +502,8 @@
/* we use noexcludes from backup.c */
if (noexcludes) goto skip_excludes;
 
-   if (S_ISDIR(st.st_mode)  !recurse) {
-   rprintf(FINFO,skipping directory %s\n,fname);
+   if (S_ISDIR(st.st_mode)  !recurse  !send_dirs) {
+   rprintf(FINFO,make_file: skipping directory %s\n,fname);
return NULL;
}

@@ -689,14 +690,16 @@
 }
 
 
-struct file_list *send_file_list(int f,int argc,char *argv[])
+static struct file_list *send_file_list_proc(int f,char *(*ffunc)(), void *opq)
 {
-   int i,l;
+   int l;
STRUCT_STAT st;
char *p,*dir,*olddir;
char lastpath[MAXPATHLEN]=;
struct file_list *flist;
int64 start_write;
+   char *in_fn;
+   extern int implicit_dirs;
 
if (verbose  recurse  !am_server  f != -1) {
rprintf(FINFO,building file list ... );
@@ -711,10 +714,10 @@
io_start_buffering(f);
}
 
-   for (i=0;iargc;i++) {
+   while ((in_fn = (*ffunc)(opq)) != NULL) {
char *fname = topsrcname;
 
-   strlcpy(fname,argv[i],MAXPATHLEN);
+   strlcpy(fname,in_fn,MAXPATHLEN);
 
l = strlen(fname);
if (l != 1  fname[l-1]

rsync hangs or exists without copying anything

2001-11-16 Thread Daniel J. Conlon


I am trying to mirror a file system using rsync. The command I am using
is of the form:

rsync -a /fs/home/6/ /usr/fs/home/6

/fs/home/6/ is an NFS file system, /usr/fs/home/6 is a local disk.

With versions 2.4.6 and 2.4.7pre1, rsync hangs at random places during
the building file list phase. I tried with and without -v option(s) and
tried breaking the file system down into smaller chunks with no
improvement. 

I have read of many similar problems with later versions of rsync so I
tried with version 2.3.2. This time rsync did not produce any errors,
but did not copy anything. /usr/fs/home/6 is at this point empty so it
should have copied the entire file system. By using the v options(s) and
by breaking the file system down into smaller and smaller sections I
found that rsync was existing at the same point during the building file
list phase, specifically, when it hit's this directory:

make_file(4,a/d/adrenalinuk/public_html/cgi-bin/misc/ib3/Database/Databa
se/Database/active_sessions/Database/Database/address_books/Database/Dat
abase/Database/authorisation/Database/Database/board_statistics/Database
/Database/categories/Database/Database/config/Database/config/Database/c
onfig/Database/config/Database/config/Database/config/Database/config/Da
tabase/config/Database/config/Database/config/Database/config/Database/c
onfig/Database/config/Database/config/Database/config/Database/config/Da
tabase/config/Database/config/Database/config/Database/config/Database/c
onfig/Database/config/Database/config/Database/config/Database/config/Da
tabase/config/Database/config/Database/Database/email_templates/Database
/Database/forum_info/Database/Database/forum_moderators/Database/Databas
e/forum_polls/Database/Database/forum_posts/Database/Database/forum_rule
s/Database/Database/forum_subscriptions/Database/Database/forum_topics/D
atabase/Database/help/Database/Database/Database/mem_groups/Database/mem
_groups) 

Is there some kind limit on the depth of file system rsync can cope
with? Or the length of the filename? If so, how can I increase it or
simply have rsync ignore paths or depths greater than the maximum,
rather than just exiting?

If there is no limit, how can I overcome this problem?

Many thanks for your help in advance.


Kind Regards,

Daniel Conlon

Using RSync with NT/98

2001-09-11 Thread Daniel . J . Botz


Hello,

I'm trying to get RSync to work between an NT workstation and a Win98
workstation.  Has anybody gotten this to work?

I know my NT workstation is set up correctly because I am able to RSync
between two NT workstations.  But when I try to RSync between an NT and
Win98 workstation I get a pipe failed on do_recv with an RSync exit
status of 10.

My NT workstation is set up to run RSync in daemon mode and it's
rsyncd.conf file looks like this:

use chroot = false
strict modes = false
hosts allow = *

[Test]
  comment = My Temp directory
  path = /cygdrive/c/Temp
  read only = yes
  list = yes


I issue this command on the Win98 workstation to start the RSync:

/bin/rsync.exe --verbose --stats --archive --one-file-system --rsync-path
='/bin/rsync.exe' --include='sitever.ini' --include='hclSetup.log'
--include='Update.zip' --exclude='*' onanmtso935::Test /cygdrive/c/Test


RSync produces the following output:

add_exclude(sitever.ini)
add_exclude(hclSetup.log)
add_exclude(Update.zip)
add_exclude(*)
local_version=24 remote_version=24
receiving file list ... recv_file_name(.)
recv_file_name(hclSetup.log)
recv_file_name(sitever.ini)
recv_file_name(Update.zip)
received 4 names
[104631079] i=0 NULL . mode=040755 len=8192
[104631079] i=1 NULL Update.zip mode=0100644 len=3200
[104631079] i=2 NULL hclSetup.log mode=0100644 len=12146
[104631079] i=3 NULL sitever.ini mode=0100644 len=205
done
recv_file_list done
get_local_name count=4 /cygdrive/c/Test
pipe failed in do_recv


Any help would be greatly appreciated.

Thanks,
Dan Botz
Cummins Power Generation

Re: RSync on NT

2001-07-06 Thread Daniel . J . Botz



O.K.  That makes sense.  I'll install Cygwin and just to make sure I've got
everything at the correct version, I'll do the configure and make.

Thanks for the help!!!




   
 
Kevin Faust
 
kevinfaust@mTo: [EMAIL PROTECTED] 
 
ail.com cc: David Bolen [EMAIL PROTECTED], 
[EMAIL PROTECTED] 
 Subject: Re: RSync on NT  
 
07/06/01   
 
05:58  
 
   
 
   
 




Even if your binary was built with Cygwin (and you have cygwin1.dll on your
path), I think you still need
cygwin installed to perform the mounts. When installed, the cygwin binary
adds its mount point(s) to the
registry.

[EMAIL PROTECTED] wrote:

 I thought the binary I down loaded was built with Cygwin but when I try
 your suggestion for the path names, RSync complains.

 Any chance I could get your binary?


 David Bolen
 db3l@fitlinxTo:
'[EMAIL PROTECTED]' [EMAIL PROTECTED]
 x.com   cc: [EMAIL PROTECTED]
  Subject: RE: RSync on NT
 07/03/01
 16:05



 [EMAIL PROTECTED] [[EMAIL PROTECTED]] writes:

  Has anybody had any luck getting RSync to work with WinNT 4.0?

 Yep.  At least compiled for use with Cygwin, it works fine.  I do use
 a local tool to make a named pipe connection to a target machine
 rather than rsh, but any old path should work.

  I am interested in using RSync in a non-daemon mode.  How do I specify
  drive/directory paths along with host names?  If I issue this command:

 You can't really use native drive specifications because as you note
 rsync uses the Unix convention of separating the system from the path
 with a colon.

 But since Rsync is built on top of Cygwin (I'm presuming that's how
 you built it) you can use the standard /cygdrive/? notation (or until
 they formally remove it, the deprecated //? notation) to select a
 drive.  You should also be able to use //system/share to access
 network shares.

 -- David

 /---\
  \   David Bolen\   E-mail: [EMAIL PROTECTED]  /
   | FitLinxx, Inc.\  Phone: (203) 708-5192|
  /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150 \
 \---/

RE: RSync on NT

2001-07-05 Thread Daniel . J . Botz



I thought the binary I down loaded was built with Cygwin but when I try
your suggestion for the path names, RSync complains.

Any chance I could get your binary?




   
 
David Bolen
 
db3l@fitlinxTo: '[EMAIL PROTECTED]' 
[EMAIL PROTECTED]  
x.com   cc: [EMAIL PROTECTED] 
 
 Subject: RE: RSync on NT  
 
07/03/01   
 
16:05  
 
   
 
   
 




[EMAIL PROTECTED] [[EMAIL PROTECTED]] writes:

 Has anybody had any luck getting RSync to work with WinNT 4.0?

Yep.  At least compiled for use with Cygwin, it works fine.  I do use
a local tool to make a named pipe connection to a target machine
rather than rsh, but any old path should work.

 I am interested in using RSync in a non-daemon mode.  How do I specify
 drive/directory paths along with host names?  If I issue this command:

You can't really use native drive specifications because as you note
rsync uses the Unix convention of separating the system from the path
with a colon.

But since Rsync is built on top of Cygwin (I'm presuming that's how
you built it) you can use the standard /cygdrive/? notation (or until
they formally remove it, the deprecated //? notation) to select a
drive.  You should also be able to use //system/share to access
network shares.

-- David

/---\
 \   David Bolen\   E-mail: [EMAIL PROTECTED]  /
  | FitLinxx, Inc.\  Phone: (203) 708-5192|
 /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150 \
\---/

RSync on NT

2001-07-03 Thread Daniel . J . Botz


Hello,

Has anybody had any luck getting RSync to work with WinNT 4.0?

I am interested in using RSync in a non-daemon mode.  How do I specify
drive/directory paths along with host names?  If I issue this command:

rsync --rsync-path=C:\programs\rsync.exe --rsh=C:\programs.rsh.exe
host:c:\dir1 c:\dir2

How does RSync distinguish between the host name and the drive letter?

Thanks,
Dan

Re: RSync directory structure only

2001-06-18 Thread Daniel . J . Botz



Hello,

Thanks for the quick response Tim!!

I tried using the excludes you mentioned but it doesn't seem to work.
Every file still gets copied.

I also tried to have the remote machine execute a command to generate a
file list but this didn't seem to work either:

rsync -av server:`find /dir1/dir2 -type d` dest

Anybody have any other ideas?

Thanks!
Dan



   
 
Tim Potter 
 
tpot@valinuxTo: [EMAIL PROTECTED] 
 
.comcc: [EMAIL PROTECTED] 
 
 Subject: Re: RSync directory structure 
only
06/17/01   
 
18:11  
 
   
 
   
 




[EMAIL PROTECTED] writes:

 Is there a way to use rsync to replicate a directory structure only and
not
 the files in the directory structure?

Try this:

rsync -av --include '*/' --exclude '*' source-dir dest-dir


Tim.

Re: RSync directory structure only

2001-06-18 Thread Daniel . J . Botz


Hello,

Just wanted to let everyone know that I figured it out.  For some reason,
when I take the single quotes away from the include and exclude options,
things seem to work as one would expect.

Thanks,
Dan


- Forwarded by Daniel J Botz/PGG/Cummins on 06/18/01 08:10 -
   
 
Daniel J Botz  
 
 To: Tim Potter [EMAIL PROTECTED] 
 
06/18/01 07:35   cc: [EMAIL PROTECTED] 
 
 Subject: Re: RSync directory structure 
only(Document link: Daniel J
 Botz) 
 
   
 



Hello,

Thanks for the quick response Tim!!

I tried using the excludes you mentioned but it doesn't seem to work.
Every file still gets copied.

I also tried to have the remote machine execute a command to generate a
file list but this didn't seem to work either:

rsync -av server:`find /dir1/dir2 -type d` dest

Anybody have any other ideas?

Thanks!
Dan



   
 
Tim Potter 
 
tpot@valinuxTo: [EMAIL PROTECTED] 
 
.comcc: [EMAIL PROTECTED] 
 
 Subject: Re: RSync directory structure 
only
06/17/01   
 
18:11  
 
   
 
   
 




[EMAIL PROTECTED] writes:

 Is there a way to use rsync to replicate a directory structure only and
not
 the files in the directory structure?

Try this:

rsync -av --include '*/' --exclude '*' source-dir dest-dir


Tim.

1 2 >

1 - 100 of 105 matches

Mail list logo