include/exclude directory question
I want to send a subset of directories sepcified in "include" arguments to a client, but I am creating all the peer and parent directories as well, although they are empty - here is basically what I'm doing. assuming I have /staging/upgrade/dir1, /staging/upgrade/dir2 and /staging/upgrade/dir3 on the source tree. --include "*/" -include upgrade/dir1/** -include upgrade/dir3/** -exclude "*" /staging/* / what I get on the client is /ugrade/dir1/* /upgrade/dir2 (an empty directory) /upgrade/dir3/* How can I get rsync to filter out and stop creating the empty dir2 directory without specifucally excluding it? I've looked through about a years worth of past postings with likely looking subject lines but can't find the answer. Thanks, Dave
Not all files synched - hard link problems???
I am sorry if this has been covered before: I have done a couple of futile searches in the bug reporting system and is there any way to search the archive? I am having a strange symptom: I am synching directories (that have very long file names, by the time the full path is specified) and a lot of hard links. It seems that the directory is being copied piecemeal - that is, if I run rsync enough times, the entire contents ultimately get copied. It seems like I am running into some hard limit in the size of the filelist or something. I am running 2.4.6 on linux - the source directory is remote mounted on solaris, the destination is linux. For instance, I have a directory that is in the tree that contains 173 files at the source - most of which are hard links - here is the effect of an ls | wc on the destination after five succesive identical runs of rsync on the source (this is in a subdirectory of one of the directories in the command below). The directory did not exist before running the sync. [root@ks-s0-107-1- SC]# ls | wc 49 491286 [root@ks-s0-107-1- SC]# ls | wc 85 852234 [root@ks-s0-107-1- SC]# ls | wc 120 1203243 [root@ks-s0-107-1- SC]# ls | wc 152 1524112 [root@ks-s0-107-1- SC]# ls | wc 173 1734739 So that it seems to have synched 49, then 36, then 35, then 32, the finally the last 21 files in the directory. (The increment seems to vary if I try it again, i.e. from 49 to 90). I get no error mesages (that I can see). In fact, the first time I run the program, it seems to notice all the files and produces 163 messages that "file blah-de-blah" is a hard link, but then doesn't seem to make the link for file "blah-de-blah" - this behavior remains constant with each succesive run. Here is the rsync command (generated by a perl script and broken into little pieces by my mailer) rsync -e 'ssh -l root -p 8989' --rsync-path /usr/bin/rsync --stats --progress -tpgoCHr -v -v --include install/ --include install/kickstart/ --include install/kickstart/zantaz_upgrade/ --include install/kickstart/zantaz_upgrade/20011121/ --include install/redhat-7.1/ --include install/redhat-7.1/zantaz_rpm_upgrade/ --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/ --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/ --include install/kickstart/zantaz_upgrade/20011121/** --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/** --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/** --exclude "*" /net/swdev/staging/* [EMAIL PROTECTED]:/ Thanks, Dave
More - cause - not all files synched - program hangs in select
The bug has something to do with verbosity - it works fine without verbosity on. The irony, of course, is that one turns verbosity on to fix these things (I have applied the hang patch, BTW). The last thing I see is the "match_report" message, then the program hangs in select. Here are the symptoms: I am syncing a number of parallel directories that contain large numbers hard links. At the source: dir a: f1 f2 f3 f4 f5 dir b: f1 f2 f3 f4 f5 dir c: f1 f2 f3 f4 f5 f1 in each directory is a hard link to every other f1, etc. (note, there are actually about 10 directories of 100 files in each directory) After my first run of rsync, I get something like this. dir a: f1 (link count 3) f2 (3) f3 (1) f4 (1) f5 (1) dir b: f1 (3) f2 (3) dir c: f1 (3) f2 (3) After my second run of rsync, I get something like this. dir a: f1 (3) f2 (3) f3 (3) f4 (1) f5 (1) dir b: f1 (3) f2 (3) f3 (3) dir c: f1 (3) f2 (3) f3 (3) After my third run of rsync, I get something like this. dir a: f1 (3) f2 (3) f3 (3) f4 (3) f5 (1) dir b: f1 (3) f2 (3) f3 (3) f4 (3) dir c: f1 (3) f2 (3) f3 (3) f4 (3) etc. So that dir a: is completely copied the first time, but only the beginnings of dir b and c. With each successive run, dirs b & c (& d & e & f) fill up, until it really is done. Again, this doesn't happen if I turn verbosity entirely off. Perhaps somebody is closing some file descriptor - I noticed that the program was hanging on the remote box in "select" - and then somebody else (probably having to do with making links) is trying to write to it. Thus, the program hangs but the timing is not always the same and sometimes a few more or less links are made before it hangs. It seems that making the hard links is done in a second phase after the real file transfers are done, at which time something must be closed. I could look through the source code and fix it myself but maybe by the time I get in tomorrow somebody that knows it well will have done so. Dave Dave Madole wrote: > I am sorry if this has been covered before: I have done a couple of > futile searches in the bug reporting system and is there any way to > search the archive? > > I am having a strange symptom: I am synching directories (that have > very long file names, by the time the full path is specified) and a lot > of hard links. It seems that the directory is being copied piecemeal - > that > is, if I run rsync enough times, the entire contents ultimately get > copied. It seems like I am running into some hard limit in the size of > the filelist or something. > > I am running 2.4.6 on linux - the source directory is remote mounted on > solaris, the destination is linux. > > For instance, I have a directory that is in the tree that contains 173 > files at the source - most of which are hard links - here is the effect > of an ls | wc on the destination after five succesive identical runs of > rsync on the source (this is in a subdirectory of one of the directories > in the command below). The directory did not exist before running the > sync. > > [root@ks-s0-107-1- SC]# ls | wc > 49 491286 > [root@ks-s0-107-1- SC]# ls | wc > 85 852234 > [root@ks-s0-107-1- SC]# ls | wc > 120 1203243 > [root@ks-s0-107-1- SC]# ls | wc > 152 1524112 > [root@ks-s0-107-1- SC]# ls | wc > 173 1734739 > > So that it seems to have synched 49, then 36, then 35, then 32, the > finally the last 21 files in the directory. (The increment seems to vary > if I try it again, i.e. from 49 to 90). > > I get no error mesages (that I can see). In fact, the first time I run > the program, it seems to notice all > the files and produces 163 messages that "file blah-de-blah" is a hard > link, but then doesn't > seem to make the link for file "blah-de-blah" - this behavior remains > constant with each succesive run. > > Here is the rsync command (generated by a perl script and broken into > little pieces by my mailer) > > rsync -e 'ssh -l root -p 8989' --rsync-path /usr/bin/rsync --stats > --progress -tpgoCHr -v -v --include install/ --include > install/kickstart/ --include install/kickstart/zantaz_upgrade/ --include > install/kickstart/zantaz_upgrade/20011121/ --include install/redhat-7.1/ > --include install/redhat-7.1/zantaz_rpm_upgrade/ --include > install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/ > --include > install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/ > --include install/kickstart/zantaz_upgrade/20011121/** --include > install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/** > --include > install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/** > --exclude "*" /net/swdev/staging/* [EMAIL PROTECTED]:/ > > Thanks, > > Dave
release 2.4.7pre4 - Thanks!
Excellent work, Martin (and any others involved). No more hangs, and the more consistent and complete messages at the different levels of verbose are great. The "why is a path included or excluded" messages at -vv should cut the traffic on this list by about 25% (I wish I'd had them a couple of weeks ago). I've been testing it all morning with a somewhat Byzantine set of perl generated options and so far no problems. (And it may be my imagination, but it even seems faster). Thanks again, Dave Madole
Re: How to avoid copying empty directories?
I agree with this - a perl script to generate a list of "includes" from a list of
directories
is not hard to write (below in "long and readable" rather than compact perly
form), and what about --python and --awk:
#cat frag.pl
#!/usr/bin/perl -w
# frag.pl - perl fragment for generating include list for rsync
my $includes = "";
my %already = ();
my @pathlist = <>;
foreach my $path (@pathlist) {
chop $path;
my @subdirs = split(/\//,$path);
my $dirs;
foreach my $dir (@subdirs) {
$dirs .= $dir."/";
unless(defined($already{$dirs}) && $already{$dirs} eq "included") {
$includes .= " --include ".$dirs;
$already{$dirs} = "included";
}
}
$includes .= " --include $path/**";
}
$includes .= " --exclude \"*\" ";
print $includes,"\n";
#cat test
dir1/local/src/dir1
dir2/local/bin
dir2/local/src
#cat test | ./frag.pl
--include dir1/ --include dir1/local/ --include dir1/local/src/ --include
dir1/local/src/dir1/ --include dir1/local/src/dir1/** --include dir2/ --include
dir2/local/ --include dir2/local/bin/ --include dir2/local/bin/** --include
dir2/local/src/ --include dir2/local/src/** --exclude "*"
Dave Dykstra wrote:
> On Thu, Nov 29, 2001 at 09:23:18AM +1100, Martin Pool wrote:
> > On 28 Nov 2001, Dave Dykstra <[EMAIL PROTECTED]> wrote:
> > > On Tue, Nov 27, 2001 at 04:03:58PM -0800, [EMAIL PROTECTED] wrote:
> > > > rsync -avu --include 'tmp1/*/*.c' --include "*/" --exclude "*" tmp1 tmp2
> > > >
> > > > The above command copies all the empty directories under tmp1/ . Is there
> > > > any way to avoid it?
> > >
> > > Currently the only way is to explicitly include the parent directories
> > > you want instead of --include "*/".
> >
> > A quick fix is to post-process the directory with
> >
> > find tmp2 -type d -empty -depth -exec rmdir {} \;
> >
> > I think the best fix is to add scripting support:
> >
> > rsync -avu --perl 'filter() { ( -d && -s ) || /\.c$/; }' here friendly:/tmp
> >
> > Anybody else like this idea?
>
> Nah, the best fix in this case is --files-from. I'm not too keen on the
> scripting idea. Seems like more bloat to me. With --files-from people
> can use whatever script they like to generate the list of files before
> sending it to rsync.
>
> - Dave
Re: problem with 2.4.7pre4
I've had similar problems with a >800M file and 2.4.7pre4. In one transfer, the program just went away, and resulted in reiserfs going into an endless loop waiting for a buffer to be freed (the machine had to be rebooted, but I'd say that even the potential for going into an endless loop like that may be a shortcoming of reiserfs). This, of course, may have had nothing to do with rsync, and, of course, this hasn't been repeatable (we've tried). Another time rsync just hung for a good minute at about 60% of the transfer and then proceeded (this may have been ssh related). Other times it is okay. It also seems that at one point rsync wasn't recognizing that "two" of the 800M files were actually hard linked together, although in the same run it did fine with smaller files (of the same name in different directories). Again, these could be operator error. Dave Jeremy Hansen wrote: > yOn Thu, 29 Nov 2001, Jeremy Hansen wrote: > > > > > rsync: connection unexpectedly closed (0 bytes read so far) > > rsync error: error in rsync protocol data stream (code 12) at io.c(139) > > Just a note here, if I move the rsync server back to 2.4.6, I'm at least > able to initiate an actual copy. 2.4.7pre4 just fails right away with the > above. > > > Also, on 2.4.6 I was getting this: > > > > access_log > > write failed on access_log : Success > > unexpected EOF in read_timeout > > > > THis is about an 800meg file I'm trying to copy. > > > > Thanks > > -jeremy > > > > -- > The trouble with being poor is that it takes up all your time.
