Hi, I don't want to start another --files-from war, but I am attaching an updated version of my patch to allow you to specify a list of files to transfer. The normal rsync syntax allows you to specify a list of SRC files to transfer on the command line. This patch adds some new options to allow you to instead supply a file that contains a list of files to transfer.
The previous version of the patch was against rsync-2.4.6; this version works for rsync-2.5.5. The only real changes relate to the use of the popt option parsing library in 2.5 (not used in 2.4). This had the minor effect of removing the possibility of using "-" to indicate stdin since the new library seems to interpret this as an option and barfs. So instead I allow the use of "/dev/stdin". By the way, this patch should also work against rsync-2.5.6pre1 except for a couple of changes relating to white space and comments. So a couple of patch hunks are rejected but are easy to fix by hand. If there is a need, I can post an updated patch. Last time we discussed this, Dave Dykstra objected to this patch for two reasons: 1. This patch only works in a single direction: when sending from a local system to a remote system. It does not handle the case where you are receiving from a remote system to a local system. 2. This capability is possible to achieve by specifying a list of files with --include-from and then adding --exclude '*' to ignore other files. While this is true, it turns out to be much slower. I have finally run a performance test to demonstrate this. Results are below. The basic idea of the patch is to handle the case where you already know a list of files that might need to be updated and don't want to use rsync's recursive directory tree scanning logic to enumerate all files. The patch adds the following options: --source-list SRC arg will be a (local) file name containing a list of files, or /dev/stdin --null used with --source-list to indicate that the file names will be separated by null (zero) bytes instead of linefeed characters; useful with gfind -print0 --send-dirs send directory entries even though not in recursive mode --no-implicit-dirs do not send implicit directories (parents of the file being sent) The --source-list option allows you to supply an explicit list of filenames to transport without using the --recursive feature and without playing around with include and exclude files. As discussed below, the same thing can be done by combining --recursive with --include-from and --exclude, but it's significantly slower and more arcane to do it that way. The --null flag allows you to handle files with embedded linefeeds. This is in the style of gnu find's -print0 operator. The --send-dirs overcomes a problem where rsync refuses to send directories unless it's in recursive mode. One needs this to make sure that even empty directories get mirrored. And the --no-implicit-dirs option turns off the default behavior in which all the parent directories of a file are transmitted before sending the file. That default behavior is very inefficient in my scenario where I am taking the responsibility for sending those directories myself. And now for a performance test: I have a directory tree containing 128219 files of which 16064 are directories. To start the test, I made a list of files that had changed in the past day: find . -mtime -1 -print > /tmp/changed (normally, my list of candidate files is generated by some other means, this is just a test example). There were 5059 entries in /tmp/changed. I used my new options to sync up these files to another host as follows: time rsync -RlHptgoD --numeric-ids --source-list \ --send-dirs --no-implicit-dirs -xz --stats /dev/stdin \ remotehost:/extra_disk/tmp/tree1 < /tmp/changed Here were the reported statistics: Number of files: 5059 Number of files transferred: 5056 Total file size: 355514100 bytes Total transferred file size: 355514100 bytes Literal data: 355514100 bytes Matched data: 0 bytes File list size: 139687 Total bytes written: 154858363 Total bytes read: 80916 wrote 154858363 bytes read 80916 bytes 364992.41 bytes/sec total size is 355514100 speedup is 2.29 And the time statistics: 112.53u 8.82s 7:03.92 28.6% I then ran the same command again (in which case there was nothing to transfer). Here's how long it took: 0.54u 0.62s 0:08.61 13.4% Now to compare with the recursive method using --include-from. First, we must create the list of files. In the case of include-from, we need to include all the parent directories as include patterns. The following gawk seems to do the job: gawk '$0 != "./" {sub(/^\.\//,"")} {while ((length > 0) && !($0 in already)) {print "/"$0; already[$0] = 1; sub(/\/[^\/]*$/,"")}}' /tmp/changed > /tmp/includes This creates a file containing 11464 lines (the original 5059 files plus all the parent directories). Now we can run the normal rsync command: time rsync -RlHptgoD --numeric-ids -r -xz \ --include-from=/dev/stdin --exclude '*' --stats . \ remotehost:/extra_disk/tmp/tree2 < /tmp/includes Here were the reported statistics: Number of files: 11464 Number of files transferred: 5056 Total file size: 355520625 bytes Total transferred file size: 355520625 bytes Literal data: 355520625 bytes Matched data: 0 bytes File list size: 250242 Total bytes written: 154970663 Total bytes read: 80916 wrote 154970663 bytes read 80916 bytes 222935.41 bytes/sec total size is 355520625 speedup is 2.29 And the time statistics: 218.45u 12.92s 11:34.48 33.3% I then repeated the command to see how long it takes when there's nothing to transfer: 106.77u 6.72s 3:12.42 58.9% I then did a recursive diff on the trees to make sure they were identical, and they were. As you can see, the --source-list feature results in a significant savings in both CPU time on the sending host and real time to complete the transfer. It should also result in some CPU time savings on the receive host since there is no need to transfer all the directories that we were forced to add to get the --include-from syntax to work properly. I think that should cover everything, let me know if there are any questions. Cheers, Andy
--- flist.c.orig Thu Mar 14 16:20:20 2002 +++ flist.c Fri Jan 10 11:10:58 2003 @@ -41,6 +41,7 @@ extern int cvs_exclude; extern int recurse; +extern int send_dirs; extern int one_file_system; extern int make_backups; @@ -662,8 +663,8 @@ if (noexcludes) goto skip_excludes; - if (S_ISDIR(st.st_mode) && !recurse) { - rprintf(FINFO, "skipping directory %s\n", fname); + if (S_ISDIR(st.st_mode) && !recurse && !send_dirs) { + rprintf(FINFO, "make_file: skipping directory %s\n", fname); return NULL; } @@ -856,14 +857,16 @@ * I *think* f==-1 means that the list should just be built in memory * and not transmitted. But who can tell? -- mbp */ -struct file_list *send_file_list(int f, int argc, char *argv[]) +static struct file_list *send_file_list_proc(int f, char *(*ffunc)(), void *opq) { - int i, l; + int l; STRUCT_STAT st; char *p, *dir, *olddir; char lastpath[MAXPATHLEN] = ""; struct file_list *flist; int64 start_write; + char *in_fn; + extern int implicit_dirs; if (show_filelist_p() && f != -1) start_filelist_progress("building file list"); @@ -876,10 +879,10 @@ io_start_buffering(f); } - for (i = 0; i < argc; i++) { + while ((in_fn = (*ffunc)(opq)) != NULL) { char *fname = topsrcname; - strlcpy(fname, argv[i], MAXPATHLEN); + strlcpy(fname, in_fn, MAXPATHLEN); l = strlen(fname); if (l != 1 && fname[l - 1] == '/') { @@ -904,8 +907,8 @@ continue; } - if (S_ISDIR(st.st_mode) && !recurse) { - rprintf(FINFO, "skipping directory %s\n", fname); + if (S_ISDIR(st.st_mode) && !recurse && !send_dirs) { + rprintf(FINFO, "send_file_list: skipping directory %s\n", +fname); continue; } @@ -922,7 +925,7 @@ dir = fname; fname = p + 1; } - } else if (f != -1 && (p = strrchr(fname, '/'))) { + } else if (f != -1 && (p=strrchr(fname,'/')) && implicit_dirs) { /* this ensures we send the intermediate directories, thus getting their permissions right */ *p = 0; @@ -1020,6 +1023,49 @@ return flist; } +struct argv_data { + int argc; + char **argv; +}; + +static char * +get_arg(struct argv_data *ad) +{ + return (ad->argc-- > 0) ? *(ad->argv++) : NULL; +} + +struct file_list *send_file_list(int f, int argc, char *argv[]) +{ + struct argv_data arg_info; + + arg_info.argc = argc; + arg_info.argv = argv; + return send_file_list_proc(f,get_arg,&arg_info); +} + +/* note that send_file_list_proc silently truncates the filename to fit + in a buffer of MAXPATHLEN characters, so we can safely truncate there */ +static char * +get_stdio(FILE *fp) +{ + static char fnbuf[MAXPATHLEN]; + char *s = fnbuf; + char *eob = &fnbuf[sizeof(fnbuf)-1]; + int cc; + extern int list_rs; + + while (((cc = getc(fp)) != list_rs) && (cc != EOF)) { + if (s < eob) + *(s++) = cc; + } + *s = '\0'; + return ((cc == EOF) && (s == fnbuf)) ? NULL : fnbuf; +} + +struct file_list *send_file_list_fp(int f,FILE *fp) +{ + return send_file_list_proc(f,get_stdio,fp); +} struct file_list *recv_file_list(int f) { --- main.c.orig Wed Mar 27 00:10:44 2002 +++ main.c Fri Jan 10 10:59:51 2003 @@ -25,6 +25,8 @@ struct stats stats; +static FILE *src_list_fp; + extern int verbose; static void show_malloc_stats(void); @@ -559,7 +561,9 @@ if (delete_mode && !delete_excluded) send_exclude_list(f_out); if (!read_batch) /* dw -- don't write to pipe */ - flist = send_file_list(f_out,argc,argv); + flist = (src_list_fp ? + send_file_list_fp(f_out,src_list_fp) : + send_file_list(f_out,argc,argv)); if (verbose > 3) rprintf(FINFO,"file list sent\n"); @@ -836,6 +840,7 @@ extern int dry_run; extern int am_daemon; extern int am_server; + extern int source_list; int ret; extern int write_batch; int orig_argc; @@ -872,6 +877,14 @@ /* FIXME: We ought to call the same error-handling * code here, rather than relying on getopt. */ option_error(); + exit_cleanup(RERR_SYNTAX); + } + + if (source_list && + ((argc != 2) || + !(src_list_fp = (strcmp(argv[0],"/dev/stdin") ? + fopen(argv[0],"r") : stdin)))) { + usage(FERROR); exit_cleanup(RERR_SYNTAX); } --- options.c.orig Tue Mar 19 15:16:42 2002 +++ options.c Fri Jan 10 11:01:10 2003 @@ -66,6 +66,10 @@ int module_id = -1; int am_server = 0; int am_sender = 0; +int source_list=0; +int list_rs='\n'; +int send_dirs=0; +int implicit_dirs=1; int recurse = 0; int am_daemon=0; int do_stats=0; @@ -266,6 +270,10 @@ rprintf(F," --bwlimit=KBPS limit I/O bandwidth, KBytes per second\n"); rprintf(F," --write-batch=PREFIX write batch fileset starting with PREFIX\n"); rprintf(F," --read-batch=PREFIX read batch fileset starting with PREFIX\n"); + rprintf(F," --source-list SRC arg will be a (local) file name +containing a list of files, or /dev/stdin\n"); + rprintf(F," --null used with --source-list to indicate that +the file names will be separated by null (zero) bytes instead of linefeed characters; +useful with gfind -print0\n"); + rprintf(F," --send-dirs send directory entries even though not in +recursive mode\n"); + rprintf(F," --no-implicit-dirs do not send implicit directories (parents +of the file being sent)\n"); rprintf(F," -h, --help show this help screen\n"); #ifdef INET6 rprintf(F," -4 prefer IPv4\n"); @@ -287,7 +295,8 @@ OPT_DELETE_AFTER, OPT_EXISTING, OPT_MAX_DELETE, OPT_BACKUP_DIR, OPT_IGNORE_ERRORS, OPT_BWLIMIT, OPT_BLOCKING_IO, OPT_NO_BLOCKING_IO, OPT_WHOLE_FILE, OPT_NO_WHOLE_FILE, - OPT_MODIFY_WINDOW, OPT_READ_BATCH, OPT_WRITE_BATCH, OPT_IGNORE_EXISTING}; + OPT_MODIFY_WINDOW, OPT_READ_BATCH, OPT_WRITE_BATCH, OPT_IGNORE_EXISTING, + OPT_SOURCE_LIST, OPT_NULL, OPT_SEND_DIRS, OPT_NO_IMPLICIT_DIRS}; static struct poptOption long_options[] = { /* longName, shortName, argInfo, argPtr, value, descrip, argDesc */ @@ -361,6 +370,10 @@ {"hard-links", 'H', POPT_ARG_NONE, &preserve_hard_links , 0, 0, 0 }, {"read-batch", 0, POPT_ARG_STRING, &batch_prefix, OPT_READ_BATCH, 0, 0 }, {"write-batch", 0, POPT_ARG_STRING, &batch_prefix, OPT_WRITE_BATCH, 0, 0 }, + {"source-list", 0, POPT_ARG_NONE, &source_list, 0, 0, 0 }, + {"null", 0, POPT_ARG_NONE, 0, OPT_NULL, 0, 0}, + {"send-dirs", 0, POPT_ARG_NONE, &send_dirs, 0, 0, 0 }, + {"no-implicit-dirs", 0, POPT_ARG_NONE, 0, OPT_NO_IMPLICIT_DIRS, 0, +0}, #ifdef INET6 {0, '4', POPT_ARG_VAL, &default_af_hint, AF_INET , 0, 0 }, {0, '6', POPT_ARG_VAL, &default_af_hint, AF_INET6 , 0, 0 }, @@ -561,6 +574,14 @@ case OPT_READ_BATCH: /* popt stores the filename in batch_prefix for us */ read_batch = 1; + break; + + case OPT_NULL: + list_rs = '\0'; + break; + + case OPT_NO_IMPLICIT_DIRS: + implicit_dirs = 0; break; default: --- proto.h.orig Sun Mar 24 22:51:17 2002 +++ proto.h Thu Jan 9 14:02:18 2003 @@ -82,6 +82,7 @@ void send_file_name(int f, struct file_list *flist, char *fname, int recursive, unsigned base_flags); struct file_list *send_file_list(int f, int argc, char *argv[]); +struct file_list *send_file_list_fp(int f,FILE *fp); struct file_list *recv_file_list(int f); int file_compare(struct file_struct **f1, struct file_struct **f2); int flist_find(struct file_list *flist, struct file_struct *f);