Hello,
I've been using rsync on some backup servers for years. In 2011 we had a situation where the FS of the backup server was
behaving strange, even thou there was enough available I/O, the fs(ext4 on 16TB partition with a lot of inodes) was
lagging. After much testing we found that rsync was hammering the fs too hard.
At that point I patched rsync to add a stupid option which will sleep for a
while before creating new file.
That helped us a lot to keep the same rate of concurrent rsyncs to the same
machine.
I would love your comments on this, generally stupid approach to the problem :)
Best regards,
Marian
>From 4c6513798c997342dc79da5531cac6e9348b5bc3 Mon Sep 17 00:00:00 2001
From: Marian Marinov <m...@yuhu.biz>
Date: Thu, 3 Apr 2014 08:23:53 +0300
Subject: [PATCH 1/2] options.c: Add the --slow-down option This option is used
to slowdown the generation of files It is needed in situation where creating
files too fast will take too much of the I/O.
---
options.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/options.c b/options.c
index dc9e62a..79d5356 100644
--- a/options.c
+++ b/options.c
@@ -123,6 +123,7 @@ int checksum_seed = 0;
int inplace = 0;
int delay_updates = 0;
long block_size = 0; /* "long" because popt can't set an int32. */
+unsigned long sleep_asec;
char *skip_compress = NULL;
item_list dparam_list = EMPTY_ITEM_LIST;
@@ -791,6 +792,7 @@ void usage(enum logcode F)
rprintf(F," --password-file=FILE read daemon-access password from FILE\n");
rprintf(F," --list-only list the files instead of copying them\n");
rprintf(F," --bwlimit=RATE limit socket I/O bandwidth\n");
+ rprintf(F," --slow-down=USECs sleep for N microseconds between directory reads\n");
#ifdef HAVE_SETVBUF
rprintf(F," --outbuf=N|L|B set output buffering to None, Line, or Block\n");
#endif
@@ -987,6 +989,7 @@ static struct poptOption long_options[] = {
{"itemize-changes", 'i', POPT_ARG_NONE, 0, 'i', 0, 0 },
{"no-itemize-changes",0, POPT_ARG_VAL, &itemize_changes, 0, 0, 0 },
{"no-i", 0, POPT_ARG_VAL, &itemize_changes, 0, 0, 0 },
+ {"slow-down", 0, POPT_ARG_LONG, &sleep_asec, 0, 0, 0 },
{"bwlimit", 0, POPT_ARG_STRING, &bwlimit_arg, OPT_BWLIMIT, 0, 0 },
{"no-bwlimit", 0, POPT_ARG_VAL, &bwlimit, 0, 0, 0 },
{"backup", 'b', POPT_ARG_VAL, &make_backups, 1, 0, 0 },
--
1.8.4
>From 664afccb75f065946b8bbf306f4ff6acd8bab401 Mon Sep 17 00:00:00 2001
From: Marian Marinov <m...@yuhu.biz>
Date: Thu, 3 Apr 2014 08:29:07 +0300
Subject: [PATCH 2/2] flist.c: Implementing the slow-down option I have put the
sleep into send_directory just before calling send_file_name. So if the
slow-down option is used, rsync will sleep for some microseconds before
creating a new file.
---
flist.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/flist.c b/flist.c
index a0f05dd..4fd6821 100644
--- a/flist.c
+++ b/flist.c
@@ -69,6 +69,7 @@ extern int sender_symlink_iconv;
extern int output_needs_newline;
extern int sender_keeps_checksum;
extern int unsort_ndx;
+extern unsigned long sleep_asec;
extern uid_t our_uid;
extern struct stats stats;
extern char *filesfrom_host;
@@ -1751,6 +1752,10 @@ static void send_directory(int f, struct file_list *flist, char *fbuf, int len,
continue;
}
+ // Sleep for a bit, to avoid hammering the disk.
+ if (sleep_asec)
+ usleep(sleep_asec);
+
send_file_name(f, flist, fbuf, NULL, flags, filter_level);
}
--
1.8.4
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
