--link-dest. Time to 'building file list' incrementing
I've been running rsync as a cygwin task on Windows Server 2008 for about two months now. I'm using the --link-dest option to do a daily 'snapshot' of the contents of a server containing about 10TB of data, about 13 million files, to a Linux based NAS server. Things started out great but I soon noticed that the time take to complete was slowly incrementing. It started at around three hours, but is now around fifteen. The command is as follows... rsync -rlptDhPR \ --password-file=password \ --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \ --Stats \ --delete \ --log-file=logfilename \ --link-dest=linkdestpath \ sourceDirectory \ rsync@192.168.1.2::destinationDirectory I'm not using the full -a option as the differences between the Windows and Linux ownership stuff messed things up. The first log file looked like this... 2018/10/01 23:00:14 [2164] building file list ...transfer file list here 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, dir: 1,499,821, link: 1) 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11) 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes 2018/10/02 02:11:30 [2164] File list size: 10.42M 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M 2018/10/02 02:11:30 [2164] sent 235.68M bytes received 7.51M bytes 21.17K bytes/sec 2018/10/02 02:11:30 [2164] total size is 10.40T speedup is 42,753.79 the most recent looks like this... 2018/11/24 23:00:15 [2924] building file list 2018/11/24 23:00:17 [2924] cd..t.. /cygdrive/ 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, dir: 1,501,780, link: 1) 2018/11/25 13:21:16 [2924] Number of created files: 0 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes 2018/11/25 13:21:16 [2924] Literal data: 0 bytes 2018/11/25 13:21:16 [2924] Matched data: 0 bytes 2018/11/25 13:21:16 [2924] File list size: 10.35M 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M 2018/11/25 13:21:16 [2924] sent 236.55M bytes received 7.51M bytes 4.72K bytes/sec 2018/11/25 13:21:16 [2924] total size is 10.49T speedup is 42,996.96 As you can see the start time is 11:00PM (23:00) in both cases. The first log shows that identifying the files to transfer took about three hours (I've omitted the file list - it's quite long), the second log takes fourteen hours to do the same job (in this case this was done at the weekend and I've include the whole log file which correctly identifies that no files have changed) The number of files is as might be expected as is everything else. It's just the time taken "building file list" is significantly larger. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --link-dest. Time to 'building file list' incrementing
Kevin The link-dest parameter is a single directory (the previous day's directory), the destination is today's directory. I haven't tried deleting a backup, there's no particular need in space terms, at the current rate there's enough space for several years of daily backups. I've reverted to daily backups on a small subset of the total; the full backup now takes around 30 hours. Clearly not practical. As the small subset takes only a few minutes to complete I can't yet see if this time is incrementing too. John On 3 Jan 2019 17:06, Kevin Korb via rsync wrote: > > It does normally take some time to analyze large trees of files. It has > to call stat() on each file to get the size and timestamp. > > However, 15 hours seems a bit excessive even though I have never tried > to do this on Windows or a NAS system. Just to be clear, is your > --link-dest parameter a single directory or are you trying to tell it to > use all of the previous backups? > > Also, have you deleted a backup yet? In my experience that takes a lot > longer than running one so if you need 15 hours to run a backup I would > expect deleting one to take about a week. > > On 1/3/19 4:23 AM, John Simpson via rsync wrote: > > > > > > I've been running rsync as a cygwin task on Windows Server 2008 for about > > two months now. I'm using the --link-dest option to do a daily 'snapshot' > > of the contents of a server containing about 10TB of data, about 13 million > > files, to a Linux based NAS server. Things started out great but I soon > > noticed that the time take to complete was slowly incrementing. It started > > at around three hours, but is now around fifteen. > > > > The command is as follows... > > > > rsync -rlptDhPR \ > > --password-file=password \ > > --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \ > > --Stats \ > > --delete \ > > --log-file=logfilename \ > > --link-dest=linkdestpath \ > > sourceDirectory \ > > rsync@192.168.1.2::destinationDirectory > > > > I'm not using the full -a option as the differences between the Windows and > > Linux ownership stuff messed things up. > > > > The first log file looked like this... > > > > 2018/10/01 23:00:14 [2164] building file list > > ...transfer file list here > > 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, > > dir: 1,499,821, link: 1) > > 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11) > > 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310 > > 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes > > 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes > > 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes > > 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes > > 2018/10/02 02:11:30 [2164] File list size: 10.42M > > 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds > > 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds > > 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M > > 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M > > 2018/10/02 02:11:30 [2164] sent 235.68M bytes received 7.51M bytes 21.17K > > bytes/sec > > 2018/10/02 02:11:30 [2164] total size is 10.40T speedup is 42,753.79 > > > > the most recent looks like this... > > > > 2018/11/24 23:00:15 [2924] building file list > > 2018/11/24 23:00:17 [2924] cd..t.. /cygdrive/ > > 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, > > dir: 1,501,780, link: 1) > > 2018/11/25 13:21:16 [2924] Number of created files: 0 > > 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0 > > 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes > > 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes > > 2018/11/25 13:21:16 [2924] Literal data: 0 bytes > > 2018/11/25 13:21:16 [2924] Matched data: 0 bytes > > 2018/11/25 13:21:16 [2924] File list size: 10.35M > > 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds > > 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds > > 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M > > 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M > > 2018/11/25 13:21:16 [2924] sent 236.55M bytes received 7.51M bytes 4.72K > > bytes/sec > > 2018/11/25 13:21:16 [2924] total size is 10.49T speedup is 42,996.96 > > > > As you can see the start time is 11:00PM (23:00) in bot
Re: --link-dest. Time to 'building file list' incrementing
Any ideas anyone? I still need at least a weekly backup of all data. The current workaround is just for the most active directories. Are there any diagnostics I can do which might shed some light on this? Thanks JohnOn 4 Jan 2019 09:53, John Simpson via rsync wrote: > > Kevin > > The link-dest parameter is a single directory (the previous day's directory), > the destination is today's directory. > > I haven't tried deleting a backup, there's no particular need in space > terms, at the current rate there's enough space for several years of daily > backups. > > I've reverted to daily backups on a small subset of the total; the full > backup now takes around 30 hours. Clearly not practical. > > As the small subset takes only a few minutes to complete I can't yet see if > this time is incrementing too. > > John On 3 Jan 2019 17:06, Kevin Korb via rsync > wrote: > > > > It does normally take some time to analyze large trees of files. It has > > to call stat() on each file to get the size and timestamp. > > > > However, 15 hours seems a bit excessive even though I have never tried > > to do this on Windows or a NAS system. Just to be clear, is your > > --link-dest parameter a single directory or are you trying to tell it to > > use all of the previous backups? > > > > Also, have you deleted a backup yet? In my experience that takes a lot > > longer than running one so if you need 15 hours to run a backup I would > > expect deleting one to take about a week. > > > > On 1/3/19 4:23 AM, John Simpson via rsync wrote: > > > > > > > > > I've been running rsync as a cygwin task on Windows Server 2008 for about > > > two months now. I'm using the --link-dest option to do a daily 'snapshot' > > > of the contents of a server containing about 10TB of data, about 13 > > > million files, to a Linux based NAS server. Things started out great but > > > I soon noticed that the time take to complete was slowly incrementing. It > > > started at around three hours, but is now around fifteen. > > > > > > The command is as follows... > > > > > > rsync -rlptDhPR \ > > > --password-file=password \ > > > --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \ > > > --Stats \ > > > --delete \ > > > --log-file=logfilename \ > > > --link-dest=linkdestpath \ > > > sourceDirectory \ > > > rsync@192.168.1.2::destinationDirectory > > > > > > I'm not using the full -a option as the differences between the Windows > > > and Linux ownership stuff messed things up. > > > > > > The first log file looked like this... > > > > > > 2018/10/01 23:00:14 [2164] building file list > > > ...transfer file list here > > > 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, > > > dir: 1,499,821, link: 1) > > > 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: > > > 11) > > > 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310 > > > 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes > > > 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes > > > 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes > > > 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes > > > 2018/10/02 02:11:30 [2164] File list size: 10.42M > > > 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds > > > 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds > > > 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M > > > 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M > > > 2018/10/02 02:11:30 [2164] sent 235.68M bytes received 7.51M bytes > > > 21.17K bytes/sec > > > 2018/10/02 02:11:30 [2164] total size is 10.40T speedup is 42,753.79 > > > > > > the most recent looks like this... > > > > > > 2018/11/24 23:00:15 [2924] building file list > > > 2018/11/24 23:00:17 [2924] cd..t.. /cygdrive/ > > > 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, > > > dir: 1,501,780, link: 1) > > > 2018/11/25 13:21:16 [2924] Number of created files: 0 > > > 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0 > > > 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes > > > 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes > > > 2018/1
Re: --link-dest. Time to 'building file list' incrementing
Thanks. I'll give this a go and report back. On 8 Jan 2019 10:03, Ben RUBSON wrote:Hi,As you are on Cygwin, you should consider the notexec & noacl mount options :https://cygwin.com/cygwin-ug-net/using.html#mount-tableThey impact stat() performance.BenOn 8 Jan 2019, at 10:56, John Simpson via rsync <rsync@lists.samba.org> wrote:Any ideas anyone?I still need at least a weekly backup of all data.The current workaround is just for the most active directories.Are there any diagnostics I can do which might shed some light on this?ThanksJohnOn 4 Jan 2019 09:53, John Simpson via rsync <rsync@lists.samba.org> wrote:KevinThe link-dest parameter is a single directory (the previous day's directory), the destination is today's directory.I haven't tried deleting a backup, there's no particular need in space terms, at the current rate there's enough space for several years of daily backups.I've reverted to daily backups on a small subset of the total; the full backup now takes around 30 hours. Clearly not practical.As the small subset takes only a few minutes to complete I can't yet see if this time is incrementing too.John On 3 Jan 2019 17:06, Kevin Korb via rsync <rsync@lists.samba.org> wrote:It does normally take some time to analyze large trees of files. It has to call stat() on each file to get the size and timestamp. However, 15 hours seems a bit excessive even though I have never tried to do this on Windows or a NAS system. Just to be clear, is your --link-dest parameter a single directory or are you trying to tell it to use all of the previous backups? Also, have you deleted a backup yet? In my experience that takes a lot longer than running one so if you need 15 hours to run a backup I would expect deleting one to take about a week. On 1/3/19 4:23 AM, John Simpson via rsync wrote: I've been running rsync as a cygwin task on Windows Server 2008 for about two months now. I'm using the --link-dest option to do a daily 'snapshot' of the contents of a server containing about 10TB of data, about 13 million files, to a Linux based NAS server. Things started out great but I soon noticed that the time take to complete was slowly incrementing. It started at around three hours, but is now around fifteen. The command is as follows... rsync -rlptDhPR \ --password-file=password \ --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \ --Stats \ --delete \ --log-file=logfilename \ --link-dest=linkdestpath \ sourceDirectory \ rsync@192.168.1.2::destinationDirectory I'm not using the full -a option as the differences between the Windows and Linux ownership stuff messed things up. The first log file looked like this... 2018/10/01 23:00:14 [2164] building file list ...transfer file list here 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, dir: 1,499,821, link: 1) 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11) 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes 2018/10/02 02:11:30 [2164] File list size: 10.42M 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M 2018/10/02 02:11:30 [2164] sent 235.68M bytes received 7.51M bytes 21.17K bytes/sec 2018/10/02 02:11:30 [2164] total size is 10.40T speedup is 42,753.79 the most recent looks like this... 2018/11/24 23:00:15 [2924] building file list 2018/11/24 23:00:17 [2924] cd..t.. /cygdrive/ 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, dir: 1,501,780, link: 1) 2018/11/25 13:21:16 [2924] Number of created files: 0 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes 2018/11/25 13:21:16 [2924] Literal data: 0 bytes 2018/11/25 13:21:16 [2924] Matched data: 0 bytes 2018/11/25 13:21:16 [2924] File list size: 10.35M 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M 2018/11/25 13:21:16 [2924] sent 236.55M bytes received 7.51M bytes 4.72K bytes/sec 2018/11/25 13:21:16 [2924] total size is 10.49T speedup is 42,996.96 As you can see the start time is 11:00PM (23:00) in both cases. The first log shows that identifying the files to transfer took about three hours (I've omitted the file list - it's quite long), the second log ta
Re: --link-dest. Time to 'building file list' incrementing
Don't know how I missed this when I was trying to figure out the cheapest solution. I don't need the complexity but it seems to be worth investigating. ThanksOn 8 Jan 2019 16:15, Andrew McGlashan via rsync wrote: > > On 8/1/19 8:56 pm, John Simpson via rsync wrote: > > Any ideas anyone? > > How about using snapshots and doing the rsync off those? > > https://www.thewindowsclub.com/vss-volume-shadow-copy-service > > https://blogs.technet.microsoft.com/josebda/2007/10/10/the-basics-of-the-volume-shadow-copy-service-vss/ > > Cheers > A. > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html