It does normally take some time to analyze large trees of files. It has to call stat() on each file to get the size and timestamp.
However, 15 hours seems a bit excessive even though I have never tried
to do this on Windows or a NAS system. Just to be clear, is your
--link-dest parameter a single directory or are you trying to tell it to
use all of the previous backups?
Also, have you deleted a backup yet? In my experience that takes a lot
longer than running one so if you need 15 hours to run a backup I would
expect deleting one to take about a week.
On 1/3/19 4:23 AM, John Simpson via rsync wrote:
>
>
> I've been running rsync as a cygwin task on Windows Server 2008 for about two
> months now. I'm using the --link-dest option to do a daily 'snapshot' of the
> contents of a server containing about 10TB of data, about 13 million files,
> to a Linux based NAS server. Things started out great but I soon noticed that
> the time take to complete was slowly incrementing. It started at around three
> hours, but is now around fifteen.
>
> The command is as follows...
>
> rsync -rlptDhPR \
> --password-file=password \
> --Chmod=Du=rwx,Dgo=rx,Fu=rw,Fgo=r \
> --Stats \
> --delete \
> --log-file=logfilename \
> --link-dest=linkdestpath \
> sourceDirectory \
> rsync@192.168.1.2::destinationDirectory
>
> I'm not using the full -a option as the differences between the Windows and
> Linux ownership stuff messed things up.
>
> The first log file looked like this...
>
> 2018/10/01 23:00:14 [2164] building file list
> ...transfer file list here
> 2018/10/02 02:11:30 [2164] Number of files: 13,759,998 (reg: 12,260,176, dir:
> 1,499,821, link: 1)
> 2018/10/02 02:11:30 [2164] Number of created files: 302 (reg: 291, dir: 11)
> 2018/10/02 02:11:30 [2164] Number of regular files transferred: 310
> 2018/10/02 02:11:30 [2164] Total file size: 10.40T bytes
> 2018/10/02 02:11:30 [2164] Total transferred file size: 664.31K bytes
> 2018/10/02 02:11:30 [2164] Literal data: 277.91K bytes
> 2018/10/02 02:11:30 [2164] Matched data: 386.40K bytes
> 2018/10/02 02:11:30 [2164] File list size: 10.42M
> 2018/10/02 02:11:30 [2164] File list generation time: 0.154 seconds
> 2018/10/02 02:11:30 [2164] File list transfer time: 0.000 seconds
> 2018/10/02 02:11:30 [2164] Total bytes sent: 235.68M
> 2018/10/02 02:11:30 [2164] Total bytes received: 7.51M
> 2018/10/02 02:11:30 [2164] sent 235.68M bytes received 7.51M bytes 21.17K
> bytes/sec
> 2018/10/02 02:11:30 [2164] total size is 10.40T speedup is 42,753.79
>
> the most recent looks like this...
>
> 2018/11/24 23:00:15 [2924] building file list
> 2018/11/24 23:00:17 [2924] cd..t...... /cygdrive/
> 2018/11/25 13:21:16 [2924] Number of files: 13,776,423 (reg: 12,274,642, dir:
> 1,501,780, link: 1)
> 2018/11/25 13:21:16 [2924] Number of created files: 0
> 2018/11/25 13:21:16 [2924] Number of regular files transferred: 0
> 2018/11/25 13:21:16 [2924] Total file size: 10.49T bytes
> 2018/11/25 13:21:16 [2924] Total transferred file size: 0 bytes
> 2018/11/25 13:21:16 [2924] Literal data: 0 bytes
> 2018/11/25 13:21:16 [2924] Matched data: 0 bytes
> 2018/11/25 13:21:16 [2924] File list size: 10.35M
> 2018/11/25 13:21:16 [2924] File list generation time: 0.316 seconds
> 2018/11/25 13:21:16 [2924] File list transfer time: 0.000 seconds
> 2018/11/25 13:21:16 [2924] Total bytes sent: 236.55M
> 2018/11/25 13:21:16 [2924] Total bytes received: 7.51M
> 2018/11/25 13:21:16 [2924] sent 236.55M bytes received 7.51M bytes 4.72K
> bytes/sec
> 2018/11/25 13:21:16 [2924] total size is 10.49T speedup is 42,996.96
>
> As you can see the start time is 11:00PM (23:00) in both cases. The first log
> shows that identifying the files to transfer took about three hours (I've
> omitted the file list - it's quite long), the second log takes fourteen hours
> to do the same job (in this case this was done at the weekend and I've
> include the whole log file which correctly identifies that no files have
> changed)
>
> The number of files is as might be expected as is everything else. It's just
> the time taken "building file list" is significantly larger.
>
>
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ke...@futurequest.net (work)
Orlando, Florida k...@sanitarium.net (personal)
Web page: https://sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
signature.asc
Description: OpenPGP digital signature
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
