Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
On 22/10/2023 09:50, Dale wrote: I've read about using rsync as a daemon. It is supposed to be faster. For some reason, I just never got around to trying it. It would likely be a good idea to work on that. It would likely be really handy when moving large amounts of data. I may try to read a howto on the Gentoo wiki or something. See what all it involves. Run the daemon on the NAS. For each file where the metadata differs, the NAS will send a bunch of block checksums to your rsync command (or the other way round). If all the checksums agree, then the files are identical, without having to send the file over the network. If only a few checksums differ, only a few blocks have to be sent. In other words, using a daemon will massively reduce the amount of data going across the network, at the expense of the NAS having to scan and checksum every "file of interest". A pretty cheap trade-off, imho. Cheers, Wol
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
Am Mon, Oct 23, 2023 at 02:29:26AM -0500 schrieb Dale: > Dale wrote: > > > > Second problem. The transfer speed is back to the old slower speed. > > I'm pretty sure I am using the same old options on both ends. Still, > > it's back to being slow again. Some info: > > > > > > <<< SNIP >>> > > Did I miss something? Typo maybe? I'm pretty sure I used copy and > > paste but still. > > > > Thanks. > > > > Dale > > > > :-) :-) > > > I been working on the speed problem again. I rebuilt the kernel on > fireball and I think some changes made a huge change. This is the > results from fireball now: > > > root@fireball / # cryptsetup benchmark > # Tests are approximate using memory only (no storage IO). > PBKDF2-sha1 931239 iterations per second for 256-bit key > PBKDF2-sha256 1356501 iterations per second for 256-bit key > PBKDF2-sha512 972705 iterations per second for 256-bit key > PBKDF2-ripemd160 648871 iterations per second for 256-bit key > PBKDF2-whirlpool 362077 iterations per second for 256-bit key > argon2i 5 iterations, 1048576 memory, 4 parallel threads (CPUs) > for 256-bit key (requested 2000 ms time) > argon2id 4 iterations, 1048576 memory, 4 parallel threads (CPUs) > for 256-bit key (requested 2000 ms time) > # Algorithm | Key | Encryption | Decryption > aes-cbc 128b 570.8 MiB/s 2045.6 MiB/s > serpent-cbc 128b 91.1 MiB/s 310.0 MiB/s > twofish-cbc 128b 198.7 MiB/s 218.9 MiB/s > aes-cbc 256b 428.8 MiB/s 1670.4 MiB/s > serpent-cbc 256b 91.6 MiB/s 309.5 MiB/s > twofish-cbc 256b 199.8 MiB/s 219.2 MiB/s > aes-xts 256b 1821.2 MiB/s 1767.1 MiB/s > serpent-xts 256b 265.9 MiB/s 270.2 MiB/s > twofish-xts 256b 201.0 MiB/s 204.2 MiB/s > aes-xts 512b 1440.0 MiB/s 1445.9 MiB/s > serpent-xts 512b 265.0 MiB/s 257.2 MiB/s > twofish-xts 512b 198.2 MiB/s 201.6 MiB/s > root@fireball / # There you go. Told ya. :) > As you can see, aes-cbc is fast now and I think that is what cryptsetup > uses. It used to be really slow I think. Cryptsetup uses aes-xts these days, I think it’s been mentioned in this thread somewhere. > Now on to the nas box. I've recompiled the kernel with some added > options. Still, it refuses to speed up. I kinda think it is the CPU > lacking support for encryption. I'm asking others just in case I'm > missing something. Also, fireball uses a older kernel, 5.14 or so. The > nas box uses 6.1 or so. The menus are different and that is why it is > hard to get them to match up. I may have missed something. Everything you need for that is in the crypto menu at the bottom. > This is the bench mark from nas box. > > nas ~ # cryptsetup benchmark > # Tests are approximate using memory only (no storage IO). > PBKDF2-sha1 700919 iterations per second for 256-bit key > PBKDF2-sha256 924670 iterations per second for 256-bit key > PBKDF2-sha512 729190 iterations per second for 256-bit key > PBKDF2-ripemd160 517559 iterations per second for 256-bit key > PBKDF2-whirlpool 359593 iterations per second for 256-bit key > argon2i 4 iterations, 1048576 memory, 4 parallel threads (CPUs) > for 256-bit key (requested 2000 ms time) > argon2id 4 iterations, 1048576 memory, 4 parallel threads (CPUs) > for 256-bit key (requested 2000 ms time) > # Algorithm | Key | Encryption | Decryption > aes-cbc 128b 63.6 MiB/s 41.6 MiB/s > serpent-cbc 128b 81.0 MiB/s 212.4 MiB/s > twofish-cbc 128b 192.5 MiB/s 222.1 MiB/s > aes-cbc 256b 47.5 MiB/s 30.0 MiB/s > serpent-cbc 256b 81.2 MiB/s 212.7 MiB/s > twofish-cbc 256b 192.3 MiB/s 221.9 MiB/s > aes-xts 256b 65.9 MiB/s 41.6 MiB/s > serpent-xts 256b 201.7 MiB/s 205.7 MiB/s > twofish-xts 256b 216.2 MiB/s 214.5 MiB/s > aes-xts 512b 48.8 MiB/s 30.0 MiB/s > serpent-xts 512b 202.7 MiB/s 205.6 MiB/s > twofish-xts 512b 216.4 MiB/s 214.0 MiB/s > nas ~ # > […] > The aes shows up on fireball. It does not on the nas box. Is the speed > above as good as I can expect with this older CPU? If not done yet, you can check whether you enabled the 64 bit versions of the crypto modules. They could push performance by a few more percent. -- Grüße | Greetings | Salut | Qapla’ Please do not share anything from, with or about me on any social network. The last chance is often the second-last, if you look close enough. signature.asc Description: PGP signature
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
On Monday, 23 October 2023 08:29:26 BST Dale wrote: > I been working on the speed problem again. I rebuilt the kernel on > fireball and I think some changes made a huge change. This is the > results from fireball now: > > > root@fireball / # cryptsetup benchmark > # Tests are approximate using memory only (no storage IO). > PBKDF2-sha1 931239 iterations per second for 256-bit key > PBKDF2-sha2561356501 iterations per second for 256-bit key > PBKDF2-sha512 972705 iterations per second for 256-bit key > PBKDF2-ripemd160 648871 iterations per second for 256-bit key > PBKDF2-whirlpool 362077 iterations per second for 256-bit key > argon2i 5 iterations, 1048576 memory, 4 parallel threads (CPUs) > for 256-bit key (requested 2000 ms time) > argon2id 4 iterations, 1048576 memory, 4 parallel threads (CPUs) > for 256-bit key (requested 2000 ms time) > # Algorithm | Key | Encryption | Decryption > aes-cbc128b 570.8 MiB/s 2045.6 MiB/s > serpent-cbc128b91.1 MiB/s 310.0 MiB/s > twofish-cbc128b 198.7 MiB/s 218.9 MiB/s > aes-cbc256b 428.8 MiB/s 1670.4 MiB/s > serpent-cbc256b91.6 MiB/s 309.5 MiB/s > twofish-cbc256b 199.8 MiB/s 219.2 MiB/s > aes-xts256b 1821.2 MiB/s 1767.1 MiB/s > serpent-xts256b 265.9 MiB/s 270.2 MiB/s > twofish-xts256b 201.0 MiB/s 204.2 MiB/s > aes-xts512b 1440.0 MiB/s 1445.9 MiB/s > serpent-xts512b 265.0 MiB/s 257.2 MiB/s > twofish-xts512b 198.2 MiB/s 201.6 MiB/s > root@fireball / # > > > As you can see, aes-cbc is fast now and I think that is what cryptsetup > uses. It used to be really slow I think. Yep, it was many times slower with your previous kernel. Now it will saturate your 1Gbps network link, as long as the other box can take it. > Now on to the nas box. I've recompiled the kernel with some added > options. Still, it refuses to speed up. I kinda think it is the CPU > lacking support for encryption. I'm asking others just in case I'm > missing something. No, you're not missing anything, although in a previous message you had posted a benchmark result for the Phenom performance being around 4 times higher than what you show below. As I mentioned then, the Phenom CPU does not have AES-NI crypto hardware acceleration. it is a rather old CPU. Assuming the MoBo has the latest OEM firmware and you have also added microcode in your kernel (CONFIG_MICROCODE_AMD=y), then probably that's all it can do. Small optimisations may show up between kernel releases, yet again patches to address CPU vulnerabilities (retpoline?) could make things a bit worse. > Also, fireball uses a older kernel, 5.14 or so. The > nas box uses 6.1 or so. The menus are different and that is why it is > hard to get them to match up. I may have missed something. This is the > bench mark from nas box. You can diff previous and current kernel config files to see what has changed. For the crypto settings you can grep for AES and for CRYPT to make sure you have not left out what you'd need for disk encryption. > nas ~ # cryptsetup benchmark > # Tests are approximate using memory only (no storage IO). > PBKDF2-sha1 700919 iterations per second for 256-bit key > PBKDF2-sha256 924670 iterations per second for 256-bit key > PBKDF2-sha512 729190 iterations per second for 256-bit key > PBKDF2-ripemd160 517559 iterations per second for 256-bit key > PBKDF2-whirlpool 359593 iterations per second for 256-bit key > argon2i 4 iterations, 1048576 memory, 4 parallel threads (CPUs) > for 256-bit key (requested 2000 ms time) > argon2id 4 iterations, 1048576 memory, 4 parallel threads (CPUs) > for 256-bit key (requested 2000 ms time) > # Algorithm | Key | Encryption | Decryption > aes-cbc128b63.6 MiB/s41.6 MiB/s > serpent-cbc128b81.0 MiB/s 212.4 MiB/s > twofish-cbc128b 192.5 MiB/s 222.1 MiB/s > aes-cbc256b47.5 MiB/s30.0 MiB/s > serpent-cbc256b81.2 MiB/s 212.7 MiB/s > twofish-cbc256b 192.3 MiB/s 221.9 MiB/s > aes-xts256b65.9 MiB/s41.6 MiB/s > serpent-xts256b 201.7 MiB/s 205.7 MiB/s > twofish-xts256b 216.2 MiB/s 214.5 MiB/s > aes-xts512b48.8 MiB/s30.0 MiB/s > serpent-xts512b 202.7 MiB/s 205.6 MiB/s > twofish-xts512b 216.4 MiB/s 214.0 MiB/s > nas ~ # How does your aes-xts 30.0 MiB/s shown above compare with your previous benchmark result? I'm sure it was quite higher than this. > I seem to recall it
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
Dale wrote: > > Second problem. The transfer speed is back to the old slower speed. > I'm pretty sure I am using the same old options on both ends. Still, > it's back to being slow again. Some info: > > > <<< SNIP >>> > Did I miss something? Typo maybe? I'm pretty sure I used copy and > paste but still. > > Thanks. > > Dale > > :-) :-) I been working on the speed problem again. I rebuilt the kernel on fireball and I think some changes made a huge change. This is the results from fireball now: root@fireball / # cryptsetup benchmark # Tests are approximate using memory only (no storage IO). PBKDF2-sha1 931239 iterations per second for 256-bit key PBKDF2-sha256 1356501 iterations per second for 256-bit key PBKDF2-sha512 972705 iterations per second for 256-bit key PBKDF2-ripemd160 648871 iterations per second for 256-bit key PBKDF2-whirlpool 362077 iterations per second for 256-bit key argon2i 5 iterations, 1048576 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time) argon2id 4 iterations, 1048576 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time) # Algorithm | Key | Encryption | Decryption aes-cbc 128b 570.8 MiB/s 2045.6 MiB/s serpent-cbc 128b 91.1 MiB/s 310.0 MiB/s twofish-cbc 128b 198.7 MiB/s 218.9 MiB/s aes-cbc 256b 428.8 MiB/s 1670.4 MiB/s serpent-cbc 256b 91.6 MiB/s 309.5 MiB/s twofish-cbc 256b 199.8 MiB/s 219.2 MiB/s aes-xts 256b 1821.2 MiB/s 1767.1 MiB/s serpent-xts 256b 265.9 MiB/s 270.2 MiB/s twofish-xts 256b 201.0 MiB/s 204.2 MiB/s aes-xts 512b 1440.0 MiB/s 1445.9 MiB/s serpent-xts 512b 265.0 MiB/s 257.2 MiB/s twofish-xts 512b 198.2 MiB/s 201.6 MiB/s root@fireball / # As you can see, aes-cbc is fast now and I think that is what cryptsetup uses. It used to be really slow I think. Now on to the nas box. I've recompiled the kernel with some added options. Still, it refuses to speed up. I kinda think it is the CPU lacking support for encryption. I'm asking others just in case I'm missing something. Also, fireball uses a older kernel, 5.14 or so. The nas box uses 6.1 or so. The menus are different and that is why it is hard to get them to match up. I may have missed something. This is the bench mark from nas box. nas ~ # cryptsetup benchmark # Tests are approximate using memory only (no storage IO). PBKDF2-sha1 700919 iterations per second for 256-bit key PBKDF2-sha256 924670 iterations per second for 256-bit key PBKDF2-sha512 729190 iterations per second for 256-bit key PBKDF2-ripemd160 517559 iterations per second for 256-bit key PBKDF2-whirlpool 359593 iterations per second for 256-bit key argon2i 4 iterations, 1048576 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time) argon2id 4 iterations, 1048576 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time) # Algorithm | Key | Encryption | Decryption aes-cbc 128b 63.6 MiB/s 41.6 MiB/s serpent-cbc 128b 81.0 MiB/s 212.4 MiB/s twofish-cbc 128b 192.5 MiB/s 222.1 MiB/s aes-cbc 256b 47.5 MiB/s 30.0 MiB/s serpent-cbc 256b 81.2 MiB/s 212.7 MiB/s twofish-cbc 256b 192.3 MiB/s 221.9 MiB/s aes-xts 256b 65.9 MiB/s 41.6 MiB/s serpent-xts 256b 201.7 MiB/s 205.7 MiB/s twofish-xts 256b 216.2 MiB/s 214.5 MiB/s aes-xts 512b 48.8 MiB/s 30.0 MiB/s serpent-xts 512b 202.7 MiB/s 205.6 MiB/s twofish-xts 512b 216.4 MiB/s 214.0 MiB/s nas ~ # I seem to recall it being said that the old CPU in the nas box lacks the aes instruction set. This is a list of the CPU flags from the nas box. nas ~ # cpuid2cpuflags CPU_FLAGS_X86: 3dnow 3dnowext mmx mmxext popcnt sse sse2 sse3 sse4a nas ~ # The aes shows up on fireball. It does not on the nas box. Is the speed above as good as I can expect with this older CPU? I can include the kernel config if needed. If you know what driver you are looking for, let me know what to grep for. If not sure, I can attach the config file. Is this it? Is this as fast as this old CPU can get? Dale :-) :-)
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
William Kenworthy wrote: > > On 22/10/23 11:23, Dale wrote: >> Frank Steinmetzger wrote: >>> Am Fri, Oct 20, 2023 at 09:20:45PM -0500 schrieb Dale: Howdy, As most know, I had to restore from backups recently. I also reworked my NAS box. I'm doing my first backup given that I have more files that need to be added to the backups. When I started the rsync, it's starting from the first file and updating each file as it goes as if all of them changed. Given that likely 95% of the files hasn't changed, I figure this is being done because of a time stamp or something. Is there a way to tell rsync to ignore the time stamp or something or if the files are the same size, just update the time stamp? Is there a way to just update the time stamps on the NAS box? Is there a option I haven't thought of to work around this? This is the old command I was using to create the backups. time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series /mnt/TV_Backup/ >>> This didn’t preserve timestamps. Hence there is one type of >>> information lost >>> from which rsync knows whether two files may be identical. So now your >>> restore has more recent timestamps the the backup. If you use -u, Rsync >>> should skip all files. >>> >>> My perfectionist self doesn’t like discarding timestamp information, >>> because >>> then my system can’t tell me how old some file is, and how old (or >>> young) I >>> was when I created it and so on. I once didn’t pay enough attention >>> when >>> restoring a backup back when I was still on Windows, which is why I >>> don’t >>> have many files left that are dated before April 2007, even though >>> they are >>> from 2000 ± x. >>> >>> BTW: -i and -v are redundant. -v will only print the file path, >>> whereas -i >>> does the same and adds the reasons colum at the beginning. >>> I tried these to try to get around it. time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series /mnt/TV_Backup/ >>> -a and -r are also redundant, as -a includes -r. >>> I looked at the man page and the options there. I don't see anything that I think will help. Is there a way around this? >>> My muscle memory uses `rsync -ai` for almost everything. And when I >>> do full >>> root file systems or stuff where I know I will need them, I use -axAHX >>> instead. Since this preserves all the usual data, I’ve never really had >>> rsync wanting to do everything all over. >>> >> Well, I can't turn back the clocks so it is what it is now. These files >> tho, I really don't worry to much about the timestamps. If I were >> backing up my OS tho, that could become a problem. >> >> So my command should be more like: >> >> rsync -ai --progress --delete /path/to/source/ path/to/target >> >> If I want to preserve all the Linux file data, then I should use this: >> >> rsync -axAHX --progress --delete /path/to/source/ path/to/target >> >> Dale >> >> :-) :-) >> >> P. S. Working on new kernel for fireball. Added some options for >> encryption stuff. I really need to update to a newer kernel. I got a >> newer one that boots but no GUI. That's not very helpful. > >> Hi Dale, I might have missed it in the thread but are you aware that >> rsync is focussed on remote filefile transfer and if its a local >> transfer it does a full copy (no delta) of the file without >> optimisations as its usually faster than all the extra operations a >> local delta requires. > > You are using NFS mounts so rsync is looking at it as a local copy - > it does not know it is a remote system. > > My recent use of moosefs (another network file system) had similar > problems using rsync - it also turns out some of the data rsync uses > for detecting changes may not be stable across a network mount - > moosefs certainly has problems with this, NFS likely to have it too. > > The workaround is to check file-size, mtimes and ctimes to figure out > which is able to be used. > > see: > > http://unix.stackexchange.com/questions/450537/ddg#450666 > > google rsync over nfs > > BillK I've read about using rsync as a daemon. It is supposed to be faster. For some reason, I just never got around to trying it. It would likely be a good idea to work on that. It would likely be really handy when moving large amounts of data. I may try to read a howto on the Gentoo wiki or something. See what all it involves. Interesting link. I find some good info on that site at times. Dale :-) :-)
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
On 22/10/23 11:23, Dale wrote: Frank Steinmetzger wrote: Am Fri, Oct 20, 2023 at 09:20:45PM -0500 schrieb Dale: Howdy, As most know, I had to restore from backups recently. I also reworked my NAS box. I'm doing my first backup given that I have more files that need to be added to the backups. When I started the rsync, it's starting from the first file and updating each file as it goes as if all of them changed. Given that likely 95% of the files hasn't changed, I figure this is being done because of a time stamp or something. Is there a way to tell rsync to ignore the time stamp or something or if the files are the same size, just update the time stamp? Is there a way to just update the time stamps on the NAS box? Is there a option I haven't thought of to work around this? This is the old command I was using to create the backups. time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series /mnt/TV_Backup/ This didn’t preserve timestamps. Hence there is one type of information lost from which rsync knows whether two files may be identical. So now your restore has more recent timestamps the the backup. If you use -u, Rsync should skip all files. My perfectionist self doesn’t like discarding timestamp information, because then my system can’t tell me how old some file is, and how old (or young) I was when I created it and so on. I once didn’t pay enough attention when restoring a backup back when I was still on Windows, which is why I don’t have many files left that are dated before April 2007, even though they are from 2000 ± x. BTW: -i and -v are redundant. -v will only print the file path, whereas -i does the same and adds the reasons colum at the beginning. I tried these to try to get around it. time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series /mnt/TV_Backup/ -a and -r are also redundant, as -a includes -r. I looked at the man page and the options there. I don't see anything that I think will help. Is there a way around this? My muscle memory uses `rsync -ai` for almost everything. And when I do full root file systems or stuff where I know I will need them, I use -axAHX instead. Since this preserves all the usual data, I’ve never really had rsync wanting to do everything all over. Well, I can't turn back the clocks so it is what it is now. These files tho, I really don't worry to much about the timestamps. If I were backing up my OS tho, that could become a problem. So my command should be more like: rsync -ai --progress --delete /path/to/source/ path/to/target If I want to preserve all the Linux file data, then I should use this: rsync -axAHX --progress --delete /path/to/source/ path/to/target Dale :-) :-) P. S. Working on new kernel for fireball. Added some options for encryption stuff. I really need to update to a newer kernel. I got a newer one that boots but no GUI. That's not very helpful. Hi Dale, I might have missed it in the thread but are you aware that rsync is focussed on remote filefile transfer and if its a local transfer it does a full copy (no delta) of the file without optimisations as its usually faster than all the extra operations a local delta requires. You are using NFS mounts so rsync is looking at it as a local copy - it does not know it is a remote system. My recent use of moosefs (another network file system) had similar problems using rsync - it also turns out some of the data rsync uses for detecting changes may not be stable across a network mount - moosefs certainly has problems with this, NFS likely to have it too. The workaround is to check file-size, mtimes and ctimes to figure out which is able to be used. see: http://unix.stackexchange.com/questions/450537/ddg#450666 google rsync over nfs BillK
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
Frank Steinmetzger wrote: > Am Fri, Oct 20, 2023 at 09:20:45PM -0500 schrieb Dale: >> Howdy, >> >> As most know, I had to restore from backups recently. I also reworked >> my NAS box. I'm doing my first backup given that I have more files that >> need to be added to the backups. When I started the rsync, it's >> starting from the first file and updating each file as it goes as if all >> of them changed. Given that likely 95% of the files hasn't changed, I >> figure this is being done because of a time stamp or something. Is >> there a way to tell rsync to ignore the time stamp or something or if >> the files are the same size, just update the time stamp? Is there a way >> to just update the time stamps on the NAS box? Is there a option I >> haven't thought of to work around this? >> >> This is the old command I was using to create the backups. >> >> time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series >> /mnt/TV_Backup/ > This didn’t preserve timestamps. Hence there is one type of information lost > from which rsync knows whether two files may be identical. So now your > restore has more recent timestamps the the backup. If you use -u, Rsync > should skip all files. > > My perfectionist self doesn’t like discarding timestamp information, because > then my system can’t tell me how old some file is, and how old (or young) I > was when I created it and so on. I once didn’t pay enough attention when > restoring a backup back when I was still on Windows, which is why I don’t > have many files left that are dated before April 2007, even though they are > from 2000 ± x. > > BTW: -i and -v are redundant. -v will only print the file path, whereas -i > does the same and adds the reasons colum at the beginning. > >> I tried these to try to get around it. >> >> time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series >> /mnt/TV_Backup/ > -a and -r are also redundant, as -a includes -r. > >> I looked at the man page and the options there. I don't see anything >> that I think will help. Is there a way around this? > My muscle memory uses `rsync -ai` for almost everything. And when I do full > root file systems or stuff where I know I will need them, I use -axAHX > instead. Since this preserves all the usual data, I’ve never really had > rsync wanting to do everything all over. > Well, I can't turn back the clocks so it is what it is now. These files tho, I really don't worry to much about the timestamps. If I were backing up my OS tho, that could become a problem. So my command should be more like: rsync -ai --progress --delete /path/to/source/ path/to/target If I want to preserve all the Linux file data, then I should use this: rsync -axAHX --progress --delete /path/to/source/ path/to/target Dale :-) :-) P. S. Working on new kernel for fireball. Added some options for encryption stuff. I really need to update to a newer kernel. I got a newer one that boots but no GUI. That's not very helpful.
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
Am Fri, Oct 20, 2023 at 09:20:45PM -0500 schrieb Dale: > Howdy, > > As most know, I had to restore from backups recently. I also reworked > my NAS box. I'm doing my first backup given that I have more files that > need to be added to the backups. When I started the rsync, it's > starting from the first file and updating each file as it goes as if all > of them changed. Given that likely 95% of the files hasn't changed, I > figure this is being done because of a time stamp or something. Is > there a way to tell rsync to ignore the time stamp or something or if > the files are the same size, just update the time stamp? Is there a way > to just update the time stamps on the NAS box? Is there a option I > haven't thought of to work around this? > > This is the old command I was using to create the backups. > > time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series > /mnt/TV_Backup/ This didn’t preserve timestamps. Hence there is one type of information lost from which rsync knows whether two files may be identical. So now your restore has more recent timestamps the the backup. If you use -u, Rsync should skip all files. My perfectionist self doesn’t like discarding timestamp information, because then my system can’t tell me how old some file is, and how old (or young) I was when I created it and so on. I once didn’t pay enough attention when restoring a backup back when I was still on Windows, which is why I don’t have many files left that are dated before April 2007, even though they are from 2000 ± x. BTW: -i and -v are redundant. -v will only print the file path, whereas -i does the same and adds the reasons colum at the beginning. > I tried these to try to get around it. > > time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series > /mnt/TV_Backup/ -a and -r are also redundant, as -a includes -r. > I looked at the man page and the options there. I don't see anything > that I think will help. Is there a way around this? My muscle memory uses `rsync -ai` for almost everything. And when I do full root file systems or stuff where I know I will need them, I use -axAHX instead. Since this preserves all the usual data, I’ve never really had rsync wanting to do everything all over. -- Grüße | Greetings | Salut | Qapla’ Please do not share anything from, with or about me on any social network. Arrogance is the art of being proud of one’s own stupidity. signature.asc Description: PGP signature
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
Daniel Pielmeier wrote: > Am 21. Oktober 2023 07:07:18 UTC schrieb Dale : >> Actually, I was looking at the man page again and that option appeared >> to me just a little bit ago. I tried it and it did several updates. >> Question tho. It finished that update just a minute ago. I tried again >> with my usual command, it wants to update everything again. I seem to >> recall that rsync goes by file size and modify time stamps. Is there a >> way to make the NAS box files newer or something so that it will not try >> to copy the same files over again? I'd like to use the default >> detection options of rsync if I can just in case something changes. >> > We are creatures of habit eh :-) > > Of course you have to specify this option every time as the timestamps remain > the same in this case. > Why not? First of all it's just video files. What are the odds that the > contents of change and the file size will be exactly the same? > > Using the defaults is fine but you should use the options which serve the > purpose. > > If you want you can use find on the backup drive in combination with exec to > run the touch command on each file in order to update the timestamps. Then > the -u option you use with rsync wont update the file on the destination > because it is newer then the source. However in the end the result is the > same as if using "--size-only". > Don't nail me on this but the command could look like the following: find > pathtobackup -type f -exec touch '{}' + > That helped. I ended up running this command, with path being longer so I shortened it a bit. find /path/to/files/* -type f -exec touch '{}' + In case anyone runs up on this, I added the wild card on the end. I ran it first without but it didn't work as expected. After I added it, it updated the modify time to the current time. I did the first run in a test directory. I'm currently rechecking the kernel config on both puter, fireball and nas. Trying to sort out the speed problem and I think the problem may be a missing driver on fireball actually. Then again, nas may have a missing driver as well. I'm working on it. Thanks for the help. Got one problem sorted at least. ;-) Dale :-) :-)
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
Not (unfortunately) speaking from my own experience, just from Googling: https://serverfault.com/questions/211005/rsync-difference-between-checksum-and-ignore-times-options discusses a —ignore-times option that may be useful. The upside is that it uses the minimum amount of network bandwidth but at the cost of checksumming everything on both sides. May take a while if you have a big stash of video files. However, there’s a —checksum-choice option that, with some testing of choices, might give you a quick-and-dirty checksum that’s good enough for your purposes. Your backup files , hopefully, are not corrupted. I was just wondering why your new backup seemed to want to transfer everything all over again. Just speculating, but maybe timestamps got changed in the NAS box work you did? I saw a post (that I can’t find now) that discussed the —modify-window option. This, I gather, is used when rsyncing between 2 different filesystems with different ideas about how precise to make the timestamps. A high precision timestamp file system’s timestamp will be a little different from a low precision timestamp file system’s timestamp, and will appear to be different timestamps to rsync, even though ls -l reports the same times. The stat command will give you full precision times, and maybe that will show a discrepancy. The —modify-window option lets you fudge subtle time differences. This is just speculation, and maybe an exotic and unlikely one, and definitely something I’ve never done myself. Another idea (again something I Googled but have never done) is to use the mtree utility. Apparently you can use this to write out the characteristics of a directory hierarchy - things like owner, permissions, checksum, size, timestamps, etc. on every file under that directory. Save this output to a file, then run mtree with this file as input on another directory hierarchy, say the backup on your NAS. Looks like you can use it to just check for differences, or you can use it to correct differences. If your timestamps were different on your video machine and your backup NAS, this looks like a way to make them match without transferring files. Hope a little of this was useful. Good luck! John Blinka
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
On Saturday, 21 October 2023 03:20:45 BST Dale wrote: > Howdy, > > As most know, I had to restore from backups recently. I also reworked > my NAS box. I'm doing my first backup given that I have more files that > need to be added to the backups. When I started the rsync, it's > starting from the first file and updating each file as it goes as if all > of them changed. Given that likely 95% of the files hasn't changed, I > figure this is being done because of a time stamp or something. Did you figure this by guessing, or by actually comparing timestamps and even running a checksum on a couple of files? > Is > there a way to tell rsync to ignore the time stamp or something or if > the files are the same size, just update the time stamp? Is there a way > to just update the time stamps on the NAS box? Is there a option I > haven't thought of to work around this? Before you ask rsync to ignore (some) differences, I suggest you investigate to establish what these differences are and why they could have occurred. I assume you'd rather fix the problem, as opposed to work around it. Options you could consider is --modify-window=1 and perhaps --size-only, but only if you are not interested in identifying and conclusively addressing the cause of the problem. > This is the old command I was using to create the backups. > > time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series > /mnt/TV_Backup/ > > I tried these to try to get around it. > > time rsync -ivr --progress --delete /home/dale/Desktop/Crypt/TV_Series > /mnt/TV_Backup/ > time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series > /mnt/TV_Backup/ > time rsync -a --progress --delete /home/dale/Desktop/Crypt/TV_Series > /mnt/TV_Backup/ The -a option on its own includes -r, as the man page explains: --archive, -aarchive mode is -rlptgoD (no -A,-X,-U,-N,-H) and --archive, -a This is equivalent to -rlptgoD. It is a quick way of saying you want re‐ cursion and want to preserve almost everything. Be aware that it does not include preserving ACLs (-A), xattrs (-X), atimes (-U), crtimes (-N), nor the finding and preserving of hardlinks (-H). The only exception to the above equivalence is when --files-from is spec‐ ified, in which case -r is not implied. > Second problem. The transfer speed is back to the old slower speed. > I'm pretty sure I am using the same old options on both ends. Still, > it's back to being slow again. Did you fix the your kernel on your PC to use hardware acceleration for crypto? Did you test the maximum network raw transfer speed to eliminate any network problem? signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
Am 21. Oktober 2023 07:07:18 UTC schrieb Dale : >Actually, I was looking at the man page again and that option appeared >to me just a little bit ago. I tried it and it did several updates. >Question tho. It finished that update just a minute ago. I tried again >with my usual command, it wants to update everything again. I seem to >recall that rsync goes by file size and modify time stamps. Is there a >way to make the NAS box files newer or something so that it will not try >to copy the same files over again? I'd like to use the default >detection options of rsync if I can just in case something changes. > We are creatures of habit eh :-) Of course you have to specify this option every time as the timestamps remain the same in this case. Why not? First of all it's just video files. What are the odds that the contents of change and the file size will be exactly the same? Using the defaults is fine but you should use the options which serve the purpose. If you want you can use find on the backup drive in combination with exec to run the touch command on each file in order to update the timestamps. Then the -u option you use with rsync wont update the file on the destination because it is newer then the source. However in the end the result is the same as if using "--size-only". Don't nail me on this but the command could look like the following: find pathtobackup -type f -exec touch '{}' + -- Best regards Daniel
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
Daniel Pielmeier wrote: >> I looked at the man page and the options there. I don't see anything >> that I think will help. Is there a way around this? > Then you didn't read properly, the man page is really exhaustive. You can try > "--size-only". > It is also possible to negate the options by prefixing them with no like e.g. > "--no-times --no-owner --no-group etc." I also had this issue when > transferring between different file systems which didn't have same concept of > file properties. > Anyway using "-a" as you tried doesn't make sense as it tries to preserve > almost everything. Actually, I was looking at the man page again and that option appeared to me just a little bit ago. I tried it and it did several updates. Question tho. It finished that update just a minute ago. I tried again with my usual command, it wants to update everything again. I seem to recall that rsync goes by file size and modify time stamps. Is there a way to make the NAS box files newer or something so that it will not try to copy the same files over again? I'd like to use the default detection options of rsync if I can just in case something changes. >> Second problem. The transfer speed is back to the old slower speed. >> I'm pretty sure I am using the same old options on both ends. Still, >> it's back to being slow again. > But the new disk is different compared to the old one? Then it needs to be > treated differently. There multiple factors which can affect performance. > Had the same issue with a new SSD drive. Unfortunately I currently don't have > access to the machine and I don't remember what needed changing but it could > well be possible that you need different mount options, format the drive > using other options, using different schedulers, etc. > Exact same drives. While adding a new drive to LVM, somehow the file system got messed up. I couldn't get it fixed so I just started over again. I might add, the restore from backups was faster. That was when the extra options were posted in another thread. I'd think it would go the same speed pretty much either way. Dale :-) :-)
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
>I looked at the man page and the options there. I don't see anything >that I think will help. Is there a way around this? Then you didn't read properly, the man page is really exhaustive. You can try "--size-only". It is also possible to negate the options by prefixing them with no like e.g. "--no-times --no-owner --no-group etc." I also had this issue when transferring between different file systems which didn't have same concept of file properties. Anyway using "-a" as you tried doesn't make sense as it tries to preserve almost everything. >Second problem. The transfer speed is back to the old slower speed. >I'm pretty sure I am using the same old options on both ends. Still, >it's back to being slow again. But the new disk is different compared to the old one? Then it needs to be treated differently. There multiple factors which can affect performance. Had the same issue with a new SSD drive. Unfortunately I currently don't have access to the machine and I don't remember what needed changing but it could well be possible that you need different mount options, format the drive using other options, using different schedulers, etc. -- Best regards Daniel
[gentoo-user] rsync options after backup restore. Transfer speed again.
Howdy, As most know, I had to restore from backups recently. I also reworked my NAS box. I'm doing my first backup given that I have more files that need to be added to the backups. When I started the rsync, it's starting from the first file and updating each file as it goes as if all of them changed. Given that likely 95% of the files hasn't changed, I figure this is being done because of a time stamp or something. Is there a way to tell rsync to ignore the time stamp or something or if the files are the same size, just update the time stamp? Is there a way to just update the time stamps on the NAS box? Is there a option I haven't thought of to work around this? This is the old command I was using to create the backups. time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series /mnt/TV_Backup/ I tried these to try to get around it. time rsync -ivr --progress --delete /home/dale/Desktop/Crypt/TV_Series /mnt/TV_Backup/ time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series /mnt/TV_Backup/ time rsync -a --progress --delete /home/dale/Desktop/Crypt/TV_Series /mnt/TV_Backup/ I looked at the man page and the options there. I don't see anything that I think will help. Is there a way around this? Second problem. The transfer speed is back to the old slower speed. I'm pretty sure I am using the same old options on both ends. Still, it's back to being slow again. Some info: nas / # mount | grep backup /dev/mapper/backup on /mnt/backup type ext4 (rw,relatime) nas / # cat /etc/exports # /etc/exports: NFS file systems being exported. See exports(5). /mnt/backup/ *(rw,async,no_subtree_check) nas / # root@fireball / # mount | grep backup 10.0.0.5:/mnt/backup on /mnt/Backup type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=4,timeo=600,retrans=2,sec=sys,clientaddr=10.0.0.2,local_lock=none,addr=10.0.0.5) root@fireball / # Did I miss something? Typo maybe? I'm pretty sure I used copy and paste but still. Thanks. Dale :-) :-)