Re: [gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-29 Thread Wol

On 22/10/2023 09:50, Dale wrote:
I've read about using rsync as a daemon.  It is supposed to be faster. 
For some reason, I just never got around to trying it.  It would likely 
be a good idea to work on that.  It would likely be really handy when 
moving large amounts of data.  I may try to read a howto on the Gentoo 
wiki or something.  See what all it involves.


Run the daemon on the NAS. For each file where the metadata differs, the 
NAS will send a bunch of block checksums to your rsync command (or the 
other way round).


If all the checksums agree, then the files are identical, without having 
to send the file over the network.


If only a few checksums differ, only a few blocks have to be sent.

In other words, using a daemon will massively reduce the amount of data 
going across the network, at the expense of the NAS having to scan and 
checksum every "file of interest". A pretty cheap trade-off, imho.


Cheers,
Wol



Re: [gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-23 Thread Frank Steinmetzger
Am Mon, Oct 23, 2023 at 02:29:26AM -0500 schrieb Dale:
> Dale wrote:
> >
> > Second problem.  The transfer speed is back to the old slower speed. 
> > I'm pretty sure I am using the same old options on both ends.  Still,
> > it's back to being slow again.  Some info:
> >
> >
> > <<< SNIP >>>
> > Did I miss something?  Typo maybe?  I'm pretty sure I used copy and
> > paste but still. 
> >
> > Thanks.
> >
> > Dale
> >
> > :-)  :-) 
> 
> 
> I been working on the speed problem again.  I rebuilt the kernel on
> fireball and I think some changes made a huge change.  This is the
> results from fireball now:
> 
> 
> root@fireball / # cryptsetup benchmark
> # Tests are approximate using memory only (no storage IO).
> PBKDF2-sha1   931239 iterations per second for 256-bit key
> PBKDF2-sha256    1356501 iterations per second for 256-bit key
> PBKDF2-sha512 972705 iterations per second for 256-bit key
> PBKDF2-ripemd160  648871 iterations per second for 256-bit key
> PBKDF2-whirlpool  362077 iterations per second for 256-bit key
> argon2i   5 iterations, 1048576 memory, 4 parallel threads (CPUs)
> for 256-bit key (requested 2000 ms time)
> argon2id  4 iterations, 1048576 memory, 4 parallel threads (CPUs)
> for 256-bit key (requested 2000 ms time)
> # Algorithm |   Key |  Encryption |  Decryption
>     aes-cbc    128b   570.8 MiB/s  2045.6 MiB/s
>     serpent-cbc    128b    91.1 MiB/s   310.0 MiB/s
>     twofish-cbc    128b   198.7 MiB/s   218.9 MiB/s
>     aes-cbc    256b   428.8 MiB/s  1670.4 MiB/s
>     serpent-cbc    256b    91.6 MiB/s   309.5 MiB/s
>     twofish-cbc    256b   199.8 MiB/s   219.2 MiB/s
>     aes-xts    256b  1821.2 MiB/s  1767.1 MiB/s
>     serpent-xts    256b   265.9 MiB/s   270.2 MiB/s
>     twofish-xts    256b   201.0 MiB/s   204.2 MiB/s
>     aes-xts    512b  1440.0 MiB/s  1445.9 MiB/s
>     serpent-xts    512b   265.0 MiB/s   257.2 MiB/s
>     twofish-xts    512b   198.2 MiB/s   201.6 MiB/s
> root@fireball / #

There you go. Told ya. :)

> As you can see, aes-cbc is fast now and I think that is what cryptsetup
> uses.  It used to be really slow I think. 

Cryptsetup uses aes-xts these days, I think it’s been mentioned in this 
thread somewhere.

> Now on to the nas box.  I've recompiled the kernel with some added
> options.  Still, it refuses to speed up.  I kinda think it is the CPU
> lacking support for encryption.  I'm asking others just in case I'm
> missing something.  Also, fireball uses a older kernel, 5.14 or so.  The
> nas box uses 6.1 or so.  The menus are different and that is why it is
> hard to get them to match up.  I may have missed something.

Everything you need for that is in the crypto menu at the bottom.

> This is the bench mark from nas box. 
> 
> nas ~ # cryptsetup benchmark
> # Tests are approximate using memory only (no storage IO).
> PBKDF2-sha1   700919 iterations per second for 256-bit key
> PBKDF2-sha256 924670 iterations per second for 256-bit key
> PBKDF2-sha512 729190 iterations per second for 256-bit key
> PBKDF2-ripemd160  517559 iterations per second for 256-bit key
> PBKDF2-whirlpool  359593 iterations per second for 256-bit key
> argon2i   4 iterations, 1048576 memory, 4 parallel threads (CPUs)
> for 256-bit key (requested 2000 ms time)
> argon2id  4 iterations, 1048576 memory, 4 parallel threads (CPUs)
> for 256-bit key (requested 2000 ms time)
> # Algorithm |   Key |  Encryption |  Decryption
>     aes-cbc    128b    63.6 MiB/s    41.6 MiB/s
>     serpent-cbc    128b    81.0 MiB/s   212.4 MiB/s
>     twofish-cbc    128b   192.5 MiB/s   222.1 MiB/s
>     aes-cbc    256b    47.5 MiB/s    30.0 MiB/s
>     serpent-cbc    256b    81.2 MiB/s   212.7 MiB/s
>     twofish-cbc    256b   192.3 MiB/s   221.9 MiB/s
>     aes-xts    256b    65.9 MiB/s    41.6 MiB/s
>     serpent-xts    256b   201.7 MiB/s   205.7 MiB/s
>     twofish-xts    256b   216.2 MiB/s   214.5 MiB/s
>     aes-xts    512b    48.8 MiB/s    30.0 MiB/s
>     serpent-xts    512b   202.7 MiB/s   205.6 MiB/s
>     twofish-xts    512b   216.4 MiB/s   214.0 MiB/s
> nas ~ #
> […]
> The aes shows up on fireball.  It does not on the nas box.  Is the speed
> above as good as I can expect with this older CPU?

If not done yet, you can check whether you enabled the 64 bit versions of 
the crypto modules. They could push performance by a few more percent.

-- 
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

The last chance is often the second-last, if you look close enough.


signature.asc
Description: PGP signature


Re: [gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-23 Thread Michael
On Monday, 23 October 2023 08:29:26 BST Dale wrote:

> I been working on the speed problem again.  I rebuilt the kernel on
> fireball and I think some changes made a huge change.  This is the
> results from fireball now:
> 
> 
> root@fireball / # cryptsetup benchmark
> # Tests are approximate using memory only (no storage IO).
> PBKDF2-sha1   931239 iterations per second for 256-bit key
> PBKDF2-sha2561356501 iterations per second for 256-bit key
> PBKDF2-sha512 972705 iterations per second for 256-bit key
> PBKDF2-ripemd160  648871 iterations per second for 256-bit key
> PBKDF2-whirlpool  362077 iterations per second for 256-bit key
> argon2i   5 iterations, 1048576 memory, 4 parallel threads (CPUs)
> for 256-bit key (requested 2000 ms time)
> argon2id  4 iterations, 1048576 memory, 4 parallel threads (CPUs)
> for 256-bit key (requested 2000 ms time)
> # Algorithm |   Key |  Encryption |  Decryption
> aes-cbc128b   570.8 MiB/s  2045.6 MiB/s
> serpent-cbc128b91.1 MiB/s   310.0 MiB/s
> twofish-cbc128b   198.7 MiB/s   218.9 MiB/s
> aes-cbc256b   428.8 MiB/s  1670.4 MiB/s
> serpent-cbc256b91.6 MiB/s   309.5 MiB/s
> twofish-cbc256b   199.8 MiB/s   219.2 MiB/s
> aes-xts256b  1821.2 MiB/s  1767.1 MiB/s
> serpent-xts256b   265.9 MiB/s   270.2 MiB/s
> twofish-xts256b   201.0 MiB/s   204.2 MiB/s
> aes-xts512b  1440.0 MiB/s  1445.9 MiB/s
> serpent-xts512b   265.0 MiB/s   257.2 MiB/s
> twofish-xts512b   198.2 MiB/s   201.6 MiB/s
> root@fireball / #
> 
> 
> As you can see, aes-cbc is fast now and I think that is what cryptsetup
> uses.  It used to be really slow I think. 

Yep, it was many times slower with your previous kernel.  Now it will saturate 
your 1Gbps network link, as long as the other box can take it.


> Now on to the nas box.  I've recompiled the kernel with some added
> options.  Still, it refuses to speed up.  I kinda think it is the CPU
> lacking support for encryption.  I'm asking others just in case I'm
> missing something.

No, you're not missing anything, although in a previous message you had posted 
a benchmark result for the Phenom performance being around 4 times higher than 
what you show below.  As I mentioned then, the Phenom CPU does not have AES-NI 
crypto hardware acceleration.  it is a rather old CPU.  Assuming the MoBo has 
the latest OEM firmware and you have also added microcode in your kernel 
(CONFIG_MICROCODE_AMD=y), then probably that's all it can do.  Small 
optimisations may show up between kernel releases, yet again patches to 
address CPU vulnerabilities (retpoline?) could make things a bit worse.


> Also, fireball uses a older kernel, 5.14 or so.  The
> nas box uses 6.1 or so.  The menus are different and that is why it is
> hard to get them to match up.  I may have missed something.  This is the
> bench mark from nas box. 

You can diff previous and current kernel config files to see what has changed.  
For the crypto settings you can grep for AES and for CRYPT to make sure you 
have not left out what you'd need for disk encryption.

> nas ~ # cryptsetup benchmark
> # Tests are approximate using memory only (no storage IO).
> PBKDF2-sha1   700919 iterations per second for 256-bit key
> PBKDF2-sha256 924670 iterations per second for 256-bit key
> PBKDF2-sha512 729190 iterations per second for 256-bit key
> PBKDF2-ripemd160  517559 iterations per second for 256-bit key
> PBKDF2-whirlpool  359593 iterations per second for 256-bit key
> argon2i   4 iterations, 1048576 memory, 4 parallel threads (CPUs)
> for 256-bit key (requested 2000 ms time)
> argon2id  4 iterations, 1048576 memory, 4 parallel threads (CPUs)
> for 256-bit key (requested 2000 ms time)
> # Algorithm |   Key |  Encryption |  Decryption
> aes-cbc128b63.6 MiB/s41.6 MiB/s
> serpent-cbc128b81.0 MiB/s   212.4 MiB/s
> twofish-cbc128b   192.5 MiB/s   222.1 MiB/s
> aes-cbc256b47.5 MiB/s30.0 MiB/s
> serpent-cbc256b81.2 MiB/s   212.7 MiB/s
> twofish-cbc256b   192.3 MiB/s   221.9 MiB/s
> aes-xts256b65.9 MiB/s41.6 MiB/s
> serpent-xts256b   201.7 MiB/s   205.7 MiB/s
> twofish-xts256b   216.2 MiB/s   214.5 MiB/s
> aes-xts512b48.8 MiB/s30.0 MiB/s
> serpent-xts512b   202.7 MiB/s   205.6 MiB/s
> twofish-xts512b   216.4 MiB/s   214.0 MiB/s
> nas ~ #

How does your aes-xts 30.0 MiB/s shown above compare with your previous 
benchmark result?  I'm sure it was quite higher than this.


> I seem to recall it 

Re: [gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-23 Thread Dale
Dale wrote:
>
> Second problem.  The transfer speed is back to the old slower speed. 
> I'm pretty sure I am using the same old options on both ends.  Still,
> it's back to being slow again.  Some info:
>
>
> <<< SNIP >>>
> Did I miss something?  Typo maybe?  I'm pretty sure I used copy and
> paste but still. 
>
> Thanks.
>
> Dale
>
> :-)  :-) 


I been working on the speed problem again.  I rebuilt the kernel on
fireball and I think some changes made a huge change.  This is the
results from fireball now:


root@fireball / # cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1   931239 iterations per second for 256-bit key
PBKDF2-sha256    1356501 iterations per second for 256-bit key
PBKDF2-sha512 972705 iterations per second for 256-bit key
PBKDF2-ripemd160  648871 iterations per second for 256-bit key
PBKDF2-whirlpool  362077 iterations per second for 256-bit key
argon2i   5 iterations, 1048576 memory, 4 parallel threads (CPUs)
for 256-bit key (requested 2000 ms time)
argon2id  4 iterations, 1048576 memory, 4 parallel threads (CPUs)
for 256-bit key (requested 2000 ms time)
# Algorithm |   Key |  Encryption |  Decryption
    aes-cbc    128b   570.8 MiB/s  2045.6 MiB/s
    serpent-cbc    128b    91.1 MiB/s   310.0 MiB/s
    twofish-cbc    128b   198.7 MiB/s   218.9 MiB/s
    aes-cbc    256b   428.8 MiB/s  1670.4 MiB/s
    serpent-cbc    256b    91.6 MiB/s   309.5 MiB/s
    twofish-cbc    256b   199.8 MiB/s   219.2 MiB/s
    aes-xts    256b  1821.2 MiB/s  1767.1 MiB/s
    serpent-xts    256b   265.9 MiB/s   270.2 MiB/s
    twofish-xts    256b   201.0 MiB/s   204.2 MiB/s
    aes-xts    512b  1440.0 MiB/s  1445.9 MiB/s
    serpent-xts    512b   265.0 MiB/s   257.2 MiB/s
    twofish-xts    512b   198.2 MiB/s   201.6 MiB/s
root@fireball / #


As you can see, aes-cbc is fast now and I think that is what cryptsetup
uses.  It used to be really slow I think. 

Now on to the nas box.  I've recompiled the kernel with some added
options.  Still, it refuses to speed up.  I kinda think it is the CPU
lacking support for encryption.  I'm asking others just in case I'm
missing something.  Also, fireball uses a older kernel, 5.14 or so.  The
nas box uses 6.1 or so.  The menus are different and that is why it is
hard to get them to match up.  I may have missed something.  This is the
bench mark from nas box. 


nas ~ # cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1   700919 iterations per second for 256-bit key
PBKDF2-sha256 924670 iterations per second for 256-bit key
PBKDF2-sha512 729190 iterations per second for 256-bit key
PBKDF2-ripemd160  517559 iterations per second for 256-bit key
PBKDF2-whirlpool  359593 iterations per second for 256-bit key
argon2i   4 iterations, 1048576 memory, 4 parallel threads (CPUs)
for 256-bit key (requested 2000 ms time)
argon2id  4 iterations, 1048576 memory, 4 parallel threads (CPUs)
for 256-bit key (requested 2000 ms time)
# Algorithm |   Key |  Encryption |  Decryption
    aes-cbc    128b    63.6 MiB/s    41.6 MiB/s
    serpent-cbc    128b    81.0 MiB/s   212.4 MiB/s
    twofish-cbc    128b   192.5 MiB/s   222.1 MiB/s
    aes-cbc    256b    47.5 MiB/s    30.0 MiB/s
    serpent-cbc    256b    81.2 MiB/s   212.7 MiB/s
    twofish-cbc    256b   192.3 MiB/s   221.9 MiB/s
    aes-xts    256b    65.9 MiB/s    41.6 MiB/s
    serpent-xts    256b   201.7 MiB/s   205.7 MiB/s
    twofish-xts    256b   216.2 MiB/s   214.5 MiB/s
    aes-xts    512b    48.8 MiB/s    30.0 MiB/s
    serpent-xts    512b   202.7 MiB/s   205.6 MiB/s
    twofish-xts    512b   216.4 MiB/s   214.0 MiB/s
nas ~ #


I seem to recall it being said that the old CPU in the nas box lacks the
aes instruction set.  This is a list of the CPU flags from the nas box.


nas ~ # cpuid2cpuflags
CPU_FLAGS_X86: 3dnow 3dnowext mmx mmxext popcnt sse sse2 sse3 sse4a
nas ~ #


The aes shows up on fireball.  It does not on the nas box.  Is the speed
above as good as I can expect with this older CPU?  I can include the
kernel config if needed.  If you know what driver you are looking for,
let me know what to grep for.  If not sure, I can attach the config file. 

Is this it?  Is this as fast as this old CPU can get? 

Dale

:-)  :-) 



Re: [gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-22 Thread Dale
William Kenworthy wrote:
>
> On 22/10/23 11:23, Dale wrote:
>> Frank Steinmetzger wrote:
>>> Am Fri, Oct 20, 2023 at 09:20:45PM -0500 schrieb Dale:
 Howdy,

 As most know, I had to restore from backups recently.  I also reworked
 my NAS box.  I'm doing my first backup given that I have more files
 that
 need to be added to the backups.  When I started the rsync, it's
 starting from the first file and updating each file as it goes as
 if all
 of them changed.  Given that likely 95% of the files hasn't changed, I
 figure this is being done because of a time stamp or something.  Is
 there a way to tell rsync to ignore the time stamp or something or if
 the files are the same size, just update the time stamp?  Is there
 a way
 to just update the time stamps on the NAS box?  Is there a option I
 haven't thought of to work around this?

 This is the old command I was using to create the backups.

 time rsync -uivr --progress --delete
 /home/dale/Desktop/Crypt/TV_Series
 /mnt/TV_Backup/
>>> This didn’t preserve timestamps. Hence there is one type of
>>> information lost
>>> from which rsync knows whether two files may be identical. So now your
>>> restore has more recent timestamps the the backup. If you use -u, Rsync
>>> should skip all files.
>>>
>>> My perfectionist self doesn’t like discarding timestamp information,
>>> because
>>> then my system can’t tell me how old some file is, and how old (or
>>> young) I
>>> was when I created it and so on. I once didn’t pay enough attention
>>> when
>>> restoring a backup back when I was still on Windows, which is why I
>>> don’t
>>> have many files left that are dated before April 2007, even though
>>> they are
>>> from 2000 ± x.
>>>
>>> BTW: -i and -v are redundant. -v will only print the file path,
>>> whereas -i
>>> does the same and adds the reasons colum at the beginning.
>>>
 I tried these to try to get around it.

 time rsync -ar --progress --delete
 /home/dale/Desktop/Crypt/TV_Series /mnt/TV_Backup/
>>> -a and -r are also redundant, as -a includes -r.
>>>
 I looked at the man page and the options there.  I don't see anything
 that I think will help.  Is there a way around this?
>>> My muscle memory uses `rsync -ai` for almost everything. And when I
>>> do full
>>> root file systems or stuff where I know I will need them, I use -axAHX
>>> instead. Since this preserves all the usual data, I’ve never really had
>>> rsync wanting to do everything all over.
>>>
>> Well, I can't turn back the clocks so it is what it is now.  These files
>> tho, I really don't worry to much about the timestamps.  If I were
>> backing up my OS tho, that could become a problem.
>>
>> So my command should be more like:
>>
>> rsync -ai --progress --delete /path/to/source/ path/to/target
>>
>> If I want to preserve all the Linux file data, then I should use this:
>>
>> rsync -axAHX --progress --delete /path/to/source/ path/to/target
>>
>> Dale
>>
>> :-)  :-)
>>
>> P. S.  Working on new kernel for fireball.  Added some options for
>> encryption stuff.  I really need to update to a newer kernel.  I got a
>> newer one that boots but no GUI.  That's not very helpful.
>
>> Hi Dale, I might have missed it in the thread but are you aware that
>> rsync is focussed on remote filefile transfer and if its a local
>> transfer it does a full copy (no delta) of the file without
>> optimisations as its usually faster than all the extra operations a
>> local delta requires.
>
> You are using NFS mounts so rsync is looking at it as a local copy -
> it does not know it is a remote system.
>
> My recent use of moosefs (another network file system) had similar
> problems using rsync - it also turns out some of the data rsync uses
> for detecting changes may not be stable across a network mount -
> moosefs certainly has problems with this, NFS likely to have it too.
>
> The workaround is to check file-size, mtimes and ctimes to figure out
> which is able to be used.
>
> see:
>
> http://unix.stackexchange.com/questions/450537/ddg#450666
>
> google rsync over nfs
>
> BillK


I've read about using rsync as a daemon.  It is supposed to be faster. 
For some reason, I just never got around to trying it.  It would likely
be a good idea to work on that.  It would likely be really handy when
moving large amounts of data.  I may try to read a howto on the Gentoo
wiki or something.  See what all it involves. 

Interesting link.  I find some good info on that site at times. 

Dale

:-)  :-) 



Re: [gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-21 Thread William Kenworthy



On 22/10/23 11:23, Dale wrote:

Frank Steinmetzger wrote:

Am Fri, Oct 20, 2023 at 09:20:45PM -0500 schrieb Dale:

Howdy,

As most know, I had to restore from backups recently.  I also reworked
my NAS box.  I'm doing my first backup given that I have more files that
need to be added to the backups.  When I started the rsync, it's
starting from the first file and updating each file as it goes as if all
of them changed.  Given that likely 95% of the files hasn't changed, I
figure this is being done because of a time stamp or something.  Is
there a way to tell rsync to ignore the time stamp or something or if
the files are the same size, just update the time stamp?  Is there a way
to just update the time stamps on the NAS box?  Is there a option I
haven't thought of to work around this?

This is the old command I was using to create the backups.

time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series
/mnt/TV_Backup/

This didn’t preserve timestamps. Hence there is one type of information lost
from which rsync knows whether two files may be identical. So now your
restore has more recent timestamps the the backup. If you use -u, Rsync
should skip all files.

My perfectionist self doesn’t like discarding timestamp information, because
then my system can’t tell me how old some file is, and how old (or young) I
was when I created it and so on. I once didn’t pay enough attention when
restoring a backup back when I was still on Windows, which is why I don’t
have many files left that are dated before April 2007, even though they are
from 2000 ± x.

BTW: -i and -v are redundant. -v will only print the file path, whereas -i
does the same and adds the reasons colum at the beginning.


I tried these to try to get around it.

time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series 
/mnt/TV_Backup/

-a and -r are also redundant, as -a includes -r.


I looked at the man page and the options there.  I don't see anything
that I think will help.  Is there a way around this?

My muscle memory uses `rsync -ai` for almost everything. And when I do full
root file systems or stuff where I know I will need them, I use -axAHX
instead. Since this preserves all the usual data, I’ve never really had
rsync wanting to do everything all over.


Well, I can't turn back the clocks so it is what it is now.  These files
tho, I really don't worry to much about the timestamps.  If I were
backing up my OS tho, that could become a problem.

So my command should be more like:

rsync -ai --progress --delete /path/to/source/ path/to/target

If I want to preserve all the Linux file data, then I should use this:

rsync -axAHX --progress --delete /path/to/source/ path/to/target

Dale

:-)  :-)

P. S.  Working on new kernel for fireball.  Added some options for
encryption stuff.  I really need to update to a newer kernel.  I got a
newer one that boots but no GUI.  That's not very helpful.


Hi Dale, I might have missed it in the thread but are you aware that 
rsync is focussed on remote filefile transfer and if its a local 
transfer it does a full copy (no delta) of the file without 
optimisations as its usually faster than all the extra operations a 
local delta requires.


You are using NFS mounts so rsync is looking at it as a local copy - it 
does not know it is a remote system.


My recent use of moosefs (another network file system) had similar 
problems using rsync - it also turns out some of the data rsync uses for 
detecting changes may not be stable across a network mount - moosefs 
certainly has problems with this, NFS likely to have it too.


The workaround is to check file-size, mtimes and ctimes to figure out 
which is able to be used.


see:

http://unix.stackexchange.com/questions/450537/ddg#450666

google rsync over nfs

BillK





Re: [gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-21 Thread Dale
Frank Steinmetzger wrote:
> Am Fri, Oct 20, 2023 at 09:20:45PM -0500 schrieb Dale:
>> Howdy,
>>
>> As most know, I had to restore from backups recently.  I also reworked
>> my NAS box.  I'm doing my first backup given that I have more files that
>> need to be added to the backups.  When I started the rsync, it's
>> starting from the first file and updating each file as it goes as if all
>> of them changed.  Given that likely 95% of the files hasn't changed, I
>> figure this is being done because of a time stamp or something.  Is
>> there a way to tell rsync to ignore the time stamp or something or if
>> the files are the same size, just update the time stamp?  Is there a way
>> to just update the time stamps on the NAS box?  Is there a option I
>> haven't thought of to work around this? 
>>
>> This is the old command I was using to create the backups.
>>
>> time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series
>> /mnt/TV_Backup/
> This didn’t preserve timestamps. Hence there is one type of information lost 
> from which rsync knows whether two files may be identical. So now your 
> restore has more recent timestamps the the backup. If you use -u, Rsync 
> should skip all files.
>
> My perfectionist self doesn’t like discarding timestamp information, because 
> then my system can’t tell me how old some file is, and how old (or young) I 
> was when I created it and so on. I once didn’t pay enough attention when 
> restoring a backup back when I was still on Windows, which is why I don’t 
> have many files left that are dated before April 2007, even though they are 
> from 2000 ± x.
>
> BTW: -i and -v are redundant. -v will only print the file path, whereas -i 
> does the same and adds the reasons colum at the beginning.
>
>> I tried these to try to get around it.
>>
>> time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series 
>> /mnt/TV_Backup/
> -a and -r are also redundant, as -a includes -r.
>
>> I looked at the man page and the options there.  I don't see anything
>> that I think will help.  Is there a way around this? 
> My muscle memory uses `rsync -ai` for almost everything. And when I do full 
> root file systems or stuff where I know I will need them, I use -axAHX 
> instead. Since this preserves all the usual data, I’ve never really had 
> rsync wanting to do everything all over.
>

Well, I can't turn back the clocks so it is what it is now.  These files
tho, I really don't worry to much about the timestamps.  If I were
backing up my OS tho, that could become a problem. 

So my command should be more like:

rsync -ai --progress --delete /path/to/source/ path/to/target

If I want to preserve all the Linux file data, then I should use this: 

rsync -axAHX --progress --delete /path/to/source/ path/to/target

Dale

:-)  :-) 

P. S.  Working on new kernel for fireball.  Added some options for
encryption stuff.  I really need to update to a newer kernel.  I got a
newer one that boots but no GUI.  That's not very helpful. 



Re: [gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-21 Thread Frank Steinmetzger
Am Fri, Oct 20, 2023 at 09:20:45PM -0500 schrieb Dale:
> Howdy,
> 
> As most know, I had to restore from backups recently.  I also reworked
> my NAS box.  I'm doing my first backup given that I have more files that
> need to be added to the backups.  When I started the rsync, it's
> starting from the first file and updating each file as it goes as if all
> of them changed.  Given that likely 95% of the files hasn't changed, I
> figure this is being done because of a time stamp or something.  Is
> there a way to tell rsync to ignore the time stamp or something or if
> the files are the same size, just update the time stamp?  Is there a way
> to just update the time stamps on the NAS box?  Is there a option I
> haven't thought of to work around this? 
> 
> This is the old command I was using to create the backups.
> 
> time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series
> /mnt/TV_Backup/

This didn’t preserve timestamps. Hence there is one type of information lost 
from which rsync knows whether two files may be identical. So now your 
restore has more recent timestamps the the backup. If you use -u, Rsync 
should skip all files.

My perfectionist self doesn’t like discarding timestamp information, because 
then my system can’t tell me how old some file is, and how old (or young) I 
was when I created it and so on. I once didn’t pay enough attention when 
restoring a backup back when I was still on Windows, which is why I don’t 
have many files left that are dated before April 2007, even though they are 
from 2000 ± x.

BTW: -i and -v are redundant. -v will only print the file path, whereas -i 
does the same and adds the reasons colum at the beginning.

> I tried these to try to get around it.
> 
> time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series 
> /mnt/TV_Backup/

-a and -r are also redundant, as -a includes -r.

> I looked at the man page and the options there.  I don't see anything
> that I think will help.  Is there a way around this? 

My muscle memory uses `rsync -ai` for almost everything. And when I do full 
root file systems or stuff where I know I will need them, I use -axAHX 
instead. Since this preserves all the usual data, I’ve never really had 
rsync wanting to do everything all over.

-- 
Grüße | Greetings | Salut | Qapla’
Please do not share anything from, with or about me on any social network.

Arrogance is the art of being proud of one’s own stupidity.


signature.asc
Description: PGP signature


Re: [gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-21 Thread Dale
Daniel Pielmeier wrote:
> Am 21. Oktober 2023 07:07:18 UTC schrieb Dale :
>> Actually, I was looking at the man page again and that option appeared
>> to me just a little bit ago.  I tried it and it did several updates. 
>> Question tho.  It finished that update just a minute ago.  I tried again
>> with my usual command, it wants to update everything again.  I seem to
>> recall that rsync goes by file size and modify time stamps.  Is there a
>> way to make the NAS box files newer or something so that it will not try
>> to copy the same files over again?  I'd like to use the default
>> detection options of rsync if I can just in case something changes. 
>>
> We are creatures of habit eh :-)
>
> Of course you have to specify this option every time as the timestamps remain 
> the same in this case.
> Why not? First of all it's just video files. What are the odds that the 
> contents of change and the file size will be exactly the same?
>
> Using the defaults is fine but you should use the options which serve the 
> purpose.
>
> If you want you can use find on the backup drive in combination with exec to 
> run the touch command on each file in order to update the timestamps. Then 
> the -u option you use with rsync wont update the file on the destination 
> because it is newer then the source. However in the end the result is the 
> same as if using "--size-only".
> Don't nail me on this but the command could look like the following: find 
> pathtobackup -type f -exec touch '{}' +
>


That helped.  I ended up running this command, with path being longer so
I shortened it a bit. 


find /path/to/files/* -type f -exec touch '{}' +


In case anyone runs up on this, I added the wild card on the end.  I ran
it first without but it didn't work as expected.  After I added it, it
updated the modify time to the current time.  I did the first run in a
test directory. 

I'm currently rechecking the kernel config on both puter, fireball and
nas.  Trying to sort out the speed problem and I think the problem may
be a missing driver on fireball actually.  Then again, nas may have a
missing driver as well.  I'm working on it. 

Thanks for the help.  Got one problem sorted at least.  ;-) 

Dale

:-)  :-) 



Re: [gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-21 Thread John Blinka
Not (unfortunately) speaking from my own experience, just from Googling:

https://serverfault.com/questions/211005/rsync-difference-between-checksum-and-ignore-times-options

discusses a —ignore-times option that may be useful. The upside is that it
uses the minimum amount of network bandwidth but at the cost of
checksumming everything on both sides. May take a while if you have a big
stash of video files. However, there’s a —checksum-choice option that, with
some testing of choices, might give you a quick-and-dirty checksum that’s
good enough for your purposes. Your backup files , hopefully, are not
corrupted.

I was just wondering why your new backup seemed to want to transfer
everything all over again. Just speculating, but maybe timestamps got
changed in the NAS box work you did? I saw a post (that I can’t find now)
that discussed the —modify-window option. This, I gather, is used when
rsyncing between 2 different filesystems with different ideas about how
precise to make the timestamps. A high precision timestamp file system’s
timestamp will be a little different from a low precision timestamp file
system’s timestamp, and will appear to be different timestamps to rsync,
even though ls -l reports the same times. The stat command will give you
full precision times, and maybe that will show a discrepancy. The
—modify-window option lets you fudge subtle time differences. This is just
speculation, and maybe an exotic and unlikely one, and definitely something
I’ve never done myself.

Another idea (again something I Googled but have never done) is to use the
mtree utility. Apparently you can use this to write out the characteristics
of a directory hierarchy - things like owner, permissions, checksum, size,
timestamps, etc. on every file under that directory. Save this output to a
file, then run mtree with this file as input on another directory
hierarchy, say the backup on your NAS. Looks like you can use it to just
check for differences, or you can use it to correct differences. If your
timestamps were different on your video machine and your backup NAS, this
looks like a way to make them match without transferring files.

Hope a little of this was useful. Good luck!

John Blinka


Re: [gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-21 Thread Michael
On Saturday, 21 October 2023 03:20:45 BST Dale wrote:
> Howdy,
> 
> As most know, I had to restore from backups recently.  I also reworked
> my NAS box.  I'm doing my first backup given that I have more files that
> need to be added to the backups.  When I started the rsync, it's
> starting from the first file and updating each file as it goes as if all
> of them changed.  Given that likely 95% of the files hasn't changed, I
> figure this is being done because of a time stamp or something.

Did you figure this by guessing, or by actually comparing timestamps and even 
running a checksum on a couple of files?


> Is
> there a way to tell rsync to ignore the time stamp or something or if
> the files are the same size, just update the time stamp?  Is there a way
> to just update the time stamps on the NAS box?  Is there a option I
> haven't thought of to work around this? 

Before you ask rsync to ignore (some) differences, I suggest you investigate 
to establish what these differences are and why they could have occurred.  I 
assume you'd rather fix the problem, as opposed to work around it.

Options you could consider is --modify-window=1 and perhaps --size-only, but 
only if you are not interested in identifying and conclusively addressing the 
cause of the problem.


> This is the old command I was using to create the backups.
> 
> time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series
> /mnt/TV_Backup/
> 
> I tried these to try to get around it.
> 
> time rsync -ivr --progress --delete /home/dale/Desktop/Crypt/TV_Series
> /mnt/TV_Backup/
> time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series
> /mnt/TV_Backup/
> time rsync -a --progress --delete /home/dale/Desktop/Crypt/TV_Series
> /mnt/TV_Backup/

The -a option on its own includes -r, as the man page explains:

--archive, -aarchive mode is -rlptgoD (no -A,-X,-U,-N,-H)

and 

--archive, -a
This is equivalent to -rlptgoD.  It is a quick way of saying you want re‐
cursion and want to preserve almost everything.  Be aware  that  it  does
not include preserving ACLs (-A), xattrs (-X), atimes (-U), crtimes (-N),
nor the finding and preserving of hardlinks (-H).

The only exception to the above equivalence is when --files-from is spec‐
ified, in which case -r is not implied.


> Second problem.  The transfer speed is back to the old slower speed. 
> I'm pretty sure I am using the same old options on both ends.  Still,
> it's back to being slow again.

Did you fix the your kernel on your PC to use hardware acceleration for 
crypto?

Did you test the maximum network raw transfer speed to eliminate any network 
problem?


signature.asc
Description: This is a digitally signed message part.


Re: [gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-21 Thread Daniel Pielmeier
Am 21. Oktober 2023 07:07:18 UTC schrieb Dale :
>Actually, I was looking at the man page again and that option appeared
>to me just a little bit ago.  I tried it and it did several updates. 
>Question tho.  It finished that update just a minute ago.  I tried again
>with my usual command, it wants to update everything again.  I seem to
>recall that rsync goes by file size and modify time stamps.  Is there a
>way to make the NAS box files newer or something so that it will not try
>to copy the same files over again?  I'd like to use the default
>detection options of rsync if I can just in case something changes. 
>

We are creatures of habit eh :-)

Of course you have to specify this option every time as the timestamps remain 
the same in this case.
Why not? First of all it's just video files. What are the odds that the 
contents of change and the file size will be exactly the same?

Using the defaults is fine but you should use the options which serve the 
purpose.

If you want you can use find on the backup drive in combination with exec to 
run the touch command on each file in order to update the timestamps. Then the 
-u option you use with rsync wont update the file on the destination because it 
is newer then the source. However in the end the result is the same as if using 
"--size-only".
Don't nail me on this but the command could look like the following: find 
pathtobackup -type f -exec touch '{}' +

-- 
Best regards
Daniel



Re: [gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-21 Thread Dale
Daniel Pielmeier wrote:
>> I looked at the man page and the options there.  I don't see anything
>> that I think will help.  Is there a way around this?
> Then you didn't read properly, the man page is really exhaustive. You can try 
> "--size-only".
> It is also possible to negate the options by prefixing them with no like e.g. 
> "--no-times --no-owner --no-group etc." I also had this issue when 
> transferring between different file systems which didn't have same concept of 
> file properties.
> Anyway using "-a" as you tried doesn't make sense as it tries to preserve 
> almost everything.

Actually, I was looking at the man page again and that option appeared
to me just a little bit ago.  I tried it and it did several updates. 
Question tho.  It finished that update just a minute ago.  I tried again
with my usual command, it wants to update everything again.  I seem to
recall that rsync goes by file size and modify time stamps.  Is there a
way to make the NAS box files newer or something so that it will not try
to copy the same files over again?  I'd like to use the default
detection options of rsync if I can just in case something changes. 


>> Second problem.  The transfer speed is back to the old slower speed. 
>> I'm pretty sure I am using the same old options on both ends.  Still,
>> it's back to being slow again.
> But the new disk is different compared to the old one? Then it needs to be 
> treated differently. There multiple factors which can affect performance.
> Had the same issue with a new SSD drive. Unfortunately I currently don't have 
> access to the machine and I don't remember what needed changing but it could 
> well be possible that you need different mount options, format the drive 
> using other options, using different schedulers, etc.
>

Exact same drives.  While adding a new drive to LVM, somehow the file
system got messed up.  I couldn't get it fixed so I just started over
again.  I might add, the restore from backups was faster.  That was when
the extra options were posted in another thread.  I'd think it would go
the same speed pretty much either way. 

Dale

:-)  :-) 



Re: [gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-21 Thread Daniel Pielmeier
>I looked at the man page and the options there.  I don't see anything
>that I think will help.  Is there a way around this?

Then you didn't read properly, the man page is really exhaustive. You can try 
"--size-only".
It is also possible to negate the options by prefixing them with no like e.g. 
"--no-times --no-owner --no-group etc." I also had this issue when transferring 
between different file systems which didn't have same concept of file 
properties.
Anyway using "-a" as you tried doesn't make sense as it tries to preserve 
almost everything.

>Second problem.  The transfer speed is back to the old slower speed. 
>I'm pretty sure I am using the same old options on both ends.  Still,
>it's back to being slow again.

But the new disk is different compared to the old one? Then it needs to be 
treated differently. There multiple factors which can affect performance.
Had the same issue with a new SSD drive. Unfortunately I currently don't have 
access to the machine and I don't remember what needed changing but it could 
well be possible that you need different mount options, format the drive using 
other options, using different schedulers, etc.

-- 
Best regards
Daniel



[gentoo-user] rsync options after backup restore. Transfer speed again.

2023-10-20 Thread Dale
Howdy,

As most know, I had to restore from backups recently.  I also reworked
my NAS box.  I'm doing my first backup given that I have more files that
need to be added to the backups.  When I started the rsync, it's
starting from the first file and updating each file as it goes as if all
of them changed.  Given that likely 95% of the files hasn't changed, I
figure this is being done because of a time stamp or something.  Is
there a way to tell rsync to ignore the time stamp or something or if
the files are the same size, just update the time stamp?  Is there a way
to just update the time stamps on the NAS box?  Is there a option I
haven't thought of to work around this? 

This is the old command I was using to create the backups.

time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series
/mnt/TV_Backup/

I tried these to try to get around it.

time rsync -ivr --progress --delete /home/dale/Desktop/Crypt/TV_Series
/mnt/TV_Backup/
time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series
/mnt/TV_Backup/
time rsync -a --progress --delete /home/dale/Desktop/Crypt/TV_Series
/mnt/TV_Backup/

I looked at the man page and the options there.  I don't see anything
that I think will help.  Is there a way around this? 


Second problem.  The transfer speed is back to the old slower speed. 
I'm pretty sure I am using the same old options on both ends.  Still,
it's back to being slow again.  Some info:


nas / # mount | grep backup
/dev/mapper/backup on /mnt/backup type ext4 (rw,relatime)
nas / # cat /etc/exports
# /etc/exports: NFS file systems being exported.  See exports(5).
/mnt/backup/  *(rw,async,no_subtree_check)
nas / #

root@fireball / # mount | grep backup
10.0.0.5:/mnt/backup on /mnt/Backup type nfs4
(rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,nocto,proto=tcp,nconnect=4,timeo=600,retrans=2,sec=sys,clientaddr=10.0.0.2,local_lock=none,addr=10.0.0.5)
root@fireball / #

Did I miss something?  Typo maybe?  I'm pretty sure I used copy and
paste but still. 

Thanks.

Dale

:-)  :-)