Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
On 22/10/23 11:23, Dale wrote: Frank Steinmetzger wrote: Am Fri, Oct 20, 2023 at 09:20:45PM -0500 schrieb Dale: Howdy, As most know, I had to restore from backups recently. I also reworked my NAS box. I'm doing my first backup given that I have more files that need to be added to the backups. When I started the rsync, it's starting from the first file and updating each file as it goes as if all of them changed. Given that likely 95% of the files hasn't changed, I figure this is being done because of a time stamp or something. Is there a way to tell rsync to ignore the time stamp or something or if the files are the same size, just update the time stamp? Is there a way to just update the time stamps on the NAS box? Is there a option I haven't thought of to work around this? This is the old command I was using to create the backups. time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series /mnt/TV_Backup/ This didn’t preserve timestamps. Hence there is one type of information lost from which rsync knows whether two files may be identical. So now your restore has more recent timestamps the the backup. If you use -u, Rsync should skip all files. My perfectionist self doesn’t like discarding timestamp information, because then my system can’t tell me how old some file is, and how old (or young) I was when I created it and so on. I once didn’t pay enough attention when restoring a backup back when I was still on Windows, which is why I don’t have many files left that are dated before April 2007, even though they are from 2000 ± x. BTW: -i and -v are redundant. -v will only print the file path, whereas -i does the same and adds the reasons colum at the beginning. I tried these to try to get around it. time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series /mnt/TV_Backup/ -a and -r are also redundant, as -a includes -r. I looked at the man page and the options there. I don't see anything that I think will help. Is there a way around this? My muscle memory uses `rsync -ai` for almost everything. And when I do full root file systems or stuff where I know I will need them, I use -axAHX instead. Since this preserves all the usual data, I’ve never really had rsync wanting to do everything all over. Well, I can't turn back the clocks so it is what it is now. These files tho, I really don't worry to much about the timestamps. If I were backing up my OS tho, that could become a problem. So my command should be more like: rsync -ai --progress --delete /path/to/source/ path/to/target If I want to preserve all the Linux file data, then I should use this: rsync -axAHX --progress --delete /path/to/source/ path/to/target Dale :-) :-) P. S. Working on new kernel for fireball. Added some options for encryption stuff. I really need to update to a newer kernel. I got a newer one that boots but no GUI. That's not very helpful. Hi Dale, I might have missed it in the thread but are you aware that rsync is focussed on remote filefile transfer and if its a local transfer it does a full copy (no delta) of the file without optimisations as its usually faster than all the extra operations a local delta requires. You are using NFS mounts so rsync is looking at it as a local copy - it does not know it is a remote system. My recent use of moosefs (another network file system) had similar problems using rsync - it also turns out some of the data rsync uses for detecting changes may not be stable across a network mount - moosefs certainly has problems with this, NFS likely to have it too. The workaround is to check file-size, mtimes and ctimes to figure out which is able to be used. see: http://unix.stackexchange.com/questions/450537/ddg#450666 google rsync over nfs BillK
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
Frank Steinmetzger wrote: > Am Fri, Oct 20, 2023 at 09:20:45PM -0500 schrieb Dale: >> Howdy, >> >> As most know, I had to restore from backups recently. I also reworked >> my NAS box. I'm doing my first backup given that I have more files that >> need to be added to the backups. When I started the rsync, it's >> starting from the first file and updating each file as it goes as if all >> of them changed. Given that likely 95% of the files hasn't changed, I >> figure this is being done because of a time stamp or something. Is >> there a way to tell rsync to ignore the time stamp or something or if >> the files are the same size, just update the time stamp? Is there a way >> to just update the time stamps on the NAS box? Is there a option I >> haven't thought of to work around this? >> >> This is the old command I was using to create the backups. >> >> time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series >> /mnt/TV_Backup/ > This didn’t preserve timestamps. Hence there is one type of information lost > from which rsync knows whether two files may be identical. So now your > restore has more recent timestamps the the backup. If you use -u, Rsync > should skip all files. > > My perfectionist self doesn’t like discarding timestamp information, because > then my system can’t tell me how old some file is, and how old (or young) I > was when I created it and so on. I once didn’t pay enough attention when > restoring a backup back when I was still on Windows, which is why I don’t > have many files left that are dated before April 2007, even though they are > from 2000 ± x. > > BTW: -i and -v are redundant. -v will only print the file path, whereas -i > does the same and adds the reasons colum at the beginning. > >> I tried these to try to get around it. >> >> time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series >> /mnt/TV_Backup/ > -a and -r are also redundant, as -a includes -r. > >> I looked at the man page and the options there. I don't see anything >> that I think will help. Is there a way around this? > My muscle memory uses `rsync -ai` for almost everything. And when I do full > root file systems or stuff where I know I will need them, I use -axAHX > instead. Since this preserves all the usual data, I’ve never really had > rsync wanting to do everything all over. > Well, I can't turn back the clocks so it is what it is now. These files tho, I really don't worry to much about the timestamps. If I were backing up my OS tho, that could become a problem. So my command should be more like: rsync -ai --progress --delete /path/to/source/ path/to/target If I want to preserve all the Linux file data, then I should use this: rsync -axAHX --progress --delete /path/to/source/ path/to/target Dale :-) :-) P. S. Working on new kernel for fireball. Added some options for encryption stuff. I really need to update to a newer kernel. I got a newer one that boots but no GUI. That's not very helpful.
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
Am Fri, Oct 20, 2023 at 09:20:45PM -0500 schrieb Dale: > Howdy, > > As most know, I had to restore from backups recently. I also reworked > my NAS box. I'm doing my first backup given that I have more files that > need to be added to the backups. When I started the rsync, it's > starting from the first file and updating each file as it goes as if all > of them changed. Given that likely 95% of the files hasn't changed, I > figure this is being done because of a time stamp or something. Is > there a way to tell rsync to ignore the time stamp or something or if > the files are the same size, just update the time stamp? Is there a way > to just update the time stamps on the NAS box? Is there a option I > haven't thought of to work around this? > > This is the old command I was using to create the backups. > > time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series > /mnt/TV_Backup/ This didn’t preserve timestamps. Hence there is one type of information lost from which rsync knows whether two files may be identical. So now your restore has more recent timestamps the the backup. If you use -u, Rsync should skip all files. My perfectionist self doesn’t like discarding timestamp information, because then my system can’t tell me how old some file is, and how old (or young) I was when I created it and so on. I once didn’t pay enough attention when restoring a backup back when I was still on Windows, which is why I don’t have many files left that are dated before April 2007, even though they are from 2000 ± x. BTW: -i and -v are redundant. -v will only print the file path, whereas -i does the same and adds the reasons colum at the beginning. > I tried these to try to get around it. > > time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series > /mnt/TV_Backup/ -a and -r are also redundant, as -a includes -r. > I looked at the man page and the options there. I don't see anything > that I think will help. Is there a way around this? My muscle memory uses `rsync -ai` for almost everything. And when I do full root file systems or stuff where I know I will need them, I use -axAHX instead. Since this preserves all the usual data, I’ve never really had rsync wanting to do everything all over. -- Grüße | Greetings | Salut | Qapla’ Please do not share anything from, with or about me on any social network. Arrogance is the art of being proud of one’s own stupidity. signature.asc Description: PGP signature
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
Daniel Pielmeier wrote:
> Am 21. Oktober 2023 07:07:18 UTC schrieb Dale :
>> Actually, I was looking at the man page again and that option appeared
>> to me just a little bit ago. I tried it and it did several updates.
>> Question tho. It finished that update just a minute ago. I tried again
>> with my usual command, it wants to update everything again. I seem to
>> recall that rsync goes by file size and modify time stamps. Is there a
>> way to make the NAS box files newer or something so that it will not try
>> to copy the same files over again? I'd like to use the default
>> detection options of rsync if I can just in case something changes.
>>
> We are creatures of habit eh :-)
>
> Of course you have to specify this option every time as the timestamps remain
> the same in this case.
> Why not? First of all it's just video files. What are the odds that the
> contents of change and the file size will be exactly the same?
>
> Using the defaults is fine but you should use the options which serve the
> purpose.
>
> If you want you can use find on the backup drive in combination with exec to
> run the touch command on each file in order to update the timestamps. Then
> the -u option you use with rsync wont update the file on the destination
> because it is newer then the source. However in the end the result is the
> same as if using "--size-only".
> Don't nail me on this but the command could look like the following: find
> pathtobackup -type f -exec touch '{}' +
>
That helped. I ended up running this command, with path being longer so
I shortened it a bit.
find /path/to/files/* -type f -exec touch '{}' +
In case anyone runs up on this, I added the wild card on the end. I ran
it first without but it didn't work as expected. After I added it, it
updated the modify time to the current time. I did the first run in a
test directory.
I'm currently rechecking the kernel config on both puter, fireball and
nas. Trying to sort out the speed problem and I think the problem may
be a missing driver on fireball actually. Then again, nas may have a
missing driver as well. I'm working on it.
Thanks for the help. Got one problem sorted at least. ;-)
Dale
:-) :-)
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
Not (unfortunately) speaking from my own experience, just from Googling: https://serverfault.com/questions/211005/rsync-difference-between-checksum-and-ignore-times-options discusses a —ignore-times option that may be useful. The upside is that it uses the minimum amount of network bandwidth but at the cost of checksumming everything on both sides. May take a while if you have a big stash of video files. However, there’s a —checksum-choice option that, with some testing of choices, might give you a quick-and-dirty checksum that’s good enough for your purposes. Your backup files , hopefully, are not corrupted. I was just wondering why your new backup seemed to want to transfer everything all over again. Just speculating, but maybe timestamps got changed in the NAS box work you did? I saw a post (that I can’t find now) that discussed the —modify-window option. This, I gather, is used when rsyncing between 2 different filesystems with different ideas about how precise to make the timestamps. A high precision timestamp file system’s timestamp will be a little different from a low precision timestamp file system’s timestamp, and will appear to be different timestamps to rsync, even though ls -l reports the same times. The stat command will give you full precision times, and maybe that will show a discrepancy. The —modify-window option lets you fudge subtle time differences. This is just speculation, and maybe an exotic and unlikely one, and definitely something I’ve never done myself. Another idea (again something I Googled but have never done) is to use the mtree utility. Apparently you can use this to write out the characteristics of a directory hierarchy - things like owner, permissions, checksum, size, timestamps, etc. on every file under that directory. Save this output to a file, then run mtree with this file as input on another directory hierarchy, say the backup on your NAS. Looks like you can use it to just check for differences, or you can use it to correct differences. If your timestamps were different on your video machine and your backup NAS, this looks like a way to make them match without transferring files. Hope a little of this was useful. Good luck! John Blinka
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
On Saturday, 21 October 2023 03:20:45 BST Dale wrote: > Howdy, > > As most know, I had to restore from backups recently. I also reworked > my NAS box. I'm doing my first backup given that I have more files that > need to be added to the backups. When I started the rsync, it's > starting from the first file and updating each file as it goes as if all > of them changed. Given that likely 95% of the files hasn't changed, I > figure this is being done because of a time stamp or something. Did you figure this by guessing, or by actually comparing timestamps and even running a checksum on a couple of files? > Is > there a way to tell rsync to ignore the time stamp or something or if > the files are the same size, just update the time stamp? Is there a way > to just update the time stamps on the NAS box? Is there a option I > haven't thought of to work around this? Before you ask rsync to ignore (some) differences, I suggest you investigate to establish what these differences are and why they could have occurred. I assume you'd rather fix the problem, as opposed to work around it. Options you could consider is --modify-window=1 and perhaps --size-only, but only if you are not interested in identifying and conclusively addressing the cause of the problem. > This is the old command I was using to create the backups. > > time rsync -uivr --progress --delete /home/dale/Desktop/Crypt/TV_Series > /mnt/TV_Backup/ > > I tried these to try to get around it. > > time rsync -ivr --progress --delete /home/dale/Desktop/Crypt/TV_Series > /mnt/TV_Backup/ > time rsync -ar --progress --delete /home/dale/Desktop/Crypt/TV_Series > /mnt/TV_Backup/ > time rsync -a --progress --delete /home/dale/Desktop/Crypt/TV_Series > /mnt/TV_Backup/ The -a option on its own includes -r, as the man page explains: --archive, -aarchive mode is -rlptgoD (no -A,-X,-U,-N,-H) and --archive, -a This is equivalent to -rlptgoD. It is a quick way of saying you want re‐ cursion and want to preserve almost everything. Be aware that it does not include preserving ACLs (-A), xattrs (-X), atimes (-U), crtimes (-N), nor the finding and preserving of hardlinks (-H). The only exception to the above equivalence is when --files-from is spec‐ ified, in which case -r is not implied. > Second problem. The transfer speed is back to the old slower speed. > I'm pretty sure I am using the same old options on both ends. Still, > it's back to being slow again. Did you fix the your kernel on your PC to use hardware acceleration for crypto? Did you test the maximum network raw transfer speed to eliminate any network problem? signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
Am 21. Oktober 2023 07:07:18 UTC schrieb Dale :
>Actually, I was looking at the man page again and that option appeared
>to me just a little bit ago. I tried it and it did several updates.
>Question tho. It finished that update just a minute ago. I tried again
>with my usual command, it wants to update everything again. I seem to
>recall that rsync goes by file size and modify time stamps. Is there a
>way to make the NAS box files newer or something so that it will not try
>to copy the same files over again? I'd like to use the default
>detection options of rsync if I can just in case something changes.
>
We are creatures of habit eh :-)
Of course you have to specify this option every time as the timestamps remain
the same in this case.
Why not? First of all it's just video files. What are the odds that the
contents of change and the file size will be exactly the same?
Using the defaults is fine but you should use the options which serve the
purpose.
If you want you can use find on the backup drive in combination with exec to
run the touch command on each file in order to update the timestamps. Then the
-u option you use with rsync wont update the file on the destination because it
is newer then the source. However in the end the result is the same as if using
"--size-only".
Don't nail me on this but the command could look like the following: find
pathtobackup -type f -exec touch '{}' +
--
Best regards
Daniel
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
Daniel Pielmeier wrote: >> I looked at the man page and the options there. I don't see anything >> that I think will help. Is there a way around this? > Then you didn't read properly, the man page is really exhaustive. You can try > "--size-only". > It is also possible to negate the options by prefixing them with no like e.g. > "--no-times --no-owner --no-group etc." I also had this issue when > transferring between different file systems which didn't have same concept of > file properties. > Anyway using "-a" as you tried doesn't make sense as it tries to preserve > almost everything. Actually, I was looking at the man page again and that option appeared to me just a little bit ago. I tried it and it did several updates. Question tho. It finished that update just a minute ago. I tried again with my usual command, it wants to update everything again. I seem to recall that rsync goes by file size and modify time stamps. Is there a way to make the NAS box files newer or something so that it will not try to copy the same files over again? I'd like to use the default detection options of rsync if I can just in case something changes. >> Second problem. The transfer speed is back to the old slower speed. >> I'm pretty sure I am using the same old options on both ends. Still, >> it's back to being slow again. > But the new disk is different compared to the old one? Then it needs to be > treated differently. There multiple factors which can affect performance. > Had the same issue with a new SSD drive. Unfortunately I currently don't have > access to the machine and I don't remember what needed changing but it could > well be possible that you need different mount options, format the drive > using other options, using different schedulers, etc. > Exact same drives. While adding a new drive to LVM, somehow the file system got messed up. I couldn't get it fixed so I just started over again. I might add, the restore from backups was faster. That was when the extra options were posted in another thread. I'd think it would go the same speed pretty much either way. Dale :-) :-)
Re: [gentoo-user] rsync options after backup restore. Transfer speed again.
>I looked at the man page and the options there. I don't see anything >that I think will help. Is there a way around this? Then you didn't read properly, the man page is really exhaustive. You can try "--size-only". It is also possible to negate the options by prefixing them with no like e.g. "--no-times --no-owner --no-group etc." I also had this issue when transferring between different file systems which didn't have same concept of file properties. Anyway using "-a" as you tried doesn't make sense as it tries to preserve almost everything. >Second problem. The transfer speed is back to the old slower speed. >I'm pretty sure I am using the same old options on both ends. Still, >it's back to being slow again. But the new disk is different compared to the old one? Then it needs to be treated differently. There multiple factors which can affect performance. Had the same issue with a new SSD drive. Unfortunately I currently don't have access to the machine and I don't remember what needed changing but it could well be possible that you need different mount options, format the drive using other options, using different schedulers, etc. -- Best regards Daniel
