Re: backing up backups
On Mon 18 Apr 2022 at 16:06:48 (-0400), Default User wrote: > BTW, I think I have narrowed the previous restore problem down to what I > believe is a "buggy" early UEFI implementation on my computer (circa 2014). > Irrelevant now; I have re-installed with BIOS (not UEFI) booting and MBR > (not GPT) partitioning. And have successfully tested restoring using both > Timeshift and Conezilla. > > And regarding learning by experience - oh, how I know. I've done so much of > that, I have a degree from the "school of hard knocks"! Just FTR, reverting to MBR should not be necessary, and you lose GPT's advantages and future-proofing. BIOS machines should be able to boot Grub from GPT because it's using the protective MBR that's left in place. And Grub should tell you if you don't leave the necessary space for its core image somewhere on a GPT disk (ie a BIOS Boot partition). Cheers, David.
Re: backing up backups
On Tue 19 Apr 2022 at 07:19:58 (+0200), DdB wrote: > So i came up with the idea to create a sort of inventory using a sparse > copy of empty files only (using mkdir, truncate + touch). The space > requirements are affordable (like 2.3M for an inventory representing > 3.5T of data). The effect being, that find will see those files by > name/directory/time/size and permissions, allowing to find duplicates > according to those attributes quite nicely. > > Since i started doing that, i always do know exactly, what's out there > and where to find it, as if i had only the inodes at hand. Suits MY > needs. :-) That sounds somewhat like my scheme for indexing my caddies, USB sticks and SD cards. When I unmount them, a script makes three indexes: the first running updatedb to a private database, the second running find to produce a list of strictly alphabetically ordered files (full name, --full-time and size) and the directories containing them, and the third, most like yours, running ls -lAR to another private database. The three indexes are propagated to my other hosts. Apart from being a plain text file, the output of ls -lAR, stored either as ….lslR or ….lslR.gz, can be navigated with Midnight Commander (mc) just as if it were a real filesystem. (Permissions don't concern me.) Cheers, David.
Re: backing up backups
Hello, Am 11.04.2022 um 04:58 schrieb Default User: > So . . . what IS the correct way to make "backups of backups"? > I don't know that for sure, but at first glance, i dont understand the complexity of your setup either. Seems to by quite elaborate, which is certainly suiting your needs. And since my base is also quite different from yours, ideas might not transfer that well... but anyhow Here is my use case: Apart from the main system, which resides on an NVME-SSD, all my storage consists of zfs pools, which allow snapshotting (instead of time-shifting). And because of the pools being hosted on raid, there is redundancy PLUS backups (also with partition images among them). Only SOME rarely used data (like movies), i do push into pools on removable media (spinning hard drives), and was interested to have their content searchable online. So i came up with the idea to create a sort of inventory using a sparse copy of empty files only (using mkdir, truncate + touch). The space requirements are affordable (like 2.3M for an inventory representing 3.5T of data). The effect being, that find will see those files by name/directory/time/size and permissions, allowing to find duplicates according to those attributes quite nicely. Since i started doing that, i always do know exactly, what's out there and where to find it, as if i had only the inodes at hand. Suits MY needs. :-) just my 2 cents DdB
Re: backing up backups
On 11/4/22 10:58, Default User wrote: So . . . what IS the correct way to make "backups of backups"? Sorry to take so long to respond. I am traveling and have only short periods that I can spend on non-pressing matters. To answer your question: the method that gets you the result you want. I have used 2 switches in rsync to create a date/time copy of files that are updated: --backup-dir=/mnt/data/rsynccBackupp/$YEAR/$MON/$TODAY/$HOUR/ source/ target creates an archive directory of older versions of files. I have noticed at times that it creates copies of files that I haven't fouched. Needs looking at, but it will something I'm doing. The variables are set in the script: YEAR=`date +%Y` MON=`date +%b` TODAY=`date +%d` NOW=`date +%Y%b%d%H` HOUR=`date +%H` rsync -avbH --suffix="."$(date +"%Y%m%d%H%M")source/ target/ replaces the standard ~ at the end of the file being updated with current time, in a numerical string. Keeps all versions of the files together. Which suits you better. As for how to 'backup your backup?' I copy everything from source documents to my backup drives, and set cron to run the script for each drive at a different time of day/hour - yes I backup current docs hourly. I'm lucky - I don't notice rsync affecting my performance. Timeshift is a great tool. I know you are correct to copy the system to a different drive. I figure that if I get to a point where I can't access the time shift files on /timeshift I'm in deep trouble. I just re-install. / has only system files on it. /home/keith is sym-linked from another partition. In any case, copying /timeshift from your system OR your first backup drive should be trivial for rsync. Out of time, but I think that's enough to think about for now -- All the best Keith Bainbridge keithrbaugro...@gmail.com
Re: backing up backups
On 4/18/22 13:06, Default User wrote: Finally, fun fact: Many years ago, at a local Linux user group meeting, Sun Microsystems put on a demonstration of their ZFS filesystem. To prove how robust it was, they pulled the power cord out of the wall socket on a running desktop computer. Then they plugged the cord back in and re-booted, with no problems! Yes, I was impressed. I bumped the USB cable of an external HDD while doing a ZFS incremental replication from my server to a backup disk. The backup pool was corrupted: 2022-03-28 18:16:25 toor@f1 ~ # zpool status -v z6000b pool: z6000b state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 0 in 0 days 09:36:15 with 0 errors on Mon Mar 28 00:09:09 2022 config: NAME STATE READ WRITE CKSUM z6000bONLINE 0 0 0 gpt/z6000b.eli ONLINE 0 0 0 cache gpt/z60a.eliONLINE 0 0 0 errors: Permanent errors have been detected in the following files: :<0x0> :<0x39> I bought a SATA 6 Gbps drive rack: https://www.startech.com/en-us/hdd/drw150satbk destroyed the backup pool, recreated the backup pool, and did a full replication. David
Re: backing up backups
On Thu, Apr 14, 2022 at 3:24 AM David Christensen wrote: > On 4/13/22 20:03, Default User wrote: > > On Wed, Apr 13, 2022 at 4:42 PM David Christensen wrote: > > >> As you find system administration commands that work, put them into > >> scripts: > >> > >> #!/bin/sh > >> sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 > >> /media/default/MSD1/ /media/default/MSD2/ > >> > >> > >> Use a version control system for system administration. Create a > >> project for every machine. Check in system configuration files, > >> scripts, partition table backups, encryption header backups, RAID header > >> backups, etc.. Maintain a plain text log file with notes of what you > >> did (e.g. console sessions), when, and why. > >> > >> > >> Put your OS on a small, fast device (e.g. SSD) and put your data on an > >> array of large devices (e.g. ZFS pool with one or more HDD mirrors). > >> Backup both as before. Additionally, take images of your OS device. > > > Yikes! > > > > David, I really think I am too old to learn all of that. But maybe I can > > learn at least some of it, over time. Please understand that I am not > > training to be a real system administrator, except that I guess anyone is > > (or should be able to be) actually the "system administrator" of their > own > > computer(s). > > > > Anyway, thanks for the advice. > > > I learned the above tools because they save time, save effort, and > provide features I want. > > > I use dd(1) and an external HDD for images. You will want to write > scripts (like the simple example I previously showed). > > > CVS has more than enough power for a single user/ system administrator, > and is simpler than Git. Here are the common use-cases: > > 1. Install CVS (and SSH) on Debian: > > # apt-get install cvs openssh-client openssh-server > > 2. Create a CVS repository: > > # mkdir -p /var/local/cvs/dpchrist > # cvs -d /var/local/cvs/dpchrist init > # chown -R dpchrist:dpchrist /var/local/cvs/dpchrist > > 3. Add CVS client environment variables to your shell (adjust host and > username as required): > > export > CVSROOT=dpchr...@cvs.tracy.holgerdanske.com:/var/local/cvs/dpchrist > export CVS_RSH=ssh > > 4. Create a project: > > $ mkdir -p import/myproject > $ cd import/myproject > $ touch .exists > $ cvs import myproject dpchrist start > > 5. Check-out a working directory of a project from the repository: > > $ cd > $ cvs co myproject > > 6. Add a file to the project working directory meta-data: > > $ cd myproject > $ vi myfile > $ cvs add myfile > > 7. See changes in the working directory compared to the repository: > > $ cvs diff > > 8. Bring in changes made elsewhere and checked-in to the repository: > > $ cvs update > > 9. Check-in working directory to the repository: > > $ cvs ci > > 10. Remove a file from the project: > > $ rm myfile > $ cvs rm myfile > > > See the GNU CVS manual for more information: > > > https://www.gnu.org/software/trans-coord/manual/cvs/html_node/index.html > > > ZFS is a new way of doing storage with disks, arrays, volumes, > filesystems, etc., including backup/ restore (snapshots and > replication). The learning curve is non-trivial. The Lucas book gave > me enough confidence to go for it: > > https://mwl.io/nonfiction/os#fmzfs > > > David > > Hey David, thanks for the information. BTW, I think I have narrowed the previous restore problem down to what I believe is a "buggy" early UEFI implementation on my computer (circa 2014). Irrelevant now; I have re-installed with BIOS (not UEFI) booting and MBR (not GPT) partitioning. And have successfully tested restoring using both Timeshift and Conezilla. And regarding learning by experience - oh, how I know. I've done so much of that, I have a degree from the "school of hard knocks"! Finally, fun fact: Many years ago, at a local Linux user group meeting, Sun Microsystems put on a demonstration of their ZFS filesystem. To prove how robust it was, they pulled the power cord out of the wall socket on a running desktop computer. Then they plugged the cord back in and re-booted, with no problems! Yes, I was impressed.
Re: backing up backups
On 4/13/22 20:03, Default User wrote: On Wed, Apr 13, 2022 at 4:42 PM David Christensen wrote: As you find system administration commands that work, put them into scripts: #!/bin/sh sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 /media/default/MSD1/ /media/default/MSD2/ Use a version control system for system administration. Create a project for every machine. Check in system configuration files, scripts, partition table backups, encryption header backups, RAID header backups, etc.. Maintain a plain text log file with notes of what you did (e.g. console sessions), when, and why. Put your OS on a small, fast device (e.g. SSD) and put your data on an array of large devices (e.g. ZFS pool with one or more HDD mirrors). Backup both as before. Additionally, take images of your OS device. Yikes! David, I really think I am too old to learn all of that. But maybe I can learn at least some of it, over time. Please understand that I am not training to be a real system administrator, except that I guess anyone is (or should be able to be) actually the "system administrator" of their own computer(s). Anyway, thanks for the advice. I learned the above tools because they save time, save effort, and provide features I want. I use dd(1) and an external HDD for images. You will want to write scripts (like the simple example I previously showed). CVS has more than enough power for a single user/ system administrator, and is simpler than Git. Here are the common use-cases: 1. Install CVS (and SSH) on Debian: # apt-get install cvs openssh-client openssh-server 2. Create a CVS repository: # mkdir -p /var/local/cvs/dpchrist # cvs -d /var/local/cvs/dpchrist init # chown -R dpchrist:dpchrist /var/local/cvs/dpchrist 3. Add CVS client environment variables to your shell (adjust host and username as required): export CVSROOT=dpchr...@cvs.tracy.holgerdanske.com:/var/local/cvs/dpchrist export CVS_RSH=ssh 4. Create a project: $ mkdir -p import/myproject $ cd import/myproject $ touch .exists $ cvs import myproject dpchrist start 5. Check-out a working directory of a project from the repository: $ cd $ cvs co myproject 6. Add a file to the project working directory meta-data: $ cd myproject $ vi myfile $ cvs add myfile 7. See changes in the working directory compared to the repository: $ cvs diff 8. Bring in changes made elsewhere and checked-in to the repository: $ cvs update 9. Check-in working directory to the repository: $ cvs ci 10. Remove a file from the project: $ rm myfile $ cvs rm myfile See the GNU CVS manual for more information: https://www.gnu.org/software/trans-coord/manual/cvs/html_node/index.html ZFS is a new way of doing storage with disks, arrays, volumes, filesystems, etc., including backup/ restore (snapshots and replication). The learning curve is non-trivial. The Lucas book gave me enough confidence to go for it: https://mwl.io/nonfiction/os#fmzfs David
Re: backing up backups
On Wed, Apr 13, 2022 at 4:42 PM David Christensen wrote: > On 4/13/22 09:20, Default User wrote: > > >> Hey guys, sorry for just getting back with you now. > >> Unfortunately, I am just now recovering from a self-inflicted computer > >> disaster. > >> > >> While fighting with rsync, I did either: > >> > >> sudo rsync -aAXHSxvv --delete --info=progress2,stats2,name2 > >> /media/default/MSD1/ /media/default/MSD2 > >> or > >> sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 > >> /media/default/MSD1/ /media/default/MSD2/ > >> > >> Just one small problem: MSD2 was not connected to my computer! > >> (Don't say it . . . ) > >> > >> Instead of giving an error message, rsync just created a directory on my > >> computer called /media/defaultMSD2, and filled it up until my / > partition > >> was full, and THEN my desktop environment (Cinnamon) popped up a > >> notification saying so. How thoughtful. > >> > >> The computer then would not reboot into the operating system. > >> > >> No problem, I say. I will just use Timeshift to restore from its backup > of > >> a few hours earlier. > >> > >> But that did not work, even after deleting the extra directory, and > trying > >> restores from multiple Timeshift backups. > >> > >> Anyway, I never could fix the problem. But I did take it as an > opportunity > >> to "start over". I put in a new(er) SSD, and did a fresh install, > replacing > >> Cinnamon with Gnome. Mistake - now I remember why I dislike Gnome, ever > >> since Gnome 3. Wish I had re-installed Cinnamon, but too late now, out > of > >> time. For now I will just have to grit my teeth and live with it. > >> > >> [BTW, yes, I do have all of my data. Backfilling it into my new setup > will > >> no doubt be an ongoing adventure.] > >> > >> Anyway, just a few notes about the rsync situation: > >> > >> 1) Having or not having a trailing / on the destination directory did > not > >> seem to make any difference in the size of the copy made, or otherwise. > >> Nevertheless, I intend to heed the advice given to have a trailing / > after > >> both source and destination, or neither, as appropriate. > >> > >> 2) Using or not using an "S" option with rsync did not seem to make any > >> difference, at least concerning the size of the copy made. > >> > >> 3) Yes, I really should check into using checksums to avoid "bot rot". > >> Good advice. > >> > >> Finally, Gnome sucks. (Did I mention that?) > >> > >> Thanks for the replies. > > > Congratulations! You now have more experience: > > "Doing things right is a matter of experience. Experience is a matter > of doing things wrong." > > > As you find system administration commands that work, put them into > scripts: > > #!/bin/sh > sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 > /media/default/MSD1/ /media/default/MSD2/ > > > Use a version control system for system administration. Create a > project for every machine. Check in system configuration files, > scripts, partition table backups, encryption header backups, RAID header > backups, etc.. Maintain a plain text log file with notes of what you > did (e.g. console sessions), when, and why. > > > Put your OS on a small, fast device (e.g. SSD) and put your data on an > array of large devices (e.g. ZFS pool with one or more HDD mirrors). > Backup both as before. Additionally, take images of your OS device. > > > David > > Yikes! David, I really think I am too old to learn all of that. But maybe I can learn at least some of it, over time. Please understand that I am not training to be a real system administrator, except that I guess anyone is (or should be able to be) actually the "system administrator" of their own computer(s). Anyway, thanks for the advice.
Re: backing up backups
On 4/13/22 09:20, Default User wrote: Hey guys, sorry for just getting back with you now. Unfortunately, I am just now recovering from a self-inflicted computer disaster. While fighting with rsync, I did either: sudo rsync -aAXHSxvv --delete --info=progress2,stats2,name2 /media/default/MSD1/ /media/default/MSD2 or sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 /media/default/MSD1/ /media/default/MSD2/ Just one small problem: MSD2 was not connected to my computer! (Don't say it . . . ) Instead of giving an error message, rsync just created a directory on my computer called /media/defaultMSD2, and filled it up until my / partition was full, and THEN my desktop environment (Cinnamon) popped up a notification saying so. How thoughtful. The computer then would not reboot into the operating system. No problem, I say. I will just use Timeshift to restore from its backup of a few hours earlier. But that did not work, even after deleting the extra directory, and trying restores from multiple Timeshift backups. Anyway, I never could fix the problem. But I did take it as an opportunity to "start over". I put in a new(er) SSD, and did a fresh install, replacing Cinnamon with Gnome. Mistake - now I remember why I dislike Gnome, ever since Gnome 3. Wish I had re-installed Cinnamon, but too late now, out of time. For now I will just have to grit my teeth and live with it. [BTW, yes, I do have all of my data. Backfilling it into my new setup will no doubt be an ongoing adventure.] Anyway, just a few notes about the rsync situation: 1) Having or not having a trailing / on the destination directory did not seem to make any difference in the size of the copy made, or otherwise. Nevertheless, I intend to heed the advice given to have a trailing / after both source and destination, or neither, as appropriate. 2) Using or not using an "S" option with rsync did not seem to make any difference, at least concerning the size of the copy made. 3) Yes, I really should check into using checksums to avoid "bot rot". Good advice. Finally, Gnome sucks. (Did I mention that?) Thanks for the replies. Congratulations! You now have more experience: "Doing things right is a matter of experience. Experience is a matter of doing things wrong." As you find system administration commands that work, put them into scripts: #!/bin/sh sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 /media/default/MSD1/ /media/default/MSD2/ Use a version control system for system administration. Create a project for every machine. Check in system configuration files, scripts, partition table backups, encryption header backups, RAID header backups, etc.. Maintain a plain text log file with notes of what you did (e.g. console sessions), when, and why. Put your OS on a small, fast device (e.g. SSD) and put your data on an array of large devices (e.g. ZFS pool with one or more HDD mirrors). Backup both as before. Additionally, take images of your OS device. David
Re: backing up backups
On Wed, Apr 13, 2022 at 12:09 PM Default User wrote: > > > On Mon, Apr 11, 2022 at 12:03 PM David Christensen < > dpchr...@holgerdanske.com> wrote: > >> On 4/10/22 22:15, to...@tuxteam.de wrote: >> > On Sun, Apr 10, 2022 at 09:44:59PM -0700, David Christensen wrote: >> >> On 4/10/22 19:58, Default User wrote: >> >>> Hello! >> >>> >> >>> My setup: >> >>> - single home x86-64 computer running Debian 11 Stable, up to date. >> >>> - one 4-Tb external usb hard drive to use as a backup device, labeled >> MSD1. >> >>> - another identical usb hard drive, labeled MSD2, to use as a copy of >> the >> >>> backups on MSD1. >> >>> - the computer and all storage devices are formatted ext4, not >> encrypted. >> >>> - two old Clonezilla disk images from when I installed Debian 11 last >> year >> >>> (probably irrelevant). >> >>> - Timeshift to daily back up system EXCEPT for data directories. >> >>> - Back in Time to daily back up data directories. >> >>> - Borgbackup to also daily back up data directories. >> >>> - Rsync to frequently backup any changed data between the daily Back >> in >> >>> Time and Borgbackup backups of data directories, using this command: >> >>> >> >>> sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 >> --exclude-from >> >>> "/home/default/rsync_exclude_list.txt" /home >> >>> /media/default/MSD1/rsync_backups_of_host_home_directory_only >> >>> >> >>> Each type of backup is in a separate subdirectory on MSD1 (Timeshift, >> Back >> >>> in TIme, Rsync, etc.). >> >>> >> >>> It "seems" to work okay, BUT . . . >> >>> >> >>> Then I try to use rsync to make an identical copy of backup device >> MSD1 on >> >>> an absolutely identical 4-Tb external usb hard drive, >> >>> labeled MSD2, using this command: >> >>> >> >>> sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 >> >>> /media/default/MSD1/ /media/default/MSD2 >> >> >> >> >> >> See 'man 1 rsync'. You have a slash at the end of SRC, but not at the >> end >> >> of DEST. I would add a slash after "MSD2": >> > >> > The only thing I find in rsync's man page about trailing slashes >> > in the `dest' argument would be relevant if MSD2 didn't exist (in >> > the OP's case it seems it does, no?) >> >> >> There are four combinations for rsync(1) SRC and DEST vs. trailing >> slashes. I use two -- trailing slashes on SRC and DEST for directories, >> and no trailing slashes on SRC and DEST for single files. The other two >> combinations may "work" under certain circumstances, but they have >> caused me grief in the past and I avoid them as a matter of habit. >> >> >> David >> >> > > Hey guys, sorry for just getting back with you now. > Unfortunately, I am just now recovering from a self-inflicted computer > disaster. > > While fighting with rsync, I did either: > > sudo rsync -aAXHSxvv --delete --info=progress2,stats2,name2 > /media/default/MSD1/ /media/default/MSD2 > or > sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 > /media/default/MSD1/ /media/default/MSD2/ > > Just one small problem: MSD2 was not connected to my computer! > (Don't say it . . . ) > > Instead of giving an error message, rsync just created a directory on my > computer called /media/defaultMSD2, and filled it up until my / partition > was full, and THEN my desktop environment (Cinnamon) popped up a > notification saying so. How thoughtful. > > The computer then would not reboot into the operating system. > > No problem, I say. I will just use Timeshift to restore from its backup of > a few hours earlier. > > But that did not work, even after deleting the extra directory, and trying > restores from multiple Timeshift backups. > > Anyway, I never could fix the problem. But I did take it as an opportunity > to "start over". I put in a new(er) SSD, and did a fresh install, replacing > Cinnamon with Gnome. Mistake - now I remember why I dislike Gnome, ever > since Gnome 3. Wish I had re-installed Cinnamon, but too late now, out of > time. For now I will just have to grit my teeth and live with it. > > [BTW, yes, I do have all of my data. Backfilling it into my new setup will > no doubt be an ongoing adventure.] > > Anyway, just a few notes about the rsync situation: > > 1) Having or not having a trailing / on the destination directory did not > seem to make any difference in the size of the copy made, or otherwise. > Nevertheless, I intend to heed the advice given to have a trailing / after > both source and destination, or neither, as appropriate. > > 2) Using or not using an "S" option with rsync did not seem to make any > difference, at least concerning the size of the copy made. > > 3) Yes, I really should check into using checksums to avoid "bot rot". > Good advice. > > Finally, Gnome sucks. (Did I mention that?) > > Thanks for the replies. > > Uh oh . . . I just realized, this was supposed to have been sent to debian-user@lists.debian,org - NOT directly to Daivd Christensen. Sorry for the mistake. (Gmail also sucks. At least as far as composing and sending
Re: backing up backups
On 4/10/22 22:15, to...@tuxteam.de wrote: On Sun, Apr 10, 2022 at 09:44:59PM -0700, David Christensen wrote: On 4/10/22 19:58, Default User wrote: Hello! My setup: - single home x86-64 computer running Debian 11 Stable, up to date. - one 4-Tb external usb hard drive to use as a backup device, labeled MSD1. - another identical usb hard drive, labeled MSD2, to use as a copy of the backups on MSD1. - the computer and all storage devices are formatted ext4, not encrypted. - two old Clonezilla disk images from when I installed Debian 11 last year (probably irrelevant). - Timeshift to daily back up system EXCEPT for data directories. - Back in Time to daily back up data directories. - Borgbackup to also daily back up data directories. - Rsync to frequently backup any changed data between the daily Back in Time and Borgbackup backups of data directories, using this command: sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 --exclude-from "/home/default/rsync_exclude_list.txt" /home /media/default/MSD1/rsync_backups_of_host_home_directory_only Each type of backup is in a separate subdirectory on MSD1 (Timeshift, Back in TIme, Rsync, etc.). It "seems" to work okay, BUT . . . Then I try to use rsync to make an identical copy of backup device MSD1 on an absolutely identical 4-Tb external usb hard drive, labeled MSD2, using this command: sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 /media/default/MSD1/ /media/default/MSD2 See 'man 1 rsync'. You have a slash at the end of SRC, but not at the end of DEST. I would add a slash after "MSD2": The only thing I find in rsync's man page about trailing slashes in the `dest' argument would be relevant if MSD2 didn't exist (in the OP's case it seems it does, no?) There are four combinations for rsync(1) SRC and DEST vs. trailing slashes. I use two -- trailing slashes on SRC and DEST for directories, and no trailing slashes on SRC and DEST for single files. The other two combinations may "work" under certain circumstances, but they have caused me grief in the past and I avoid them as a matter of habit. David
Re: backing up backups
On Sun, Apr 10, 2022 at 09:44:59PM -0700, David Christensen wrote: > On 4/10/22 19:58, Default User wrote: > > Hello! > > > > My setup: > > - single home x86-64 computer running Debian 11 Stable, up to date. > > - one 4-Tb external usb hard drive to use as a backup device, labeled MSD1. > > - another identical usb hard drive, labeled MSD2, to use as a copy of the > > backups on MSD1. > > - the computer and all storage devices are formatted ext4, not encrypted. > > - two old Clonezilla disk images from when I installed Debian 11 last year > > (probably irrelevant). > > - Timeshift to daily back up system EXCEPT for data directories. > > - Back in Time to daily back up data directories. > > - Borgbackup to also daily back up data directories. > > - Rsync to frequently backup any changed data between the daily Back in > > Time and Borgbackup backups of data directories, using this command: > > > > sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 --exclude-from > > "/home/default/rsync_exclude_list.txt" /home > > /media/default/MSD1/rsync_backups_of_host_home_directory_only > > > > Each type of backup is in a separate subdirectory on MSD1 (Timeshift, Back > > in TIme, Rsync, etc.). > > > > It "seems" to work okay, BUT . . . > > > > Then I try to use rsync to make an identical copy of backup device MSD1 on > > an absolutely identical 4-Tb external usb hard drive, > > labeled MSD2, using this command: > > > > sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 > > /media/default/MSD1/ /media/default/MSD2 > > > See 'man 1 rsync'. You have a slash at the end of SRC, but not at the end > of DEST. I would add a slash after "MSD2": The only thing I find in rsync's man page about trailing slashes in the `dest' argument would be relevant if MSD2 didn't exist (in the OP's case it seems it does, no?) Cheers -- t signature.asc Description: PGP signature
Re: backing up backups
On 4/10/22 19:58, Default User wrote: Hello! My setup: - single home x86-64 computer running Debian 11 Stable, up to date. - one 4-Tb external usb hard drive to use as a backup device, labeled MSD1. - another identical usb hard drive, labeled MSD2, to use as a copy of the backups on MSD1. - the computer and all storage devices are formatted ext4, not encrypted. - two old Clonezilla disk images from when I installed Debian 11 last year (probably irrelevant). - Timeshift to daily back up system EXCEPT for data directories. - Back in Time to daily back up data directories. - Borgbackup to also daily back up data directories. - Rsync to frequently backup any changed data between the daily Back in Time and Borgbackup backups of data directories, using this command: sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 --exclude-from "/home/default/rsync_exclude_list.txt" /home /media/default/MSD1/rsync_backups_of_host_home_directory_only Each type of backup is in a separate subdirectory on MSD1 (Timeshift, Back in TIme, Rsync, etc.). It "seems" to work okay, BUT . . . Then I try to use rsync to make an identical copy of backup device MSD1 on an absolutely identical 4-Tb external usb hard drive, labeled MSD2, using this command: sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 /media/default/MSD1/ /media/default/MSD2 See 'man 1 rsync'. You have a slash at the end of SRC, but not at the end of DEST. I would add a slash after "MSD2": sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 /media/default/MSD1/ /media/default/MSD2/ I have experienced USB connection failures with external drives, resulting in filesystem corruption. I prefer mobile racks and caddies for my backup drives, and black SATA 6 GBps cables with locking connectors: https://www.startech.com/en-us/hdd/drw150satbk https://www.cablematters.com/pc-187-156-3-pack-straight-60-gbps-sata-iii-cable.aspx Here's the problem. Although the size and number of items in each subdirectory on MSD1 and MSD2 seem to be identical, more space in total seems to be taken up on MSD2 than on MSD1: df -h /dev/sdb1 Filesystem Size Used Avail Use% Mounted on /dev/sdb1 3.6T 68G 3.4T 2% /media/default/MSD1 df -h /dev/sdc1 Filesystem Size Used Avail Use% Mounted on /dev/sdc1 3.6T 69G 3.4T 2% /media/default/MSD2 df /dev/sdb1 Filesystem 1K-blocks Used AvailableUse% Mounted on /dev/sdb1384451788071255588 35778967642% /media/default/MSD1 df /dev/sdc1 Filesystem 1K-blocksUsed AvailableUse% Mounted on /dev/sdc1384451788071906088 35772462642% /media/default/MSD2 I have tried "everything", even re-formatted MSD2 and re-done: sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 /media/default/MSD1/ /media/default/MSD2 but the results were exactly the same. I doubt that this is a problem with hard links, as I am using the "H" option in rsync. Now , pardon me for thinking that a copy of backups should not take up any more or less space than the original. And I consider backups to be much too important to not be absolutely "correct". So, what SHOULD be done so that MSD2 will always be EXACTLY the same as MSD1? Is there a way to do this using rsync? Do you pause Timeshift, Back in Time, Borgbackup, etc., before using rsync(1) to copy MSD1 to MSD2? Have you compared MSD1 to MSD2 using diff(1) afterwards? If diff(1) is happy, I would call it good. If diff(1) finds differences, figure out why. I suppose I could just make images of MSD1, using dd for example. But then each time wouldn't I just be backing up not just the changed data, but all the data, AND backing up all of the free space on MSD1 also. Aside from wasting TONS of space, each time it could take many hours or even days to do. Which means it just won't get done. So much for redundancy! I agree that dd(1) is overkill and impractical for this use-case. So . . . what IS the correct way to make "backups of backups"? For ext4 backups and ext4 copies of backups, I use rsync(1). Are you doing any integrity checking? E.g. generating a checksum file of the backup contents? Without such, you are exposed to bit rot. David
Re: backing up backups
On Sun, Apr 10, 2022 at 11:13 PM David wrote: > On Mon, 11 Apr 2022 at 12:59, Default User > wrote: > > > Then I try to use rsync to make an identical copy of backup device MSD1 > on an absolutely identical 4-Tb external usb hard drive, > > labeled MSD2, using this command: > > > > sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 > /media/default/MSD1/ /media/default/MSD2 > > > > Here's the problem. Although the size and number of items in each > subdirectory on MSD1 and MSD2 seem to be identical, more space in total > seems to be taken up on MSD2 than on MSD1: > > I suggest to try adding option -S to rsync, and see if that makes > any difference. It might only affect the creation of new files, I'm > unsure about that aspect. > Hi David, thanks for the reply. I really don't know anything about sparse blocks, but I will check it out and give it a try.
Re: backing up backups
On Mon, 11 Apr 2022 at 12:59, Default User wrote: > Then I try to use rsync to make an identical copy of backup device MSD1 on an > absolutely identical 4-Tb external usb hard drive, > labeled MSD2, using this command: > > sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 > /media/default/MSD1/ /media/default/MSD2 > > Here's the problem. Although the size and number of items in each > subdirectory on MSD1 and MSD2 seem to be identical, more space in total seems > to be taken up on MSD2 than on MSD1: I suggest to try adding option -S to rsync, and see if that makes any difference. It might only affect the creation of new files, I'm unsure about that aspect.
backing up backups
Hello! My setup: - single home x86-64 computer running Debian 11 Stable, up to date. - one 4-Tb external usb hard drive to use as a backup device, labeled MSD1. - another identical usb hard drive, labeled MSD2, to use as a copy of the backups on MSD1. - the computer and all storage devices are formatted ext4, not encrypted. - two old Clonezilla disk images from when I installed Debian 11 last year (probably irrelevant). - Timeshift to daily back up system EXCEPT for data directories. - Back in Time to daily back up data directories. - Borgbackup to also daily back up data directories. - Rsync to frequently backup any changed data between the daily Back in Time and Borgbackup backups of data directories, using this command: sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 --exclude-from "/home/default/rsync_exclude_list.txt" /home /media/default/MSD1/rsync_backups_of_host_home_directory_only Each type of backup is in a separate subdirectory on MSD1 (Timeshift, Back in TIme, Rsync, etc.). It "seems" to work okay, BUT . . . Then I try to use rsync to make an identical copy of backup device MSD1 on an absolutely identical 4-Tb external usb hard drive, labeled MSD2, using this command: sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 /media/default/MSD1/ /media/default/MSD2 Here's the problem. Although the size and number of items in each subdirectory on MSD1 and MSD2 seem to be identical, more space in total seems to be taken up on MSD2 than on MSD1: df -h /dev/sdb1 Filesystem Size Used Avail Use% Mounted on /dev/sdb1 3.6T 68G 3.4T 2% /media/default/MSD1 df -h /dev/sdc1 Filesystem Size Used Avail Use% Mounted on /dev/sdc1 3.6T 69G 3.4T 2% /media/default/MSD2 df /dev/sdb1 Filesystem 1K-blocks Used AvailableUse% Mounted on /dev/sdb1384451788071255588 35778967642% /media/default/MSD1 df /dev/sdc1 Filesystem 1K-blocksUsed AvailableUse% Mounted on /dev/sdc1384451788071906088 35772462642% /media/default/MSD2 I have tried "everything", even re-formatted MSD2 and re-done: sudo rsync -aAXHxvv --delete --info=progress2,stats2,name2 /media/default/MSD1/ /media/default/MSD2 but the results were exactly the same. I doubt that this is a problem with hard links, as I am using the "H" option in rsync. Now , pardon me for thinking that a copy of backups should not take up any more or less space than the original. And I consider backups to be much too important to not be absolutely "correct". So, what SHOULD be done so that MSD2 will always be EXACTLY the same as MSD1? Is there a way to do this using rsync? I suppose I could just make images of MSD1, using dd for example. But then each time wouldn't I just be backing up not just the changed data, but all the data, AND backing up all of the free space on MSD1 also. Aside from wasting TONS of space, each time it could take many hours or even days to do. Which means it just won't get done. So much for redundancy! So . . . what IS the correct way to make "backups of backups"?
