Re: Backup.

[Thomas Schmitt]

Hi,

pe...@easthope.ca wrote:
> Thanks Thomas.  Ideally I should find time to follow your suggestions
> but already overcommitted to volunteer activities.  I might have to wait
> until -toc_info_type is in a Debian release.

Yeah. Everybody has to follow the own timetable.


So without any implied expectation of a test report by you:

I uploaded the new development snapshot

   https://www.gnu.org/software/xorriso/xorriso-1.5.7.tar.gz

with the new time oriented table-of-content features.

They are for situations where the usually shown volume ids are fewly
informative (like "ISOIMAGE" in each session) but also if it is difficult
to create a search expression for the desired volume id when mounting.

Time oriented table-of-content:

  $ xorriso -indev /dev/sr0 -toc_info_type mtime -toc
  ...
  TOC layout   : Idx ,  sbsector ,   Size , Modification Time
  ISO session  :   1 ,32 ,   1240808s , 2024.06.20.232334
  ISO session  :   2 ,   1240864 , 29797s , 2024.06.21.220651
  ISO session  :   3 ,   1270688 , 20484s , 2024.06.23.225019
  ISO session  :   4 ,   1291200 , 28928s , 2024.06.24.224429
  ISO session  :   5 ,   1320128 , 21352s , 2024.06.25.223943
  ISO session  :   6 ,   1341504 , 30352s , 2024.06.26.223934
  ISO session  :   7 ,   1371872 , 29023s , 2024.06.27.225617
  ISO session  :   8 ,   1400896 , 31463s , 2024.06.28.232751
  ISO session  :   9 ,   1432384 , 34008s , 2024.06.30.223451
  ISO session  :  10 ,   1466400 , 20329s , 2024.07.01.220901
  Media summary: 10 sessions, 1486544 data blocks, 2903m data, 1579m free

Creating mount command lines by time constraints:

  $ xorriso -mount_cmd /dev/sr0 not_after 2024.06.26.06 /mnt/iso
  ...
  mount -t iso9660 -o nodev,noexec,nosuid,ro,sbsector=1320128 '/dev/sr0' 
'/mnt/iso'

The defined time constraints are:
  "at_time", "before", "after", "not_before", "not_after"

Time can be given in various formats, among them 'June 26' and
'June 26 2024'. See man xorriso, command -alter_date and "Examples of
input timestrings". If you become too perky, expect rejections like:
  xorriso : SORRY : -mount: Cannot decode timestring 'Rapunzel'

Complaints about unfulfillable time constraints will tell the internally
used seconds since 1970:
  $ xorriso -mount_cmd /dev/sr0 before "June 01 2024" /mnt/iso
  ...
  libisoburn: FAILURE : Failed to find "before" "=1717192800"

"=" is the time_t indicator for xorriso.
Program "date" digests the string if "=" gets replaced by "@":
  $ date -d @1717192800
  Sat Jun  1 00:00:00 CEST 2024


Have a nice day :)

Thomas

Re: Backup.

[eben]

On 6/30/24 10:42, pe...@easthope.ca wrote:
>  From: e...@gmx.us
>  Date: Thu, 27 Jun 2024 15:52:44 -0400
>> On one computer I use rsync ...
>
> See reply to Eduardo.

Ah, you mean this one:

On 6/30/24 10:36, pe...@easthope.ca wrote:
>  From: Eduardo M KALINOWSKI 
>  Date: Thu, 27 Jun 2024 16:06:18 -0300
>> rnapshot
>
>>From https://rsnapshot.org/
>> rsnapshot is a filesystem snapshot utility ...
>
> Rather than a snapshot of the extant file system, I want to keep a
> history of the files in the file system.

rsync does do individual files.  Not sure what it's on about.  I can do
cp /backup/2024-06-25_13:55:24/x/y/z .
(assuming I did a backup then).

--
Scientist A: A matterbaby is a very unstable particle.
Scientist B: What's a matterbaby?
Scientist A: I'm doing fine honey, how you doing? -- mrshowrules on Fark

Re: Backup

[Andy Smith]

On Sun, Jun 30, 2024 at 08:19:54AM -0700, pe...@easthope.ca wrote:
> From: Andy Smith 
> Date: Sun, 30 Jun 2024 14:21:45 +
> > What is this person doing?
> 
> Keeping a historical backup by an efficient method. 

While refusing to look into any modern backup system designed by
experienced people for that express goal, and instead writing tomes
of navel gazing to debian-user. It's not my cup of tea, but have
fun; as this and other threads demonstrate you will still find a
large number of willing playmates.

Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Backup

[peter]

From: Andy Smith 
Date: Sun, 30 Jun 2024 14:21:45 +
> What is this person doing?

Keeping a historical backup by an efficient method. 
https://lists.debian.org/debian-user/2024/06/msg00780.html

> Just use borgbackup, restic, amanda or even rsnapshot (ancient but
> still functional).
 
Ref. URL above.  I've used xorriso for several years.  If you advise 
against it, please explain why.

Thx, ... P.


-- 
VoIP:   +1 604 670 0140
work: https://en.wikibooks.org/wiki/User:PeterEasthope

Re: Backup.

[Anssi Saari]

pe...@easthope.ca writes:

> I'm beginning to learn Git.  So I wonder about another approach where 
> files are in a local Git repository.  That would allow  tracing the 
> history of any file.  A backup of the extant repository would still be 
> necessary.  

bup is a backup application using git. I like it because it can add
error correction codes generated by par2 in the backup.

Re: Backup.

[peter]

From: e...@gmx.us
Date: Thu, 27 Jun 2024 15:52:44 -0400
> On one computer I use rsync ...

See reply to Eduardo.

Thx,   ... P.

-- 
VoIP:   +1 604 670 0140
work: https://en.wikibooks.org/wiki/User:PeterEasthope

Re: Backup.

[peter]

From: Jerome BENOIT 
Date: Thu, 27 Jun 2024 21:53:44 +0200
> why did you not use something as backup2l ?

Unaware of it.

From: https://github.com/gkiefer/backup2l
> The restore function allows to easily restore the state of the file 
> system or arbitrary directories/files of previous points in time. 
> ...
> An integrated split-and-collect function allows to comfortably 
> transfer all or selected archives to a set of CDs or other removable 
> media.

Appears relevant.  To catch subtleties, will need to work with it. Two 
phases of workflow may be required.

(1) backup2l records filesystem to rewritable medium such as HDD.

(2) Copy to optical medium.  Each copy may require a fresh optical 
disk. Thomas Schmidt's xorriso appends data to a disk.  Medium is used 
more efficiently..

Thanks,   ... P.

-- 
VoIP:   +1 604 670 0140
work: https://en.wikibooks.org/wiki/User:PeterEasthope

Re: Backup.

[tomas]

On Sun, Jun 30, 2024 at 02:21:45PM +, Andy Smith wrote:
> Hello,

[...]

> Git has some properties that are desirable for general backup
> purposes, but also some fairly huge downsides. For example:
> 
> - It's not efficient or performant for storing large binary files.

[...]

Plus, it doesn't record file ownership (for general backup,
this *is* important).

I'm a fan of rsync. If you want to keep more than one generation,
--link-dest or --compare-dest, as has been stated elsewhere in
this thread, are your friends.

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: Backup.

[Andy Smith]

Hello,

On Sun, Jun 30, 2024 at 06:40:03AM -0700, pe...@easthope.ca wrote:
> I'm beginning to learn Git.  So I wonder about another approach where 
> files are in a local Git repository.  That would allow  tracing the 
> history of any file.  A backup of the extant repository would still be 
> necessary.  
> 
> I don't know the software well enough to compare the two approaches.

Git has some properties that are desirable for general backup
purposes, but also some fairly huge downsides. For example:

- It's not efficient or performant for storing large binary files.
  As a result, several extensions and external programs around git
  exist for getting large binary files into git. Trying to use git
  for general purpose backups will run up against this unless you
  never want to back up large binary files.

- Git stores full (compressed) copies of every version of every
  file. Most backup solutions do better on space.

- Git has no built in way to purge old content. It keeps it all. A
  typical requirement for backup software is to have a bounded limit
  on the oldest versions that will be kept, and quite often there
  are more complex requirements such as "keep daily copies for a
  month, week;y copies for 6 months, monthly copies for 6 years"
  etc. Very hard to do with git.

My first thought when I read the post that started this thread was,

"What is this person doing? If the goal is to have a real world
project to learn some programming techniques and have fun, fair
enough, but if the goal here is to have a decent backup scheme
why are they not using any of the existing excellent solutions
that have thought of and solved so many of the problems in this
space?"

That did not seem like it would be a welcome response at the time so
I held my tongue, but if you are now thinking of looking in to using
git for the purpose, I think it's a wrong step, and in saying so I
might as well say the other as well.

Just use borgbackup, restic, amanda or even rsnapshot (ancient but
still functional).

Unless you are wanting to have a first hand learning experience
about why you should just use borgbackup, restic, amanda or even
rsnapshot.

(Also I think learning about git is best done by using it for what
it was designed for: managing source code in git.)

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Backup.

[Michael Kjörling]

On 30 Jun 2024 06:40 -0700, from pe...@easthope.ca:
>> You could give the backups volume ids which tell the date.
>> 
>>  -volid BOB_"$(date '+%Y_%m_%d_%H%M%S')"
>> 
>> (BOB = Backup Of Backup :))
> 
> I'm beginning to learn Git.  So I wonder about another approach where 
> files are in a local Git repository.  That would allow  tracing the 
> history of any file.  A backup of the extant repository would still be 
> necessary.  

That sounds a lot like etckeeper, except on a larger scale.

-- 
Michael Kjörling  https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

Re: Backup.

[peter]

From: "Thomas Schmitt" 
Date: Fri, 28 Jun 2024 23:35:31 +0200
> You could give the backups volume ids which tell the date.
> 
>  -volid BOB_"$(date '+%Y_%m_%d_%H%M%S')"
> 
> (BOB = Backup Of Backup :))

I'm beginning to learn Git.  So I wonder about another approach where 
files are in a local Git repository.  That would allow  tracing the 
history of any file.  A backup of the extant repository would still be 
necessary.  

I don't know the software well enough to compare the two approaches.

Thx,  ... P.

-- 
VoIP:   +1 604 670 0140
work: https://en.wikibooks.org/wiki/User:PeterEasthope

Re: Backup.

[peter]

From: "Thomas Schmitt" 
Date: Fri, 28 Jun 2024 23:35:31 +0200
> I am working on a solution for your non-unique volume id situation
> by optionally referring to modification timestamps.
> A new command -toc_info_type can switch -toc away from showing volume ids:
> 
>   $ xorriso -indev /dev/sr0  -toc_info_type mtime  -toc
>   xorriso 1.5.7 : RockRidge filesystem manipulator, libburnia project.
>   ...
>   Media current: DVD+RW
>   ...
>   Volume id: 'HOME_Z_2024_06_27_225526'
>   ...
>   TOC layout   : Idx ,  sbsector ,   Size , Modification Time
>   ISO session  :   1 ,32 ,   1240808s , 2024.06.20.232334
>   ISO session  :   2 ,   1240864 , 29797s , 2024.06.21.220651
>   ISO session  :   3 ,   1270688 , 20484s , 2024.06.23.225019
>   ISO session  :   4 ,   1291200 , 28928s , 2024.06.24.224429
>   ISO session  :   5 ,   1320128 , 21352s , 2024.06.25.223943
>   ISO session  :   6 ,   1341504 , 30352s , 2024.06.26.223934
>   ISO session  :   7 ,   1371872 , 29023s , 2024.06.27.225617
>   Media summary: 7 sessions, 1400744 data blocks, 2736m data, 1746m free
> 
> This is a zisofs compressed backup which happens every evening except
> saturdays.
> Note the time difference between 2024_06_27_225526 and 2024.06.27.225617.
> These 51 seconds where spent between program start and begin of writing.
> 
> This program enhancement is already committed to git.
> In a few days there will be a new GNU xorriso 1.5.7 tarball, which is
> easy to build and to test without any danger of frankendebianing.

Thanks Thomas.  Ideally I should find time to follow your suggestions  
but already overcommitted to volunteer activities.  I might have to wait 
until -toc_info_type is in a Debian release.

Thx,  ... P.

-- 
VoIP:   +1 604 670 0140
work: https://en.wikibooks.org/wiki/User:PeterEasthope

Re: Backup.

[Thomas Schmitt]

Hi,

pe...@easthope.ca wrote:
> >   xorriso -for_backup -dev /dev/sr0 \
> > Finding a file as it existed months or years ago can be tedious

I wrote:
> > You could give the backups volume ids which tell the date.

pe...@easthope.ca wrote:
> Thanks.  I should have added that when you mentioned a few
> years ago.

I am working on a solution for your non-unique volume id situation
by optionally referring to modification timestamps.
A new command -toc_info_type can switch -toc away from showing volume ids:

  $ xorriso -indev /dev/sr0  -toc_info_type mtime  -toc
  xorriso 1.5.7 : RockRidge filesystem manipulator, libburnia project.
  ...
  Media current: DVD+RW
  ...
  Volume id: 'HOME_Z_2024_06_27_225526'
  ...
  TOC layout   : Idx ,  sbsector ,   Size , Modification Time
  ISO session  :   1 ,32 ,   1240808s , 2024.06.20.232334
  ISO session  :   2 ,   1240864 , 29797s , 2024.06.21.220651
  ISO session  :   3 ,   1270688 , 20484s , 2024.06.23.225019
  ISO session  :   4 ,   1291200 , 28928s , 2024.06.24.224429
  ISO session  :   5 ,   1320128 , 21352s , 2024.06.25.223943
  ISO session  :   6 ,   1341504 , 30352s , 2024.06.26.223934
  ISO session  :   7 ,   1371872 , 29023s , 2024.06.27.225617
  Media summary: 7 sessions, 1400744 data blocks, 2736m data, 1746m free

This is a zisofs compressed backup which happens every evening except
saturdays.
Note the time difference between 2024_06_27_225526 and 2024.06.27.225617.
These 51 seconds where spent between program start and begin of writing.

This program enhancement is already committed to git.
In a few days there will be a new GNU xorriso 1.5.7 tarball, which is
easy to build and to test without any danger of frankendebianing.
I'll give you a note.

Currently i am working on a mount gesture which shall automate the task
of picking the youngest backup before a given time point:

   -mount /dev/sr0 not_after 2024.06.22.12 /mnt/iso

I think "not_after" is a concise way to say "younger_or_exactly_that_age".
Up to the next release i could take better proposals. :))

(I am having my fun with time zones. The ISO 9660 timestamps can have a
time zone from their maker and the computer running xorriso has a local
time zone. Argh ...)


> >  -assert_volid 'BOB_*' fatal

> Prevents me from appending to a DVD from another backup system. But I
> need to add it when beginning a blank DVD.

You may use it also on a DVD which already has sessions with non-unique
ids. This needs two editing operations in your script.

Begin by adding the volume id command to the backup-of-backup run:

  -volid BOB_"$(date '+%Y_%m_%d_%H%M%S')"

Then do an add-on backup to the DVD with the boring ids.

After this session add the check command:

  -assert_volid 'BOB_*' fatal

In the further backup runs, the DVD is supposed to pass this check.


Have a nice day :)

Thomas

Re: Backup.

[peter]

Hello Thomas & all,

From: "Thomas Schmitt" 
Date: Thu, 27 Jun 2024 22:49:10 +0200
> You could give the backups volume ids which tell the date.

Thanks.  I should have added that when you mentioned a few 
years ago.

> This would also make it possible to verify that the medium is either an
> appendable BOB or blank. Before -dev you would insert:
> 
>  -assert_volid 'BOB_*' fatal

Prevents me from appending to a DVD from another backup system. But I 
need to add it when beginning a blank DVD.

Thanks.  Will be sure that works before unwrapping one of the 500 GB 
drives.

   ... P.

-- 
VoIP:   +1 604 670 0140
work: https://en.wikibooks.org/wiki/User:PeterEasthope

Re: Backup.

[eben]

On 6/27/24 15:52, e...@gmx.us wrote:
> When I boot the file server (possibly today but definitely tomorrow) I'll
> post my backup script.

OK, it's pretty long so I won't post the whole thing, but the important
lines are

rsyncoptions="--archive --progress --verbose --recursive"

"$rsync" $rsyncoptions --hard-links --link-dest "$previous" "$source" "$current"

I have it create destination dirs like

/backup/2024-01-04_22:26:35/
/backup/2024-01-12_20:21:02/
/backup/2024-01-20_23:19:11/
/backup/2024-01-26_23:04:31/
/backup/2024-02-01_22:13:38/
/backup/2024-02-14_00:12:37/
/backup/2024-03-25_23:52:52/
/backup/2024-05-06_22:47:31/
/backup/2024-05-12_22:08:31/
/backup/2024-06-12_23:39:44/

so $previous and $current are two of those, and $source is /files .

--
"Never go off on tangents, which are lines that intersect a [circle] at
only one point and were discovered by Euclid, who lived in the [4th C
BC], which was an era dominated by the Goths, who lived in what we now
know as Poland." - from the Nov. 1998 issue of _Infosystems Executive_.

Re: Backup.

[CHRIS M]

On Thursday 27 June 2024 03:49:03 PM (-05:00), Thomas Schmitt wrote:

> Hi,
> 
> pe...@easthope.ca wrote:
> > This function is applied every week or two to write to a DVD.
> >  xorriso -for_backup -dev /dev/sr0 \
> >  -update_r . / \
> >  -commit \
> >  -toc -check_md5 failure -- \
> >  -eject all ;
> >
> > Finding a file as it existed months or years ago can be tedious.
> 
> You could give the backups volume ids which tell the date.
> 
>  -volid BOB_"$(date '+%Y_%m_%d_%H%M%S')"
> 
> (BOB = Backup Of Backup :))
> This would also make it possible to verify that the medium is either an
> appendable BOB or blank. Before -dev you would insert:
> 



< SNIP >

Did anyone else get this reply twice?

Thanks,
Chris
-- 
Sent with Vivaldi Mail. Download Vivaldi for free at vivaldi.com

Re: Backup.

[Thomas Schmitt]

Hi,

pe...@easthope.ca wrote:
> This function is applied every week or two to write to a DVD.
>  xorriso -for_backup -dev /dev/sr0 \
>  -update_r . / \
>  -commit \
>  -toc -check_md5 failure -- \
>  -eject all ;
>
> Finding a file as it existed months or years ago can be tedious.

You could give the backups volume ids which tell the date.

 -volid BOB_"$(date '+%Y_%m_%d_%H%M%S')"

(BOB = Backup Of Backup :))
This would also make it possible to verify that the medium is either an
appendable BOB or blank. Before -dev you would insert:

 -assert_volid 'BOB_*' fatal

As a whole:

  xorriso -for_backup \
  -assert_volid 'BOB_*' fatal \
  -dev /dev/sr0 \
  -volid BOB_"$(date '+%Y_%m_%d_%H%M%S')" \
  -update_r . / \
  -commit \
  -toc -check_md5 failure -- \
  -eject all

Then -toc can later tell the backup date.
I do daily backups with "HOME_" instead of "BOB_":

  Media current: BD-RE
  ...
  ISO session  :   1 ,32 ,   2129457s , HOME_2024_02_10_152530
  ISO session  :   2 ,   2129504 , 26203s , HOME_2024_02_11_151813
  ...
  ISO session  : 138 ,   8090176 , 34330s , HOME_2024_06_26_152446
  ISO session  : 139 ,   8124512 , 50534s , HOME_2024_06_27_152907
  Media summary: 139 sessions, 8172847 data blocks, 15.6g data, 7643m free

which gives me the opportunity to guess the volume id by date:

  sudo osirrox -mount /dev/sr4 volid '*_2024_04_30_*' /mnt/iso

With less regular dates it would be helpful if the user could wish for
"youngest_before_given_date". I will ponder ...

For now you would have to apply manual work or your own shell magic based
on the output of
  xorriso -indev /dev/sr0 -toc
in order to find the appropriate sbsector number, unless you can describe
the date string by a shell parser expression.


> Now I have a pair of 500 GB external USB drives.  Large compared to my
> working data of ~3 GB.  Please suggest improvements to my backup
> system by exploiting these drives.  I can imagine a complete copy of A
> onto an external drive for each backup; but with most files in A not
> changing during the backup interval, that is inefficient.

You could let xorriso do a daily update.
E.g. on a file of the mounted read-write filesystem on the external disk

  destination=/mnt/backup_disk/xorriso_daily.iso
  ...
  xorriso ... \
  -dev "$destination" \
  ...
  -not_leaf 'Trap*' \
  -not_leaf '*.mp3' \
  -not_leaf '*.mp4' \
  -update_r "$source" / \
  ...

(-eject all would be futile. :))

You could easily have one or more ISOs with history and one or more rsync
mirror trees in the same filesystem. It is always good to keep one valid
backup untouched when the other gets endangered by writing.

(If xorriso -update_r is too slow compared to rsync, consider xorriso
command -disk_dev_ino as in the man page example.)

You could also dedicate the whole plain disk device without any prepared
partitions or filesystems (i do this with USB sticks):
  xorriso ... -dev stdio:/dev/sdd ...
or a partition without prepared filesystem:
  xorriso ... -dev stdio:/dev/sdd1 ...
In the beginning you would have to pseudo-blank the device file, so that
xorriso does not find data in its first few kB which would let it declare
it as "closed":
  xorriso -outdev stdio:/dev/sdd -blank fast
(dd-ing 64 KiB of /dev/zero would have the same effect.)


Have a nice day :)

Thomas

Re: Backup.

[Paul M Foster]

On Thu, Jun 27, 2024 at 04:06:18PM -0300, Eduardo M KALINOWSKI wrote:

> On 27/06/2024 15:23, pe...@easthope.ca wrote:
> > Now I have a pair of 500 GB external USB drives.  Large compared to my
> > working data of ~3 GB.  Please suggest improvements to my backup
> > system by exploiting these drives.  I can imagine a complete copy of A
> > onto an external drive for each backup; but with most files in A not
> > changing during the backup interval, that is inefficient.
> 
> rnapshot

Rsnapshot is written in Perl and is based on an article:
http://www.mikerubel.org/computers/rsync_snapshots/

I read that article a while back, and not knowing of the existence of
rsnapshot, I modified my existing bash/rsync backup script to use this
guy's methods. In essence, rather than backing up the same file again with
rsync, you just make a hard link to it. All "copies" of that file on the
backup link to that one copy. Saves a tremendous amount of room. I keep 7
backup directories on my backup drives (one for each day), and a cron job
fires of the backup each day. 

You can take a look at the script at:

https://gitlab.com/paulmfoster/bkp

Paul

-- 
Paul M. Foster
Personal Blog: http://noferblatz.com
Company Site: http://quillandmouse.com
Software Projects: https://gitlab.com/paulmfoster

Re: Backup.

[Jerome BENOIT]

Hello,
why did you not use something as backup2l ?
Best wishes,
Jerome


On 27/06/2024 20:23, pe...@easthope.ca wrote:

Hi,

My working data is in a directory we can refer to as A. A is on a
removable flash store.  "du -hs /home/me/A" reports 3.0G.  I want a
reliable backup of most files A/*.

I created a directory "Backup" on the HDD and apply this shell
function whenever motivated.

Backup() { \
   if [ $# -gt 1 ]; then
 echo "Too many arguments.";
   else
 echo "0 or 1 arguments are OK.";
 source="/home/me/A/*";
 echo "source is $source.";
 if [ $# -eq 0 ]; then
   echo "0 arguments is OK.";
   destination=/home/me/Backup;
   echo "destination is $destination.";
 else
   echo "1 argument is OK.";
   destination=/home/me/$1;
   echo "destination is $destination.";
 fi;
   echo "Executing sync and rsync.";
   sync;
   rsync \
   --exclude='Trap*' \
   --exclude='*.mp3' \
   --exclude='*.mp4' \
   -auv $source $destination ;
  /bin/ls -ld $destination/MailMessages;
  printf "du -hs $destination => ";
  du -hs $destination;
   fi;
}

When the flash store fails, work since the last execution of Backup
can be lost.

In case the Backup directory on the HDD is lost or I want to see an
old file not current in A, I want backups of Backup.  This function is
applied every week or two to write to a DVD.

FilesToDVD () { \
   printf "Insert open or new DVD-R.";
   read t;
   startPath=$PWD;
   echo "startPath is $startPath";
   source=/home/me/Backup;
   echo "source is $source";
   cd $source;
   xorriso -for_backup -dev /dev/sr0 \
   -update_r . / \
   -commit \
   -toc -check_md5 failure -- \
   -eject all ;
   cd $startPath ;
   echo "  xorriso -dev /dev/sr0 -toc";
   echo "  mount -o sbsector=nn  /dev/sr0 /mnt/iso"; }
   
Finding a file as it existed months or years ago can be tedious.  For

example, find A/MailMessages as it was at 2023.02.07.  Otherwise the
backup system works well.

Now I have a pair of 500 GB external USB drives.  Large compared to my
working data of ~3 GB.  Please suggest improvements to my backup
system by exploiting these drives.  I can imagine a complete copy of A
onto an external drive for each backup; but with most files in A not
changing during the backup interval, that is inefficient.

Thanks,... Peter E.

Re: Backup.

[Michael Kjörling]

On 27 Jun 2024 16:06 -0300, from edua...@kalinowski.com.br (Eduardo M 
KALINOWSKI):
>> Now I have a pair of 500 GB external USB drives.  Large compared to my
>> working data of ~3 GB.  Please suggest improvements to my backup
>> system by exploiting these drives.  I can imagine a complete copy of A
>> onto an external drive for each backup; but with most files in A not
>> changing during the backup interval, that is inefficient.
> 
> rnapshot

Yes, rsnapshot.

Which is essentially a front-end to rsync --link-dest; so, for
mostly-static data, very efficient.

-- 
Michael Kjörling  https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

Re: Backup.

[eben]

On 6/27/24 14:23, pe...@easthope.ca wrote:


Finding a file as it existed months or years ago can be tedious.  For
example, find A/MailMessages as it was at 2023.02.07.  Otherwise the
backup system works well.


On one computer I use rsync to do what appear to be complete backups, only
files identical to the previous run are hard-linked.  This appears to fail
sometimes (not sure under what conditions), so I wrote scripts to find
identical files and link them.  I think the scripts are too complex for my
pea brain to debug them successfully, so there are errors.  But they seem to
work.


I can imagine a complete copy of A onto an external drive for each
backup; but with most files in A not changing during the backup interval,
that is inefficient.


When rsync works, that is exactly what it fixes.

When I boot the file server (possibly today but definitely tomorrow) I'll
post my backup script.

--
Most people don't even know what sysadmins do, but trust me,
if they all took a lunch break at the same time they wouldn't
make it to the deli before you ran out of bullets protecting
your canned goods from roving bands of mutants. -- Peter Welch

Re: Backup.

[Eduardo M KALINOWSKI]

On 27/06/2024 15:23, pe...@easthope.ca wrote:

Now I have a pair of 500 GB external USB drives.  Large compared to my
working data of ~3 GB.  Please suggest improvements to my backup
system by exploiting these drives.  I can imagine a complete copy of A
onto an external drive for each backup; but with most files in A not
changing during the backup interval, that is inefficient.


rnapshot


--
Mike:   "The Fourth Dimension is a shambles?"
Bernie: "Nobody ever empties the ashtrays.  People are SO inconsiderate."
-- Gary Trudeau, "Doonesbury"

Eduardo M KALINOWSKI
edua...@kalinowski.com.br

Re: Backup système

[Sébastien Dinot]

Bonsoir,

Alex PADOLY a écrit :
> Je recherche donc un outil qui privilégie, la fiabilité de la
> sauvegarde, la simplicité, la prise en charge du MBR et donc de GRUB
> et bien entendu la restauration.

Ayant lu cette phrase, je vais me permettre de ne pas répondre à la
question et d'interroger le besoin. Si vous souhaitez sauvegarder le MBR
et tutti quanti, c'est que vous souhaitez faire une image du disque, et
non sauvegarder le système. Or, de manière générale, faire une image du
disque me semble être la plus mauvaise des solutions.

Certes, en sauvegardant l'image du disque, vous sauvegardez tout, donc
vous n'oubliez rien. Mais vous sauvegardez l'utile, comme l'inutile, les
données comme ce qui n'en est pas. Ce faisant, la sauvegarde va
surconsommer de l'espace disque sur le système de sauvegarde. Elle va
être bien plus longue qu'une sauvegarde intelligente de plus haut niveau
qui, en ayant une compréhension de ce qu'elle sauvegarde, pourrait être
sélective et optimiser le processus.

Du coup, pourquoi vouloir agir à si bas niveau ?

Le seul cas de figure où un « ghost » me semble pertinent est celui de
l'administration d'un parc de machines semblables, sur lesquelles on
souhaite pouvoir (ré)installer un système à l'identique à partir de
l'image du disque de la première machine installée. Dans ce cas, on peut
même décider de restaurer périodiquement le système, même en l'absence
de problème déclaré, pour nettoyer le système de potentielles scories ou
cochonneries laissées par les utilisateurs précédents. Mais les gens qui
adoptent cette approche ne le font pas de manière récurrente. Et du
coup, soit ils ne souhaitent sauvegarder ni le dernier état du système,
ni les éventuelles données enregistrées par les utilisateurs, soit ces
données sont sauvegardées d'une autre manière (et le ghost ne traite
alors pas le problème le plus essentiel qui est la sauvegarde des
données).

Je comprends bien que la restauration « bête et méchante » d'une image
disque est plus rapide que la réinstallation classique d'un système,
même automatisée. Mais en revanche, la copie du disque ne peut se faire
qu'à froid. Elle nécessite l'arrêt de la machine. Cela me semble un peu
« lourdingue » et pas automatisable du tout.

À titre personnel, même si la restauration (évènement exceptionnel)
prend plus de temps, je préfère au quotidien disposer d'un système de
sauvegarde :

* efficace et performant, qui ne sauvegarde que ce qui doit l'être ;
* dédupliquant les données, afin d'optimiser la sauvegarde en temps et
  en disque ;
* me permettant d'effectuer des restauration sélectives (besoin le plus
  courant dans la pratique) ;
* ne nécessitant pas l'arrêt du système et affecte le moins possible le
  fonctionnement des services (même si l'arrêt des services le temps de
  sauvegarder les données associées peut être nécessaire dans certains
  cas pour assure l'intégrité de ces données) ;
* qui me permette d'avoir un long historique de sauvegarde ;
* qui chiffre mes données ;
* qui est automatisable.

Sébastien

-- 
Sébastien Dinot, sebastien.di...@free.fr
http://www.palabritudes.net/
Ne goutez pas au logiciel libre, vous ne pourriez plus vous en passer !

Re: Backup système

[benoit]

Le mardi 5 décembre 2023 à 18:18, jafiv  a écrit :

> TimeShift, idem en graphique

J'utilise TimeShift, pour mon usage personnel, c'est super facile et j'ai déjà 
eu besoin de restaurer mon système, ça s'est très bien passé !

Benoit

>

Re: Backup système

[Erwann Le Bras]

j'avais compris que c'était pour pouvoir restaurer une machine entière ; 
un outil universel en ligne de commande est l'idéal.


*Déjà Dup* pourrait t'intéresser. Je ne l'utilise pas, mais il serait 
dans les dépôts. https://launchpad.net/deja-dup
même si plus rien ne fonctionne, il permettrait d'utiliser duplicity en 
ligne de commande pour restaurer.


Erwann

Le 06/12/2023 à 20:57, Belaïd a écrit :

Duplicati, duplicity mais pour les gens qui préfèrent le mode graphique !

Le mer. 6 déc. 2023 à 18:11, Erwann Le Bras 
 a écrit :


bonjour

j'utilise duplicity en ligne de commande. Disponible dans les
dépôts, simple d'utilisation.

Le 05/12/2023 à 16:28, Alex PADOLY a écrit :


Bonsoir à tous,


Sous Debian, j'ai installé et configuré un serveur LTSP, j'ai
appris beaucoup de choses.

Cela n'a pas été facile, c'est pour cette raison que je souhaite
effectuer un backup de mon système

pour pouvoir le restaurer à l'identique en cas de problème ou si
un paquet pose des problèmes.


Il existe beaucoup de logiciels, mais je souhaite me reposer sur
vos expériences personnelles et professionnelles.

CLONEZILLA ne m'a pas donné de satisfaction en termes de simplicité.

Je recherche donc un outil qui privilégie, la fiabilité de la
sauvegarde, la simplicité, la prise en charge du MBR et donc de
GRUB et bien entendu la restauration.

Outils ne signifie pas obligatoirement un logiciel, cela peut
être aussi des commandes GNU.


Merci pour vos conseils.

Re: Backup système

[Belaïd]

Duplicati, duplicity mais pour les gens qui préfèrent le mode graphique !

Le mer. 6 déc. 2023 à 18:11, Erwann Le Bras  a
écrit :

> bonjour
>
> j'utilise duplicity en ligne de commande. Disponible dans les dépôts,
> simple d'utilisation.
> Le 05/12/2023 à 16:28, Alex PADOLY a écrit :
>
> Bonsoir à tous,
>
>
> Sous Debian, j'ai installé et configuré un serveur LTSP, j'ai appris
> beaucoup de choses.
>
> Cela n'a pas été facile, c'est pour cette raison que je souhaite effectuer
> un backup de mon système
>
> pour pouvoir le restaurer à l'identique en cas de problème ou si un paquet
> pose des problèmes.
>
>
> Il existe beaucoup de logiciels, mais je souhaite me reposer sur vos
> expériences personnelles et professionnelles.
>
> CLONEZILLA ne m'a pas donné de satisfaction en termes de simplicité.
>
> Je recherche donc un outil qui privilégie, la fiabilité de la sauvegarde,
> la simplicité, la prise en charge du MBR et donc de GRUB et bien entendu la
> restauration.
>
> Outils ne signifie pas obligatoirement un logiciel, cela peut être aussi
> des commandes GNU.
>
>
> Merci pour vos conseils.
>
>
>
>

Re: Backup système

[jafiv]

Le 05/12/2023 à 16:39, Klaus Becker a écrit :

Fsarchiver ? Très simple et fiable, en ligne de commande.

Bye
Klaus


TimeShift, idem en graphique

Mes 2 cts

Re: backup inmutable

[Luis Muñoz Fuente]

El 14/9/23 a las 13:15, Luciano Andino escribió:
> ¡Buenas! Bueno resulta que los ransomware están a full y parece que no
> respetan ningún sistema operativo. Mi consulta es: ¿conocen una solución
> libre (o de última no tan costosa) para poder realizar backups que sean
> inmutables?

Hola:
Yo tengo un guión de bash que con rsync mantiene un espejo del
directorio a respaldar, y todos los archivos que borre o modifique van a
un directorio que se crea en cada copia. De esta manera puedo recuperar
fácilmente los datos directamente desde el espejo y también tengo copias
de los ficheros borrados o modificados con la fecha de cada respaldo.
Así, si borro accidentalmente mi disco y luego hago la copia, tendría
todos los ficheros en el directorio de la copia. No es inmutable pero
mantiene muchas copias, porque lo ejecuto todos los días con anacron.

Saludos

Re: backup inmutable

[Gonzalo Rivero]

El 14/9/23 a las 08:15, Luciano Andino escribió:
¡Buenas! Bueno resulta que los ransomware están a full y parece que no 
respetan ningún sistema operativo. Mi consulta es: ¿conocen una 
solución libre (o de última no tan costosa) para poder realizar 
backups que sean inmutables?


no entiendo bien a que te referís con inmutable ¿o sea como hacer una 
copia para archivar y no cambie nunca? pues volvería a algún medio de 
solo lectura, tipo cdr o dvdr.


No se que tan práctico sea pero si están almacenados correctamente 
deberían durar mucho, o al menos lo suficiente para que sea un backup 
demasiado viejo como para ser útil.

Re: Backup systems

[David Christensen]

On 9/5/23 07:34, Michael Kjörling wrote:

On 4 Sep 2023 13:57 -0700, from dpchr...@holgerdanske.com (David Christensen):

* I am using zfs-auto-snapshot(8) for snapsnots.  Are you using rsnapshot(1)
for snapshots?


No. I'm using ZFS snapshots on the source, but not for backup
purposes. (I have contemplated doing that, but it would increase
complexity a fair bit.) The backup target is not snapshotted at the > block 
storage or file system level; however, rsync --link-dest uses
hardlinks to deduplicate whole files.



+1 for complexity of ZFS backups via snapshots and replication.


My question was incongruous, as "snapshot" has different meanings for 
ZFS and rsnapshot(1):


*   https://docs.oracle.com/cd/E18752_01/html/819-5461/ftyue.html

snapshot

A read-only copy of a file system or volume at a given point in
time.

*   https://rsnapshot.org/rsnapshot/docs/docbook/rest.html

Using rsnapshot, it is possible to take snapshots of your
filesystems at different points in time.


As I understand your network topology and backup strategy, it appears 
that you are using rsnapshot(1) for snapshots (in the rsnapshot(1) sense 
of the term).




* du(1) of the backup file system matches ZFS properties 'referenced' and
'usedbydataset'.


This would be expected, depending on exact specifics (what data du
traverses over and what your ZFS dataset layout is). To more closely
match the the _apparent_ size of the files, you'd look at e.g.
logicalreferenced or logicalused.


* I am unable to correlate du(1) of the snapshots to any ZFS properties --
du(1) reports much more storage than ZFS 'usedbysnapshots', even when scaled
by 'compressratio'.


This would also be expected, as ZFS snapshots are copy-on-write and
thus in effect only bookkeep a delta, whereas du counts the apparent
size of all files accessible under a path and ZFS snapshots allow
access to all files within the file system as they appeared at the
moment the snapshot was created. There are nuances and caveats
involved but, as a first approximation, immediately after taking a ZFS
snapshot the size of the snapshot is zero (plus a small amount of
metadata overhead for the snapshot itself) regardless of the size of
the underlying dataset, and the apparent size of the snapshot grows as
changes are made to the underlying dataset which cause some data to be
referenced only by the snapshot.

In general, ZFS disk space usage accounting for snapshots is really
rather non-intuitive, but it does make more sense when you consider
that ZFS is a copy-on-write file system and that snapshots largely
boil down to an atomic point-in-time marker for dataset state.



Okay.  My server contains one backup ZFS file system for each host on my 
network.  So, the 'logicalreferenced', 'logicalused', and 
'usedbysnapshots' properties I posted for one host's backup file system 
are affected by the ZFS pool aggregate COW, compression, and/or 
deduplcation features.




(In ZFS, a dataset can be either a file system optionally exposed at a
directory mountpoint or a volume exposed as a block device.)



I try to use ZFS vocabulary per the current Oracle WWW documentation 
(but have found discrepancies).  I wonder if ZFS-on-Linux and/or OpenZFS 
have diverged (e.g. 'man zfs' on Debian, etc.):


https://docs.oracle.com/cd/E18752_01/html/819-5461/ftyue.html

"A generic name for the following ZFS components: clones, file
systems, snapshots, and volumes."


David

Re: Backup systems

[Michael Kjörling]

On 4 Sep 2023 13:57 -0700, from dpchr...@holgerdanske.com (David Christensen):
> * I am using zfs-auto-snapsnot(8) for snapsnots.  Are you using rsnapsnot(1)
> for snapshots?

No. I'm using ZFS snapshots on the source, but not for backup
purposes. (I have contemplated doing that, but it would increase
complexity a fair bit.) The backup target is not snapshotted at the
block storage or file system level; however, rsync --link-dest uses
hardlinks to deduplicate whole files.


> * du(1) of the backup file system matches ZFS properties 'referenced' and
> 'usedbydataset'.

This would be expected, depending on exact specifics (what data du
traverses over and what your ZFS dataset layout is). To more closely
match the the _apparent_ size of the files, you'd look at e.g.
logicalreferenced or logicalused.


> * I am unable to correlate du(1) of the snapshots to any ZFS properties --
> du(1) reports much more storage than ZFS 'usedbysnapshots', even when scaled
> by 'compressratio'.

This would also be expected, as ZFS snapshots are copy-on-write and
thus in effect only bookkeep a delta, whereas du counts the apparent
size of all files accessible under a path and ZFS snapshots allow
access to all files within the file system as they appeared at the
moment the snapshot was created. There are nuances and caveats
involved but, as a first approximation, immediately after taking a ZFS
snapshot the size of the snapshot is zero (plus a small amount of
metadata overhead for the snapshot itself) regardless of the size of
the underlying dataset, and the apparent size of the snapshot grows as
changes are made to the underlying dataset which cause some data to be
referenced only by the snapshot.

In general, ZFS disk space usage accounting for snapshots is really
rather non-intuitive, but it does make more sense when you consider
that ZFS is a copy-on-write file system and that snapshots largely
boil down to an atomic point-in-time marker for dataset state.

(In ZFS, a dataset can be either a file system optionally exposed at a
directory mountpoint or a volume exposed as a block device.)

-- 
Michael Kjörling  https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

Re: Backup systems

[David Christensen]

On 9/4/23 00:53, Michael Kjörling wrote:

On 3 Sep 2023 14:20 -0700, from dpchr...@holgerdanske.com (David Christensen):

Without seeing a console session, I am unsure what you mean by "physically
stored", "total logical (excluding effects of compression) data", and "hot
current logical data ... (excluding things like ZFS snapshots and
compression)".


"Physically stored" is how much data, after compression and including
file system metadata, is actually written to disk and necessary for
all data to be accessible; it's the relevant metric for whether I need
to add disk space.

"Logical" is the sum of all apparent file sizes as visible to userland
utilities e.g. through stat(2).

Something like `dd if=/dev/zero of=$(mktemp) bs=1M count=1M` would
result in a large logical size but, because of compression, a very
small amount of physically stored data.

"Hot" is perhaps better referred to as the "current" data set; since
snapshots (and earlier backups) can include data which has since been
deleted, and is thus no longer current but still exists on disk.



What partitioning scheme, volume manager, file system, compression, etc., do
you use on your backup server?


ZFS within LUKS containers. If I recall correctly, the backup pool is
set to use zstd compression.



I had thought you were using rsnapsnot/ rsync --link-dest, but you also
mention ZFS snapshots.  Please clarify.


Mostly ZFS with a rotating snapshot schedule on the source (the root
file system is ext4); copied using rsync --link-dest (through
rsnapshot) to a ZFS file system which doesn't use snapshots on the
backup target. Most of the ZFS file systems are set up to use
compression; there are a few where I know _a priori_ that the data is
in effect completely incompressible so there's no point in using CPU
to even try to compress that data, so those have compression turned
off.

(In ZFS, creating a file system is barely any more involved than
creating a directory, and all file systems come out of the same "pool"
which is a collection of >=1 storage devices set up with some
particular method of redundancy, possibly none. In more traditional
*nix parlace, a *nix file system is conceptually closer to a ZFS
pool.)

Hopefully this is more clear.



So for backup storage:

* We are both using ZFS with default compression.

* You are using 'rsync --link-dest' (via rsnapshot(1)) for deduplication 
and I am using ZFS for deduplication.


Related:

* I am using zfs-auto-snapsnot(8) for snapsnots.  Are you using 
rsnapsnot(1) for snapshots?



Here are the current backups for my current daily driver:

2023-09-04 13:26:15 toor@f3 ~
# zfs get -o property,value 
compression,compressratio,dedup,logicalreferenced,logicalused,refcompressratio,referenced,used,usedbydataset,usedbysnapshots 
p3/backup/taz.tracy.holgerdanske.com

PROPERTY   VALUE
compressionlz4
compressratio  2.14x
dedup  verify
logicalreferenced  6.59G
logicalused48.7G
refcompressratio   1.83x
referenced 3.89G
used   23.4G
usedbydataset  3.89G
usedbysnapshots19.5G

2023-09-04 13:26:36 toor@f3 ~
# ls -1 /var/local/backup/taz.tracy.holgerdanske.com/.zfs/snapshot | wc -l
 186

2023-09-04 13:27:15 toor@f3 ~
# du -hs /var/local/backup/taz.tracy.holgerdanske.com/ 
/var/local/backup/taz.tracy.holgerdanske.com/.zfs

3.9G/var/local/backup/taz.tracy.holgerdanske.com/
722G/var/local/backup/taz.tracy.holgerdanske.com/.zfs

2023-09-04 13:28:02 toor@f3 ~
# crontab -l
 9 3 * * * /usr/local/sbin/zfs-auto-snapshot -k d 40
21 3 1 * * /usr/local/sbin/zfs-auto-snapshot -k m 99
27 3 1 1 * /usr/local/sbin/zfs-auto-snapshot -k y 99


Observations:

* du(1) of the backup file system matches ZFS properties 'referenced' 
and 'usedbydataset'.


* I am unable to correlate du(1) of the snapshots to any ZFS properties 
-- du(1) reports much more storage than ZFS 'usedbysnapshots', even when 
scaled by 'compressratio'.



David

Re: Backup systems

[Michael Kjörling]

On 3 Sep 2023 14:20 -0700, from dpchr...@holgerdanske.com (David Christensen):
>> 8.07 TiB physically stored on one backup drive holding 174 backups;
>> 11.4 TiB total logical (excluding effects of compression) data on the
>> source; 7.83 TiB hot current logical data on the source (excluding
>> things like ZFS snapshots and compression).
> 
> Without seeing a console session, I am unsure what you mean by "physically
> stored", "total logical (excluding effects of compression) data", and "hot
> current logical data ... (excluding things like ZFS snapshots and
> compression)".

"Physically stored" is how much data, after compression and including
file system metadata, is actually written to disk and necessary for
all data to be accessible; it's the relevant metric for whether I need
to add disk space.

"Logical" is the sum of all apparent file sizes as visible to userland
utilities e.g. through stat(2).

Something like `dd if=/dev/zero of=$(mktemp) bs=1M count=1M` would
result in a large logical size but, because of compression, a very
small amount of physically stored data.

"Hot" is perhaps better referred to as the "current" data set; since
snapshots (and earlier backups) can include data which has since been
deleted, and is thus no longer current but still exists on disk.


> What partitioning scheme, volume manager, file system, compression, etc., do
> you use on your backup server?

ZFS within LUKS containers. If I recall correctly, the backup pool is
set to use zstd compression.


> I had thought you were using rsnapsnot/ rsync --link-dest, but you also
> mention ZFS snapshots.  Please clarify.

Mostly ZFS with a rotating snapshot schedule on the source (the root
file system is ext4); copied using rsync --link-dest (through
rsnapshot) to a ZFS file system which doesn't use snapshots on the
backup target. Most of the ZFS file systems are set up to use
compression; there are a few where I know _a priori_ that the data is
in effect completely incompressible so there's no point in using CPU
to even try to compress that data, so those have compression turned
off.

(In ZFS, creating a file system is barely any more involved than
creating a directory, and all file systems come out of the same "pool"
which is a collection of >=1 storage devices set up with some
particular method of redundancy, possibly none. In more traditional
*nix parlace, a *nix file system is conceptually closer to a ZFS
pool.)

Hopefully this is more clear.

-- 
Michael Kjörling  https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

Re: Backup systems

[David Christensen]

On 9/3/23 03:02, Michael Kjörling wrote:

8.07 TiB physically stored on one backup drive holding 174 backups;
11.4 TiB total logical (excluding effects of compression) data on the
source; 7.83 TiB hot current logical data on the source (excluding
things like ZFS snapshots and compression).

Which by your way of calculating seems to work out to an about 246:1
savings compared to simply keeping every single copy in full and
uncompressed.



Without seeing a console session, I am unsure what you mean by 
"physically stored", "total logical (excluding effects of compression) 
data", and "hot current logical data ... (excluding things like ZFS 
snapshots and compression)".



What partitioning scheme, volume manager, file system, compression, 
etc., do you use on your backup server?



I had thought you were using rsnapsnot/ rsync --link-dest, but you also 
mention ZFS snapshots.  Please clarify.



David

Re: Backup systems

[Michael Kjörling]

On 2 Sep 2023 14:49 -0700, from dpchr...@holgerdanske.com (David Christensen):
> So, 693 GB backup size, 98 backups, 67 TB apparent total backup storage, and
> 777 GB actual total backup storage.  So, a savings of about 88:1.
> 
> What statistics are other readers seeing for similar use-cases and their
> backup solutions?

8.07 TiB physically stored on one backup drive holding 174 backups;
11.4 TiB total logical (excluding effects of compression) data on the
source; 7.83 TiB hot current logical data on the source (excluding
things like ZFS snapshots and compression).

Which by your way of calculating seems to work out to an about 246:1
savings compared to simply keeping every single copy in full and
uncompressed. (Which would require almost 2 PB of storage.) But this
figure is a bit exaggerated since there are parts of the backups that
I prune after a while while keeping the rest of that backup, so let's
be very generous and call it maybe a 100:1 savings in practice.

Which is still pretty good for something that only does raw copying
with whole-file deduplication.

I have a wide mix of file sizes and content types; everything from
tiny Maildir message files through photos in the tens of megabytes
range to VM disk image files in the tens of gigabytes range, ranging
from highly compressible to essentially incompressible, and ranging
from files that practically never change after I initially store them
to ones that change all the time.

-- 
Michael Kjörling  https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

Re: Backup systems

[David Christensen]

On 9/2/23 15:26, Michel Verdier wrote:

On 2023-09-02, David Christensen wrote:


What statistics are other readers seeing for similar use-cases and
their backup solutions?


I have 83 backups resulting to 130% of data. So a ratio of 63:1.



Nice.



But because of performance limitation I don't use compression on
backup server.



What partitioning scheme, volume manager, file system, etc., do you use 
on your backup server?



What is the performance limitation?


If you wanted compression on the backup server, how would you implement it?



And also rsync deduplication is less consuming than zfs.



Please define your system and metrics.


David

Re: Backup systems

[David Christensen]

On 9/2/23 12:15, Michel Verdier wrote:

On 2023-09-02, Stefan Monnier wrote:


I switched to Bup a few years ago and saw a significant reduction in the
size of my backups that is partly due to the deduplication *between*
machines (I backup several Debian machines to the same backup
repository) as well as because the deduplication occurs even when I move
files around (most obvious when I move directories filled with large
files like videos or music).


I setup deduplication between hosts with rsnapshot as you do. But it was
a small gain in my case as the larger part was users data, logs and the
like. So always different between hosts. I gain only on system
files. Mainly /etc as I don't backup binaries and libs.
I almost never move large directories. But if needed it's easy to move it
also in rsnapshot directories.



I have a SOHO LAN:

* My primary workstation is Debian Xfce on a 60GB  2.5" SATA SSD with 1G 
boot, 1G swap, and 12G root partitions.  It has one user (myself) with 
minimal home data (e-mail and CVS working directories).  I backup boot 
and root.


* I keep the vast majority of my data on a FreeBSD server with Samba and 
the CVS repository (via SSH) on a ZFS stripe of two mirrors containing 
two 3TB 3.5" SATA HDD's each (e.g. 6TB RAID10).  I backup the Samba data.


* I run rsync(1) and homebrew shell/ Perl scripts on the server to 
backup the various LAN sources to backup destination file system tree on 
the server.  I have enabled ZFS compression on the pool and enabled 
deduplication on the backup tree.



I ran some statistics for the daily driver backups in March.  The 
results were 4.9 GB backup size, 258 backups, 1.2 TB apparent total 
backup storage, and 29.0 GB actual total backup storage.  So, a savings 
of about 42:1:


https://www.mail-archive.com/debian-user@lists.debian.org/msg789807.html


Today, I collected some statistics for the backups of my data on the 
file server:


2023-09-02 14:10:30 toor@f3 ~
# du -hsx /jail/samba/var/local/samba/dpchrist
693G/jail/samba/var/local/samba/dpchrist

2023-09-02 14:11:09 toor@f3 ~
# ls /jail/samba/var/local/samba/dpchrist/.zfs/snapshot | wc -l
  98

2023-09-02 14:13:50 toor@f3 ~
# du -hs /jail/samba/var/local/samba/dpchrist/.zfs/snapshot
 67T/jail/samba/var/local/samba/dpchrist/.zfs/snapshot

2023-09-02 14:19:24 toor@f3 ~
# zfs get 
compression,compressratio,dedup,used,usedbydataset,usedbysnapshots 
p3/ds2/samba/dpchrist | sort

NAME   PROPERTY VALUE  SOURCE
p3/ds2/samba/dpchrist  compression  lz4inherited from p3
p3/ds2/samba/dpchrist  compressratio1.02x  -
p3/ds2/samba/dpchrist  dedupoffdefault
p3/ds2/samba/dpchrist  used 777G   -
p3/ds2/samba/dpchrist  usedbydataset693G   -
p3/ds2/samba/dpchrist  usedbysnapshots  84.2G  -


So, 693 GB backup size, 98 backups, 67 TB apparent total backup storage, 
and 777 GB actual total backup storage.  So, a savings of about 88:1.



What statistics are other readers seeing for similar use-cases and their 
backup solutions?



David

Re: backup de windows avant formatage pour debian

[Daniel Caillibaud]

Le 15/06/23 à 12:23, didier gaumet  a écrit :
> Alors ma réponse ne va probablement pas te plaire, 

Une réponse aussi détaillée, ça me plaît forcément ;-)

> - quand je veux faire de la place sur un PC Windows pour installer une 
> distro, si on veut garder le windows installé, je pense que le plus sûr 
> moyen de faire de la place est de réduire la ou les partitions système 
> et données (pas les fameuses partitions cachées) de Windows, directement 
> dans Windows (de mémoire un truc du genre (à peu près): menu > outils 
> Microsoft > gestion de l'ordinateur > gestion des disques).
> J'ai pas fait ça depuis longtemps (généralement, je supprime windows et 
> je le réinstalle sur mon PC perso) et par le passé y avait peut-êtr des 
> limitations, fallait faire ça en plusieurs passes parce que Win dows ne 
> voulait pas te laisser trop réduire (me souviens plus)

C'est p'tet là que j'ai déconné sur mon PC actuel, me rappelle plus trop si 
j'avais réduit sous
windows ou linux. En tout cas une fois réduit et debian+grub installé il a 
jamais rebooté (il
apparaît bien dans le menu de boot, ça fait semblant de démarrer et ça plante 
plus loin, pas
cherché davantage car j'en avais pas besoin).

> - pour essayer de rendre de nouveau opérationnel ton Windows actuel 
> lement installé, tu peux créer (si tu ne l'as pas déjà) un CD/DVD 
> d'installation/dépannage ou une clé USB. Attention pour la clé , ça doit 
> être fait sous Windows et je ne crois pas avoir par le passé essayé avec 
> succès de transformer sous Linux une ISO Windows téléchargée en clé USB 
> via un outil

Ah ben ça c'est bon à savoir. J'ai passé un bon moment y'a qq mois sur un autre 
PC pour
remettre en route un windows7 qui avait été tassé pour faire de la place à 
linux y'a une paire
d'années, et j'ai jamais réussi à booter sur la clé.

Mon pb est que j'ai pas de windows sous la main, je vais essayer de trouver un 
voisin qui a ça
quand j'en aurai besoin, ça m'évitera de perdre des heures à tenter de le faire 
depuis un
linux, merci.

> La page Microsoft qui te permet de télécharger une ISO Win10:
> https://www.microsoft.com/fr-fr/software-download/windows10ISO
> Une fois que tu ton CD ou ta clé, tu démarres dessus et tu répares les 
> PC plutôt que de faire une installation Windows. Si ça marche: bingo. 
> Sinon, tu peux toi-même utiliser la CLI (Powershell) du CD/clé pour 
> réinstaller le chargeur de démarrage Windows

Alors ça je vais même pas essayer, avec moi du shell windows c'est des heures 
de perdues pour
un résultat nul quasi assuré.

> - pour savoir comment créer un point de sauvegarde système, faire une 
> restauration système, créer une sauvegarde donnes perso, restaurer ces 
> données perso, tout ça sous Windows, tu peux consulter le support 
> Microsoft, quand on creuse un peu, y a pas que support premier niveau, y 
> a aussi des articles assez détaillés:
> https://support.microsoft.com/fr-fr/windows

Ok, donc faut que j'abandonne mon idée de faire un backup (table de partitions 
+ partitions
utilise) du PC windows depuis un live linux… Dommage, ça je savais faire et ça 
prenait pas
plus de 15min.

Je vais pas me lancer dans du backup depuis windows, j'en ai pour des heures à 
trouver comment
ça marche et je vais sûrement oublier un truc qui planterait la restauration.

Je vais plutôt essayer de conserver une clé d'activation retrouvée (je verrai 
quand j'aurai le
PC concerné sous la main).

> - je n'ai pas l'explication détaillée des partitions installées soit par 
> l'installateur Windows, soit par le vendeur (constructeur du PC), mais 
> je pense (hypothèse) qu'il y a une partition de données de récupération 
> (à partir des points de restauration créés)(concerne le système, pas les 
> données utilisateur). Je pense aussi souvent (créée par le constricteur) 
> qu'il y a une partition dédiée pour utiliser une procédure de 
> récupération (l'équivalent du média d'installation utilisé en mode 
> réparation/dépannage)

Merci pour toutes ces explications détaillées !

-- 
Daniel

Echouer, c'est avoir la possibilité de recommencer de manière plus intelligente.
Henry Ford

Re: backup de windows avant formatage pour debian

[Daniel Caillibaud]

Le 15/06/23 à 10:40, Hugues Larrive  a écrit :

> Bonjour,
> 
> Pour récupérer la clé de windows OEM depuis linux, pas besoin de sauvegarde, 
> depuis qu'elle
> n'est plus sur le sticker de licence elle est inscrite dans une mémoire de la 
> carte-mère :
> strings /sys/firmware/acpi/tables/MSDM

MERCI !

J'ai bien une clé.

Donc si je dois rendre mon PC actuel avec son windows, je peux tout formater et 
refaire une
install à partir d'une ISO MS (mises sur une clé depuis un windows) en 
indiquant cette clé (il
la trouvera d'ailleurs peut-être tout seul).

-- 
Daniel

Il faut choisir, dans la vie, entre gagner de l'argent et le dépenser, 
on n'a pas le temps de faire les deux. 
Edouard Bourdet

Re: backup de windows avant formatage pour debian

[didier gaumet]

Le 15/06/2023 à 12:03, Daniel Caillibaud a écrit :
[...]

Mais je vais voir, avec un peu de chance c'est une licence qui permet une 
réinstallation
standard si on a la clé d'activation (dans ce cas j'ai que cette clé à 
conserver, avec une
iso d'install windows10 éventuellement au cas où MS arrêterait de les 
distribuer un jour), je
me pose ces questions justement depuis que j'ai mis HS un windows en réduisant 
sa partition (et
que je sais pas le restaurer car c'est un oem sans clé, seulement un truc 
d'activation mis qq
part dans une partition).

[...]

les explications sur les systèmes d'activation Windows 10:
https://support.microsoft.com/fr-fr/windows/activer-windows-c39005d4-95ee-b91e-b399-2820fda32227#WindowsVersion=Windows_10

Re: backup de windows avant formatage pour debian

[Hugues Larrive]

Bonjour,

Pour récupérer la clé de windows OEM depuis linux, pas besoin de sauvegarde, 
depuis qu'elle n'est plus sur le sticker de licence elle est inscrite dans une 
mémoire de la carte-mère :
strings /sys/firmware/acpi/tables/MSDM

@+
Hugues


--- Original Message ---
Le jeudi 15 juin 2023 à 12:03, Daniel Caillibaud  a écrit 
:


> 

> 

> Le 15/06/23 à 11:32, "ajh-valmer" ajh.val...@free.fr a écrit :
> 

> > On Thursday 15 June 2023 11:08:01 Daniel Caillibaud wrote:
> > 

> > > Avant d'installer debian sur un PC je voudrais faire un backup
> > > du windows qui est dessus, pour le remettre "comme il était
> > > avant" si besoin.
> > 

> > Attention, si on réinstalle Windows à partir de son backup,
> > il ne redémarrera plus (sauf si, peut-être, il est installé
> > exactement sur le même disque dur et partition).
> 

> 

> Oui, c'est pour ça que je veux faire le backup de la table des partitions et 
> des partitions
> indispensables pour pouvoir restaurer, reste à savoir ce qui est 
> indispensable (car en cas
> d'oubli c'est trop tard quand on le réalise).
> 

> Mais je vais voir, avec un peu de chance c'est une licence qui permet une 
> réinstallation
> standard si on a la clé d'activation (dans ce cas j'ai que cette clé à 
> conserver, avec une
> iso d'install windows10 éventuellement au cas où MS arrêterait de les 
> distribuer un jour), je
> me pose ces questions justement depuis que j'ai mis HS un windows en 
> réduisant sa partition (et
> que je sais pas le restaurer car c'est un oem sans clé, seulement un truc 
> d'activation mis qq
> part dans une partition).
> 

> Mais je m'affole peut-être pour rien, j'ai vu qu'on pouvait légalement 
> acheter des clés pour
> pas grand chose, donc je peux jeter le windows existant et on verra bien un 
> jour pour en
> remettre un autre si la question se pose (peut-être jamais).
> 

> --
> Daniel
> 

> Les femmes seront l'égal de l'homme
> le jour où elles accepteront d'être
> chauves et de trouver ça distingué.
> Coluche

publickey - hlarrive@pm.me - 0xE9429B87.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Re: backup de windows avant formatage pour debian

[didier gaumet]

Le 15/06/2023 à 11:08, Daniel Caillibaud a écrit :

Bonjour,

Avant d'installer debian sur un PC je voudrais faire un backup du windows qui 
est dessus, pour
le remettre "comme il était avant" si besoin.

Je sais pas encore s'il n'a qu'une partition système ou plusieurs (je pense à 
une partition de
restauration), en plus de la partition efi (probable, il est pas très vieux).

J'imaginais booter sur une clé usb d'installeur debian (à priori debian11, 
quitte à
upgrader en 12 ensuite) ou SystemRescueCd ou clonezilla live si besoin d'outils 
plus pointus, et
de mettre les images des partitions sur une 2e clé usb (j'en ai une de 16G, 
j'espère que ça
suffira).

À priori je pensais faire
- backup table des partitions avec sgdisk
- backup partition efi et partition windows avec partclone (normalement il ne 
backup que les
   block utilisés, j'espère rentrer dans mes 16G)

Ça suffirait ?

Merci




PS: Question subsidiaire liée à la précédente, la restauration d'un windows 
chafoin depuis une
install debian…
(c'est ça qui m'a amené à la quesstion précédente, si windows marche plus quand on le 
"tasse",
inutile de le garder dans un coin avec un double boot, autant le virer d'office)

Si la procédure précédente fonctionne, il faudrait que je rende mon PC actuel, 
si possible
comme il était avant qu'on me le file.

C'est un dell avec un windows10 oem (sans n° de licence, il "reconnait" qu'il 
est sur le bon
hardware et peut s'activer au boot, je sais plus comment ça s'appelle chez MS), 
qui veut plus
booter depuis que je lui ai réduit sa partition pour mettre debian. Je ne m'en 
suis jamais
préoccupé parce que j'en avais pas besoin, mais va falloir s'y coller…

Voici la table de partition du disque ssd (y'en a un 2e qui ne contient que du 
linux, ajouté
après l'install oem de windows)

Model: BC511 NVMe SK hynix 512GB (nvme)
Disk /dev/nvme0n1: 512GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End SizeFile system  Name  Flags
  1  1049kB  263MB   262MB   fat32EFI system partition  
boot, esp
  2  263MB   397MB   134MBMicrosoft reserved partition  
msftres
  3  397MB   71,6GB  71,2GB  ntfs Basic data partition  
msftdata
  7  71,6GB  72,1GB  500MB   ext4
  8  72,1GB  491GB   419GB
  4  491GB   492GB   1038MB  ntfs   
hidden, diag
  5  492GB   510GB   18,3GB  ntfs   
hidden, diag
  6  510GB   512GB   1710MB  ntfs   
hidden, diag

7 est /boot et 8 une partition luks (avec du lvm dedans) contenant debian.

Supprimer 7 & 8 puis agrandir 3 avec l'espace libéré suffirait à ce que windows 
retrouve ses
esprits ? (comme un idiot j'ai pas fait de backup de la table des partitions 
avant d'avoir
tassé windows)

Une idée de ce que peuvent être les partitions 2/4/5/6 ? (une partition de 
restauration je veux
bien, mais les 3 autres c'est quoi ?)



Bonjour,

Alors ma réponse ne va probablement pas te plaire, mais mon expérience 
personnelle (donc pas universelle ni forcément reproductible) me conduit 
à généralement gérer les problèmes Linux sous Linux, les problèmes BSD 
sous BSD et les problèmes Windows sous Windows, sous peine d'ennuis 
supplémentaires


- Windows 10 en 64 bits demande 32G (et généralement on parle plutôt de 
40G) d'espace disque minimum:

https://www.microsoft.com/fr-fr/windows/windows-10-specifications

- quand je veux faire de la place sur un PC Windows pour installer une 
distro, si on veut garder le windows installé, je pense que le plus sûr 
moyen de faire de la place est de réduire la ou les partitions système 
et données (pas les fameuses partitions cachées) de Windows, directement 
dans Windows (de mémoire un truc du genre (à peu près): menu > outils 
Microsoft > gestion de l'ordinateur > gestion des disques).
J'ai pas fait ça depuis longtemps (généralement, je supprime windows et 
je le réinstalle sur mon PC perso) et par le passé y avait peut-êtr des 
limitations, fallait faire ça en plusieurs passes parce que Win dows ne 
voulait pas te laisser trop réduire (me souviens plus)


- pour essayer de rendre de nouveau opérationnel ton Windows actuel 
lement installé, tu peux créer (si tu ne l'as pas déjà) un CD/DVD 
d'installation/dépannage ou une clé USB. Attention pour la clé , ça doit 
être fait sous Windows et je ne crois pas avoir par le passé essayé avec 
succès de transformer sous Linux une ISO Windows téléchargée en clé USB 
via un outil

La page Microsoft qui te permet de télécharger une ISO Win10:
https://www.microsoft.com/fr-fr/software-download/windows10ISO
Une fois que tu ton CD ou ta clé, tu démarres dessus et tu répares les 
PC plutôt que de faire une installation Windows. Si ça marche: bingo. 
Sinon, tu peux toi-même utiliser la CLI (Powershell) du CD/clé pour 
réinstaller le chargeur de démarrage Windows


- pour savoir 

Re: backup de windows avant formatage pour debian

[Daniel Caillibaud]

Le 15/06/23 à 11:32, "ajh-valmer"  a écrit :
> On Thursday 15 June 2023 11:08:01 Daniel Caillibaud wrote:
> > Avant d'installer debian sur un PC je voudrais faire un backup 
> > du windows qui est dessus, pour le remettre "comme il était 
> > avant" si besoin.  
> 
> Attention, si on réinstalle Windows à partir de son backup,
> il ne redémarrera plus (sauf si, peut-être, il est installé
> exactement sur le même disque dur et partition).

Oui, c'est pour ça que je veux faire le backup de la table des partitions et 
des partitions
indispensables pour pouvoir restaurer, reste à savoir ce qui est indispensable 
(car en cas
d'oubli c'est trop tard quand on le réalise).

Mais je vais voir, avec un peu de chance c'est une licence qui permet une 
réinstallation
standard si on a la clé d'activation (dans ce cas j'ai que cette clé à 
conserver, avec une
iso d'install windows10 éventuellement au cas où MS arrêterait de les 
distribuer un jour), je
me pose ces questions justement depuis que j'ai mis HS un windows en réduisant 
sa partition (et
que je sais pas le restaurer car c'est un oem sans clé, seulement un truc 
d'activation mis qq
part dans une partition).

Mais je m'affole peut-être pour rien, j'ai vu qu'on pouvait légalement acheter 
des clés pour
pas grand chose, donc je peux jeter le windows existant et on verra bien un 
jour pour en
remettre un autre si la question se pose (peut-être jamais).

-- 
Daniel

Les femmes seront l'égal de l'homme
le jour où elles accepteront d'être
chauves et de trouver ça distingué.
Coluche

Re: backup de windows avant formatage pour debian

[ajh-valmer]

On Thursday 15 June 2023 11:08:01 Daniel Caillibaud wrote:
> Avant d'installer debian sur un PC je voudrais faire un backup 
> du windows qui est dessus, pour le remettre "comme il était 
> avant" si besoin.

Attention, si on réinstalle Windows à partir de son backup,
il ne redémarrera plus (sauf si, peut-être, il est installé
exactement sur le même disque dur et partition).

Bonne journée.

Re: Backup ... men hvordan

[Povl Ole Haarlev Olsen]

On Mon, 19 Dec 2022, Flemming Bjerke wrote:
Mit spørgsmål var i virkeligheden: Hvor længe gemmer I backups, efter hvilket 
skema? Og hvor tit laver I full backup, og hvor tit incrementel?


Det kommer nok helt an på hvad det er for nogle data og hvilke regler, man 
er underlagt.


Er der ikke noget med, at firmaer skal gemme nogle regneskabsdata 5 eller 
10 år tilbage?


Af nogle filesystems på nogle af mine maskiner, laver jeg en daglig backup 
med mit backup script. Af /home på nogle af mine maskiner, laver jeg en 
backup hver time. Jeg bruger rsync --link-dest, så jeg ved ikke helt om du 
vil kalde det en full backup eller incremental. Hver backup dir har alle 
data fra det givne tidspunkt, men pladsen per backup dir (undtagen den 
første backup) svarer til en incremental backup.


Det er forskelligt hvor længe jeg gemmer data. Filesystemer, der næsten 
ikke ændrer sig, koster ikke ret meget mere plads, når hver backup er med 
--link-dest til forrige backup, så der kan jeg uden store problemer have 
backups liggende i et år eller mere. Mere aktive filesystemer, hvor filer 
tit ændrer sig eller bliver flyttet, koster lidt mere plads, så det er 
sværere at retfærdigøre pladsforbruget.


Som et par modspørgsmål til dit spørgsmål er:

Hvilken type data?

Er der særlig regler for de data, som du skal overholde?

Hvor tit er der nye/ændrede data?

Hvor meget/lidt plads vil du bruge?

--
Povl Ole

Re: Backup ... men hvordan

[Flemming Bjerke]

Mange tak, for tips som jeg benytter.

Maria-backup er ret simpel at bruge:

To perform additional incremental backups, you can then use the target 
directory of the previous incremental backup as the incremental base 
directory of the next incremental backup. For example:


$ mariabackup --backup \
   --target-dir=/var/mariadb/inc2/ \
   --incremental-basedir=/var/mariadb/inc1/
https://mariadb.com/kb/en/incremental-backup-and-restore-with-mariabackup/

Man skal bare have lavet mappen 'inc2' først, og det jo ikke 
raketvidenskab med cronjob at lave sådan en incrementel kæde af mapper 
som maria-backup løbende hælder filer i.


Mit spørgsmål var i virkeligheden: Hvor længe gemmer I backups, efter 
hvilket skema? Og hvor tit laver I full backup, og hvor tit incrementel?


Jeg kan iøvrigt ikke forestille mig at CVS-problemet ville opstå i 
dag(?) Jeg mindes det vist var forholdsvis sjældent problemet opstod, 
men det var til gengæld ødelæggende - og jeg opgav derfor CVS.


Flemming

Re: Backup ... men hvordan

[Povl Ole Haarlev Olsen]

On Mon, 19 Dec 2022, Flemming Bjerke wrote:

Den 19.12.2022 kl. 05.38 skrev Povl Ole Haarlev Olsen:

On Mon, 19 Dec 2022, Flemming Bjerke wrote:
Men jeg glemmer aldrig da jeg havde brugt srv til backup, og så pludselig 
kunne jeg ikke tilgå de gamle versioner ... og hvorfor ... pga. en eller 
anden kendt bug som ingen havde gjort noget ved. Jeg ville aldrig turde 
lægge
Har du lyst til at fortælle hvilket program, du brugte og hvad bug'en var, 
så vi andre lettere kan checke, om vi måske også har et problem?
Undskyld, det var CVS, og det var dengang git begyndte at blive rigtig 
udbredt ... altså 10-15 år siden ... så det er historie, og jeg husker 
overhovedet ikke hvad buggen gik ud på. Blot at den var beskrevet, og at jeg 
ikke fandt nogen løsninger på nettet, men derimod andre som heller ikke havde 
fundet nogen løsninger.


Interessant, især fordi jeg stadig bruge CVS.

min endelige backup i en krypteret fil som kun et program kan tilgå, og 
hvis
Det lyder meget som noget hjemmerullet kryptering. Man skal selvfølgelig 
bruge en eller anden standard, som nogle kloge mennesker har tænkt længe 
over og som findes i mere end een implementation.
Hjemmerullet kryptering ... nej, så selvovervurderende er jeg alligevel ikke 
;-) Og, det må være rigtigt at hvis man skal have alt liggende krypteret så 
må man have mindst 2 uafhængige implementationer.


Jeg mente ikke nødvendigvis hjemmerullet af dig, men af dem, der har lavet 
det een program, som kan tilgå backup'en.


jeg mistede nøglen ... Jeg får ondt i maven ved tanken. Jeg tænker ikke at 
jeg afskaffer mine to fysiske bakop-harddiske som ikke er koblet på min 
computer til daglig, og hvor intet er krypteret.
Du kunne gemme nøglen på dine to backup harddiske. Medmindre du altså 
regner med at smide alle tre kopier væk samtidig. Men du kan jo også skrive 
nøglen på et papir, som kan ligge i din bankboks. Eller 2/3 af din nøgle 
til tre af dine venner, så ingen af dem kan tilgå backup'en alene. Eller... 
Der er mange løsninger for at undgå, at man mister sin key.

Tak, for gode ideer.


Hvis du vil kigge nærmere på det der med en splittet key, hvor der skal X 
dele til at lave en hel key, så tag et kig på "" pakken.


Umiddelbart virker det som om du har tænkt dig, at lave en kopi af 
filesystemet, hvor mariadb har sine data liggende. Hvis du gør det mens 
mariadb kører, skal du være opmærksom på, at du måske ikke kan bruge din 
backup til noget, f.eks. fordi mariadb kan have ting cachet, som endnu ikke 
er lagt ud i filerne eller fordi der går noget tid mens du kopiere filerne 
og de første filer derfor kan nå at ændre sig, mens du stadig er i gang med 
at kopiere de sidste filer.
Som beskrevet tidligere, havde jeg tænkt at bruge maria-backup som har alt 
muligt indbygget omkring at databsen kører, osv. Problemet er at man for hver 
incrementel backup selv skal lave en ny mappe hvori maria-backup kan lægge 
diverse filer og linke til den tidligere incrementelle mappe, som linker til 
forrige mappe, osv., osv., indtil en full-backup. Jeg kunne så ikke finde 
nogle bud på hvordan man på nettet byggede en klog bakop af mariadb på det 
grundlag. Der er vel en trade-off mellem simpel teknisk bakop og andre hensyn 
såsom praktisk anvendelighed og sårbarhed, eksemplificeret med at jeg ikke 
lige forestiller mig at nogen ville lave en kæde på 730 mapper med 
incrementel bakop af mariadb over 2 år. Men der tager jeg måske fejl?


Jeg bruger, som sagt, ikke mariadb og kender derfor ikke maria-backup, men 
hvis den ikke selv understøtter det, så lyder det som du "bare" skal 
diff'e filerne fra dagens backup med filerne fra igår. De filer, der er 
ens skal så bare erstattes af hardlinks til gårdagens backup (som så igen 
kan være hardlinks til dem fra i forgårs osv.)


Derefter kunne

rsync -H --link-dest ${LAST_TIMESTAMP} ...

være din ven.

Problematikken gælder vel generelt. Vælger man f.eks. 2 års = 730 dages 
incrementel bakop med rdiff-bakop? Eller hvordan gør I andre som jo er 
professionelle?


Ovenstående rsync kommando plus lidt flere options er en del af nogle af 
mine backup scripts.


Den korte version af de scripts er noget a'la:

[- Quote -]
TIMESTAMP=$(date -u +%Y-%m-%d-T-%H-%M)

BACKUP_ID=$1
shift

SRC_DIR=$1
shift

BACKUP_SERVER=$1
shift

DEST_DIR=$1
shift

BACKUP_LOG_DIR=/var/local/log

mkdir -p ${BACKUP_LOG_DIR}

LAST_TIMESTAMP_FILE=${BACKUP_LOG_DIR}/${BACKUP_ID}.timestamp

if [ -e ${LAST_TIMESTAMP_FILE} ]
then
LAST_TIMESTAMP=$(cat ${LAST_TIMESTAMP_FILE})

LINK_DEST="--link-dest ${DEST_DIR}/${LAST_TIMESTAMP}"
fi

rsync \
-HPSavx \
${LINK_DEST} \
${SRC_DIR}/ \
${BACKUP_SERVER}:${DEST_DIR}/${TIMESTAMP}/ \
>${BACKUP_LOG_DIR}/${BACKUP_ID}.${TIMESTAMP}.log 2>&1

echo ${TIMESTAMP} > ${LAST_TIMESTAMP_FILE}
[- End quote -]

Ovenstående forudsætter at den, der kører scriptet (root?) har en ssh-key, 
der kan logge ind på ${BACKUP_SERVER} uden password.


Spørg gerne, hvis der er noget i scriptet, 

Re: Backup ... men hvordan

[Flemming Bjerke]

Den 19.12.2022 kl. 05.38 skrev Povl Ole Haarlev Olsen:

On Mon, 19 Dec 2022, Flemming Bjerke wrote:
Men jeg glemmer aldrig da jeg havde brugt srv til backup, og så 
pludselig kunne jeg ikke tilgå de gamle versioner ... og hvorfor ... 
pga. en eller anden kendt bug som ingen havde gjort noget ved. Jeg 
ville aldrig turde lægge


Har du lyst til at fortælle hvilket program, du brugte og hvad bug'en 
var, så vi andre lettere kan checke, om vi måske også har et problem?


Undskyld, det var CVS, og det var dengang git begyndte at blive rigtig 
udbredt ... altså 10-15 år siden ... så det er historie, og jeg husker 
overhovedet ikke hvad buggen gik ud på. Blot at den var beskrevet, og at 
jeg ikke fandt nogen løsninger på nettet, men derimod andre som heller 
ikke havde fundet nogen løsninger.





min endelige backup i en krypteret fil som kun et program kan tilgå, 
og hvis


Det lyder meget som noget hjemmerullet kryptering. Man skal 
selvfølgelig bruge en eller anden standard, som nogle kloge mennesker 
har tænkt længe over og som findes i mere end een implementation.


Hjemmerullet kryptering ... nej, så selvovervurderende er jeg alligevel 
ikke ;-) Og, det må være rigtigt at hvis man skal have alt liggende 
krypteret så må man have mindst 2 uafhængige implementationer.





jeg mistede nøglen ... Jeg får ondt i maven ved tanken. Jeg tænker 
ikke at jeg afskaffer mine to fysiske bakop-harddiske som ikke er 
koblet på min computer til daglig, og hvor intet er krypteret.


Du kunne gemme nøglen på dine to backup harddiske. Medmindre du altså 
regner med at smide alle tre kopier væk samtidig. Men du kan jo også 
skrive nøglen på et papir, som kan ligge i din bankboks. Eller 2/3 af 
din nøgle til tre af dine venner, så ingen af dem kan tilgå backup'en 
alene. Eller... Der er mange løsninger for at undgå, at man mister sin 
key.


Tak, for gode ideer.




Men det løser ikke mit problem med hvordan man bakker mariadb op. 
Hvad synes I om følgende løsning?


1. Hvert kvartal full backup. Slettes efter 5/4 år.
2. Hver måned incrementel bakop ift. seneste kvartals full backup. 
Slettes efter et år.
3. Hver uge incrementel bakop ift. seneste kvartals full backup, 
slettes efter 1½ måned.
4. Daglig incrementel bakup i forhold til seneste kvartals full 
backup. Slettes efter 10 dage.


Grundlaget skulle være et lille simpelt skript i stil med vedhæftede.


Umiddelbart virker det som om du har tænkt dig, at lave en kopi af 
filesystemet, hvor mariadb har sine data liggende. Hvis du gør det 
mens mariadb kører, skal du være opmærksom på, at du måske ikke kan 
bruge din backup til noget, f.eks. fordi mariadb kan have ting cachet, 
som endnu ikke er lagt ud i filerne eller fordi der går noget tid mens 
du kopiere filerne og de første filer derfor kan nå at ændre sig, mens 
du stadig er i gang med at kopiere de sidste filer.


Som beskrevet tidligere, havde jeg tænkt at bruge maria-backup som har 
alt muligt indbygget omkring at databsen kører, osv. Problemet er at man 
for hver incrementel backup selv skal lave en ny mappe hvori 
maria-backup kan lægge diverse filer og linke til den tidligere 
incrementelle mappe, som linker til forrige mappe, osv., osv., indtil en 
full-backup. Jeg kunne så ikke finde nogle bud på hvordan man på nettet 
byggede en klog bakop af mariadb på det grundlag. Der er vel en 
trade-off mellem simpel teknisk bakop og andre hensyn såsom praktisk 
anvendelighed og sårbarhed, eksemplificeret med at jeg ikke lige 
forestiller mig at nogen ville lave en kæde på 730 mapper med 
incrementel bakop af mariadb over 2 år. Men der tager jeg måske fejl?


Problematikken har jo helt konkret relevans. Jeg lavede en gang en 
wordpress-hjemmeside sammen med nogle andre. Men siden blev cracket, og 
det havde den faktisk været i mere end 14 dage før vi opdagede det. 
Webhotellet (som en af de andre insisterede på var supergode til at 
bakke op) havde desværre kun 14 dages bakop liggende. (Jeg undrede mig 
over hvorfor der ikke en månedlig bakop, f.eks. bare 3 måneder tilbage.) 
Nu var det nogle fjolser der havde cracket siden, så jeg løste problemet 
uden det helt store bøvl ... men det var jo bare heldigt. Så vidt jeg 
har bemærket, er det ikke ualmindeligt at webhoteludbydere tilbyder 
bakop som i virkeligheden kun er få dages bakop.


Problematikken gælder vel generelt. Vælger man f.eks. 2 års = 730 dages 
incrementel bakop med rdiff-bakop? Eller hvordan gør I andre som jo er 
professionelle?


Flemming

Re: Backup ... men hvordan

[Povl Ole Haarlev Olsen]

On Mon, 19 Dec 2022, Flemming Bjerke wrote:
Men jeg glemmer aldrig da jeg havde brugt srv til backup, og så pludselig 
kunne jeg ikke tilgå de gamle versioner ... og hvorfor ... pga. en eller 
anden kendt bug som ingen havde gjort noget ved. Jeg ville aldrig turde lægge


Har du lyst til at fortælle hvilket program, du brugte og hvad bug'en var, 
så vi andre lettere kan checke, om vi måske også har et problem?



min endelige backup i en krypteret fil som kun et program kan tilgå, og hvis


Det lyder meget som noget hjemmerullet kryptering. Man skal selvfølgelig 
bruge en eller anden standard, som nogle kloge mennesker har tænkt længe 
over og som findes i mere end een implementation.


jeg mistede nøglen ... Jeg får ondt i maven ved tanken. Jeg tænker ikke at 
jeg afskaffer mine to fysiske bakop-harddiske som ikke er koblet på min 
computer til daglig, og hvor intet er krypteret.


Du kunne gemme nøglen på dine to backup harddiske. Medmindre du altså 
regner med at smide alle tre kopier væk samtidig. Men du kan jo også 
skrive nøglen på et papir, som kan ligge i din bankboks. Eller 2/3 af din 
nøgle til tre af dine venner, så ingen af dem kan tilgå backup'en alene. 
Eller... Der er mange løsninger for at undgå, at man mister sin key.


Men det løser ikke mit problem med hvordan man bakker mariadb op. Hvad synes 
I om følgende løsning?


1. Hvert kvartal full backup. Slettes efter 5/4 år.
2. Hver måned incrementel bakop ift. seneste kvartals full backup. Slettes 
efter et år.
3. Hver uge incrementel bakop ift. seneste kvartals full backup, slettes 
efter 1½ måned.
4. Daglig incrementel bakup i forhold til seneste kvartals full backup. 
Slettes efter 10 dage.


Grundlaget skulle være et lille simpelt skript i stil med vedhæftede.


Umiddelbart virker det som om du har tænkt dig, at lave en kopi af 
filesystemet, hvor mariadb har sine data liggende. Hvis du gør det mens 
mariadb kører, skal du være opmærksom på, at du måske ikke kan bruge din 
backup til noget, f.eks. fordi mariadb kan have ting cachet, som endnu 
ikke er lagt ud i filerne eller fordi der går noget tid mens du kopiere 
filerne og de første filer derfor kan nå at ændre sig, mens du stadig er i 
gang med at kopiere de sidste filer.


Jeg bruger ikke selv mariadb, men i f.eks. postgresql er der et tool til 
at lave backups af en database, der er i aktiv brug. Jeg regner med, at 
mariadb har et tool i samme stil. Det tool vil dog lave en full backup af 
din database, ikke incremental.


TL;DR: Enten stopper du databasen hverken du laver en backup eller du 
laver en fuld backup hver gang. Eller du har en måske ubrugelig backup...


--
Povl Ole

Re: Backup ... men hvordan

[Flemming Bjerke]

Jeg synes at restic ser godt ud.

Men jeg glemmer aldrig da jeg havde brugt srv til backup, og så 
pludselig kunne jeg ikke tilgå de gamle versioner ... og hvorfor ... 
pga. en eller anden kendt bug som ingen havde gjort noget ved. Jeg ville 
aldrig turde lægge min endelige backup i en krypteret fil som kun et 
program kan tilgå, og hvis jeg mistede nøglen ... Jeg får ondt i maven 
ved tanken. Jeg tænker ikke at jeg afskaffer mine to fysiske 
bakop-harddiske som ikke er koblet på min computer til daglig, og hvor 
intet er krypteret.


Men jeg synes restic ser ud til at være det rigtige til at cron-job 
bakke op i skyen. En Hetzner-storage-box på 1 TB til 3,81€ /md med 
restic-bakop, er da et meget interessant alternativ til alle mulige 
internationale tjenester som næppe fortjener så lige megen tillid som 
restic-udviklerne (og som NSA har adgang til pga. DMCA). Mange tak, for 
den løsning! Og hvad jeg rigtig godt kan lide: Bakop hos Hetzner og 
web/mail-server hos Contabo. Og mail-webserveren smides også lige hjem 
til min raspberry pie! Mine egentlige arbejdsdata kan så med cron-job 
ryge til Hetzner krypteret - og ukrypteret til mine to harddisk.


Men det løser ikke mit problem med hvordan man bakker mariadb op. Hvad 
synes I om følgende løsning?


1. Hvert kvartal full backup. Slettes efter 5/4 år.
2. Hver måned incrementel bakop ift. seneste kvartals full backup. 
Slettes efter et år.
3. Hver uge incrementel bakop ift. seneste kvartals full backup, slettes 
efter 1½ måned.
4. Daglig incrementel bakup i forhold til seneste kvartals full backup. 
Slettes efter 10 dage.


Grundlaget skulle være et lille simpelt skript i stil med vedhæftede.

Bedste hilsener

Flemming



Den 17.12.2022 kl. 14.05 skrev Thomas Damgaard:


Hej Flemming,


Jeg bruger restic (https://restic.net/)

Du kan apt install'e det, men du kan også hente den seneste statiske 
binary fra github.


Restic er backup "gjort rigtigt".

Selve backuppen er krypteret, repository'et sørger for deduplication, 
det understøtter et hav af protokoller (særligt hvis du kombinerer det 
med rclone).


Ift. selve storage-delen, så bruger jeg selv en fysisk server jeg har 
co-located i et datacenter. Men du kan leje forholdsvist billige 
storage boxe, hvis du har brug for meget plads. F.eks. hos Hetzner. 
Jeg har også en enkelt maskine som faktisk bruger Mega som storage 
backend til restic. Det fungerer også fint nok.


Mvh

Thomas



On 2022-12-17 08:16, Flemming Bjerke wrote:


Kære Alle

Hvordan bakker i op? Tidligere har jeg bakket op med rsync skripts. 
Men nu har jeg tænkt mig at bruge rdiff-backup og mariabackup. Det er 
sådan set nemt nok. Bortset fra at mariabackup får en til at tænke 
over hvad man skal med f.eks. 2 års backup bestående af 730 mapper 
der alle linker tilbage til den forrige incrementelle backup og til 
sidst til den oprindelige full backup - og det forkommer også at være 
en ret sårbar backup. Et lignende problem er der vel også med 
rdiff-backup.


Nu er jeg nok ikke den første der har tænkt på dette ;-) Alligevel 
har jeg ledt forgæves efter relevante skrifts o. lign. på nettet.


Hvad gør I?

Flemming

PS: Jeg er indtil videre tilfreds med contabo.com: 60 kr/md for 300 
MB debian 11 VPS med ganske meget ram og cpu. Supporten er ret 
hjælpsom, og jeg har ikke noget at udsætte på den. Jeg havde dog et 
seriøst problem. Jeg havde løbende lavet bakop af /etc, og da noget 
var gået galt, ville jeg lige gå et par dage tilbage, og så purgede 
jeg relevante programpakker. Og så slettede jeg lige (med hovedet 
under armen) /etc og erstattede den med en 2 dage gammel /etc. Det 
skulle jeg ikke have gjort, for så kunne jeg ikke genstarte. Det var 
noget med at boot-processen ikke kunne finde min virtuelle harddisk. 
Jeg (gennem VNC) og supporten prøvede at forklare fstab at den skulle 
bare bruge min virtuelle harddisk, men det ville den ikke høre tale 
om. Det syntes hurtigt lettere at starte forfra og resette debian. Og 
jeg vidste jo faktisk godt at jeg netop ikke kunne regne med at bare 
kunne kopiere hele en gammel version af /etc.



--
Mvh
Thomas




bakop-map.sh
Description: application/shellscript

Re: Backup ... men hvordan

[Thomas Damgaard]

Hej Flemming,


Jeg bruger restic (https://restic.net/)

Du kan apt install'e det, men du kan også hente den seneste statiske 
binary fra github.


Restic er backup "gjort rigtigt".

Selve backuppen er krypteret, repository'et sørger for deduplication, 
det understøtter et hav af protokoller (særligt hvis du kombinerer det 
med rclone).


Ift. selve storage-delen, så bruger jeg selv en fysisk server jeg har 
co-located i et datacenter. Men du kan leje forholdsvist billige storage 
boxe, hvis du har brug for meget plads. F.eks. hos Hetzner. Jeg har også 
en enkelt maskine som faktisk bruger Mega som storage backend til 
restic. Det fungerer også fint nok.


Mvh

Thomas



On 2022-12-17 08:16, Flemming Bjerke wrote:


Kære Alle

Hvordan bakker i op? Tidligere har jeg bakket op med rsync skripts. 
Men nu har jeg tænkt mig at bruge rdiff-backup og mariabackup. Det er 
sådan set nemt nok. Bortset fra at mariabackup får en til at tænke 
over hvad man skal med f.eks. 2 års backup bestående af 730 mapper der 
alle linker tilbage til den forrige incrementelle backup og til sidst 
til den oprindelige full backup - og det forkommer også at være en ret 
sårbar backup. Et lignende problem er der vel også med rdiff-backup.


Nu er jeg nok ikke den første der har tænkt på dette ;-) Alligevel har 
jeg ledt forgæves efter relevante skrifts o. lign. på nettet.


Hvad gør I?

Flemming

PS: Jeg er indtil videre tilfreds med contabo.com: 60 kr/md for 300 MB 
debian 11 VPS med ganske meget ram og cpu. Supporten er ret hjælpsom, 
og jeg har ikke noget at udsætte på den. Jeg havde dog et seriøst 
problem. Jeg havde løbende lavet bakop af /etc, og da noget var gået 
galt, ville jeg lige gå et par dage tilbage, og så purgede jeg 
relevante programpakker. Og så slettede jeg lige (med hovedet under 
armen) /etc og erstattede den med en 2 dage gammel /etc. Det skulle 
jeg ikke have gjort, for så kunne jeg ikke genstarte. Det var noget 
med at boot-processen ikke kunne finde min virtuelle harddisk. Jeg 
(gennem VNC) og supporten prøvede at forklare fstab at den skulle bare 
bruge min virtuelle harddisk, men det ville den ikke høre tale om. Det 
syntes hurtigt lettere at starte forfra og resette debian. Og jeg 
vidste jo faktisk godt at jeg netop ikke kunne regne med at bare kunne 
kopiere hele en gammel version af /etc.



--
Mvh
Thomas

Re: backup directory/file exclusion pattern list for borgbackup

[Default User]

On Sun, Oct 3, 2021 at 2:55 AM Andrei POPESCU  wrote:
>
> On Sb, 02 oct 21, 19:56:02, Default User wrote:
> >
> > And . . .  I make a special point of excluding:
> > /media
> > /media/*
> > /mnt
> > /mnt/*
> >
> > Why?
> >
> > Because long ago, when I was just learning to use rsync, I tried to
> > use it to do a full-system backup.  Since nobody told me that /mnt and
> > /media had to be specifically excluded, rsync did exactly what it was
> > told to do, recursively backing up everything, filling up an entire
> > 1Tb hard drive, stopping only when it ran out of room!
> >
> > I do not ever want that to happen again.
> >
> > Fun fact:
> > I can use the same exclude list to mirror the same user stuff to a
> > different backup directory, with obsessive frequency, using rsync:
> >
> > sudo rsync -avHx --delete --stats --exclude-from
> > "/home/debian-user/rsync_exclude_list.txt" /home
> > /media/debian-user/backup_drive/backups_of_host_home_directory_only
>
> In case of system backups the top-level directories themselves should
> probably be included because they are needed as mount points. rsync's -x
> parameter should already take care of excluding the (special) mounts
> *under* /mnt, /media, /dev, /sys, /run etc.
>
>
> Hope this helps,
> Andrei
> --
> http://wiki.debian.org/FAQsFromDebianUser


Hi Andrei, thanks for the tip!

Re: backup directory/file exclusion pattern list for borgbackup

[Andrei POPESCU]

On Sb, 02 oct 21, 19:56:02, Default User wrote:
> 
> And . . .  I make a special point of excluding:
> /media
> /media/*
> /mnt
> /mnt/*
> 
> Why?
> 
> Because long ago, when I was just learning to use rsync, I tried to
> use it to do a full-system backup.  Since nobody told me that /mnt and
> /media had to be specifically excluded, rsync did exactly what it was
> told to do, recursively backing up everything, filling up an entire
> 1Tb hard drive, stopping only when it ran out of room!
> 
> I do not ever want that to happen again.
> 
> Fun fact:
> I can use the same exclude list to mirror the same user stuff to a
> different backup directory, with obsessive frequency, using rsync:
> 
> sudo rsync -avHx --delete --stats --exclude-from
> "/home/debian-user/rsync_exclude_list.txt" /home
> /media/debian-user/backup_drive/backups_of_host_home_directory_only

In case of system backups the top-level directories themselves should 
probably be included because they are needed as mount points. rsync's -x 
parameter should already take care of excluding the (special) mounts 
*under* /mnt, /media, /dev, /sys, /run etc.


Hope this helps,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser


signature.asc
Description: PGP signature

Re: backup directory/file exclusion pattern list for borgbackup

[Default User]

On Tue, Sep 28, 2021 at 1:54 AM deloptes  wrote:
>
> Default User wrote:
>
> > Hello!
> >
> > I want to try using borgbackup to do backups of my (only) user directory:
> > /home/debian-user
> >
> > I just want to do so using Vorta, a GUI for borgbackup.
> >
> > But I just need a good, general list of directory and file type
> > exclusions that I can just cut and paste into the Exclude Patterns
> > window in Vorta.  Something like the default list of exclusions that
> > appears by default in the Backintime backup program.
> >
>
> I use this. For file type I do not know
>
> borg create --progress --stats --compression zstd,10 \
> -e 'pp:/sys' \
> -e 'pp:/proc' \
> -e 'pp:/dev' \
> -e 'pp:/run' \
> -e 'pp:/tmp' \
> -e 'pp:/var/tmp' \
> -e 'pp:/var/log'
>

---

Hi!

Just an update.

Here is what I came up with, cobbled together, from a number of sources:
*~
*.backup*
**/.cache
/boot/*
/BORG/*
/BORG/.?*
/dev
/dev/*
/dev/.?*
/etc/mtab
/home/*/.cache/
#  /home/*/.cache/chromium/*
/home/*/.cache/mozilla/*
/home/*/.cache/mozilla/firefox/*
#  /home/*/.claws-mail/tmp/*
/home/*/.gvfs
/home/*/.gvfs/*
/home/*/.gvfs/.?*
#  /home/*/.googleearth/Cache/*
/home/*/.opt/
/home/*/.thumbnails/*
/lost+found
/lost+found/*
/lost+found/.?*
/media
/media/*
/mnt
/mnt/*
/proc
/proc/*
/root/.gvfs/*
/root/.gvfs/.?*
/run
/run/*
/sys
/sys/*
/tmp
/tmp/*
/usr/tmp/*
/var/backups/*
/var/cache/*
/var/cache/apt/archives/*.deb
/var/lib/dhcpcd/*
/var/tmp/*
~/.adobe/Flash_Player/AssetCache
~/.cache
~/.ccache
~/.gvfs
#  ~/.local/share/Steam
~/.Private
~/.recent-applications.xbel
~/.recently-used.xbel
#  ~/snap/*/*/.cache
#  ~/.steam/root
~/.thumbnails
~/.var/app/*/cache
~/.xsession-errors
.cache
.cache/*
#  .dropbox*
.gvfs
lost+found/*
.Private
.thumbnails*

This is saved as a text file, which can be altered as needed at any
time, and is copied and pasted into the Exclude Patterns window under
the Sources tab of Vorta.  I want to use Vorta, at least for now,
because it is much more user-friendly  than raw borgbackup.

Note that some of the entries are commented out, as they are not
needed currently, but are there to be uncommented as needed.

Also, there are probably more entries than are really necessary.  I
hope to improve the exclude list over time.

Also note that there are directory entries that probably do not apply
to my immediate use case (backing up user stuff, from
/home/debian-user on down).  For example:
/dev
/dev/*
/dev/.?*
/media
/media/*
/mnt
/mnt/*
/proc
/proc/*
/run
/run/*
/sys
/sys/*
/tmp
/tmp/*
/usr/tmp/*

And some things I just don't know if I need to back them up or not.  Examples:

/BORG/*
/BORG/.?*
/etc/mtab
/home/*/.opt/
/lost+found
/lost+found/*
/lost+found/.?*
lost+found/*
.Private
~/.xsession-errors

Further, the Archlinux wiki rsync article states:
"If GVFS is installed [it is!], /home/*/.gvfs must be excluded to
prevent rsync errors."  Thus, these are on the exclude list:
/home/*/.gvfs
/home/*/.gvfs/*
/home/*/.gvfs/.?*
/root/.gvfs/*
/root/.gvfs/.?*
~/.gvfs
.gvfs

And . . .  I make a special point of excluding:
/media
/media/*
/mnt
/mnt/*

Why?

Because long ago, when I was just learning to use rsync, I tried to
use it to do a full-system backup.  Since nobody told me that /mnt and
/media had to be specifically excluded, rsync did exactly what it was
told to do, recursively backing up everything, filling up an entire
1Tb hard drive, stopping only when it ran out of room!

I do not ever want that to happen again.

Fun fact:
I can use the same exclude list to mirror the same user stuff to a
different backup directory, with obsessive frequency, using rsync:

sudo rsync -avHx --delete --stats --exclude-from
"/home/debian-user/rsync_exclude_list.txt" /home
/media/debian-user/backup_drive/backups_of_host_home_directory_only

Finally, it has been suggested that I could do full system backups
using borgbackup, and doing full system restores, if needed, using
borgbackup from a live SystemRescue cd/usb.
I may try that in the future, as I learn more about borg. It is
complicated, or at least
complex!  But for now I feel secure knowing that I can restore my system
(non-data stuff) at any time, using Timeshift. And 1+ here also, for
SystemRescue.

Well, I don't know if any of this was helpful or even interesting.
But here it is, FWIW.

Thanks for the replies!

Re: backup directory/file exclusion pattern list for borgbackup

[deloptes]

Default User wrote:

> Hello!
> 
> I want to try using borgbackup to do backups of my (only) user directory:
> /home/debian-user
> 
> I just want to do so using Vorta, a GUI for borgbackup.
> 
> But I just need a good, general list of directory and file type
> exclusions that I can just cut and paste into the Exclude Patterns
> window in Vorta.  Something like the default list of exclusions that
> appears by default in the Backintime backup program.
> 

I use this. For file type I do not know

borg create --progress --stats --compression zstd,10 \
-e 'pp:/sys' \
-e 'pp:/proc' \
-e 'pp:/dev' \
-e 'pp:/run' \
-e 'pp:/tmp' \
-e 'pp:/var/tmp' \
-e 'pp:/var/log'

Re: backup directory/file exclusion pattern list for borgbackup

[Kushal Kumaran]

[slightly re-arranged segments below]

On Sat, Sep 25 2021 at 09:03:37 PM, Default User  
wrote:
>
> In Vorta, under the "Sources" tab, there is an area (window) for input
> into which you can type or paste text, such as:
>
> **/.cache
>
> to denote exclusions, that is, things you do not want to back up.
> This is from /home/debian_user/.config/backintime/config:
>
> . . .
> profile1.snapshots.exclude.1.value=.gvfs
> profile1.snapshots.exclude.2.value=.cache/*
> profile1.snapshots.exclude.3.value=.thumbnails*
> profile1.snapshots.exclude.4.value=.local/share/[Tt]rash*
> profile1.snapshots.exclude.5.value=*.backup*
> profile1.snapshots.exclude.6.value=*~
> profile1.snapshots.exclude.7.value=.dropbox*
> profile1.snapshots.exclude.8.value=/proc/*
> profile1.snapshots.exclude.9.value=/sys/*
> profile1.snapshots.exclude.10.value=/dev/*
> profile1.snapshots.exclude.11.value=/run/*
> profile1.snapshots.exclude.12.value=/etc/mtab
> profile1.snapshots.exclude.13.value=/var/cache/apt/archives/*.deb
> profile1.snapshots.exclude.14.value=lost+found/*
> profile1.snapshots.exclude.15.value=/tmp/*
> profile1.snapshots.exclude.16.value=/var/tmp/*
> profile1.snapshots.exclude.17.value=/var/backups/*
> profile1.snapshots.exclude.18.value=.Private
> . . .
>
> Of course that is expressed in backintime's own configuration
> "language", and would probably need to be translated into borgbackup's
> equivalent "language".
> ...
> And that isn't even (afaik) Fnmatch!
> (BTW, I have read what you referenced as ' Run "borg help patterns" '.
> I'm afraid it wasn't very helpful to me.)

I mean you should run the command "borg help patterns" (without quotes)
from a terminal.  That produces detailed explanation of what kinds of
patterns borg supports, including examples.  The same content is also
available in the borg-patterns manpage.

It also mentions the --dry-run option that you can use to try your
patterns out.  fnmatch is similar to shell pattern matching, like the
backintime configuration fragment you've shown above.  You can take
those as-is if you want all of those to apply.  The patterns above that
start with / will not apply to your scenario, where you're backing up
/home/debian-user.

>> [https://defuse.ca/audits/encfs.htm] talks about issues with encfs
>> design.  There is nothing backintime can do to fix those.
>>
>> borg can encrypt its backup images, and it recommendeds enabling that.
>> So an adversary would not get access to the encfs ciphertext directly.
>> They could get access to borg ciphertext instead, which may or may not
>> be vulnerable to the same problems.  AFAIK there hasn't been a security
>> audit of borgbackup itself.  The page at
>> https://borgbackup.readthedocs.io/en/stable/internals/security.html#borgcrypto
>> describes the design of borg security.
>>
>
> I shall take for granted that backintime developers do not code encfs.
> Fine.  But after 7 years (at least), why haven't they replaced encfs
> with a "safer" encryption scheme, or at least just removed it and
> simply not replaced it at all?  IMHO, either option would seem far
> better than the status quo.

I had misunderstood the scenario.  I'd read it as you using backintime
to backup encfs-encrypted content, not realizing that backintime uses
encfs to provide encrypted backups.

-- 
regards,
kushal

Re: backup directory/file exclusion pattern list for borgbackup

[Marco Möller]

On 26.09.21 00:24, Default User wrote:

Hello!

I want to try using borgbackup to do backups of my (only) user directory:
/home/debian-user

I just want to do so using Vorta, a GUI for borgbackup.

But I just need a good, general list of directory and file type
exclusions that I can just cut and paste into the Exclude Patterns
window in Vorta.  Something like the default list of exclusions that
appears by default in the Backintime backup program.

Note 1: borgbackup uses a matching pattern called "Fnmatch" with which
I am not familiar, and don't want to learn by trial and error, losing
data in the process.  Which is why I am looking for a "drop-in" basic
exclude list.

Note 2: I am not intending to use borgbackup to back up the whole
system; just /home/debian-user and its subdirectories.  I am using
timeshift to back up the rest of the system.  Timeshift uses a huge
amount of disk space, but it . . .  works.

Note 3: I am aware that some use backintime to back up user data, and
I have tried it myself.  But it just seems to have some "problems".
For example, the built-in "diff" utility does not seem to do anything.
It seems old and gives the impression of not being heavily developed.
The documentation is "adequate" but mediocre. And what really grinds
my gears about backintime, a problem apparently known as far back as
2014:

"Warning: A recent security audit revealed several possible attack
vectors for EncFs.


From https://defuse.ca/audits/encfs.htm:


EncFS is probably safe as long as the adversary only gets one copy of
the ciphertext and nothing more. EncFS is not safe if the adversary
has the opportunity to see two or more snapshots of the ciphertext at
different times. EncFS attempts to protect files from malicious
modification, but there are serious problems with this feature.

This might be a problem with Back In Time snapshots."

Gee . . .  think so?



The following is an example for what could be in an exclude file for the 
borg command being used at the CLI with the option:

--exclude-from myExcludeFile

,

# The following items will be excluded from the borg backup
# use absolute paths like in: "borg create repo::archive /home/someUserName"
# do NOT use relative paths like in "borg create repo::archive ."
#
# a slash as the last character excludes all contents but not the dir 
name itself

# like this the softlinks are preserved


/home/someUserName/.cache/
/home/someUserName/Downloads/
/home/someUserName/TEMP/

/home/someUserName/.julia/artifacts/
/home/someUserName/.julia/compiled/
/home/someUserName/.julia/conda/
/home/someUserName/.julia/packages/
/home/someUserName/.julia/registries/


/home/someUserName/.opt/



You will see that I personally decided to not include in my Backup some 
quit common folders:

.cache
Downloads
TEMP

You will also see, that I did not exclude a particular single file, only 
complete directories. You could do so, you could include in teh list 
particular files, if of interest to you.


I then have some folders to which I install software relevant only to 
this user, and as this user could anytime reinstall this software, the 
content of these folders do not contain user data or configuration data 
of importance, I decided to not fill my backups with the huge and often 
changing content of these folders:


.opt
.julia/selectionOfReinstallableJuliaFolders

Note that I did not include the complete tree ".julia", because in other 
sub-directories of Julia there is important user data and configuration 
data which I do want to become backuped!


I wouldn't know about a general recommendation about folders which by 
default are recommended for exclusion. You will have to go for the 
effort and personally decide for your very own situation.
If you have an exclusion list which you are happy with from other 
software, you mentioned backintime, maybe you can learn from my above 
example about the borg syntax and reuse the exclusions which backintime 
has configured for you?
Note the in my above example there are comment lines included, 
everything behind the sign "#" is a comment and the comment ends at the 
end of line. These line can be part of the exclude file and do not harm, 
they will simply be ignored when borg searches for the entries of to be 
excluded files or directories. These comments point out some frecuentley 
parts of particular interest when populating a borg exclusion file. 
However, nothing beets reading the original documentation.


Good Luck!
Marco

Re: backup directory/file exclusion pattern list for borgbackup

[Andrei POPESCU]

On Sb, 25 sep 21, 21:03:37, Default User wrote:
> 
> to denote exclusions, that is, things you do not want to back up.
> This is from /home/debian_user/.config/backintime/config:
> 
> . . .
> profile1.snapshots.exclude.1.value=.gvfs
> profile1.snapshots.exclude.2.value=.cache/*
> profile1.snapshots.exclude.3.value=.thumbnails*
> profile1.snapshots.exclude.4.value=.local/share/[Tt]rash*
> profile1.snapshots.exclude.5.value=*.backup*
> profile1.snapshots.exclude.6.value=*~
> profile1.snapshots.exclude.7.value=.dropbox*
> profile1.snapshots.exclude.8.value=/proc/*
> profile1.snapshots.exclude.9.value=/sys/*
> profile1.snapshots.exclude.10.value=/dev/*
> profile1.snapshots.exclude.11.value=/run/*
> profile1.snapshots.exclude.12.value=/etc/mtab
> profile1.snapshots.exclude.13.value=/var/cache/apt/archives/*.deb
> profile1.snapshots.exclude.14.value=lost+found/*
> profile1.snapshots.exclude.15.value=/tmp/*
> profile1.snapshots.exclude.16.value=/var/tmp/*
> profile1.snapshots.exclude.17.value=/var/backups/*
> profile1.snapshots.exclude.18.value=.Private
> . . .
 
Half of those are system directories, so they are irrelevant for your 
use case (backing up a /home directory).
 
> Borg takes up about 4.2 Gb of user data only.
> 
> Backintime uses about 4.4Gb to back up the same user data.  

If I understand correctly Borg takes less space *without* exclusions to 
backup the same data as timeshift *with* exclusions.

What problem are you trying to solve?

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser


signature.asc
Description: PGP signature

Re: backup directory/file exclusion pattern list for borgbackup

[Default User]

On Sat, Sep 25, 2021 at 8:04 PM Kushal Kumaran  wrote:
>
> On Sat, Sep 25 2021 at 06:24:12 PM, Default User  
> wrote:
> > Hello!
> >
> > I want to try using borgbackup to do backups of my (only) user directory:
> > /home/debian-user
> >
> > I just want to do so using Vorta, a GUI for borgbackup.
> >
> > But I just need a good, general list of directory and file type
> > exclusions that I can just cut and paste into the Exclude Patterns
> > window in Vorta.  Something like the default list of exclusions that
> > appears by default in the Backintime backup program.
> >
>
> I don't understand what a general list of exclusions would look like.
> Do you have examples of what backintime excludes by default?  My own
> borgbackup runs backup everything on disk; I don't feel the need to
> exclude anything.
>
> > Note 1: borgbackup uses a matching pattern called "Fnmatch" with which
> > I am not familiar, and don't want to learn by trial and error, losing
> > data in the process.  Which is why I am looking for a "drop-in" basic
> > exclude list.
> >
>
> Run "borg help patterns" to see explanation of how borgbackup deals with
> patterns.  It has this to say about fnmatch:
>
> This is the default style for --exclude and --exclude-from.  These
> patterns use a variant of shell pattern syntax, with '*' matching
> any number of characters, '?' matching any single character, '[...]'
> matching any single character specified, including ranges, and
> '[!...]'  matching any character not specified. For the purpose of
> these patterns, the path separator (backslash for Windows and '/' on
> other systems) is not treated specially. Wrap meta-characters in
> brackets for a literal match (i.e. [?] to match the literal
> character ?). For a path to match a pattern, the full path must
> match, or it must match from the start of the full path to just
> before a path separator. Except for the root path, paths will never
> end in the path separator when matching is attempted.  Thus, if a
> given pattern ends in a path separator, a '*' is appended before
> matching is attempted.
>
> > Note 2: I am not intending to use borgbackup to back up the whole
> > system; just /home/debian-user and its subdirectories.  I am using
> > timeshift to back up the rest of the system.  Timeshift uses a huge
> > amount of disk space, but it . . .  works.
> >
>
> I don't know how timeshift stores backups.  borg uses deduplicated
> storage that avoids storing identical data multiple times.  My own borg
> backups results in ~1G of new data every week (and about the same amount
> being deleted from expiring backups).  There is no significant increase
> in repository size week-over-week.  That obviously would not be the same
> for everyone, but if you're bothered by the amount of disk space used
> you can try it out.
>
> > Note 3: I am aware that some use backintime to back up user data, and
> > I have tried it myself.  But it just seems to have some "problems".
> > For example, the built-in "diff" utility does not seem to do anything.
> > It seems old and gives the impression of not being heavily developed.
> > The documentation is "adequate" but mediocre. And what really grinds
> > my gears about backintime, a problem apparently known as far back as
> > 2014:
> >
> > "Warning: A recent security audit revealed several possible attack
> > vectors for EncFs.
> >
> >>From https://defuse.ca/audits/encfs.htm:
> >
> > EncFS is probably safe as long as the adversary only gets one copy of
> > the ciphertext and nothing more. EncFS is not safe if the adversary
> > has the opportunity to see two or more snapshots of the ciphertext at
> > different times. EncFS attempts to protect files from malicious
> > modification, but there are serious problems with this feature.
> >
> > This might be a problem with Back In Time snapshots."
> >
> > Gee . . .  think so?
>
> That report talks about issues with encfs design.  There is nothing
> backintime can do to fix those.
>
> borg can encrypt its backup images, and it recommendeds enabling that.
> So an adversary would not get access to the encfs ciphertext directly.
> They could get access to borg ciphertext instead, which may or may not
> be vulnerable to the same problems.  AFAIK there hasn't been a security
> audit of borgbackup itself.  The page at
> https://borgbackup.readthedocs.io/en/stable/internals/security.html#borgcrypto
> describes the design of borg security.
>
> --
> regards,
> kushal
>


Hi, Kushal.

In Vorta, under the "Sources" tab, there is an area (window) for input
into which you can type or paste text, such as:

**/.cache

to denote exclusions, that is, things you do not want to back up.
This is from /home/debian_user/.config/backintime/config:

. . .
profile1.snapshots.exclude.1.value=.gvfs
profile1.snapshots.exclude.2.value=.cache/*
profile1.snapshots.exclude.3.value=.thumbnails*
profile1.snapshots.exclude.4.value=.local/share/[Tt]rash*

Re: backup directory/file exclusion pattern list for borgbackup

[Kushal Kumaran]

On Sat, Sep 25 2021 at 06:24:12 PM, Default User  
wrote:
> Hello!
>
> I want to try using borgbackup to do backups of my (only) user directory:
> /home/debian-user
>
> I just want to do so using Vorta, a GUI for borgbackup.
>
> But I just need a good, general list of directory and file type
> exclusions that I can just cut and paste into the Exclude Patterns
> window in Vorta.  Something like the default list of exclusions that
> appears by default in the Backintime backup program.
>

I don't understand what a general list of exclusions would look like.
Do you have examples of what backintime excludes by default?  My own
borgbackup runs backup everything on disk; I don't feel the need to
exclude anything.

> Note 1: borgbackup uses a matching pattern called "Fnmatch" with which
> I am not familiar, and don't want to learn by trial and error, losing
> data in the process.  Which is why I am looking for a "drop-in" basic
> exclude list.
>

Run "borg help patterns" to see explanation of how borgbackup deals with
patterns.  It has this to say about fnmatch:

This is the default style for --exclude and --exclude-from.  These
patterns use a variant of shell pattern syntax, with '*' matching
any number of characters, '?' matching any single character, '[...]'
matching any single character specified, including ranges, and
'[!...]'  matching any character not specified. For the purpose of
these patterns, the path separator (backslash for Windows and '/' on
other systems) is not treated specially. Wrap meta-characters in
brackets for a literal match (i.e. [?] to match the literal
character ?). For a path to match a pattern, the full path must
match, or it must match from the start of the full path to just
before a path separator. Except for the root path, paths will never
end in the path separator when matching is attempted.  Thus, if a
given pattern ends in a path separator, a '*' is appended before
matching is attempted.

> Note 2: I am not intending to use borgbackup to back up the whole
> system; just /home/debian-user and its subdirectories.  I am using
> timeshift to back up the rest of the system.  Timeshift uses a huge
> amount of disk space, but it . . .  works.
>

I don't know how timeshift stores backups.  borg uses deduplicated
storage that avoids storing identical data multiple times.  My own borg
backups results in ~1G of new data every week (and about the same amount
being deleted from expiring backups).  There is no significant increase
in repository size week-over-week.  That obviously would not be the same
for everyone, but if you're bothered by the amount of disk space used
you can try it out.

> Note 3: I am aware that some use backintime to back up user data, and
> I have tried it myself.  But it just seems to have some "problems".
> For example, the built-in "diff" utility does not seem to do anything.
> It seems old and gives the impression of not being heavily developed.
> The documentation is "adequate" but mediocre. And what really grinds
> my gears about backintime, a problem apparently known as far back as
> 2014:
>
> "Warning: A recent security audit revealed several possible attack
> vectors for EncFs.
>
>>From https://defuse.ca/audits/encfs.htm:
>
> EncFS is probably safe as long as the adversary only gets one copy of
> the ciphertext and nothing more. EncFS is not safe if the adversary
> has the opportunity to see two or more snapshots of the ciphertext at
> different times. EncFS attempts to protect files from malicious
> modification, but there are serious problems with this feature.
>
> This might be a problem with Back In Time snapshots."
>
> Gee . . .  think so?

That report talks about issues with encfs design.  There is nothing
backintime can do to fix those.

borg can encrypt its backup images, and it recommendeds enabling that.
So an adversary would not get access to the encfs ciphertext directly.
They could get access to borg ciphertext instead, which may or may not
be vulnerable to the same problems.  AFAIK there hasn't been a security
audit of borgbackup itself.  The page at
https://borgbackup.readthedocs.io/en/stable/internals/security.html#borgcrypto
describes the design of borg security.

-- 
regards,
kushal

Re: Backup debconf state

[Sven Joachim]

On 2021-01-19 16:09 +0100, Erwan David wrote:

> Hello everybody
>
> If I want to be able to fast reinstall a debian after a crash, I
> already backup /etc (including /etc/apt), a file with the output of
> apt-show manual
> to get the list of manually installed packages, /etc, but It would be
> handy to have the state of debconf (with all the answers I already
> gave).
> Is a backup of /var/cache debconf sufficient for this ?

Yes, that should be enough.  Note that maintainer scripts which use
debconf should first consult the configuration files in /etc and only
fall back to the debconf database if those do not deliver the necessary
information.  That's why the debconf database is located under
/var/cache and not under /var/lib.

Cheers,
   Sven

Re: Backup debconf state

[Erwan David]

Le 19/01/2021 à 16:23, Andy Smith a écrit :

Hi Erwan,

On Tue, Jan 19, 2021 at 04:09:47PM +0100, Erwan David wrote:

It would be handy to have the state of debconf (with all the
answers I already gave).


I do:

dpkg --get-selections \* > /var/lib/dpkg_selections
debconf-get-selections > /var/lib/debconf_selections

(and then back up those files)

You can restore them with the corresponding --set-selections.


Is a backup of /var/cache debconf sufficient for this ?


I think that stuff lives in /var/lib, but it's better to export it
in a format where it can be re-imported.

Cheers,
Andy




Thanks for the debconf-get-selections.
the dpkg --get-selections is an old command, but I prefer the apt-show 
manual wich gives only the manually installed packages, installing them 
will install the dependencies, no problem, but they will be marked as such.

Re: Backup debconf state

[Andy Smith]

Hi Erwan,

On Tue, Jan 19, 2021 at 04:09:47PM +0100, Erwan David wrote:
> It would be handy to have the state of debconf (with all the
> answers I already gave).

I do:

dpkg --get-selections \* > /var/lib/dpkg_selections
debconf-get-selections > /var/lib/debconf_selections

(and then back up those files)

You can restore them with the corresponding --set-selections.

> Is a backup of /var/cache debconf sufficient for this ?

I think that stuff lives in /var/lib, but it's better to export it
in a format where it can be re-imported.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Backup ideas

[Default User]

Hi to all.

I want to thank you all very much  for taking the time to reply to this
thread.

Your replies have indeed been most thoughtful and informative, and you have
given me much to consider.

The subject of data protection and preservation seems to be not only quite
extensive, but also seems to be something of an art, as well as a science.
It's hard to get right, but easy to get wrong.

So, not wanting to get it wrong, I have rolled up my sleeves, am studying,
and practicing, practicing, practicing.

Please let me apologize for not replying more promptly to your posts. It
isn't out of disinterest, it's that I have been busy doing rather than
talking.

Again, thank you for your interest and assistance.  I really do appreciate
it!

Re: Backup ideas

[Alessandro Baggi]

3 - Now on to the subject of backing up a system. I am using rsync as
my primary backup tool.  Celejar, thank you for suggesting borg.  I
did install it and look at it briefly.  But for several reasons, I'm
not sure I want to use that right now.

It is of course one more thing to learn, use, and maintain.

I get the sense that it stores data as a mass of chunks, rather than
as individual files, so if something goes wrong it could really go
wrong?  It is newer, and perhaps less proven than some older
alternatives.
Hi, I would spend some words for borgbackup. borgbackup is an amazing 
tool (all of us know that it does).


Don't worry about deduplication. Deduplication is a feature. 
Deduplication currently is taking great place in many envs because it is 
a secure practices if well done and the amount of data is increased in 
last years.
The problem you stated about "if something goes wrong" is a real problem 
today and loosing data is bad. I would say that things could go wrong 
using zfs, vdo, rsync + hardlink, borgbackup, restic and any other 
tool/fs that provides deduplication, all in the same way. Suppose that 
you have a broken deduped chuck, all files pointing to this chunk are 
faulty. This type of problem could happen also with any devices that 
encounter bad blocks, bit rotting and so on. Deduplication is a features 
that permit you to save space. "If something should go bad, It will go 
bad with/without deduplication" (the biggest errors are human errors) 
and you will know that your backup is really working well when you 
try/need restore (this is why a periodical restore test is needed). You 
could not use deduplication but you could lost all data because your 
drive dies. So nothing changed, it is a features. About how borgbackup 
saves data inside the repository I would say that you can encounter this 
problem also using tool like bacula, bareos or with rsync + hardlink. 
Imagine that using bacula you saved 3 backup cycles and having 3 full 
backups of 1TB plus 30 daily incremental for each full backup (monthly 
cycle). If the volume that contains the full backup get faulty, all 
backups based on this full backup are not usefull, so the first backup 
cycle is incosistent and red flagged. To avoid problem with data 
corruption, you should use raid, better hardware, ECC ram to avoid data 
corruption.


Compressed archives have the same problem so you should not compress 
your data.


In the case of borgbackup, you have the ability to perform checks on 
repository and archives inside repository and check if something is broken.
rsync + hardlink is an inefficient way to do deduplication..it works, 
save some space on devices but nothing more.(I'm not saying that rsync 
is not a very good tool).



Also, the borg website seems to suggest that they will be developing
aggressively, and breaking compatibility over time.  Where else have I
heard that lately?  (*cough* SystemD *cough* . . .)
About this. Backup is a practice that permits you to protect your data. 
If you really care your data you should use a stable system and you 
should not use the latest version when released. If your data are 
important and you update too much frequently the software, you will 
encounter at 100% some problems. This is why stable distribution like 
debian, centos, ubuntu and other maintain the same software for many 
years with changing the version and push security or critical fixes. 
Always about borg when a new release is released the maintainer will 
report any incompability with previous version and always if you care 
your data you should read notes before any update. This is a good 
practice and not only for backup.


And, I always worry that today's free/libre can be tomorrow's
proprietary/unfree.


About this it is not really a problem. If borgbackup will not be open 
source/free, someone probably will fork the latest open source version 
and that's all. [I know that this is off-topic for debian but this is 
only an example, sorry] For example when big blue bought redhat, many 
centos users were worried about the future of centos. If big blue (is 
some case) will drop centos, the community will fork it without any 
problem and again if your preferred software, in this case borgbackup, 
will be non-free you could use it until you deploy another solution with 
another free/open source software and replace it (or you could pay it 
considering as a support donation). I don't think that actually, when 
open-source is a business model, that a software, released as 
open-source, will change its license. Generally they release open source 
software and services like support or other. borgbackup is around since 
2010 and many companies are selling storage space service for borgbackup 
like rsync.net and other (please don't consider this as spam is only an 
example).



Andy, you suggested having off-site backups.  You are correct that
local-only backups do not protect against things like fire or theft.

But 

Re: Backup ideas

[Celejar]

On Tue, 28 Apr 2020 20:37:27 +
Andy Smith  wrote:

> Hello,
> 
> On Tue, Apr 28, 2020 at 11:43:22AM -0400, Default User wrote:

...

> The thing about cloud storage is that it is generic and it is easy.
> Once you come up with a means to backup to an S3-compatible storage,
> should your data disappear you can just point it at another
> S3-compatible storage. Amazon's S3 was of course the original but
> there are many competitors to Amazon now that use the same protocol.

And furthermore, there's the wonderful rclone, which supports literally
dozens of cloud types in a single utility:

https://rclone.org/

Celejar

Re: Backup ideas

[Celejar]

On Tue, 28 Apr 2020 11:43:22 -0400
Default User  wrote:

> On Fri, Apr 24, 2020 at 12:28 PM Celejar  wrote:

...

> 3 - Now on to the subject of backing up a system. I am using rsync as
> my primary backup tool.  Celejar, thank you for suggesting borg.  I
> did install it and look at it briefly.  But for several reasons, I'm
> not sure I want to use that right now.
> 
> It is of course one more thing to learn, use, and maintain.
> 
> I get the sense that it stores data as a mass of chunks, rather than
> as individual files, so if something goes wrong it could really go
> wrong?  It is newer, and perhaps less proven than some older
> alternatives.

All true enough, but note that my impression at least is that it's not
some kind of small, hobby project, but a very sophisticated, highly
polished, widely used piece of software.

Celejar

Re: Backup ideas

[Bob Weber]

On 4/28/20 3:15 PM, Default User wrote:

On 2020-04-28 [TU] at 14:18 EDT, Bob Weber  said:


According to the manual the -x option is:

-x, --one-file-system   don't cross filesystem boundaries

Question:

When you use rsync, do you ever do it on a live, mounted filesystem
from within said machine/filesystem (that is, using the same machine)?

Or do you do it on a "dormant" unmounted filesystem, either from
another machine or from a "live [usb or .iso] utility distribution or
boot disk from which you have booted the same machine?

Most references to rsync I have seen just seem to accept as a given,
that you are doing it remotely, from across a LAN (or across the
world).

And don't seem to address whether the machine/filesystem they are
rsyncing to/from is "live" (mounted), or can/should be unmounted (like
it would be when imaging a disk with dd or Clonezilla, for example.

Yes that is the way I use rsnapshot (which uses rsync) ... on a live system.  I 
do this before an upgrade.  rsnapshot copies the files to a directory under 
/home.  I have a very big /home filesystem.  The files I am interested in are 
under /usr /etc and the various bin opt and lib directories.  These files will 
just be open for reading but not for writing.  There are log files and mail 
files under /var that may be copied in an open for write state but I can loose 
those files if necessary.  Also, files under /tmp and /run may be lost but those 
directories are usually cleared on reboot.  I do run PostgreSQL but the data 
files are under /home (mounted from another partition)  ... these are the kind 
of files that should be backed up on a "dormant" filesystem or after PostgreSQL 
is shut down.  I also run several virtual machines but their files are also 
under /home.


To restore the rsnapshot backup if I need to I run sysrescuecd. I mount the 
filesystems on /root/src and /root/dst and use rsync with the correct options 
including  --delete to get rid of any extra files that were just upgraded.  I 
usually use  -aHXA --one-file-system --progress for rsync backup options.  That 
way I can be sure the file attributes will be preserved.


A note about borgbackup.  I used it a few years ago but I found that ALL 
attributes were not backed up.  Maybe I didn't use the correct options.   The 
many options can be confusing.  The backup medium should be a journaling 
filesystem according to the docs. rsnapshot just duplicates the file structure 
of the source so you can just get one file back if you want rather than having 
to use borg to get the backed up file.



--


*...Bob*

Re: Backup ideas

[Andy Smith]

Hello,

On Tue, Apr 28, 2020 at 11:43:22AM -0400, Default User wrote:
> Andy, you mentioned restic, which I am not familiar with.  Similar
> considerations would seem to apply to that also.  But I might also try
> that out later.

Note that you did not state any of these requirements for only using
tools you are familiar with, so we cannot really be blamed for
suggesting things just because we think they are good. You didn't
really state many requirements at all, only asked for pointers as to
what makes a good backup system, so only generalities were possible.
I did say there are many many different solutions.

> 4 - So I am still using rsync.  I back up a single system to a local
> external usb drive.  That drive is then backed up to another local
> external usb drive. So I do have some redundancy.  I could go even
> further, backing up to a third drive, and using the --checksum option.
> But that might be excessive, and I do have a life to live.

If you wish to continue using rsync then I would suggest looking at
rsnapshot. It uses rsync. It provides for incremental backups. It
stores things as the actual files which makes them easy to
investigate and restore.

When you come to features like deduplication and encryption either
the remote filesystem has to support it, or else the backup tool has
to support it. That is why so many of these solutions store data in
big blocks in their own format, so they can provide those features.

rsnapshot gives you some limited form of deduplication by hard
linking identical versions of each file together. It is far from
perfect but is better than no deduplication.

> Andy, you suggested having off-site backups.  You are correct that
> local-only backups do not protect against things like fire or theft.
> 
> But off-site backups require a place for them to be, the ongoing
> effort of getting them there (and back, in a timely manner, if
> needed), and some cost involved.  And the storage place is also
> subject to "disasters" , or just going out of business overnight.

The thing about cloud storage is that it is generic and it is easy.
Once you come up with a means to backup to an S3-compatible storage,
should your data disappear you can just point it at another
S3-compatible storage. Amazon's S3 was of course the original but
there are many competitors to Amazon now that use the same protocol.

Anyone using cloud ("someone else's computer") services needs to
think about what they will do if/when the cloud ("someone else's
computer") goes away. The usual answer is "switch everything over to
another cloud".

So the concern when using such a service usually isn't whether they
are going to go out of business, but more often about keeping
control of costs. At some point there may be a time where it's
cheaper to do yourself, or scale up the amount of responsibility you
take. For example, if you require many terabytes of storage then it
will be cheaper for you to rent a whole server from someone instead.
That server will almost certainly have a lower reliability than the
cloud storage vendor, but if it's only for backups then maybe that
is not as important to you as the cost saving.

I would not suggest having "the cloud" as one's ONLY backup of
important data. Multiple different clouds may be safe enough, though
the advantage of having a copy of the data local is large.

> On-line backups have the same problems, in addition to the fact that
> once your data is "out there", even if it is encrypted, it can be
> "cracked" by governments, corporations, or individuals, at their
> leisure.

Any organisation that wants to spend that amount of resources
accessing your data has already been in your home and cloned your
storage while you were out/asleep.

> I'm not sure my threat model really justifies off-site backups at this point.

Fair enough. Fire and theft seem pretty common to me, but it's your
call.

> 5 - You also mentioned encrypted backups.  Of course, you have to to
> trust the encryption methodology.  And it should not be such a hassle
> that you don't actually use it.

It is perfectly feasible to have it be entirely automated. I would
not recommend any kind of backup that's not automated!

> Finally, after well over seven hours into rsyncing (with no end in
> sight) from the first external usb drive to the second one (both are
> HDD), I am beginning to wonder if that is a good idea.  Those first
> full backups always take forever . . .

…and USB is a fairly awful way to use storage beyond personal
gadgets (camera, phone, etc)…

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Backup ideas

[David Christensen]

On 2020-04-28 12:15, Default User wrote:


I don't recall the specific error code I got; just that it refused to
do a sync using the -x option.  


I keep a plain text system administration log file for each computer in 
CVS.  I cut and paste console sessions into it.  If a given command 
produces a lot of output, I paste it into a separate file or create the 
file with tee(1).




And I can't try it now, as I am now
into the 11th or 12th hour of rsyncing one external usb drive to
another.  At this rate it could take days to complete!  I had no idea.


Use df(1) to find out how much data is on the source disk.


Install nmon(1) and use it to find out the transfer rate.  If you use 
the Xfce desktop, install the Disk Performance Monitor panel applet.



Divide the data size by the transfer rate to get an estimate of how long 
the transfer will take.  Understand that the transfer rate of HDD's 
slows down as the block number increases (transfer rate of the last 
block is about half that of the first block).



Unfortunately, I only have one computer. 


Ouch.  You need another computer.  I would suggest a used server with 
ECC memory and a decent number of drive bays.  But, a $20 garage sale 
used tower is better than nothing.



I am not on a LAN 


Assuming you have a residential broadband Internet connection, many 
gateways include a 4 port Gigabit switch and Wi-Fi.  If you have more 
than one device connected to the gateway/ Internet, you have a LAN.



and have no NAS, etc. 


NAS devices give you functionality without having to understand the 
details.  If you want to understand the details, building your own file 
server with Debian and Samba is an intermediate-level project.




And raid setups are "above my pay grade" anyway.


Setting up a mirror with md(4) and mdadm(8) is a beginner/ 
intermediate-level project.  If your one computer is a desktop and it 
has two spare bays, you should consider it.




Question:

When you use rsync, do you ever do it on a live, mounted filesystem
from within said machine/filesystem (that is, using the same machine)?

Or do you do it on a "dormant" unmounted filesystem, either from
another machine or from a "live [usb or .iso] utility distribution or
boot disk from which you have booted the same machine?

Most references to rsync I have seen just seem to accept as a given,
that you are doing it remotely, from across a LAN (or across the
world).

And don't seem to address whether the machine/filesystem they are
rsyncing to/from is "live" (mounted), or can/should be unmounted (like
it would be when imaging a disk with dd or Clonezilla, for example.


rsync(1) is for mounted filesystems.  If you need consistency of files 
and/or directories, the source filesystem should be mounted read-only. 
I use rsync(1) for data and for system configuration files.  I do not 
use rsync(1) to backup, restore, or migrate operating system images.



I use dd(1) to backup, restore, and migrate operating system raw drive 
images.  To make this practical, I keep my system images small enough to 
fit on a "16 GB" device (USB flash drive, SSD, HDD), and I do not use RAID.



It is useful to have a live Debian system on a USB flash drive for doing 
these kinds of operations.



David

Re: Backup ideas

[Default User]

On 2020-04-28 [TU] at 14:18 EDT, Bob Weber  said:

> According to the manual the -x option is:
>
> -x, --one-file-system   don't cross filesystem boundaries
>
> I use that option all the time to keep from backing up my large home > 
> directory when I only
> want the system directories under root.  It even > keeps rsync from copying 
> system directories
> like /proc /dev and
> /sys.

> Before I do a system update/upgrade I run rsnapshot (debian
> package) on the root system directories so I can get the system
> back in case of major failure in some of the updates (I run testing so > I 
> have to be careful).  I do > run this on a live system and on 3 or 4
> occasions I have had to restore from the snapshot successfully
> getting my system back alive.

> Do you have another system you could backup to?  I can get around > 50 mega 
> bytes per
> second transfer over 1Gb Ethernet so you might > try that.

> My main backup is done by backuppc on a dedicated server .  I
> have 4 or 5 systems that get unattended daily backups with versions > going 
> back about a year. > All my systems use a 2 drive raid1  array >
> so I can survive a single disk crash without having to resort to restoring a 
> backup.  Every few
> months I install an extra drive in the backuppc server and have raid sync it 
> to the 2 drives in the > server.  After syncing I pull the drive and put it 
> in my barn for offsite storage.  Since it is a raid1
> full copy you can take that drive and mount it on another system and get the 
> files back if you
> need to (running the raid array in a degraded mode).


Hi, Bob!

I don't recall the specific error code I got; just that it refused to
do a sync using the -x option.  And I can't try it now, as I am now
into the 11th or 12th hour of rsyncing one external usb drive to
another.  At this rate it could take days to complete!  I had no idea.

Unfortunately, I only have one computer.  I am not on a LAN and have
no NAS, etc.  And raid setups are "above my pay grade" anyway.

Question:

When you use rsync, do you ever do it on a live, mounted filesystem
from within said machine/filesystem (that is, using the same machine)?

Or do you do it on a "dormant" unmounted filesystem, either from
another machine or from a "live [usb or .iso] utility distribution or
boot disk from which you have booted the same machine?

Most references to rsync I have seen just seem to accept as a given,
that you are doing it remotely, from across a LAN (or across the
world).

And don't seem to address whether the machine/filesystem they are
rsyncing to/from is "live" (mounted), or can/should be unmounted (like
it would be when imaging a disk with dd or Clonezilla, for example.

Re: Backup ideas

[Default User]

On Tue, Apr 28, 2020 at 12:16 PM Andrei POPESCU
 wrote:
>
> On Ma, 28 apr 20, 11:43:22, Default User wrote:
> > > Andy Smith  wrote:
> >
> > 1 - First, regarding:
> > > [vast mounts of quoted text snipped - please don't quote too much!]
> >
> > I didn't realize some trimming might be needed.  I guess I just
> > figured you can't scroll through an email for information that has
> > been trimmed.  And on some lists, like OpenBSD and Arch, people will
> > actually yell at you for trimming or not providing enough information.
>
> Ideally every message should be as short as possible while still being
> as much as possible self-contained.
>
> Proper attribution of quotes is also much appreciated.
>
> > And, I always worry that today's free/libre can be tomorrow's
> > proprietary/unfree.
>
> See "Tentacles of Evil" test.
>
> https://en.wikipedia.org/wiki/Debian_Free_Software_Guidelines#debian-legal_tests_for_DFSG_compliance
>
> Short version: once a software has been released under a free software
> license not even the author can change it. They can only change the
> license for future versions. Anybody else is free to fork from the last
> free version.
>
> This is not just theoretical, it has already happened in several cases
> and will likely happen again if the software in question has any
> significant user base.
>
> > But off-site backups require a place for them to be, the ongoing
> > effort of getting them there (and back, in a timely manner, if
> > needed), and some cost involved.  And the storage place is also
> > subject to "disasters" , or just going out of business overnight.
>
> At the same time with a "local" disaster? Highly unlikely.
>
> > On-line backups have the same problems, in addition to the fact that
> > once your data is "out there", even if it is encrypted, it can be
> > "cracked" by governments, corporations, or individuals, at their
> > leisure.
>
> [citation needed]
>
> With current encryption methods it's probably easier to compromise the
> source computer instead. Or https://xkcd.com/538/.
>
> Kind regards,
> Andrei
> --
> http://wiki.debian.org/FAQsFromDebianUser



Hi Andrei.

On 2020-04-28 [TU] at 12:16 EDT, Andrei POPESCU
 wrote:
>
> Ideally every message should be as short as possible while still being
> as much as possible self-contained.
>
> Proper attribution of quotes is also much appreciated.
>

Okay.  Do you mean like this?

>
> See "Tentacles of Evil" test.
>
> https://en.wikipedia.org/wiki/Debian_Free_Software_Guidelines#debian-legal_tests_for_DFSG_compliance
>

Noted. And read.

>
> > But off-site backups require a place for them to be, the ongoing
> > effort of getting them there (and back, in a timely manner, if
> > needed), and some cost involved.  And the storage place is also
> > subject to "disasters" , or just going out of business overnight.
>
> At the same time with a "local" disaster? Highly unlikely.
>

Well, things can (and do) happen, anywhere at any time.  And things
that are away from me are not under my control.

("Just because I'm paranoid doesn't mean they're not out to get me!")

>
> > On-line backups have the same problems, in addition to the fact that
> > once your data is "out there", even if it is encrypted, it can be
> > "cracked" by governments, corporations, or individuals, at their
> > leisure.
>
> [citation needed]
>

I would have thought that by now, this would be almost universally
accepted and considered virtually self-evident.

("Just because I'm paranoid doesn't mean they're not out to get me!")

>
> With current encryption methods it's probably easier to compromise the
> source computer instead. Or https://xkcd.com/538/.
>

There is truth in that.  Yes, I am familiar with "lead-pipe"
cryptanalysis.  But:
("Just because I'm paranoid doesn't mean they're not out to get me!")
- and these days, they don't even have to get out of their chair to
get information . . .

Re: Backup ideas

[Bob Weber]

6 - Finally, using rsync I actually am doing two separate backups:

date; sudo rsync -avvzHAXPSish --delete --stats
--exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/home/default/*"}
/ /media/default/USBHD005/Backup_of_Dell_Debian_dimwit/root_partition

[This backs up the filesystem EXCEPT for the home directory.]

And:

date; sudo rsync -avvzHAXPSish --delete --stats
--exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"}
/home/default 
/media/default/USBHD005/Backup_of_Dell_Debian_dimwit/home_partition

[This backs up the home directory.]

Note: my home directory is on a separate (extended) partition from my
root directory.

Another note: rsync did NOT like the -x option.  I wanted to use that
to prevent getting into a recursive loop while backing up. Rsync just
refused, complaining with a sparse non-informative error code message.
But without it, it seems to work okay.  Go figure.


According to the manual the -x option is:

-x, --one-file-system   don't cross filesystem boundaries

I use that option all the time to keep from backing up my large home directory 
when I only want the system directories under root.  It even keeps rsync from 
copying system directories like /proc /dev and /sys.


Before I do a system update/upgrade I run rsnapshot (debian package) on the root 
system directories so I can get the system back in case of major failure in some 
of the updates (I run testing so I have to be careful).  I do run this on a live 
system and on 3 or 4 occasions I have had to restore from the snapshot 
successfully getting my system back alive.


Do you have another system you could backup to?  I can get around 50 mega bytes 
per second transfer over 1Gb Ethernet so you might try that.


My main backup is done by backuppc on a dedicated server .  I have 4 or 5 
systems that get unattended daily backups with versions going back about a 
year.  All my systems use a 2 drive raid1  array so I can survive a single disk 
crash without having to resort to restoring a backup.  Every few months I 
install an extra drive in the backuppc server and have raid sync it to the 2 
drives in the server.  After syncing I pull the drive and put it in my barn for 
offsite storage.  Since it is a raid1 full copy you can take that drive and 
mount it on another system and get the files back if you need to (running the 
raid array in a degraded mode).



I have been using rsync to backup live, from within Debian.  Maybe not
a good idea.  I could instead try using rsync from a live usb, such as
SystemRescueCD, etc.  I'll try that later.  After all, it does seem to
make more sense to back up a dead filesystem from outside it than a
live filesystem from inside it.

Finally, after well over seven hours into rsyncing (with no end in
sight) from the first external usb drive to the second one (both are
HDD), I am beginning to wonder if that is a good idea.  Those first
full backups always take forever . . .



--


*...Bob*

Re: Backup ideas

[Andrei POPESCU]

On Ma, 28 apr 20, 11:43:22, Default User wrote:
> > Andy Smith  wrote:
> 
> 1 - First, regarding:
> > [vast mounts of quoted text snipped - please don't quote too much!]
> 
> I didn't realize some trimming might be needed.  I guess I just
> figured you can't scroll through an email for information that has
> been trimmed.  And on some lists, like OpenBSD and Arch, people will
> actually yell at you for trimming or not providing enough information.

Ideally every message should be as short as possible while still being 
as much as possible self-contained.

Proper attribution of quotes is also much appreciated.
 
> And, I always worry that today's free/libre can be tomorrow's
> proprietary/unfree.

See "Tentacles of Evil" test. 

https://en.wikipedia.org/wiki/Debian_Free_Software_Guidelines#debian-legal_tests_for_DFSG_compliance

Short version: once a software has been released under a free software 
license not even the author can change it. They can only change the 
license for future versions. Anybody else is free to fork from the last 
free version.

This is not just theoretical, it has already happened in several cases 
and will likely happen again if the software in question has any 
significant user base.
 
> But off-site backups require a place for them to be, the ongoing
> effort of getting them there (and back, in a timely manner, if
> needed), and some cost involved.  And the storage place is also
> subject to "disasters" , or just going out of business overnight.

At the same time with a "local" disaster? Highly unlikely.

> On-line backups have the same problems, in addition to the fact that
> once your data is "out there", even if it is encrypted, it can be
> "cracked" by governments, corporations, or individuals, at their
> leisure.

[citation needed]

With current encryption methods it's probably easier to compromise the 
source computer instead. Or https://xkcd.com/538/.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser


signature.asc
Description: PGP signature

Re: Backup ideas

[Default User]

On Fri, Apr 24, 2020 at 12:28 PM Celejar  wrote:
>
> On Fri, 24 Apr 2020 15:34:24 +
> Andy Smith  wrote:
>
> ...
>
> > Something like restic backup is a very good all around solution.
>
> OP: also look at borg.
>
> > You could consider backing up to a cloud like Amazon S3. Using the
> > Infrequent Access or Glacier storage classes you can store 100s of
> > gigabytes of data for just a few US cents per day. There are many
>
> It should just be noted that anyone considering Glacier should also
> price in its considerable retrieval costs.
>
> Celejar


Hi Andy and Celejar!

Sorry it took several days to get back to this.  I have had some
personal things going on, and I have also been spending some quality
time with Clonezilla and rsync.

1 - First, regarding:
"[vast mounts of quoted text snipped - please don't quote too much!]"

I didn't realize some trimming might be needed.  I guess I just
figured you can't scroll through an email for information that has
been trimmed.  And on some lists, like OpenBSD and Arch, people will
actually yell at you for trimming or not providing enough information.

2 - I had originally posted in another thread about checking an ssd
and replacing it if needed, imaging the old system and applying the
image(s) to the new drive.  I did test Clonezilla, and it did image
and restore all partitions to the same ssd.  I have not tested
restoring to a different drive, though.

Clonezilla (using partclone by default) does work.  But the user
interface is really ugly, and how to use it is somewhat non-intuitive
(not good if you are using it for the first  time in an emergency).

3 - Now on to the subject of backing up a system. I am using rsync as
my primary backup tool.  Celejar, thank you for suggesting borg.  I
did install it and look at it briefly.  But for several reasons, I'm
not sure I want to use that right now.

It is of course one more thing to learn, use, and maintain.

I get the sense that it stores data as a mass of chunks, rather than
as individual files, so if something goes wrong it could really go
wrong?  It is newer, and perhaps less proven than some older
alternatives.

Also, the borg website seems to suggest that they will be developing
aggressively, and breaking compatibility over time.  Where else have I
heard that lately?  (*cough* SystemD *cough* . . .)

And, I always worry that today's free/libre can be tomorrow's
proprietary/unfree.

But I still might try it out later.

Andy, you mentioned restic, which I am not familiar with.  Similar
considerations would seem to apply to that also.  But I might also try
that out later.

4 - So I am still using rsync.  I back up a single system to a local
external usb drive.  That drive is then backed up to another local
external usb drive. So I do have some redundancy.  I could go even
further, backing up to a third drive, and using the --checksum option.
But that might be excessive, and I do have a life to live.

Andy, you suggested having off-site backups.  You are correct that
local-only backups do not protect against things like fire or theft.

But off-site backups require a place for them to be, the ongoing
effort of getting them there (and back, in a timely manner, if
needed), and some cost involved.  And the storage place is also
subject to "disasters" , or just going out of business overnight.

On-line backups have the same problems, in addition to the fact that
once your data is "out there", even if it is encrypted, it can be
"cracked" by governments, corporations, or individuals, at their
leisure.

I'm not sure my threat model really justifies off-site backups at this point.

5 - You also mentioned encrypted backups.  Of course, you have to to
trust the encryption methodology.  And it should not be such a hassle
that you don't actually use it.

And . . .   don't lose your key!  That could really ruin your day.  Or life.

I have actually considered (and still am) encrypting the backups,
perhaps using full disk encryption.  Maybe LUKS or something similar.
But . . .  I do remember using disk encryption on the home partition
of my system, years ago.  I worked fine for a short time, and then it
just didn't. Not a lost password, it just stopped working.  Lost data
forever.  Not fun.  Never forgot that.

6 - Finally, using rsync I actually am doing two separate backups:

date; sudo rsync -avvzHAXPSish --delete --stats
--exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/home/default/*"}
/ /media/default/USBHD005/Backup_of_Dell_Debian_dimwit/root_partition

[This backs up the filesystem EXCEPT for the home directory.]

And:

date; sudo rsync -avvzHAXPSish --delete --stats
--exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"}
/home/default 
/media/default/USBHD005/Backup_of_Dell_Debian_dimwit/home_partition

[This backs up the home directory.]

Note: my home directory is on a separate (extended) partition from my
root directory.

Another note: 

Re: Backup ideas

[Celejar]

On Fri, 24 Apr 2020 15:34:24 +
Andy Smith  wrote:

...

> Something like restic backup is a very good all around solution.

OP: also look at borg.

> You could consider backing up to a cloud like Amazon S3. Using the
> Infrequent Access or Glacier storage classes you can store 100s of
> gigabytes of data for just a few US cents per day. There are many

It should just be noted that anyone considering Glacier should also
price in its considerable retrieval costs.

Celejar

Re: Backup Times on a Linux desktop

[Charles Curley]

Thanks for the feedback.

On Tue, 05 Nov 2019 23:35:05 +0100
Linux-Fan  wrote:

> Charles Curley writes:
> 

> > https://charlescurley.com/blog/posts/2019/Nov/02/backups-on-linux/index.html
> >   
> 
> [...]
> 
> Thanks for sharing! I appreciate that I am not the only one with a
> backup system composed of multiple tools with different timings and
> occasions of invocation :)

They just metastasize over the years.

> 
> One point where my opinion is slightly different (might boil down to
> taste, but that's part of the feedback?). Quoting from the blog:
> 
> > Some stuff isn't worth the disk space to back up because you can
> > regenerate it or re-install it just as easily. Caches, such as a web
> > proxy's. Executables you can re-install, like your office suite.  
> 
> I personally think it is (especially today) not so easy to keep track
> of all the programs one actually needs and where to get them.
> Additionally, one should take into consideration, whether the
> avaiability of Internet access (needed for software re-installation
> unless other measures are taken) is really part of the assumptions
> for backup restoring? I try to put some effort into
> 100%-offline-restoration.

I see your point. I certainly expect to do bare metal restoration with
local resources only, and see to it that everything I need to do that
is available.

> 
> At the same time, I try to avoid "disk image"-style backups, because
> they are hard to make (usually the system needs to be offline for
> this) and they are hard to restore: What if my server with 4x2T HDDs
> just dies. By tomorrow, I will not have another server, a humble
> laptop with 500 GB HDD might be all there is for the moment.
> Restoring images is infeasible in that situation, a normal
> "reinstallation" is less (but might be: consider borrowing a computer
> frome someone else for some time. In that case it will likely be
> impossible to change the OS and thus the software installation might
> be limited...)

Both good points. That is part of your disaster recovery planning: can
you get a replacement box quickly enough. I've had clients buy a spare
box and keep it off site. Another key part of disaster recovery is: how
quickly do you have to be back up and running?

One reason I like amanda is that you can restore without having amanda
on the machine with the files. I've never actually had to do that,
fortunately. Also amanda lets you select individual files to restore.


> 
> YMMV
> Linux-Fan
> 



-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Re: Backup Times on a Linux desktop

[Linux-Fan]

Charles Curley writes:


On Sat, 02 Nov 2019 20:24:52 +0100
Konstantin Nebel  wrote:


[...]


> So now I am thinking. How should I approach backups. On windows it
> does magically backups and remind me when they didnt run for a while.
> I like that attitude.
>
> On linux with all that decision freedom it can be good and bad cause
> you have to think about things :D

I started writing a reply to this several days ago, and realized it
would make a good blog entry. I'd appreciate feedback.

https://charlescurley.com/blog/posts/2019/Nov/02/backups-on-linux/index.html


[...]

Thanks for sharing! I appreciate that I am not the only one with a backup
system composed of multiple tools with different timings and occasions of
invocation :)

One point where my opinion is slightly different (might boil down to taste,
but that's part of the feedback?). Quoting from the blog:


Some stuff isn't worth the disk space to back up because you can
regenerate it or re-install it just as easily. Caches, such as a web
proxy's. Executables you can re-install, like your office suite.


I personally think it is (especially today) not so easy to keep track of all
the programs one actually needs and where to get them. Additionally, one
should take into consideration, whether the avaiability of Internet access
(needed for software re-installation unless other measures are taken) is
really part of the assumptions for backup restoring? I try to put some
effort into 100%-offline-restoration.

At the same time, I try to avoid "disk image"-style backups, because they
are hard to make (usually the system needs to be offline for this) and they
are hard to restore: What if my server with 4x2T HDDs just dies. By
tomorrow, I will not have another server, a humble laptop with 500 GB HDD
might be all there is for the moment. Restoring images is infeasible in that
situation, a normal "reinstallation" is less (but might be: consider
borrowing a computer frome someone else for some time. In that case it will
likely be impossible to change the OS and thus the software installation
might be limited...)

YMMV
Linux-Fan

Re: Backup Times on a Linux desktop

[Charles Curley]

On Sat, 02 Nov 2019 20:24:52 +0100
Konstantin Nebel  wrote:

> Now i attached a 4 tb drive to my pi and I decided what the heck, why
> not doing backups now.
> 
> So now I am thinking. How should I approach backups. On windows it
> does magically backups and remind me when they didnt run for a while.
> I like that attitude.
> 
> On linux with all that decision freedom it can be good and bad cause
> you have to think about things :D

I started writing a reply to this several days ago, and realized it
would make a good blog entry. I'd appreciate feedback.

https://charlescurley.com/blog/posts/2019/Nov/02/backups-on-linux/index.html

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/


pgpO87Gr8WjdW.pgp
Description: OpenPGP digital signature

Re: Backup Times on a Linux desktop

[Stefan Monnier]

> On linux with all that decision freedom it can be good and bad cause you have
> to think about things :D

All the answers I've seen mention the use of "cron" but I'm not sure
what they mean by that, nor am I sure what is your typical use of the
desktop (e.g. is it always ON?), so I think it's worth mentioning the
use of /etc/cron.daily and /etc/cron.weekly as well as `anacron`
(nowadays provided by `systemd-cron`) which will run those tasks "when
possible" rather than at fixed times.

Regarding backup software I use `bup` for some systems and `rsync`
for others.


Stefan

Re: Backup Times on a Linux desktop

[Stefan Monnier]

> Suppose that you backup 2000 files in a day and inside this backup a chunk
> is deduped and referenced by 300 files. If the deduped chunk is broken
> I think you will lost it on 300 referenced files/chunks. This is not good
> for me.

I don't know what other backup software does, but at least `bup`
addresses this risk by recommending the use of `par2` (and
of course Git's content-addresses storage makes it easy to detect
corruption).


Stefan

Re: Backup Times on a Linux desktop

[Alessandro Baggi]

On 04/11/19 20:43, deloptes wrote:

Alessandro Baggi wrote:


If I'm not wrong deduplication "is a technique for eliminating duplicate
copies of repeating data".

I'm not a borg expert and it performs deduplication on data chunk.

Suppose that you backup 2000 files in a day and inside this backup a
chunk is deduped and referenced by 300 files. If the deduped chunk is
broken I think you will lost it on 300 referenced files/chunks. This is
not good for me.



Look at the explanation by Linux-Fan. I think it is pretty good. It fits one
scenario, however if your backup system (disks or whatever) is broken - it
can not be considered as backup system at all.



Linux-Fan reply is interesting but there is not nothing new for me.


I think deduplication is a great thing nowdays - People need to backup TBs,
take care of retention etc. I do not share your concerns at all.


if your main dataset has a broken file, no problem, you can recovery
from backups.

If your saved deduped chunk is broken all files that has reference to it
could be broken. I think also that the same chunk will be used for
successive backups (always for deduplication) so this single chunk could
be used from backup1 to backupN.



This is not true.



What is not true?
The same single chunk will not be used inside other backups? So dedup 
chunk is related only to one backup?




It has also integrity check but don't know if check this. I read also
that integrity check on bigsized dataset could require too much time.

In my mind a backup is a copy of file in window time and if needed in
another window time another copy could be picked but it could not be a
reference to a previous copy. Today there are people that make backups
on tape (expensive) for reliability. I run backups on disks. Disks are
cheap so compression (that require time in backup and restore) and
deduplication (that add complexity) are not needed for me and they don't
affect really my free disk space because I can add a disk.



I think it depends how far you want to go - how precious is the data.
Magnetic disk and tapes can be destroyed by EMP or similar. SSD despite its
price can fail and if it fails - it can not recover anything.
So ... there are some rules in securely preserving backups - but all of this
is very expensive.



EMP or similar? You are right but I have seen only one case in my 
experience where a similar event broken a memory and was a laptop disk 
near a radar station. How many times could happen this?



Rsnapshot uses hardlink that is similar.

All this solutions are valid if them fit your needs. You must choose how
important are data inside your backups and if losing a chunk deduped
could make damage to your backup dataset in a timeline.



No unless the corruption is on the backup server, but if it happens ... well
you should consider the backup server broken - I do not think it has
anything with deduplication.


Ah if you have multiple server to backup, I prefer bacula because can
pull data from hosts and can backup multiple server from the same point
(maybe using for each client a separated bacula-sd daemon with dedicated
storage).



.

Re: Backup Times on a Linux desktop

[Charles Curley]

On Mon, 4 Nov 2019 06:01:54 -1000
Joel Roth  wrote:

> These days I use rsync with the --link-dest option to make
> complete Time-Machine(tm) style backups using hardlinks to
> avoid file duplication in the common case.  In this
> scenario, the top-level directory is typically named based
> on date and time, e.g. back-2019.11.04-05:32:06.

Take a look at rsnapshot. You have pretty well described it.


> 
> I usually make backups while the system is running, although
> I'm not sure it's considered kosher. It takes around 10% of
> CPU on my i5 system.

It's kosher except in a few places where referential integrity is an
issue. The classic here is a database that extends across multiple
files, which means almost all of them.

Referential integrity means keeping the data consistent. Suppose you
send an INSERT statement to a SQL database, and it affects multiple
files. The database writes to the first file. Then your backup comes
along and grabs the files for backup. Then your database writes the
other files. Your backups are broken, and you won't know it until you
restore and test.

There are work-arounds. Shut the database down during backups, or make
it read only during backups. Or tell it to accept writes from clients
but not actually write them out to the files until the backup is over.

Obviously this requires some sort of co-ordination between the backup
software and the software maintaining the files.

Or use Sqlite, which I believe avoids this issue entirely.


-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Re: Backup Times on a Linux desktop

[Joel Roth]

On Mon, Nov 04, 2019, Charles Curley wrote:
> On Mon, 4 Nov 2019 06:01:54 -1000
> Joel Roth  wrote:
> 
> > These days I use rsync with the --link-dest option to make
> > complete Time-Machine(tm) style backups using hardlinks to
> > avoid file duplication in the common case.  In this
> > scenario, the top-level directory is typically named based
> > on date and time, e.g. back-2019.11.04-05:32:06.
> 
> Take a look at rsnapshot. You have pretty well described it.

Looks like a featureful, capable, and thoroughly debugged
front end to rsync with the --link-dest option. 

Thanks, I'll fool around with this. 

Also for the explanations about file integrity issues when
databases are involved. 

--
Joel Roth

Re: Backup Times on a Linux desktop

[deloptes]

Alessandro Baggi wrote:

> If I'm not wrong deduplication "is a technique for eliminating duplicate
> copies of repeating data".
> 
> I'm not a borg expert and it performs deduplication on data chunk.
> 
> Suppose that you backup 2000 files in a day and inside this backup a
> chunk is deduped and referenced by 300 files. If the deduped chunk is
> broken I think you will lost it on 300 referenced files/chunks. This is
> not good for me.
> 

Look at the explanation by Linux-Fan. I think it is pretty good. It fits one
scenario, however if your backup system (disks or whatever) is broken - it
can not be considered as backup system at all.

I think deduplication is a great thing nowdays - People need to backup TBs,
take care of retention etc. I do not share your concerns at all.

> if your main dataset has a broken file, no problem, you can recovery
> from backups.
> 
> If your saved deduped chunk is broken all files that has reference to it
> could be broken. I think also that the same chunk will be used for
> successive backups (always for deduplication) so this single chunk could
> be used from backup1 to backupN.
> 

This is not true.

> It has also integrity check but don't know if check this. I read also
> that integrity check on bigsized dataset could require too much time.
> 
> In my mind a backup is a copy of file in window time and if needed in
> another window time another copy could be picked but it could not be a
> reference to a previous copy. Today there are people that make backups
> on tape (expensive) for reliability. I run backups on disks. Disks are
> cheap so compression (that require time in backup and restore) and
> deduplication (that add complexity) are not needed for me and they don't
> affect really my free disk space because I can add a disk.
> 

I think it depends how far you want to go - how precious is the data.
Magnetic disk and tapes can be destroyed by EMP or similar. SSD despite its
price can fail and if it fails - it can not recover anything.
So ... there are some rules in securely preserving backups - but all of this
is very expensive.

> Rsnapshot uses hardlink that is similar.
> 
> All this solutions are valid if them fit your needs. You must choose how
> important are data inside your backups and if losing a chunk deduped
> could make damage to your backup dataset in a timeline.
> 

No unless the corruption is on the backup server, but if it happens ... well
you should consider the backup server broken - I do not think it has
anything with deduplication.

> Ah if you have multiple server to backup, I prefer bacula because can
> pull data from hosts and can backup multiple server from the same point
> (maybe using for each client a separated bacula-sd daemon with dedicated
> storage).

Re: Backup Times on a Linux desktop

[Joel Roth]

On Sat, Nov 02, 2019, Konstantin Nebel wrote:
> So now I am thinking. How should I approach backups. On windows it does
> magically backups and remind me when they didnt run for a while. I like that
> attitude.
(...) 
>  I like to turn off
> my computer at night. So a backup running in night is not really an option
> unless I do wake on lan and run backup and then turn off. 
(...)

Someone already recommended setting up a cron job for triggering backups
on a regular schedule. That takes care of the automagic part.

These days I use rsync with the --link-dest option to make
complete Time-Machine(tm) style backups using hardlinks to
avoid file duplication in the common case.  In this
scenario, the top-level directory is typically named based
on date and time, e.g. back-2019.11.04-05:32:06.

I usually make backups while the system is running, although
I'm not sure it's considered kosher. It takes around 10% of
CPU on my i5 system.

> Whoever read till the end Im thankful and ready to hear your opinion.
 
> Cheers
> Konstantin

--
Joel Roth

Re: Backup Times on a Linux desktop

[Alessandro Baggi]

On 04/11/19 15:41, deloptes wrote:

Not sure if true - for example you make daily, weekly and monthly backups
(classical) Lets focus on the daily part. On day 3 the files is broken.
You have to recover from day 2. The file is not broken for day 2 - correct?!


If I'm not wrong deduplication "is a technique for eliminating duplicate 
copies of repeating data".


I'm not a borg expert and it performs deduplication on data chunk.

Suppose that you backup 2000 files in a day and inside this backup a 
chunk is deduped and referenced by 300 files. If the deduped chunk is 
broken I think you will lost it on 300 referenced files/chunks. This is 
not good for me.


if your main dataset has a broken file, no problem, you can recovery 
from backups.


If your saved deduped chunk is broken all files that has reference to it 
could be broken. I think also that the same chunk will be used for 
successive backups (always for deduplication) so this single chunk could 
be used from backup1 to backupN.


It has also integrity check but don't know if check this. I read also 
that integrity check on bigsized dataset could require too much time.


In my mind a backup is a copy of file in window time and if needed in 
another window time another copy could be picked but it could not be a 
reference to a previous copy. Today there are people that make backups 
on tape (expensive) for reliability. I run backups on disks. Disks are 
cheap so compression (that require time in backup and restore) and 
deduplication (that add complexity) are not needed for me and they don't 
affect really my free disk space because I can add a disk.


Rsnapshot uses hardlink that is similar.

All this solutions are valid if them fit your needs. You must choose how 
important are data inside your backups and if losing a chunk deduped 
could make damage to your backup dataset in a timeline.


Ah if you have multiple server to backup, I prefer bacula because can 
pull data from hosts and can backup multiple server from the same point 
(maybe using for each client a separated bacula-sd daemon with dedicated 
storage).

Re: Backup Times on a Linux desktop

[Linux-Fan]

deloptes writes:


Alessandro Baggi wrote:

> Borg seems very promising but I performs only push request at the moment
> and I need pull request. It offers deduplication, encryption and much
> more.
>
> One word on deduplication: it is a great feature to save space, with
> deduplication compression ops (that could require much time) are avoided
> but remember that with deduplication for multiple backups only one
> version of this files is deduplicated. So if this file get corrupted
> (for every reason) it will be compromised on all  previous backups jobs
> performed, so the file is lost. For this I try to avoid deduplication on
> important backup dataset.

Not sure if true - for example you make daily, weekly and monthly backups
(classical) Lets focus on the daily part. On day 3 the files is broken.
You have to recover from day 2. The file is not broken for day 2 - correct?!


[...]

I'd argue that you are both right about this. It just depends on where the
file corruption occurs.

Consider a deduplicated system which stores backups in /fs/backup and reads
the input files from /fs/data. Then if a file in /fs/data is corrupted, you
could always extract it from the backup successfully. If that file were
changed and corrupted, the backup system would no longer consider it a
"duplicate" and thus store the corrupted content of the file as a new
version. Effectively, while the newest version of the file is corrupted and
thus not useful, it is still possible to recover the old version of the file
from the (deduplicated or not) backup.

The other consideration is a corruption on the backup storage volume like
some files in /fs/backup go bad. In a deduplicated setting, if a single
piece of data in /fs/backup corresponds to a lot of restored files with the
same contents, all of these files are no longer successfully recoverable,
because the backup's internal structure contains corrupted data.

In a non-deduplicated (so to say: redundant) backup system, if parts of the
backup store become corrupted, the damage is likely (but not necessarily)
restricted to only some files upon restoration and as there is no
deduplication, it is likely that the "amount of data non-restorable" is
somehow related to the "amount of data corrupted"...

as these considerations about a corrupted backup store are mostly on such a
blurry level as described, the benefit from avoiding deduplication because
of the risk of losing more files upon corruption of the backup store is
possibly limited. However, given some concrete systems, the picture might
change entirely. A basic file-based (e.g. rsync) backup is as tolerant to
corruption as the original "naked" files. For any system maintaining its own
filesystem, the respective system needs to be studied extensively to find
out how partial corruption affects restorability. In theory, it could have
additional redundancy data to restore files even in the presence of a
certain level of corruption (e.g. in percent bytes changed or similar).

This whole thing was actually a reason for writing my own system: File-based
rsync-backup was slow, space inefficient and did not provide encryption.
However, more advanced systems (like borg, obnam?) split files into
multiple chunks and maintain their own filesystem. For me it is not really
obvious how a partially corrupted backup restores with these systems. For
my tool, I chose an approach between these: I store only "whole" files and
do not deduplicate them in any way. However, I put multipls small files into
archives such that I can compress and encrypt them. In my case, a partial
corruption would exactly lose the files from the corrupted archives which
establishes a relation between the amount of data corrupted and lost
(although in the worst case: "each archive slightly corrupted", all is
lost... to avoid that one needs error correction, but my tool does not do it
[yet?])

HTH
Linux-Fan

Re: Backup Times on a Linux desktop

[deloptes]

Alessandro Baggi wrote:

> Borg seems very promising but I performs only push request at the moment
> and I need pull request. It offers deduplication, encryption and much
> more.
> 
> One word on deduplication: it is a great feature to save space, with
> deduplication compression ops (that could require much time) are avoided
> but remember that with deduplication for multiple backups only one
> version of this files is deduplicated. So if this file get corrupted
> (for every reason) it will be compromised on all  previous backups jobs
> performed, so the file is lost. For this I try to avoid deduplication on
> important backup dataset.

Not sure if true - for example you make daily, weekly and monthly backups
(classical) Lets focus on the daily part. On day 3 the files is broken.
You have to recover from day 2. The file is not broken for day 2 - correct?!

> but remember that with deduplication for multiple backups only one
> version of this files is deduplicated.

I do not know how you come to the conclusion regarding this. This is not how
deduplication works. At least not according my understanding. The
documentation describes the process of backing up and deduplication such
that file chunks are being read and compared. If they are different the new
chunk is backuped. Remember this is done for each backup. If you want to
restore a previous one obviously the file will be reconstructed based on
previously store/backuped information.

regards

Re: Backup Times on a Linux desktop

[Jonathan Dowland]

On Sun, Nov 03, 2019 at 02:47:46AM -0500, Gene Heskett wrote:

Just 4 or 5 days ago, I had to recover the linuxcnc configs from a backup
of the pi3, making a scratch dir here at home, then scanned my database
for the last level0 of the pi3b, pulled that out with amrecover then
copied what I needed back to the rpi4 now living in my Sheldon Lathes
control box. File moving done by an old friend, mc, and sshfs mounts.
Totally painless,


As a former Amanda user in a professional setting (thankfully now deep
in my past), I read most of this with a mixed sense of nostalgia (oh yes
I remember that) and pleasure that I am no longer having to put up with
it, although once I got to "totally painless" I almost spat out my tea. 


Since you are all set up already and it's working great for you I
wouldn't suggest you change anything but for anyone who isn't already
invested in Amanda, the process you describe there is considerably more
awkward than that offered by many more modern tools.

Re: Backup Times on a Linux desktop

[Jonathan Dowland]

I'll respond on the issue of triggering the backup, rather than the
specific backup software itself, because my solution for triggering
is separate from the backup software I use (rdiff-backup).

I trigger (some) backup jobs via systemd units, that are triggered by
the insertion of my removeable backup drive. So, I would suggest that
instead of doing a network backup to your 4T drive on the other side of
your pi, you could attach the drive directly to your Computer when you
want to initiate a backup. This doesn't address your desire to have it
happen in the background, though, because you would still need to
remember (or prompt yourself) to attach the drive. I provide the details
anyway just in case they are interesting.

My "backup-exthdd.service" is what performs the actual backup job:

   [Unit]
   OnFailure=status-email-user@%n.service blinkstick-fail.service
   Requires=systemd-cryptsetup@extbackup.service
   After=systemd-cryptsetup@extbackup.service

   [Service]
   Type=oneshot
   ExecStart=/bin/mount /extbackup
   ExecStart=
   ExecStop=/bin/umount /extbackup
   ExecStop=/usr/local/bin/blinkstick --index 1 --limit 10 --set-color green

   [Install]
   
WantedBy=dev-disk-by\x2duuid-e0eed9b6\x2d03f1\x2d41ed\x2d80a4\x2dc7cc4ff013c3.device

(the mount and umount Execs there shouldn't be needed, they should be
addressed by systemd unit dependencies, but in practice they were
necessary when I set this up. This was a while ago and systemd may
perform differently now.)

My external backup disk has an encrypted partition on it. So, the job
above actually depends upon the decrypted partition. The job
"systemd-cryptsetup@extbackup.service" handles that. The skeleton of the
job was written by systemd-cryptsetup-generator automatically, based on
content in /etc/crypttab; I then had to adapt it further. The entirety
of it is:

   [Unit]
   Description=Cryptography Setup for %I
   SourcePath=/etc/crypttab
   DefaultDependencies=no
   Conflicts=umount.target
   BindsTo=dev-mapper-%i.device
   IgnoreOnIsolate=true
   After=systemd-readahead-collect.service systemd-readahead-replay.service 
cryptsetup-pre.target
   Before=cryptsetup.target
   
BindsTo=dev-disk-by\x2duuid-e0eed9b6\x2d03f1\x2d41ed\x2d80a4\x2dc7cc4ff013c3.device
   
After=dev-disk-by\x2duuid-e0eed9b6\x2d03f1\x2d41ed\x2d80a4\x2dc7cc4ff013c3.device
   Before=umount.target
   StopWhenUnneeded=true

   [Service]
   Type=oneshot
   RemainAfterExit=yes
   TimeoutSec=0
   ExecStart=/lib/systemd/systemd-cryptsetup attach 'extbackup' 
'/dev/disk/by-uuid/e0eed9b6-03f1-41ed-80a4-c7cc4ff013c3' '/root/exthdd.key' 
'luks,noauto'
   ExecStop=/lib/systemd/systemd-cryptsetup detach 'extbackup'

So when the remote disk device with the UUID
e0eed9b6-03f1-41ed-80a4-c7cc4ff013c3 appears on the system, its
appearance causes systemd to start the "backup-exthdd.service" job,
which depends upon the bits to enable the encrypted volume.

(the "blinkstick-fail.service" and ExecStop=/usr/local/bin/blinkstick…
line relate to a notification system I have: this is my headless NAS,
and the "blinkstick" is a little multicolour LED attached via USB. In
normal circumstances it is switched off. When a job is running it
changes to a particular colour; when the job finished successfully, it's
green - indicating I can unplug the drive (it's all unmounted etc.), if
anything goes wrong it turns red.)

Re: Backup Times on a Linux desktop

[Alessandro Baggi]

On 02/11/19 20:24, Konstantin Nebel wrote:

Hi,

this is basically a question, what you guys prefer and do. I have a Linux
destkop and recently I decided to buy a raspberry pi 4 (great device) and
already after a couple days I do not know how I lived without it. So why
Raspberrypi.

In the past I decided not to do backups on purpose. I decided that my data on
my local Computer is not important and to store my important stuff in a
nextcloud I host for myself and do backups of that. And for a long period of
time I was just fine with it.

Now i attached a 4 tb drive to my pi and I decided what the heck, why not
doing backups now.

So now I am thinking. How should I approach backups. On windows it does
magically backups and remind me when they didnt run for a while. I like that
attitude.

On linux with all that decision freedom it can be good and bad cause you have
to think about things :D

(SKIP THIS IF U DONT WANT TO READ TOO MUCH) ;)
So I could do the backup on logout for example but I am not sure if that is
not annoying so I'd like to have your opinion. Oh and yeah. I like to turn off
my computer at night. So a backup running in night is not really an option
unless I do wake on lan and run backup and then turn off. But right now I have
dual boot and Windows on default (for games, shame on me) and I might switch
cause first Gaming on Linux is really becoming rly good and second I could buy
second GPU for my Linux and then forward my GPU to a windows VM running my
games in 3d... Especially after buying Ryzen 3900X (that a monster of cpu)

Whoever read till the end Im thankful and ready to hear your opinion.


Cheers
Konstantin



Hi Konstantin,
In my linux experience I found several solution for backup.
First of all rsync.

Scripted rsync is well suited for your situation. Remember that rsync is 
not a backup tool/system alone, it is very helpfull when you need to 
sync file between hosts. Over this you can user --backup option that 
saves the last copy before it will be overwritten by the new copy in a 
different dir. You can use SSH to add encryption during transfer. If you 
add a catalog and configuration you can use it for multiple client.
In then past I ran my scripted rsync backup tool, with catalog, 
prejob/postjob script etc.



Then I encountered bacula. bacula is a beast, complex, hard to configure 
in the first time but it is very powerfull. It permit pooling, 
scheduling, mailing, encryption, multiple clients, prejob/postjob script 
on server and on client, storage on tape or disks, has its own scheduler 
like cron that works very well, volume recycling, Client GUI, Windows 
Client, Web Interface and much more.
I used it for several server and works great. In some situation I prefer 
run rsync to a local machine before run backup because on large datasets 
it requires more times and more overhead on network bandwidth plus all 
operation like stopping services + create lvm snapshot ecc With 
large datasets rsync permit to sync file very quickly so I can block my 
service for very small amount of time and the perform backup locally on 
synced dataset.



There are also other backup tool like rsnapshot (based on rsync) and I 
think this is the best solution for you. There is bareOS (a clone of 
bacula), amanda, restic, duplicity, BackupPC and borg.


Borg seems very promising but I performs only push request at the moment 
and I need pull request. It offers deduplication, encryption and much more.


One word on deduplication: it is a great feature to save space, with 
deduplication compression ops (that could require much time) are avoided 
but remember that with deduplication for multiple backups only one 
version of this files is deduplicated. So if this file get corrupted 
(for every reason) it will be compromised on all  previous backups jobs 
performed, so the file is lost. For this I try to avoid deduplication on 
important backup dataset.


My 2 cents.

Re: Backup Times on a Linux desktop

[Gene Heskett]

On Sunday 03 November 2019 01:49:15 ghe wrote:

> > On Nov 2, 2019, at 05:42 PM, Linux-Fan  wrote:
> >
> > Konstantin Nebel writes:
> >> this is basically a question, what you guys prefer and do. I have a
> >> Linux destkop and recently I decided to buy a raspberry pi 4 (great
> >> device) and
> >
> > [...]
> >
> >> Now i attached a 4 tb drive to my pi and I decided what the heck,
> >> why not doing backups now.
> >>
> >> So now I am thinking. How should I approach backups. On windows it
> >> does magically backups and remind me when they didnt run for a
> >> while. I like that attitude.
>
> I've used Amanda (in a shell script like Gene does) for going on 20
> years. It's been rock solid. I use it with tape, but I hear it backs
> up to disks too. The only thing I don't like about it (coming from
> memory of the experience when I configured it these many years ago) is
> that it's pretty difficult to get going. Now that it's going, though,
> I easily change things all the time.
>
> Recovering from a backup is a reasonable job. I don't do it very
> often, but the recovery software is pretty good about asking questions
> and providing help if you haven't used it for a few months.

Just 4 or 5 days ago, I had to recover the linuxcnc configs from a backup 
of the pi3, making a scratch dir here at home, then scanned my database 
for the last level0 of the pi3b, pulled that out with amrecover then 
copied what I needed back to the rpi4 now living in my Sheldon Lathes 
control box. File moving done by an old friend, mc, and sshfs mounts. 
Totally painless, and all the magic I had written on the pi3 is now 
running considerably better on the pi4.  Saves me from having to 
re-invent at least 2 wheels you won't find on any other cnc lathes, that 
took me over 6 months to develop.

> It doesn't do magic, like Winders does, though. You have to tell it
> what kind of magic you'd like. I have a cron job that runs the backup
> every couple days (in the middle of the night). And another that
> reminds me to change the tape -- amanda whines when I ignore the
> reminder/forget to change the tape. Very thoughtful and well done
> software.
>
> Oh, and it backs up all the computers on my LAN, including my 'Pi. And
> as best I know, it's strictly CLI.


I don't use tapes, but virtual tapes, 60 of them on a 2TB drive, and 
which are nothing but a directory entry on the disk. So I have backups 
up to 60 days old instantly available. Using tapes nearly bankrupted me 
as the hard drives will outlast tapes by at least 1000/1. Amanda ages 
out the old ones and reuses the space on the drives with no drama.  This 
wouldn't do for business records of course, but is ideal for a home 
workshop. I started out with a 1T drive but after several years it was 
getting crowded, so it got replaced with a 2T about a year back.  The 
retired 1T has nearly 80,000 spinning hours on it with a perfect bill of 
health from smartctl.
Cheers, Gene Heskett
-- 
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis
Genes Web page 

Re: Backup Times on a Linux desktop

[ghe]

> On Nov 2, 2019, at 05:42 PM, Linux-Fan  wrote:
> 
> Konstantin Nebel writes:
> 
>> this is basically a question, what you guys prefer and do. I have a Linux
>> destkop and recently I decided to buy a raspberry pi 4 (great device) and
> 
> [...]
> 
>> Now i attached a 4 tb drive to my pi and I decided what the heck, why not
>> doing backups now.
>> 
>> So now I am thinking. How should I approach backups. On windows it does
>> magically backups and remind me when they didnt run for a while. I like that
>> attitude.

I've used Amanda (in a shell script like Gene does) for going on 20 years. It's 
been rock solid. I use it with tape, but I hear it backs up to disks too. The 
only thing I don't like about it (coming from memory of the experience when I 
configured it these many years ago) is that it's pretty difficult to get going. 
Now that it's going, though, I easily change things all the time.

Recovering from a backup is a reasonable job. I don't do it very often, but the 
recovery software is pretty good about asking questions and providing help if 
you haven't used it for a few months.

It doesn't do magic, like Winders does, though. You have to tell it what kind 
of magic you'd like. I have a cron job that runs the backup every couple days 
(in the middle of the night). And another that reminds me to change the tape -- 
amanda whines when I ignore the reminder/forget to change the tape. Very 
thoughtful and well done software.

Oh, and it backs up all the computers on my LAN, including my 'Pi. And as best 
I know, it's strictly CLI.

-- 
Glenn English

Re: Backup Times on a Linux desktop

[elvis]

On 3/11/19 5:24 am, Konstantin Nebel wrote:

Hi,

this is basically a question, what you guys prefer and do. I have a Linux
destkop and recently I decided to buy a raspberry pi 4 (great device) and
already after a couple days I do not know how I lived without it. So why
Raspberrypi.

In the past I decided not to do backups on purpose. I decided that my data on
my local Computer is not important and to store my important stuff in a
nextcloud I host for myself and do backups of that. And for a long period of
time I was just fine with it.

Now i attached a 4 tb drive to my pi and I decided what the heck, why not
doing backups now.

So now I am thinking. How should I approach backups. On windows it does
magically backups and remind me when they didnt run for a while. I like that
attitude.

On linux with all that decision freedom it can be good and bad cause you have
to think about things :D

(SKIP THIS IF U DONT WANT TO READ TOO MUCH) ;)
So I could do the backup on logout for example but I am not sure if that is
not annoying so I'd like to have your opinion. Oh and yeah. I like to turn off
my computer at night. So a backup running in night is not really an option
unless I do wake on lan and run backup and then turn off. But right now I have
dual boot and Windows on default (for games, shame on me) and I might switch
cause first Gaming on Linux is really becoming rly good and second I could buy
second GPU for my Linux and then forward my GPU to a windows VM running my
games in 3d... Especially after buying Ryzen 3900X (that a monster of cpu)
an
Whoever read till the end Im thankful and ready to hear your opinion.


I use Bacula which runs the backups on a schedule, but you can also 
trigger them with scripting if you want them done during the day.






Cheers
Konstantin


--
Auntie Em:  Hate you, Hate Kansas, Taking the dog.  -Dorothy.

Re: Backup Times on a Linux desktop

[Rick Thomas]

On Sat, Nov 2, 2019, at 1:30 PM, Konstantin Nebel wrote:
> Hi,
> 
> > Anyway from my experience borg is the best and I can recommend it wormly.
> 
> I appreciate you answering in the fullest how you do backups and I used borg
> in the past which I can recommend as well. But I really want to focus on how
> to trigger the backup in an automated way and not in which tool is recommended
> to use.
> 
> > My use case is quite opposite. I shut down the backup server as I do only
> > weekly and monthly backups.
> 
> I assume u trigger them manually then? I would like to do them automatically
> and hopefully forget about them that they exist and do the occasional check if
> they work or not :)

For this I use "rsnapshot," which schedules it's backup runs via Linux "cron".  
It's pretty much "fire and forget" once you've done the configuration details.

Hope that helps!
Rick

Re: Backup Times on a Linux desktop

[Linux-Fan]

Konstantin Nebel writes:


this is basically a question, what you guys prefer and do. I have a Linux
destkop and recently I decided to buy a raspberry pi 4 (great device) and


[...]


Now i attached a 4 tb drive to my pi and I decided what the heck, why not
doing backups now.

So now I am thinking. How should I approach backups. On windows it does
magically backups and remind me when they didnt run for a while. I like that
attitude.


[...]


So I could do the backup on logout for example but I am not sure if that is
not annoying so I'd like to have your opinion. Oh and yeah. I like to turn


[...]


Whoever read till the end Im thankful and ready to hear your opinion.


[...]

My opinion on the matter is this: Go for a good (fast) tool and trigger it
often :) Borg has been mentioned and might be very good (I wrote my backup
tool myself but that is probably less good :) )

About the times of triggering: For me (also as a "Desktop" user of sorts), I
actually do a variant of backup on logout which is actually "backup before
shutdown". I do it by using a custom script called `mahalt` which I invoke
to shutdown my computer. Before triggering the actual shutdown, it invokes
the backup procedure.

As a "Laptop" user (current situation), I can really not be sure that I will
have the time to await the backup (on my "Desktop" it takes about 2 minutes
or so which is really acceptable for shutdown). Thus for the "Laptop" usage,
I trigger backup manually once per day (usually as the last action of the
day's computer usage) and it runs slightly faster because everything is on
SSD.

If that triggering intervals are good for you also depends on the amount of
data. If it is much (as per >50 GiB or so), the detection of changed files
(even with good tools) will take some considerable amount of time and thus a
"more rare" triggering interval might be good. Still, triggering it entirely
in background (so as to have enough "time" for the backup) should be
considered with extreme care, because unexpectedly triggered backups can not
only impair system performance but also backup an "inconsistent" state of
sorts (e.g. opened files or partial directory structures if large parts of
the structure are just being copied/renamed etc.).

Btw. as I am paranoid when it comes to backups I always do multi-level
backup: The first copy is on the very same computer's HDD (same for
Laptop/Desktop), the second copy goes to a "mini" computer similar to
Raspberry Pi. Then for the "Desktop" I usually have all HDDs on RAID1 and
the "mini" computer synchronizes to an online file storage service. For the
"Laptop" I sync to the "mini" computer and to a separate SD card (via rsync
at the moment and also triggered manually once per day). In any case, at
least one of the storage locations is offsite...

HTH
Linux-Fan

Re: Backup Times on a Linux desktop

[Charles Curley]

On Sat, 02 Nov 2019 21:30:32 +0100
Konstantin Nebel  wrote:

> But I really want to focus on how
> to trigger the backup in an automated way and not in which tool is
> recommended to use.

cron, unless the program you select has a built-in equivalent.

But it's hard to get good backups while the machine is shut down.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/


pgpT5XPhp2E4R.pgp
Description: OpenPGP digital signature

Re: Backup Times on a Linux desktop

[deloptes]

Konstantin Nebel wrote:

>> Anyway from my experience borg is the best and I can recommend it wormly.
> 
> I appreciate you answering in the fullest how you do backups and I used
> borg in the past which I can recommend as well. But I really want to focus
> on how to trigger the backup in an automated way and not in which tool is
> recommended to use.
> 

I think it is not only me that misunderstood the point of the question.

>> My use case is quite opposite. I shut down the backup server as I do only
>> weekly and monthly backups.
> 
> I assume u trigger them manually then? I would like to do them
> automatically and hopefully forget about them that they exist and do the
> occasional check if they work or not :)

Yes, on my list is still to find a way to trigger one single backup in a
time window when the backup server is available. I want to also explore the
snapshot function of LVM. This is my use case.

In any case a backup is time consuming process and what many suggest is to
sync the content you are interested in permanently to a local machine and
then do your regular backup from there.

This would mean you use something like inotify and sync the files to the PI4
and have a cronjob that does the backup at some point of time from the
local directory. To me it looks very clean and simple.
https://linuxhint.com/inotofy-rsync-bash-live-backups/

there are couple of other articles

regards

Re: Backup Times on a Linux desktop

[Dominic Knight]

On Sat, 2019-11-02 at 20:24 +0100, Konstantin Nebel wrote:
> Hi,
>
>

> dual boot and Windows on default (for games, shame on me) and I might
> switch
> cause first Gaming on Linux is really becoming rly good and second I
> could buy

Missing out answers to your question :)
Yes Gaming on Linux is becoming really good, running Flatpak Steam on
Debian testing and only one game for Linux I have is dead (and that
might well work if I really bothered), less than 0.5 %. Proton also
works very well, the few Windows games I have run as well and some
times better than their target OS.

Cheers,
Dom.