Re: [GTALUG] What Not To Backup

2016-12-27 Thread Blaise Alleyne via talk 
On 23/12/16 12:11 PM, John Moniz via talk wrote:
> Hi everyone,
> 
> I'm backing up my system on a more regular basis and am trying to fine tune 
> the
> files that I backup. I am looking for advice on what NOT to bother to backup 
> on
> the /home directory.
> 
> I am using rsync (took a long time and lots of trials to figure out the man 
> page
> - and still don't know 90% of it) and presently have the following on my
> exclude_list.txt:
> (Note: multiple items shown on one line are just for readability, each line in
> the file only has one item)
> 
> tmp* TMP*
> .cache* cache* Cache* CACHE* *CACHE *Cache *cache
> .cookies* cookies*
> Trash Trash* TRASH*
> Junk* junk*
> .gvfs
> Backups backups
> Crash*
> .xsession-errors*
> .macromedia
> .thumbnails
> .mozilla/firefox/*/thumbnails
> *.corrupt
> minidumps
> .local/share/gvfs*
> 
> I'd love to exclude things that perhaps one would never use from a backup to
> rebuild a system after an accidental clean wipe of all data.
> 

Personally, I wouldn't be comfortable with so many wildcards in an rsync
exclude. I compiled mine through trial and error, running manual backups
frequently at first and finding directories that constantly had new stuff to
backup with names like tmp or cache and excluding those.

Here's my rsync exclude file (might be a little out of date, as I constructed in
a few years back on an Ubuntu machine, now using Debian):
# These files are not necessary to backup
*.swp
.cache/
.config/banshee/covers/
.evolution/cache/
.gnash/
.gnome2/epiphany/favicon_cache/
.gnome2/epiphany/mozilla/epiphany/Cache/
.gvfs/
.liferea_1.8/cache/
.local/share/Trash/files/
.macromedia/
.mozilla/firefox*/Profiles/*.default/Cache/
.mozilla/firefox*/Profiles/*.default/*.sqlite
.mozilla/firefox*/Profiles/*.default/weave/
Trash.msf
.mythtv/themecache/
.pulse/
.thumbnails/
.icedove/*.default/Cache/
.icedove/*.default/ImapMail/
.wine/
.wapi
.xchat2/scrollback/


Now, some things in there are conscious decisions, like I'm excluding my
IceDove/Thunderbird ImapMail folder because I don't want to constantly rsync my
cache of my ImapMail folders -- I have proper backups directly from my mail
server instead.


I also have a bunch of other custom folders excluded on any given machine,
usually ~/Downloads/ or some kind of directory where I may download large files
like ISOs which I have no need or desire to be going through my backup system.


> Similarly, any recommendations of what I should back up outside of /home? I am
> thinking of things like /etc/fstab, files that would make it easier to recover
> from a crash or to upgrade a distro.
> 

Here's the script I put in /usr/local/bin/backup to run every hour or so on all
my laptops/N900:

#!/bin/sh
START=`date +%s`
LOCAL_HOST='192.168.2.160'
REMOTE_HOST='myhome.domain.tld'
DEST_DIR="backups/thinkpad-x60"  # this would be different on each client


# Try to connect to host locally, otherwise use remote connection
if ssh -q $LOCAL_HOST exit;
then
HOST=$LOCAL_HOST
else
HOST=$REMOTE_HOST
fi

echo "== Backup to $HOST =="
date

DESTINATION="$HOST:$DEST_DIR"

# Backup home directory (Note: this is only for primary user!)
echo "-- ${HOME} --"
rsync -e ssh -avz --delete --delete-excluded
--exclude-from=${HOME}/.rsync.exclude --numeric-ids --relative ${HOME}
${DESTINATION}

rsync -e ssh -avz --relative --delete --exclude-from=${HOME}/.rsync.etc.exclude
/etc/ ${DESTINATION}/
rsync -e ssh -avz --relative --delete /usr/local/bin/ ${DESTINATION}/

# Calculate elapsed time
END=`date +%s`
ELAPSEDTIME=`expr $END - $START`
echo Finished at: `date` - "It took $ELAPSEDTIME seconds"



I can run manually with `backup`, but I have it set to run every hour.

Not I'm backing up most of /etc/ too. I have an exclude file that leaves out
some stuff there (also trial and error by running the backup manually and seeing
what perhaps wasn't necessary).


Some important notes!

Are there sensitive files you don't want bac

As other people have mentioned, this just mirrors your home directory to a
backup server somewhere, which is awesome, but not a real backup because you
can't go back to older versions.

So what I do is I use rsnapshot for versioned backups.

I have a server running at my apartment, and a server running at my parents' 
place.

- laptops and mobile devices do rsync mirrors to the local server
- the remote server and the opposite place does a nightly rsnapshot of the
backup directories (and other stuff on the servers)

This way, my laptops/mobiles mirror ~hourly to my living room server, but if
anything ever went wrong, I could go back to the last 7 days, last 4 weeks or
last 3-6 months in the rsnapshot backup. And with that being at my parents'
place, it's also in a separate physical location in the even of fire, flood,
theft, etc.

(If there was some kind of nuclear bomb or natural disaster or something that
took out physical locations across Toronto, then I wouldn't be covered, but I
also

Re: [GTALUG] What Not To Backup

2016-12-23 Thread John Moniz via talk 

On 12/23/2016 04:58 PM, William Park via talk wrote:

On Fri, Dec 23, 2016 at 04:13:48PM -0500, Alvin Starr via talk wrote:

Yes.  I do
  telinit 1
  mount -o remount,ro /
before running 'dd'.

Remounting read only will work but that is not what you said to do.
You will need to make sure all file systems are remounted read only.
It will work but could have you taking your system offline for quite a while
to do the full copy if you have something like a 10TB drive.

Our audience is GTALUG members, so they know what we're talking about.
:-)  I have everything in one filesystem.  I do backup of that.  Then,
I take snapshots of the backup.
William, some of us GTALug members are nothing more than home desktop 
users. ;-)

---
Talk Mailing List
talk@gtalug.org
https://gtalug.org/mailman/listinfo/talk

Re: [GTALUG] What Not To Backup

2016-12-23 Thread John Moniz via talk 

On 12/23/2016 04:11 PM, James Knott via talk wrote:

On 12/23/2016 12:11 PM, John Moniz via talk wrote:

I'd love to exclude things that perhaps one would never use from a
backup to rebuild a system after an accidental clean wipe of all data.

Well, you could probably pass on /dev/null.  ;-)


I always back up /dev/null because it never takes up any space. :-)
---
Talk Mailing List
talk@gtalug.org
https://gtalug.org/mailman/listinfo/talk

Re: [GTALUG] What Not To Backup

2016-12-23 Thread William Park via talk 
On Fri, Dec 23, 2016 at 04:13:48PM -0500, Alvin Starr via talk wrote:
> > Yes.  I do
> >  telinit 1
> >  mount -o remount,ro /
> > before running 'dd'.
> 
> Remounting read only will work but that is not what you said to do.
> You will need to make sure all file systems are remounted read only.
> It will work but could have you taking your system offline for quite a while
> to do the full copy if you have something like a 10TB drive.

Our audience is GTALUG members, so they know what we're talking about.
:-)  I have everything in one filesystem.  I do backup of that.  Then,
I take snapshots of the backup.
-- 
William
---
Talk Mailing List
talk@gtalug.org
https://gtalug.org/mailman/listinfo/talk

Re: [GTALUG] What Not To Backup

2016-12-23 Thread Alvin Starr via talk 

On 12/23/2016 03:56 PM, William Park via talk wrote:

On Fri, Dec 23, 2016 at 02:32:17PM -0500, Alvin Starr via talk wrote:

On 12/23/2016 02:21 PM, William Park via talk wrote:

My recommendations...

1.  Backup entire disk to another disk, verbatim.  That is,
dd if=/dev/sdX of=/dev/sdY bs=10M

  First, you don't have to waste time figuring out what to back up.
  Second, if disk fails, you can just swap the disks, and copy over
  only the new files since last dd.

Generally this is a bad idea if your copying a live file system because you
end up with a disk image that is corrupted to some extent.

Yes.  I do
 telinit 1
 mount -o remount,ro /
before running 'dd'.


Remounting read only will work but that is not what you said to do.
You will need to make sure all file systems are remounted read only.
It will work but could have you taking your system offline for quite a 
while to do the full copy if you have something like a 10TB drive.





--
Alvin Starr   ||   voice: (905)513-7688
Netvel Inc.   ||   Cell:  (416)806-0133
al...@netvel.net  ||

---
Talk Mailing List
talk@gtalug.org
https://gtalug.org/mailman/listinfo/talk

Re: [GTALUG] What Not To Backup

2016-12-23 Thread James Knott via talk 
On 12/23/2016 12:11 PM, John Moniz via talk wrote:
> I'd love to exclude things that perhaps one would never use from a
> backup to rebuild a system after an accidental clean wipe of all data.

Well, you could probably pass on /dev/null.  ;-)
---
Talk Mailing List
talk@gtalug.org
https://gtalug.org/mailman/listinfo/talk

Re: [GTALUG] What Not To Backup

2016-12-23 Thread William Park via talk 
On Fri, Dec 23, 2016 at 02:32:17PM -0500, Alvin Starr via talk wrote:
> On 12/23/2016 02:21 PM, William Park via talk wrote:
> > My recommendations...
> > 
> > 1.  Backup entire disk to another disk, verbatim.  That is,
> > dd if=/dev/sdX of=/dev/sdY bs=10M
> > 
> >  First, you don't have to waste time figuring out what to back up.
> >  Second, if disk fails, you can just swap the disks, and copy over
> >  only the new files since last dd.
> Generally this is a bad idea if your copying a live file system because you
> end up with a disk image that is corrupted to some extent.

Yes.  I do 
telinit 1
mount -o remount,ro /
before running 'dd'.
-- 
William
---
Talk Mailing List
talk@gtalug.org
https://gtalug.org/mailman/listinfo/talk

Re: [GTALUG] What Not To Backup

2016-12-23 Thread Howard Gibson via talk 
On Fri, 23 Dec 2016 12:11:15 -0500
John Moniz via talk  wrote:

> Hi everyone,
> 
> I'm backing up my system on a more regular basis and am trying to fine 
> tune the files that I backup. I am looking for advice on what NOT to 
> bother to backup on the /home directory.

John,

   I back up my entire /home directory every night.  I also back up /etc, /root 
and /usr/local, but very much less frequently.  My backup device is a 4TB hard 
drive.  Periodically, I burn my /home backup to Blu-ray.  If I have to recover 
something, I use my latest backup.  I have written a backup script that uses 
tar.  

   KISS. 

-- 
Howard Gibson 
hgib...@eol.ca
howard.gib...@teledyne.com 
jhowardgib...@gmail.com
http://home.eol.ca/~hgibson
---
Talk Mailing List
talk@gtalug.org
https://gtalug.org/mailman/listinfo/talk

Re: [GTALUG] What Not To Backup

2016-12-23 Thread Alvin Starr via talk 

On 12/23/2016 02:21 PM, William Park via talk wrote:

My recommendations...

1.  Backup entire disk to another disk, verbatim.  That is,
dd if=/dev/sdX of=/dev/sdY bs=10M

 First, you don't have to waste time figuring out what to back up.
 Second, if disk fails, you can just swap the disks, and copy over
 only the new files since last dd.
Generally this is a bad idea if your copying a live file system because 
you end up with a disk image that is corrupted to some extent.




2.  Backup entire /home to another disk.  That is,
mount /home/sdZ /mnt/home
rsync -aHxv -SAX /home/ /mnt/home/ --delete --exclude='/*/.gvfs'

 Again, I would copy the whole things.  If you have somethings that
 you don't want to backup, then you can put them in
 /home/dont_backup, and add it to exclude list,
--exclude='/dont_backup'
If your keeping rsync backups make sure you have some way to rotate the 
backups so you have more than just 1.
A single backup only works when the oops/problem... happened since the 
last backup and you catch it before the next backup.




3.  If your /home is on BTRFS filesystem, you can take snapshots, like
btrfs subvolume snapshot -r home home--$(date +%F)

 Harddisk is "cheap", and snapshot is "cheaper".

LVM snapshots are also handy.
It will be nice when btrfs becomes the linux default file system because 
the snapshot feature is just so damn handy.


--
Alvin Starr   ||   voice: (905)513-7688
Netvel Inc.   ||   Cell:  (416)806-0133
al...@netvel.net  ||

---
Talk Mailing List
talk@gtalug.org
https://gtalug.org/mailman/listinfo/talk

Re: [GTALUG] What Not To Backup

2016-12-23 Thread Mauro Souza via talk 
I use backintime for my local backups. It keeps history of files, so even
if you accidentally alter a file and discovers a month later, you can
recover the last version.
If you have a VPS somewhere, like me, you can use syncthing or owncloud
too. They are very good for backups.

I keep a local backintime backup, rsync to a raspberry pi every night, and
sync it to the VPS every week.

On Dec 23, 2016 5:21 PM, "William Park via talk"  wrote:

> My recommendations...
>
> 1.  Backup entire disk to another disk, verbatim.  That is,
> dd if=/dev/sdX of=/dev/sdY bs=10M
>
> First, you don't have to waste time figuring out what to back up.
> Second, if disk fails, you can just swap the disks, and copy over
> only the new files since last dd.
>
> 2.  Backup entire /home to another disk.  That is,
> mount /home/sdZ /mnt/home
> rsync -aHxv -SAX /home/ /mnt/home/ --delete --exclude='/*/.gvfs'
>
> Again, I would copy the whole things.  If you have somethings that
> you don't want to backup, then you can put them in
> /home/dont_backup, and add it to exclude list,
> --exclude='/dont_backup'
>
> 3.  If your /home is on BTRFS filesystem, you can take snapshots, like
> btrfs subvolume snapshot -r home home--$(date +%F)
>
> Harddisk is "cheap", and snapshot is "cheaper".
> --
> William
>
>
> On Fri, Dec 23, 2016 at 12:11:15PM -0500, John Moniz via talk wrote:
> > Hi everyone,
> >
> > I'm backing up my system on a more regular basis and am trying to fine
> tune
> > the files that I backup. I am looking for advice on what NOT to bother to
> > backup on the /home directory.
> >
> > I am using rsync (took a long time and lots of trials to figure out the
> man
> > page - and still don't know 90% of it) and presently have the following
> on
> > my exclude_list.txt:
> > (Note: multiple items shown on one line are just for readability, each
> line
> > in the file only has one item)
> >
> > tmp* TMP*
> > .cache* cache* Cache* CACHE* *CACHE *Cache *cache
> > .cookies* cookies*
> > Trash Trash* TRASH*
> > Junk* junk*
> > .gvfs
> > Backups backups
> > Crash*
> > .xsession-errors*
> > .macromedia
> > .thumbnails
> > .mozilla/firefox/*/thumbnails
> > *.corrupt
> > minidumps
> > .local/share/gvfs*
> >
> > I'd love to exclude things that perhaps one would never use from a
> backup to
> > rebuild a system after an accidental clean wipe of all data.
> >
> > Similarly, any recommendations of what I should back up outside of
> /home? I
> > am thinking of things like /etc/fstab, files that would make it easier to
> > recover from a crash or to upgrade a distro.
> >
> > Thanks for any advice.
> >
> > John.
> > ---
> > Talk Mailing List
> > talk@gtalug.org
> > https://gtalug.org/mailman/listinfo/talk
> ---
> Talk Mailing List
> talk@gtalug.org
> https://gtalug.org/mailman/listinfo/talk
>
---
Talk Mailing List
talk@gtalug.org
https://gtalug.org/mailman/listinfo/talk

Re: [GTALUG] What Not To Backup

2016-12-23 Thread William Park via talk 
My recommendations...

1.  Backup entire disk to another disk, verbatim.  That is,
dd if=/dev/sdX of=/dev/sdY bs=10M

First, you don't have to waste time figuring out what to back up.
Second, if disk fails, you can just swap the disks, and copy over
only the new files since last dd.

2.  Backup entire /home to another disk.  That is,
mount /home/sdZ /mnt/home
rsync -aHxv -SAX /home/ /mnt/home/ --delete --exclude='/*/.gvfs'

Again, I would copy the whole things.  If you have somethings that
you don't want to backup, then you can put them in
/home/dont_backup, and add it to exclude list,
--exclude='/dont_backup'

3.  If your /home is on BTRFS filesystem, you can take snapshots, like
btrfs subvolume snapshot -r home home--$(date +%F)

Harddisk is "cheap", and snapshot is "cheaper".
-- 
William


On Fri, Dec 23, 2016 at 12:11:15PM -0500, John Moniz via talk wrote:
> Hi everyone,
> 
> I'm backing up my system on a more regular basis and am trying to fine tune
> the files that I backup. I am looking for advice on what NOT to bother to
> backup on the /home directory.
> 
> I am using rsync (took a long time and lots of trials to figure out the man
> page - and still don't know 90% of it) and presently have the following on
> my exclude_list.txt:
> (Note: multiple items shown on one line are just for readability, each line
> in the file only has one item)
> 
> tmp* TMP*
> .cache* cache* Cache* CACHE* *CACHE *Cache *cache
> .cookies* cookies*
> Trash Trash* TRASH*
> Junk* junk*
> .gvfs
> Backups backups
> Crash*
> .xsession-errors*
> .macromedia
> .thumbnails
> .mozilla/firefox/*/thumbnails
> *.corrupt
> minidumps
> .local/share/gvfs*
> 
> I'd love to exclude things that perhaps one would never use from a backup to
> rebuild a system after an accidental clean wipe of all data.
> 
> Similarly, any recommendations of what I should back up outside of /home? I
> am thinking of things like /etc/fstab, files that would make it easier to
> recover from a crash or to upgrade a distro.
> 
> Thanks for any advice.
> 
> John.
> ---
> Talk Mailing List
> talk@gtalug.org
> https://gtalug.org/mailman/listinfo/talk
---
Talk Mailing List
talk@gtalug.org
https://gtalug.org/mailman/listinfo/talk

Re: [GTALUG] What Not To Backup

2016-12-23 Thread Alvin Starr via talk 

On 12/23/2016 12:11 PM, John Moniz via talk wrote:

Hi everyone,

I'm backing up my system on a more regular basis and am trying to fine 
tune the files that I backup. I am looking for advice on what NOT to 
bother to backup on the /home directory.


I am using rsync (took a long time and lots of trials to figure out 
the man page - and still don't know 90% of it) and presently have the 
following on my exclude_list.txt:
(Note: multiple items shown on one line are just for readability, each 
line in the file only has one item)


tmp* TMP*
.cache* cache* Cache* CACHE* *CACHE *Cache *cache
.cookies* cookies*
Trash Trash* TRASH*
Junk* junk*
.gvfs
Backups backups
Crash*
.xsession-errors*
.macromedia
.thumbnails
.mozilla/firefox/*/thumbnails
*.corrupt
minidumps
.local/share/gvfs*

I'd love to exclude things that perhaps one would never use from a 
backup to rebuild a system after an accidental clean wipe of all data.


Similarly, any recommendations of what I should back up outside of 
/home? I am thinking of things like /etc/fstab, files that would make 
it easier to recover from a crash or to upgrade a distro.


Thanks for any advice.

Take a look at rdiff-backup I prefer it over rsync because it keeps a 
history as diffs from the current backup.
Backups are not just disaster recovery sometimes you need to go back a 
week to find that file you accidentally deleted.


As a rule I do full backups because I have been burned by backing up 
just the "critical" information only to find that there was other 
information pushed all over the system.


Database backups need to be handled somewhat special because most 
databases do not guarantee that the binary file backup will not be 
corrupted  if the server is active while the data is being backed up.


I will exclude things like /tmp and /var/tmp but I make sure I keep the 
/var/log/* files because the log files are the first thing someone who 
compromises your system will delete.


--
Alvin Starr   ||   voice: (905)513-7688
Netvel Inc.   ||   Cell:  (416)806-0133
al...@netvel.net  ||

---
Talk Mailing List
talk@gtalug.org
https://gtalug.org/mailman/listinfo/talk

Re: [GTALUG] What Not To Backup

2016-12-23 Thread Giles Orr via talk 
On 23 December 2016 at 12:11, John Moniz via talk  wrote:
> I'm backing up my system on a more regular basis and am trying to fine tune
> the files that I backup. I am looking for advice on what NOT to bother to
> backup on the /home directory.
>
> I am using rsync (took a long time and lots of trials to figure out the man
> page - and still don't know 90% of it) and presently have the following on
> my exclude_list.txt:
> (Note: multiple items shown on one line are just for readability, each line
> in the file only has one item)
>
> tmp* TMP*
> .cache* cache* Cache* CACHE* *CACHE *Cache *cache
> .cookies* cookies*
> Trash Trash* TRASH*
> Junk* junk*
> .gvfs
> Backups backups
> Crash*
> .xsession-errors*
> .macromedia
> .thumbnails
> .mozilla/firefox/*/thumbnails
> *.corrupt
> minidumps
> .local/share/gvfs*
>
> I'd love to exclude things that perhaps one would never use from a backup to
> rebuild a system after an accidental clean wipe of all data.
>
> Similarly, any recommendations of what I should back up outside of /home? I
> am thinking of things like /etc/fstab, files that would make it easier to
> recover from a crash or to upgrade a distro.
>
> Thanks for any advice.

If you only have very limited space for backups, that looks like a
pretty good list (very similar to one I had a few years back).  If you
can afford it, I'd recommend just backing up your entire /home/ (and
rsync is a great way to do it).  It's easier because A) you don't have
to maintain that list of exceptions, and B) you don't find out AFTER
your HD crashes that you really did need that .cookies folder to get
back into your bank's account page.  That latter is just a random
example: if you don't back it all up, you'll find out that you needed
something that's no longer there.

Several further recommendations:
- if you have a directory full of Linux ISOs (as I suspect many of us
do), that's a good directory to not back up ... unless you
custom-build them
- back up /etc/ in its entirety
- back up /var/ if you have the space - /var/ is usually small,
although Apt, Docker, and possibly your web server are given to
seriously fattening it up
- the options I use for local copies with rsync are: "--verbose
--recursive --update --times --progress --stats --delete --perms
--hard-links --links" (not necessarily right for you, but may help)
- use the long-form options in scripts: it's a LOT easier to read and
remember what you were trying to achieve
- keep at least three rotating backups, preferably one off-site
- keep your backups on encrypted media if you can

-- 
Giles
http://www.gilesorr.com/
giles...@gmail.com
---
Talk Mailing List
talk@gtalug.org
https://gtalug.org/mailman/listinfo/talk

[GTALUG] What Not To Backup

2016-12-23 Thread John Moniz via talk 

Hi everyone,

I'm backing up my system on a more regular basis and am trying to fine 
tune the files that I backup. I am looking for advice on what NOT to 
bother to backup on the /home directory.


I am using rsync (took a long time and lots of trials to figure out the 
man page - and still don't know 90% of it) and presently have the 
following on my exclude_list.txt:
(Note: multiple items shown on one line are just for readability, each 
line in the file only has one item)


tmp* TMP*
.cache* cache* Cache* CACHE* *CACHE *Cache *cache
.cookies* cookies*
Trash Trash* TRASH*
Junk* junk*
.gvfs
Backups backups
Crash*
.xsession-errors*
.macromedia
.thumbnails
.mozilla/firefox/*/thumbnails
*.corrupt
minidumps
.local/share/gvfs*

I'd love to exclude things that perhaps one would never use from a 
backup to rebuild a system after an accidental clean wipe of all data.


Similarly, any recommendations of what I should back up outside of 
/home? I am thinking of things like /etc/fstab, files that would make it 
easier to recover from a crash or to upgrade a distro.


Thanks for any advice.

John.
---
Talk Mailing List
talk@gtalug.org
https://gtalug.org/mailman/listinfo/talk