Re: [Hampshire] backup migrations

2012-02-24 Thread Andy Smith 
Hello,

On Fri, Feb 24, 2012 at 09:34:06PM +, James Courtier-Dutton wrote:
> I have looked at rsnapshot.
> It uses hardlinks, so if I need to copy the backup data itself, I will
> have to find a special copy program that preserves hard links.

GNU cp, GNU tar, rsync, …

> I am sure they exist, but not found an answer on google yet. I know
> how to handle copy softlnks, just not hardlinks yet.

You don't have to do anything special unless you want to preserve
the hardlink. i.e. if you used tar or cp or rsync to backup your
backup but forgot to make them preserve hardlinks, you'd still have
a functioning backup but it would be many times bigger as it would
contain multiple copies of each file. So not the end of the world if
you did forget.

> The final requirement I have is to be able to select a file, and see
> the dates when it changed.

$ ls -ilh */clamps.bitfolk.com/home/minecraft/minecraft/world/level.dat
8635494 -rw-r- 1 1002 1002 250 2012-02-24 08:00 
daily.0/clamps.bitfolk.com/home/minecraft/minecraft/world/level.dat
8635924 -rw-r- 1 1002 1002 249 2012-02-23 07:59 
daily.1/clamps.bitfolk.com/home/minecraft/minecraft/world/level.dat
8635927 -rw-r- 1 1002 1002 248 2012-02-22 08:14 
daily.2/clamps.bitfolk.com/home/minecraft/minecraft/world/level.dat
8635849 -rw-r- 1 1002 1002 246 2012-02-21 07:58 
daily.3/clamps.bitfolk.com/home/minecraft/minecraft/world/level.dat
8635723 -rw-r- 1 1002 1002 245 2012-02-20 07:35 
daily.4/clamps.bitfolk.com/home/minecraft/minecraft/world/level.dat
8635560 -rw-r- 1 1002 1002 249 2012-02-19 07:57 
daily.5/clamps.bitfolk.com/home/minecraft/minecraft/world/level.dat
8635520 -rw-r- 1 1002 1002 248 2012-02-18 08:15 
daily.6/clamps.bitfolk.com/home/minecraft/minecraft/world/level.dat
1859665 -rw-r- 1 1002 1002 229 2012-01-02 07:43 
monthly.0/clamps.bitfolk.com/home/minecraft/minecraft/world/level.dat
8636046 -rw-r- 1 1002 1002 247 2012-02-13 07:35 
weekly.0/clamps.bitfolk.com/home/minecraft/minecraft/world/level.dat
8635793 -rw-r- 1 1002 1002 250 2012-02-06 07:39 
weekly.1/clamps.bitfolk.com/home/minecraft/minecraft/world/level.dat
1859634 -rw-r- 1 1002 1002 246 2012-01-30 07:35 
weekly.2/clamps.bitfolk.com/home/minecraft/minecraft/world/level.dat
1859659 -rw-r- 1 1002 1002 246 2012-01-23 07:40 
weekly.3/clamps.bitfolk.com/home/minecraft/minecraft/world/level.dat

Versions that are the same would have identical inode number (first
number). All these are different so every instance is a unique file.

$ ls -ilh */clamps.bitfolk.com/home/minecraft/.viminfo
8299529 -rw--- 5 1002 1002 5.0K 2012-02-19 23:02 
daily.0/clamps.bitfolk.com/home/minecraft/.viminfo
8299529 -rw--- 5 1002 1002 5.0K 2012-02-19 23:02 
daily.1/clamps.bitfolk.com/home/minecraft/.viminfo
8299529 -rw--- 5 1002 1002 5.0K 2012-02-19 23:02 
daily.2/clamps.bitfolk.com/home/minecraft/.viminfo
8299529 -rw--- 5 1002 1002 5.0K 2012-02-19 23:02 
daily.3/clamps.bitfolk.com/home/minecraft/.viminfo
8299529 -rw--- 5 1002 1002 5.0K 2012-02-19 23:02 
daily.4/clamps.bitfolk.com/home/minecraft/.viminfo
8299539 -rw--- 6 1002 1002 4.8K 2012-01-12 16:18 
daily.5/clamps.bitfolk.com/home/minecraft/.viminfo
8299539 -rw--- 6 1002 1002 4.8K 2012-01-12 16:18 
daily.6/clamps.bitfolk.com/home/minecraft/.viminfo
8299524 -rw--- 1 1002 1002 4.8K 2011-12-28 09:54 
monthly.0/clamps.bitfolk.com/home/minecraft/.viminfo
8299539 -rw--- 6 1002 1002 4.8K 2012-01-12 16:18 
weekly.0/clamps.bitfolk.com/home/minecraft/.viminfo
8299539 -rw--- 6 1002 1002 4.8K 2012-01-12 16:18 
weekly.1/clamps.bitfolk.com/home/minecraft/.viminfo
8299539 -rw--- 6 1002 1002 4.8K 2012-01-12 16:18 
weekly.2/clamps.bitfolk.com/home/minecraft/.viminfo
8299539 -rw--- 6 1002 1002 4.8K 2012-01-12 16:18 
weekly.3/clamps.bitfolk.com/home/minecraft/.viminfo

There's actually only three different files there.

IMHO disadvantages of rsnapshot:

- Very shonky do-it-yourself approach to scheduling of backup runs.
  You trigger them from cron; if they overrun it can get very
  confusing. Other backup systems have fancier scheduling facilities.

- Can only use hardlinks within the backup space of one host, for
  same path.

  So for example two instances of
  /home/minecraft/minecraft/world/level.dat on clamps.bitfolk.com
  that are identical would be hardlinked, but if that same content
  was elsewhere on that same host, or elsewhere on another host, it
  wouldn't be hardlinked. Other backup systems (like backuppc) can
  deduplicate on a per-file basis regardless of where the file is.

- Will break a hardlink even on a metadata change.

  For example if you change ownership or permissions of a file then
  you'll get two copies stored even if the contents are identical.
  Other backup systems (like backuppc) store the metadata separately
  from the content and handle this.

- Any time a file changes a complete copy of both versions is
  stored, not diffs. This is inherent to the hardlink design c

Re: [Hampshire] backup migrations

2012-02-24 Thread Anton Piatek 
afraid.org is good because you can use your own domains for free with them.
Few others do.

Anton
--
Anton Piatek
(sent from my phone, please excuse any typos)
email: an...@piatek.co.uk
blog/photos: http://www.strangeparty.com
pgp: [74B1FA37] (http:// www.strangeparty.com/anton.asc)

No trees were destroyed in the sending of this message, however, a
significant number of electrons were terribly inconvenienced.
On Feb 24, 2012 9:34 PM, "James Courtier-Dutton" 
wrote:

> On 15 February 2012 12:38, Imran Chaudhry  wrote:
> > Another vote for rsnapshot here. As Leo says it works over rsync and you
> can
> > just feed rsnapshot the rsync options you are already using. It supports
> > daily, weekly, monthly out the box so it's just a matter of commenting
> out
> > the relevant lines in the config. My approach would be to set-up
> rsnapshot
> > to do an immediate backup of something small so the directory structure
> is
> > correct, then copy your backup data into the relevant place, then
> perform a
> > "dry run" to confirm that only the diffs are transferred.
> >
> > The only snag would be that the backups are not encrypted, in which case
> > duplicity might be your best bet. However it comes at the cost of
> complexity
> > and I found that restoring backups becomes non-trivial. Another HantsLUG
> > member wrote a good guide to duplicity a while back on his personal wiki.
> >
> >
> > On 14 February 2012 19:36, Leo  wrote:
> >>
> >> Have you seen rsnapshot? That might be able to do what you want and
> works
> >> on rsync, so you might be able to avoid copying everything again.
> >>
> >> Leo
> >>
>
> I have looked at rsnapshot.
> It uses hardlinks, so if I need to copy the backup data itself, I will
> have to find a special copy program that preserves hard links.
> I am sure they exist, but not found an answer on google yet. I know
> how to handle copy softlnks, just not hardlinks yet.
> I have also looked further at duplicity, and found that I can convert
> my current backup into a duplicity one without having to copy
> everything across the WAN/Internet again.
> The final requirement I have is to be able to select a file, and see
> the dates when it changed.
> I don't think duplicity or rsnapshot has this feature but I am still
> looking.
>
> --
> Please post to: Hampshire@mailman.lug.org.uk
> Web Interface: https://mailman.lug.org.uk/mailman/listinfo/hampshire
> LUG URL: http://www.hantslug.org.uk
> --
>
--
Please post to: Hampshire@mailman.lug.org.uk
Web Interface: https://mailman.lug.org.uk/mailman/listinfo/hampshire
LUG URL: http://www.hantslug.org.uk
--

Re: [Hampshire] backup migrations

2012-02-24 Thread James Courtier-Dutton 
On 15 February 2012 12:38, Imran Chaudhry  wrote:
> Another vote for rsnapshot here. As Leo says it works over rsync and you can
> just feed rsnapshot the rsync options you are already using. It supports
> daily, weekly, monthly out the box so it's just a matter of commenting out
> the relevant lines in the config. My approach would be to set-up rsnapshot
> to do an immediate backup of something small so the directory structure is
> correct, then copy your backup data into the relevant place, then perform a
> "dry run" to confirm that only the diffs are transferred.
>
> The only snag would be that the backups are not encrypted, in which case
> duplicity might be your best bet. However it comes at the cost of complexity
> and I found that restoring backups becomes non-trivial. Another HantsLUG
> member wrote a good guide to duplicity a while back on his personal wiki.
>
>
> On 14 February 2012 19:36, Leo  wrote:
>>
>> Have you seen rsnapshot? That might be able to do what you want and works
>> on rsync, so you might be able to avoid copying everything again.
>>
>> Leo
>>

I have looked at rsnapshot.
It uses hardlinks, so if I need to copy the backup data itself, I will
have to find a special copy program that preserves hard links.
I am sure they exist, but not found an answer on google yet. I know
how to handle copy softlnks, just not hardlinks yet.
I have also looked further at duplicity, and found that I can convert
my current backup into a duplicity one without having to copy
everything across the WAN/Internet again.
The final requirement I have is to be able to select a file, and see
the dates when it changed.
I don't think duplicity or rsnapshot has this feature but I am still looking.

--
Please post to: Hampshire@mailman.lug.org.uk
Web Interface: https://mailman.lug.org.uk/mailman/listinfo/hampshire
LUG URL: http://www.hantslug.org.uk
--

Re: [Hampshire] backup migrations

2012-02-15 Thread Imran Chaudhry 
Another vote for rsnapshot here. As Leo says it works over rsync and you
can just feed rsnapshot the rsync options you are already using. It
supports daily, weekly, monthly out the box so it's just a matter of
commenting out the relevant lines in the config. My approach would be to
set-up rsnapshot to do an immediate backup of something small so the
directory structure is correct, then copy your backup data into the
relevant place, then perform a "dry run" to confirm that only the diffs are
transferred.

The only snag would be that the backups are not encrypted, in which case
duplicity might be your best bet. However it comes at the cost of
complexity and I found that restoring backups becomes non-trivial. Another
HantsLUG member wrote a good guide to duplicity a while back on his
personal wiki.

On 14 February 2012 19:36, Leo  wrote:

> Have you seen rsnapshot? That might be able to do what you want and works
> on rsync, so you might be able to avoid copying everything again.
>
> Leo
>
>
> --
> Please post to: Hampshire@mailman.lug.org.uk
> Web Interface: 
> https://mailman.lug.org.uk/**mailman/listinfo/hampshire
> LUG URL: http://www.hantslug.org.uk
> --**--**--
>



-- 
GPG Key fingerprint = B323 477E F6AB 4181 9C65  F637 BC5F 7FCC 9CC9 CC7F

“Live as if you were to die tomorrow. Learn as if you were to live
forever.” - Indian political and spiritual leader Mahatma Gandhi (1869-1948)
--
Please post to: Hampshire@mailman.lug.org.uk
Web Interface: https://mailman.lug.org.uk/mailman/listinfo/hampshire
LUG URL: http://www.hantslug.org.uk
--

Re: [Hampshire] backup migrations

2012-02-14 Thread Anton Piatek 
I just use cp with hardlinks to make a copy every month. Similar to
rsnapshot in many ways.

Anton
-
Anton Piatek
(sent from my phone, please excuse any typos)
email: an...@piatek.co.uk
blog/photos: http://www.strangeparty.com
pgp: [74B1FA37] (http:// www.strangeparty.com/anton.asc)

No trees were destroyed in the sending of this message, however, a
significant number of electrons were terribly inconvenienced.
On Feb 14, 2012 7:36 PM, "Leo"  wrote:

> Have you seen rsnapshot? That might be able to do what you want and works
> on rsync, so you might be able to avoid copying everything again.
>
> Leo
>
> --
> Please post to: Hampshire@mailman.lug.org.uk
> Web Interface: 
> https://mailman.lug.org.uk/**mailman/listinfo/hampshire
> LUG URL: http://www.hantslug.org.uk
> --**--**--
>
--
Please post to: Hampshire@mailman.lug.org.uk
Web Interface: https://mailman.lug.org.uk/mailman/listinfo/hampshire
LUG URL: http://www.hantslug.org.uk
--

Re: [Hampshire] backup migrations

2012-02-14 Thread Leo 
Have you seen rsnapshot? That might be able to do what you want and 
works on rsync, so you might be able to avoid copying everything again.


Leo

--
Please post to: Hampshire@mailman.lug.org.uk
Web Interface: https://mailman.lug.org.uk/mailman/listinfo/hampshire
LUG URL: http://www.hantslug.org.uk
--

[Hampshire] backup migrations

2012-02-14 Thread James Courtier-Dutton 
Hi,

I currently use rsync to backup data from one server to another over a
slow internet link.
So, at the moment, both sides are in sync with each other.
I wish to move to a differences backup, so that the remote server
keeps a view of what the local server looked like 6 months ago, on a
weekly basis.
I have looked at deja-dup/duplicity, but I have a problem.
How do I migrate the remote rsync copied data into a duplicity remote
backup without having to copy everything over the slow internet link
again?

Kind Regards

James

--
Please post to: Hampshire@mailman.lug.org.uk
Web Interface: https://mailman.lug.org.uk/mailman/listinfo/hampshire
LUG URL: http://www.hantslug.org.uk
--