Re: Glacier Storage
On 02/23/15 05:28, John Gamble wrote: >> Understood. And for this, the best solution is probably to use something >> other than Tarsnap. Given that you're going to upload them once and never >> touch them again (hopefully) there are lots of tools which will do the job >> for you -- removing the need for deduplication makes everything much simpler. > > Can you suggest what other service might be useful in this situation? I'm > curious to know, as it might be something I'd want to look into at some point. If I was going to put "cold" data into Glacier, I would probably encrypt it using scrypt and then upload it directly. I think the AWS Console even lets you do this from a web browser. The key issue here is that when you take deduplication out of the picture, you can encrypt and upload separately; which means that as long as you do the encryption using a good tool, the tool you use to upload your data doesn't need to be written with any awareness of security or cryptography. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
Re: Glacier Storage
Hi Colin, > Understood. And for this, the best solution is probably to use something > other than Tarsnap. Given that you're going to upload them once and never > touch them again (hopefully) there are lots of tools which will do the job > for you -- removing the need for deduplication makes everything much simpler. Can you suggest what other service might be useful in this situation? I'm curious to know, as it might be something I'd want to look into at some point. Regards, John ## On 23 Feb 2015, at 03:32, Colin Percival wrote: > Hi Hugo & list, > > On 02/22/15 14:51, Hugo Osvaldo Barrera wrote: >> On 2015-02-22 11:56, Colin Percival wrote: >>> I discuss this in some detail in the blog post which Marcin linked to, >>> but the short answer is: It's not possible to mark particular files for >>> "cold storage" due to tarsnap's deduplication; it would theoretically >>> be possible to mark all the files stored with a particular key as being >>> frozen (glaciated?); but the implementation would be a pain given the >>> way that the tarsnap server works right now. >>> >>> This is something I want to support eventually, but it's a long way >>> off. >> >> Thanks for sending out an official response to this. I'd not crossed the >> above mentioned blog post before my initial email to this list. >> >> Yes, I understand that freezing a single file is extremely inconvenient >> and would make duplication a pain/expensive. > > It's not just that -- the tarsnap server doesn't know which blocks were reused > in an archive, so if "cold" and "warm" data was deduplicated together, you > could find that restoring a "warm" archive needed to read a "cold" block of > data. > >> What I had is mind is something like "Backup up 200G of photos that I >> have in my home NAS. I'll only want these if my NAS blows up, which will >> hopefully be never." > > Understood. And for this, the best solution is probably to use something > other than Tarsnap. Given that you're going to upload them once and never > touch them again (hopefully) there are lots of tools which will do the job > for you -- removing the need for deduplication makes everything much simpler. > >> For the rest of my stuff and files I might want access to, tarsnap works >> fine for me. > > Good! And just to reinforce what I say very often: Tarsnap is *a* solution, > and it is the best solution *some* of the time, but most people will want to > use Tarsnap to back up only the most sensitive and/or "value dense" (cost of > losing data divided by the size of the data) material. > > -- > Colin Percival > Security Officer Emeritus, FreeBSD | The power to serve > Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid John Gamble Senior Computer Biologist Cancer Genome Project Wellcome Trust Sanger Institute Cambridge, UK CB10 1SA Tel: +44 (0)1223 - 834244 Ext: 7703 j...@sanger.ac.uk -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
Re: Glacier Storage
Hi Hugo & list, On 02/22/15 14:51, Hugo Osvaldo Barrera wrote: > On 2015-02-22 11:56, Colin Percival wrote: >> I discuss this in some detail in the blog post which Marcin linked to, >> but the short answer is: It's not possible to mark particular files for >> "cold storage" due to tarsnap's deduplication; it would theoretically >> be possible to mark all the files stored with a particular key as being >> frozen (glaciated?); but the implementation would be a pain given the >> way that the tarsnap server works right now. >> >> This is something I want to support eventually, but it's a long way >> off. > > Thanks for sending out an official response to this. I'd not crossed the > above mentioned blog post before my initial email to this list. > > Yes, I understand that freezing a single file is extremely inconvenient > and would make duplication a pain/expensive. It's not just that -- the tarsnap server doesn't know which blocks were reused in an archive, so if "cold" and "warm" data was deduplicated together, you could find that restoring a "warm" archive needed to read a "cold" block of data. > What I had is mind is something like "Backup up 200G of photos that I > have in my home NAS. I'll only want these if my NAS blows up, which will > hopefully be never." Understood. And for this, the best solution is probably to use something other than Tarsnap. Given that you're going to upload them once and never touch them again (hopefully) there are lots of tools which will do the job for you -- removing the need for deduplication makes everything much simpler. > For the rest of my stuff and files I might want access to, tarsnap works > fine for me. Good! And just to reinforce what I say very often: Tarsnap is *a* solution, and it is the best solution *some* of the time, but most people will want to use Tarsnap to back up only the most sensitive and/or "value dense" (cost of losing data divided by the size of the data) material. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
Re: Glacier Storage
On Sun, Feb 22, 2015 at 07:51:27PM -0300, Hugo Osvaldo Barrera wrote: > What I had is mind is something like "Backup up 200G of photos that I have in > my home NAS. I'll only want these if my NAS blows up, which will hopefully be > never." git-annex works well with Amazon Glacier for this purpose. What you lose compared to Tarsnap is whole-archive duplication. For photos though, it should work fine, as it effectively deduplicates individual files that never change since it's content addressed (it uses a hash).
Re: Glacier Storage
On 2015-02-22 11:56, Colin Percival wrote: > Hi Hugo & list, > > On 02/21/15 22:14, Hugo Osvaldo Barrera wrote: > > I need to store some backups files for really long-term (eg: I've no > > intention > > of checking them out unless I have some serious hardware crashes). > > > > I realize that tarsnap uses AWS for file storage, and these sort of files > > could > > really be stored in Amazon Glacier (which has lower storage costs). I'm > > wondering if you guys (eg: tarsnap devs) would ever be willing to implement > > a > > a feature to mark a certain file (or maybe all the files stored with a > > certain > > key) as frozen/long term storge. > > > > I'm not just thinking about reducing costs to the end users; some of the > > savings can be kept by tarsnap itself in these scenarios, and, unless the > > implementation is a pain, there might end up being some profit on both > > sides. > > > > What do you guys think? Is this doable, or is there some technical > > limitation? > > I discuss this in some detail in the blog post which Marcin linked to, but the > short answer is: It's not possible to mark particular files for "cold storage" > due to tarsnap's deduplication; it would theoretically be possible to mark > all the files stored with a particular key as being frozen (glaciated?); but > the implementation would be a pain given the way that the tarsnap server works > right now. > > This is something I want to support eventually, but it's a long way off. > Hi, Thanks for sending out an official response to this. I'd not crossed the above mentioned blog post before my initial email to this list. Yes, I understand that freezing a single file is extremely inconvenient and would make duplication a pain/expensive. What I had is mind is something like "Backup up 200G of photos that I have in my home NAS. I'll only want these if my NAS blows up, which will hopefully be never." For the rest of my stuff and files I might want access to, tarsnap works fine for me. Cheers, and thanks for the great service! 👍 -- Hugo Osvaldo Barrera A: Because we read from top to bottom, left to right. Q: Why should I start my reply below the quoted text? signature.asc Description: PGP signature
Re: Glacier Storage
Hi Hugo & list, On 02/21/15 22:14, Hugo Osvaldo Barrera wrote: > I need to store some backups files for really long-term (eg: I've no intention > of checking them out unless I have some serious hardware crashes). > > I realize that tarsnap uses AWS for file storage, and these sort of files > could > really be stored in Amazon Glacier (which has lower storage costs). I'm > wondering if you guys (eg: tarsnap devs) would ever be willing to implement a > a feature to mark a certain file (or maybe all the files stored with a certain > key) as frozen/long term storge. > > I'm not just thinking about reducing costs to the end users; some of the > savings can be kept by tarsnap itself in these scenarios, and, unless the > implementation is a pain, there might end up being some profit on both sides. > > What do you guys think? Is this doable, or is there some technical limitation? I discuss this in some detail in the blog post which Marcin linked to, but the short answer is: It's not possible to mark particular files for "cold storage" due to tarsnap's deduplication; it would theoretically be possible to mark all the files stored with a particular key as being frozen (glaciated?); but the implementation would be a pain given the way that the tarsnap server works right now. This is something I want to support eventually, but it's a long way off. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
Re: Glacier Storage
On Sun, 22 Feb 2015, Hugo Osvaldo Barrera wrote: > Hi, > > I need to store some backups files for really long-term (eg: I've no intention > of checking them out unless I have some serious hardware crashes). > > I realize that tarsnap uses AWS for file storage, and these sort of files > could > really be stored in Amazon Glacier (which has lower storage costs). I'm > wondering if you guys (eg: tarsnap devs) would ever be willing to implement a > a feature to mark a certain file (or maybe all the files stored with a certain > key) as frozen/long term storge. > > I'm not just thinking about reducing costs to the end users; some of the > savings can be kept by tarsnap itself in these scenarios, and, unless the > implementation is a pain, there might end up being some profit on both sides. > > What do you guys think? Is this doable, or is there some technical limitation? There is a post from Colin related to the subject: http://www.daemonology.net/blog/2012-09-04-why-tarsnap-doesnt-use-glacier.html //Marcin
