Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

2014-10-23 Thread Duncan
Robert White posted on Wed, 22 Oct 2014 22:18:09 -0700 as excerpted: On 10/22/2014 09:30 PM, Chris Murphy wrote: Sure. So if Btrfs is meant to address scalability, then perhaps at the moment it's falling short. As it's easy to add large drives and get very large multiple device volumes, the

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

2014-10-23 Thread Zygo Blaxell
On Wed, Oct 22, 2014 at 10:18:09PM -0700, Robert White wrote: On 10/22/2014 09:30 PM, Chris Murphy wrote: Sure. So if Btrfs is meant to address scalability, then perhaps at the moment it's falling short. As it's easy to add large drives and get very large multiple device volumes, the

Re: 5 _thousand_ snapshots? even 160?

2014-10-23 Thread Arnaud Kapp
Hello, First, I'd like to thank you for this is interesting discussion and for pointing efficient snapshotting strategies. My 5k snapshots actually come from 4 subvolumes. I create 8 snapshots per hour because I actually create both a read-only and writable snapshots for each of my volume. Yeah

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

2014-10-22 Thread Tomasz Chmielewski
But 5000 snapshots? Why? Are you *TRYING* to test btrfs until it breaks, or TRYING to demonstrate a balance taking an entire year? Remember a given btrfs filesystem is not necessarily a backup destination for data from one source. It can be, say, 30 or 60 daily snapshots, plus several

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

2014-10-22 Thread Duncan
Tomasz Chmielewski posted on Wed, 22 Oct 2014 09:14:14 +0200 as excerpted: Remember a given btrfs filesystem is not necessarily a backup destination for data from one source. It can be, say, 30 or 60 daily snapshots, plus several monthly, for each data source * number of data sources. So

Re: 5 _thousand_ snapshots? even 160?

2014-10-22 Thread Austin S Hemmelgarn
On 2014-10-21 21:10, Robert White wrote: I don't think balance will _ever_ move the contents of a read only snapshot. I could be wrong. I think you just end up with an endlessly fragmented storage space and balance has to take each chunk and search for someplace else it might better fit. Which

Re: 5 _thousand_ snapshots? even 160?

2014-10-22 Thread Goffredo Baroncelli
On 10/22/2014 03:10 AM, Robert White wrote: Each snapshot is effectively stapling down one version of your entire metadata tree, right ? On the best of my knowledge, I cannot confirm that. I understood (please, be free to correct me if I am wrong) that each snapshot create a copy of the

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

2014-10-22 Thread Zygo Blaxell
On Wed, Oct 22, 2014 at 07:41:32AM +, Duncan wrote: Tomasz Chmielewski posted on Wed, 22 Oct 2014 09:14:14 +0200 as excerpted: Tho that is of course per subvolume. If you have multiple subvolumes on the same filesystem, that can still end up being a thousand or two snapshots per

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

2014-10-22 Thread Robert White
On 10/22/2014 01:08 PM, Zygo Blaxell wrote: I have datasets where I record 14000+ snapshots of filesystem directory trees scraped from test machines and aggregated onto a single server for deduplication...but I store each snapshot as a git commit, not as a btrfs snapshot or even subvolume. We

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

2014-10-22 Thread Zygo Blaxell
On Wed, Oct 22, 2014 at 01:37:15PM -0700, Robert White wrote: On 10/22/2014 01:08 PM, Zygo Blaxell wrote: I have datasets where I record 14000+ snapshots of filesystem directory trees scraped from test machines and aggregated onto a single server for deduplication...but I store each snapshot

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

2014-10-22 Thread Chris Murphy
On Oct 22, 2014, at 4:08 PM, Zygo Blaxell zblax...@furryterror.org wrote: If you have one subvolume per user and 1000 user directories on a server, it's only 5 snapshots per user (last hour, last day, last week, last month, and last year). Sure. So if Btrfs is meant to address

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

2014-10-22 Thread Robert White
On 10/22/2014 09:30 PM, Chris Murphy wrote: Sure. So if Btrfs is meant to address scalability, then perhaps at the moment it's falling short. As it's easy to add large drives and get very large multiple device volumes, the snapshotting needs to scale also. I'd say per user, it's reasonable to

5 _thousand_ snapshots? even 160? (was: device balance times)

2014-10-21 Thread Robert White
That's an unmanageably large and probably pointless number of snapshots guys. I mean 150 is a heck of a lot, and 5000 is almost unfathomable in terms of possible usefulness. Snapshots are cheap but they aren't free. Each snapshot is effectively stapling down one version of your entire

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

2014-10-21 Thread Zygo Blaxell
On Tue, Oct 21, 2014 at 06:10:27PM -0700, Robert White wrote: That's an unmanageably large and probably pointless number of snapshots guys. I mean 150 is a heck of a lot, and 5000 is almost unfathomable in terms of possible usefulness. Snapshots are cheap but they aren't free. This could

Re: 5 _thousand_ snapshots? even 160? (was: device balance times)

2014-10-21 Thread Duncan
Robert White posted on Tue, 21 Oct 2014 18:10:27 -0700 as excerpted: Each snapshot is effectively stapling down one version of your entire metadata tree, right? So imagine leaving tape spikes (little marks on the floor to keep track of where something is so you can put it back) for the last