On Mon, Sep 15, 2014 at 3:06 PM, Christophe Pettus <[email protected]> wrote: > So, an interesting problem has presented itself, and I'm not quite sure how > to solve it. > > One of the (theoretically) nice things about using WAL-E to archive to S3 is > that you can set up a lifecycle rule to move stuff to Glacier over time, so > that old backups remain available while not paying the full S3 price. (And, > of course, they can be deleted entirely after a while.) > > This works great... except for the timeline's .history file. Since S3 -> > Glacier rules are based on creation date, eventually that file will be > migrated to Glacier, and WAL-E restores start failing because it can't get at > it. Since S3 lifecycle rules are path-prefix-based, there's no way to set a > different rule for that file in particular. > > So, some thoughts: > > 1. I'm misunderstanding how this work, and there's an easy way that I've > missed.
I don't use glacier so my expertise is not as great but I don't think you've missed anything obvious. > 2. WAL-E could have an option to put the .history file in a different prefix, > so different lifecycle rules could apply. Feels hackish. I have considered putting history metadata elsewhere for rapid recall of the members of of a timeline, as prefixes aid that in S3's indexing scheme. Right now I have no downstream features to benefit from such a storage-format-upgrade, but this would be one...admittedly, a bit minor. Another entity that should get the same treatment are the "sentinel" (metadata) files for backups. > 3. A different solution that I'm missing. Maybe think a bit more, but there's some reasons to cordon off history files into a prefix. The thing that is most worrisome about this is how to deal with breaking storage compatibility. It's probably not insurmountable but it's a bit of a thing. -- You received this message because you are subscribed to the Google Groups "wal-e" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
