Re: [Openstack] Enabling data deduplication on Swift

2012-03-12 Thread Caitlin Bestler
Andi abes asked: > Doesn't that depend on the ratios of read vs write? > In a read tilted environment (e.g. CDN's, image stores etc), being able to > dedup at the block level in the > relatively rare write case seems a boon. The simplification this could allow > - performing localized dedup >

Re: [Openstack] Enabling data deduplication on Swift

2012-03-11 Thread andi abes
On Sun, Mar 11, 2012 at 3:49 PM, Caitlin Bestler wrote: > Restricting fingerprinting to blocks would make block level compares > possible, but as I noted on an earlier reply > it would *always* require that the blocks be transferred to perform the > calculation. It is a lot harder to double > Ne

Re: [Openstack] Enabling data deduplication on Swift

2012-03-11 Thread Caitlin Bestler
Joe Gordon asked: > Can SHA-1 collisions be generated? If so can you point me to the article? Check Wikipedia on cryptographic hashing and especially "preimage attack" To summarize, SHA-256 is effectively immune from a pre image attack. Even MD5 is effectiely i

Re: [Openstack] Enabling data deduplication on Swift

2012-03-11 Thread Caitlin Bestler
Restricting fingerprinting to blocks would make block level compares possible, but as I noted on an earlier reply it would *always* require that the blocks be transferred to perform the calculation. It is a lot harder to double Network bandwidth than to double storage. Deduplication that only sav

Re: [Openstack] Enabling data deduplication on Swift

2012-03-10 Thread Maru Newby
first then everyone who downloads that ISO will get my altered >>> copy. >>> >>> >>> >>> SHA-256 is really needed to make this type of attack infeasible. >>> >>> >>> >>> I also think that distributed deduplication wor

Re: [Openstack] Enabling data deduplication on Swift

2012-03-10 Thread andi abes
would be great to hear. >> >> >> >> From: openstack-bounces+caitlin.bestler=nexenta@lists.launchpad.net >> [mailto:openstack-bounces+caitlin.bestler=nexenta@lists.launchpad.net] >> On Behalf Of Paulo Ricardo Motta Gomes >> Sent: Thursday, March 08, 2012 1

Re: [Openstack] Enabling data deduplication on Swift

2012-03-10 Thread Maru Newby
a@lists.launchpad.net > [mailto:openstack-bounces+caitlin.bestler=nexenta@lists.launchpad.net] On > Behalf Of Paulo Ricardo Motta Gomes > Sent: Thursday, March 08, 2012 1:19 PM > To: openstack@lists.launchpad.net > > > Subject: [Openstack] Enabling data dedupl

Re: [Openstack] Enabling data deduplication on Swift

2012-03-10 Thread Joe Gordon
exenta@lists.launchpad.net] *On > Behalf Of *Paulo Ricardo Motta Gomes > *Sent:* Thursday, March 08, 2012 1:19 PM > *To:* openstack@lists.launchpad.net > > *Subject:* [Openstack] Enabling data deduplication on Swift > > ** ** > > Hello everyone, > > ** ** >

Re: [Openstack] Enabling data deduplication on Swift

2012-03-09 Thread Caitlin Bestler
er=nexenta@lists.launchpad.net [mailto:openstack-bounces+caitlin.bestler=nexenta@lists.launchpad.net] On Behalf Of Paulo Ricardo Motta Gomes Sent: Thursday, March 08, 2012 1:19 PM To: openstack@lists.launchpad.net Subject: [Openstack] Enabling data deduplication on Swift Hello everyone, I'

[Openstack] Enabling data deduplication on Swift

2012-03-08 Thread Paulo Ricardo Motta Gomes
Hello everyone, I'm a student of the European Master in Distributed Computing (EMDC) currently working on my master thesis on distributed content-addressable storage/deduplication. I'm happy to announce I will be contributing the outcome of my thesis work to OpenStack by enabling both object-leve