On Wed, Jun 12, 2013 at 8:13 AM, Daniel Flores <[email protected]> wrote:
> Hello everyone, > > My name is Daniel Flores and my proposal called “HAMMER2 compression > feature” was accepted for this year’s Google Summer of Code. I already > posted the draft of my proposal [1] in this kernel list, so I will not > repeat much of it, but instead I want to focus on some design decisions > that I’ve already made. Since I’m an inexperienced developer at this point, > I’d be happy to receive your suggestions and criticism. > > The feature itself consists in that it attempts to compress a HAMMER2 > block, which is of 64KB in size. The result should be a block of 32KB, > 16KB, 8KB, etc. in size. > > Currently I’m searching for the algorithms that are the most appropriate > for a file system. Particularly I’m searching for algorithms that are very > fast; don’t require a lot of memory and processing power and offer fairly > good compression rate (but not necessarily the best compression rate out of > all). Right now I have two candidates – DEFLATE [2] and LZO [3]. Both of > them have some available implementations which I intend to use, as Alex > suggested in his review of my proposal. > > DEFLATE seems to be a good choice, because it works with small amounts of > data and has a sliding window of 32KB – just nice for a 64KB block. It is > based on another algorithm – LZ77, which is successfully used in > compression feature for NTFS, so hopefully DEFLATE would be good as well. > > LZO seems to be a good choice, because, similarly, it works on small > amounts of data, it is as fast as DEFLATE and was specifically designed to > have a very fast decompression speed. > You might want to consider LZ4 [1]. It was just recently added to ZFS [2]. It's multi-threaded-aware, extremely fast for both compression and decompression, and gives very good compression ratios. It also includes a short-circuit design to prevent wasting time trying to compress already compressed blocks or highly incompressible blocks. It's also OSS and several reference implementations are available in various languages. [1] http://code.google.com/p/lz4/ [2] http://wiki.illumos.org/display/illumos/LZ4+Compression -- Freddie Cash [email protected]
