Hi Sergi, Didn't know you were cooking this dish in the background ) Excellent. Just to be sure, that's part of this IEP, right? https://cwiki.apache.org/confluence/display/IGNITE/IEP-20%3A+Data+Compression+in+Ignite#IEP-20:DataCompressioninIgnite-Withoutin-memorycompression
Since we can release only full file system blocks which are typically 4k > size, user must configure page size to be at least multiple FS blocks, e.g. > 8k or 16k. It also means that max compression ratio here is fsBlockSize / > pageSize = 4k / 16k = 0.25 How to we handle the case if the page size is not a multiple of 4K? What is the most optimal page size if the user wants to get the best compression? Probably, we can adjust the default page size automatically if it's a clean deployment. There will be 2 new properties on CacheConfiguration > (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk page > compression. How about setting it at DataRegionConfiguration level as well so that it's applied for all the caches/tables from there? -- Denis On Mon, Nov 19, 2018 at 2:01 AM Sergi Vladykin <[email protected]> wrote: > Folks, > > I've implemented page compression for persistent store and going to merge > it to master. > > https://github.com/apache/ignite/pull/5200 > > Some design notes: > > It employs "hole punching" approach, it means that the pages are kept > uncompressed in memory, > but when they get written to disk, they will be compressed and all the > extra file system blocks for the page will be released. Thus the storage > files become sparse. > > Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY and > SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE, which > basically just takes only meaningful data from half-filled pages but does > not apply any compression. It is easy to add more if needed. > > Since we can release only full file system blocks which are typically 4k > size, user must configure page size to be at least multiple FS blocks, e.g. > 8k or 16k. It also means that max compression ratio here is fsBlockSize / > pageSize = 4k / 16k = 0.25 > > It is possible to enable compression for existing databases if they were > configured for large enough page size. In this case pages will be written > to disk in compressed form when updated, and the database will become > compressed gradually. > > There will be 2 new properties on CacheConfiguration > (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk page > compression. > > Compression dictionaries are not supported at the time, but may in the > future. IMO it should be added as a separate feature if needed. > > The only supported platform for now is Linux. Since all popular file > systems support sparse files, it must be relatively easy to support more > platforms. > > Please take a look and provide your thoughts and suggestions. > > Thanks! > > Sergi >
