Hi Sergi, It is not clear for me will your changes affect PageSnapshot WAL record. Is it possible to add compression support for PageSnapshot WAL record as well, to reduce WAL size?
Thanks. On Mon, Nov 19, 2018 at 1:01 PM Sergi Vladykin <sergi.vlady...@gmail.com> wrote: > Folks, > > I've implemented page compression for persistent store and going to merge > it to master. > > https://github.com/apache/ignite/pull/5200 > > Some design notes: > > It employs "hole punching" approach, it means that the pages are kept > uncompressed in memory, > but when they get written to disk, they will be compressed and all the > extra file system blocks for the page will be released. Thus the storage > files become sparse. > > Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY and > SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE, which > basically just takes only meaningful data from half-filled pages but does > not apply any compression. It is easy to add more if needed. > > Since we can release only full file system blocks which are typically 4k > size, user must configure page size to be at least multiple FS blocks, e.g. > 8k or 16k. It also means that max compression ratio here is fsBlockSize / > pageSize = 4k / 16k = 0.25 > > It is possible to enable compression for existing databases if they were > configured for large enough page size. In this case pages will be written > to disk in compressed form when updated, and the database will become > compressed gradually. > > There will be 2 new properties on CacheConfiguration > (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk page > compression. > > Compression dictionaries are not supported at the time, but may in the > future. IMO it should be added as a separate feature if needed. > > The only supported platform for now is Linux. Since all popular file > systems support sparse files, it must be relatively easy to support more > platforms. > > Please take a look and provide your thoughts and suggestions. > > Thanks! > > Sergi > -- Best regards, Andrey V. Mashenkov
