Hello! You have zstd default level of 3. In my tests, zstd usually performed much better with compression level 2. Please consider.
I admire your effort! Regards, -- Ilya Kasnacheev пн, 19 нояб. 2018 г. в 14:02, Sergi Vladykin <[email protected]>: > Right now the functionality has nothing to do with WAL, but your idea > definitely makes sense and worth being implemented as a next step. > > Sergi > > пн, 19 нояб. 2018 г. в 13:58, Andrey Mashenkov <[email protected] > >: > > > Hi Sergi, > > > > It is not clear for me will your changes affect PageSnapshot WAL record. > > Is it possible to add compression support for PageSnapshot WAL record as > > well, to reduce WAL size? > > > > Thanks. > > > > On Mon, Nov 19, 2018 at 1:01 PM Sergi Vladykin <[email protected] > > > > wrote: > > > > > Folks, > > > > > > I've implemented page compression for persistent store and going to > merge > > > it to master. > > > > > > https://github.com/apache/ignite/pull/5200 > > > > > > Some design notes: > > > > > > It employs "hole punching" approach, it means that the pages are kept > > > uncompressed in memory, > > > but when they get written to disk, they will be compressed and all the > > > extra file system blocks for the page will be released. Thus the > storage > > > files become sparse. > > > > > > Right now we will support 4 compression methods: ZSTD, LZ4, SNAPPY and > > > SKIP_GARBAGE. All of them are self-explaining except SKIP_GARBAGE, > which > > > basically just takes only meaningful data from half-filled pages but > does > > > not apply any compression. It is easy to add more if needed. > > > > > > Since we can release only full file system blocks which are typically > 4k > > > size, user must configure page size to be at least multiple FS blocks, > > e.g. > > > 8k or 16k. It also means that max compression ratio here is > fsBlockSize / > > > pageSize = 4k / 16k = 0.25 > > > > > > It is possible to enable compression for existing databases if they > were > > > configured for large enough page size. In this case pages will be > written > > > to disk in compressed form when updated, and the database will become > > > compressed gradually. > > > > > > There will be 2 new properties on CacheConfiguration > > > (setDiskPageCompression and setDiskPageCompressionLevel) to setup disk > > page > > > compression. > > > > > > Compression dictionaries are not supported at the time, but may in the > > > future. IMO it should be added as a separate feature if needed. > > > > > > The only supported platform for now is Linux. Since all popular file > > > systems support sparse files, it must be relatively easy to support > more > > > platforms. > > > > > > Please take a look and provide your thoughts and suggestions. > > > > > > Thanks! > > > > > > Sergi > > > > > > > > > -- > > Best regards, > > Andrey V. Mashenkov > > >
