Dinesh & Abe, Thank you very much for your feedback. The algorithm used by this HW compressor is compatible with Deflate but there is a constraint of 4K window size. Therefore the concern is that existing data may not decompress correctly as is. That is why we chose the path of adding a new compressor. Another reason is that, there are some additional features available in the hardware which are not compatible with zlib. With this approach we could enable those features as well.
We are also planning to accelerate existing compressors, if that is the preferred approach we will try to come up with a solution to work around the 4k window limitation. Thank you, Shylaja From: Dinesh Joshi <djo...@apache.org> Sent: Monday, January 22, 2024 11:18 AM To: dev@cassandra.apache.org Subject: Re: CASSANDRA-19268: Improve Cassandra compression performance using hardware accelerators Shylaja, Cassandra uses ZStd, LZ4 and other compression libraries via JNI to compress data. The intel hardware accelerator support is integrated into those libraries and we can benefit from it. If there are special parameters that need to be passed in to these libraries we can make those changes on the database but as such Cassandra does not directly implement the compression algorithms itself. Dinesh
