JigaoLuo commented on issue #8358: URL: https://github.com/apache/arrow-rs/issues/8358#issuecomment-5181917154
Just sharing a blog post that was published today. It's nice to see that we're not the only ones thinking along these lines, and that our intuition seems to be heading in the right direction. CedarDB uses a tunable threshold (which I assume was chosen empirically): > For CedarDB, we employ zstd compression if it will further reduce the encoded data on disk by at least 20%. > Why Not Both? https://cedardb.com/blog/encoding_vs_compression/ *: (In my GPU Parquet work, I ended up choosing 10%. The main reason is that GPU decompression is significantly faster than CPU decompression, so the GPU can afford to take a more aggressive approach and decompress more data without hurting performance much.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
