On Thu, 20 Jan 2022 14:42:25 -0700 Chris Nuernberger <[email protected]> wrote: > OK, well, for the record tech.ml.dataset supports three major features the > official SDK does not - mmap, JDK-17 and just now compression as a > user-accessible option during write > - > file:///home/chrisn/dev/tech.all/tech.ml.dataset/docs/tech.v3.libs.arrow.html. > > Other lz4 compressors are faster than apache but regardless zstd gets the > best compression ratio for the very simple test files I tested.
If better compression speed in Java Arrow is desired, then opening a JIRA + a Github PR would be most welcome. Regards Antoine.
