"Is https://github.com/lz4/lz4-java the fast Java lz4 library in question? The incompleteness of this implementation is a known problem for other user communities, not only Arrow. It would be a great public service to improve it so that it fully implements the lz4 frame specification."
Very much +1. On Thu, Mar 11, 2021 at 6:08 PM Steve Kim <chairm...@gmail.com> wrote: > I prefer the lz4 frame format for the reasons that Antoine stated. > > To be friendly to users, the Arrow IPC documentation could mention > that lz4 compression may break Java interoperability. If block > dependency is the only obstacle to Java interoperability, the Arrow > IPC implementation could disable this flag as an _implementation > detail_. But I think that the _specification_ of lz4 compression in > the Arrow IPC format should not carve out exceptions to accommodate > current (temporary, I hope) implementation shortfalls in lz4 > libraries. > > Is https://github.com/lz4/lz4-java the fast Java lz4 library in > question? The incompleteness of this implementation is a known problem > for other user communities, not only Arrow. It would be a great public > service to improve it so that it fully implements the lz4 frame > specification. > > Steve >
