Over the wire helps for sure, but keeping memory pressure down is the real goal.
Let me know if anyone decides to take this. Elliott Bradshaw *Co-Founder | Principal* *Tectonix, LLC* m: 443-285-9224 a: Columbia, MD w: www.tectonix.com e: [email protected] CONFIDENTIALITY NOTICE: The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. On Tue, Oct 24, 2023 at 8:36 AM Antoine Pitrou <[email protected]> wrote: > > If the concern is simply data size on the wire, in the meantime, you > can simply enable buffer compression. > > However, if you're also concerned about data size in memory, then > indeed Run-End Encoding would help. > > Regards > > Antoine. > > > On Mon, 23 Oct 2023 14:13:25 -0400 > Elliott Bradshaw <[email protected]> wrote: > > Just as a simple example, if we’re shipping a table of events that has a > > year, month, and day column, and it’s sorted by time, then each column > will > > have long runs of duplicate numbers that would benefit greatly from > > compaction. > > > > It looks like REE is already built into the standard and supported by the > > C, Go and Python apis. Would be awesome to have this support in Java as > > well! > > > > Thanks, > > Elliott > > > > On Mon, Oct 23, 2023 at 10:55 AM Felipe Oliveira Carvalho < > > [email protected]> wrote: > > > > > Hi Elliott, > > > > > > Not that I know of. But do you have concrete numbers and a practical > case > > > that could motivate someone to tackle the project? > > > > > > -- > > > Felipe > > > > > > On Sun, Oct 22, 2023 at 10:05 AM Elliott Bradshaw < > [email protected]> > > > wrote: > > > > > >> Hi Arrow Team, > > >> > > >> We love your work. Wondering if support for Run End Encoded Vectors > is > > >> on the horizon in Java? It would make it much more efficient to ship > > >> column sorted tables over the wire. > > >> > > >> Thanks! > > >> Elliott > > >> > > > > > > > > >
