I created https://github.com/apache/arrow/issues/39015 and will try tackling it in bits and pieces over time.
On Wed, Oct 25, 2023 at 8:47 AM Elliott Bradshaw <elli...@tectonix.com> wrote: > Over the wire helps for sure, but keeping memory pressure down is the real > goal. > > Let me know if anyone decides to take this. > > Elliott Bradshaw > *Co-Founder | Principal* > *Tectonix, LLC* > m: 443-285-9224 > a: Columbia, MD > w: www.tectonix.com e: elli...@tectonix.com > CONFIDENTIALITY NOTICE: > The contents of this email message and any attachments are intended solely > for the addressee(s) and may contain confidential and/or privileged > information and may be legally protected from disclosure. If you are not > the intended recipient of this message or their agent, or if this message > has been addressed to you in error, please immediately alert the sender by > reply email and then delete this message and any attachments. If you are > not the intended recipient, you are hereby notified that any use, > dissemination, copying, or storage of this message or its attachments is > strictly prohibited. > > > On Tue, Oct 24, 2023 at 8:36 AM Antoine Pitrou <anto...@python.org> wrote: > >> >> If the concern is simply data size on the wire, in the meantime, you >> can simply enable buffer compression. >> >> However, if you're also concerned about data size in memory, then >> indeed Run-End Encoding would help. >> >> Regards >> >> Antoine. >> >> >> On Mon, 23 Oct 2023 14:13:25 -0400 >> Elliott Bradshaw <elli...@tectonix.com> wrote: >> > Just as a simple example, if we’re shipping a table of events that has a >> > year, month, and day column, and it’s sorted by time, then each column >> will >> > have long runs of duplicate numbers that would benefit greatly from >> > compaction. >> > >> > It looks like REE is already built into the standard and supported by >> the >> > C, Go and Python apis. Would be awesome to have this support in Java as >> > well! >> > >> > Thanks, >> > Elliott >> > >> > On Mon, Oct 23, 2023 at 10:55 AM Felipe Oliveira Carvalho < >> > felipe...@gmail.com> wrote: >> > >> > > Hi Elliott, >> > > >> > > Not that I know of. But do you have concrete numbers and a practical >> case >> > > that could motivate someone to tackle the project? >> > > >> > > -- >> > > Felipe >> > > >> > > On Sun, Oct 22, 2023 at 10:05 AM Elliott Bradshaw < >> elli...@tectonix.com> >> > > wrote: >> > > >> > >> Hi Arrow Team, >> > >> >> > >> We love your work. Wondering if support for Run End Encoded Vectors >> is >> > >> on the horizon in Java? It would make it much more efficient to ship >> > >> column sorted tables over the wire. >> > >> >> > >> Thanks! >> > >> Elliott >> > >> >> > > >> > >> >> >> >>
