Re: Arrow Flight + Go, Arrow for Realtime

2020-08-15 Thread Pierre Belzile
Mark, Dis you take a look at finos perspective? It seems to have some interesting overlaps with your goals. I've come across it but have not digged in. Be curious to get your thoughts on it . Cheers On Sat., Aug. 15, 2020, 13:05 , wrote: > David, > > Still investigating, but I suspect for

RE: Arrow Flight + Go, Arrow for Realtime

2020-08-15 Thread mark
David, Still investigating, but I suspect for streaming I may have to fall back to some form of "custom" Flight implementation over Websockets. Assuming Arrow/Flight actually makes sense for that link, which will probably depend on how well it compresses. However it will be very nice if it

Re: [DISCUSS] Plasma appears to have been forked, consider deprecating pyarrow.serialization

2020-08-15 Thread Wes McKinney
On Fri, Aug 14, 2020 at 11:56 PM Micah Kornfield wrote: > > > > > Regarding Plasma, you're right we should have started this conversation > > earlier! The way it's being developed in Ray currently isn't useful as a > > standalone project. We realized that tighter integration with Ray's object > >

Re: Arrow Flight + Go, Arrow for Realtime

2020-08-15 Thread David Li
I am curious what you accomplish with Arrow + Flight from the browser. Right now, Flight is all gRPC-based, and browser compatibility is a bit mixed. I expect the various transcoders/gRPC-Web can handle GetFlightInfo/DoGet fine, though IIRC for DoGet, at least some of the transcoders would have to

Re: [DISCUSS] Support of higher bit-width Decimal type

2020-08-15 Thread Wes McKinney
On Fri, Aug 14, 2020 at 11:17 PM Micah Kornfield wrote: > > Hi Jacques, > > Do we have a good definition of what is necessary to add a new data type? > > Adding a type but not pulling it through most of the code seems less than > > ideal since it means one part of Arrow doesn't work with another

[NIGHTLY] Arrow Build Report for Job nightly-2020-08-15-0

2020-08-15 Thread Crossbow
Arrow Build Report for Job nightly-2020-08-15-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-08-15-0 Failed Tasks: - test-conda-python-3.7-hdfs-2.9.2: URL: