Mark, Dis you take a look at finos perspective? It seems to have some interesting overlaps with your goals. I've come across it but have not digged in.
Be curious to get your thoughts on it . Cheers On Sat., Aug. 15, 2020, 13:05 , <m...@markfarnan.com> wrote: > David, > > Still investigating, but I suspect for streaming I may have to fall back > to some form of "custom" Flight implementation over Websockets. > > Assuming Arrow/Flight actually makes sense for that link, which will > probably depend on how well it compresses. However it will be very nice > if it does, to allow common format everywhere. > > The data I need to move around is highly variable in 'type', (Arrays of > Floats, Ints & Strings with occasional Binary, or vector (array of an array > of floats in my case) but the number of columns, and their type vary by > dataset and visualization choices. So far arrow seems a good choice rather > than any 'roll your own', and it will be nice to use same format on Client > side as well as in the Server system. > > My use case is primarily 'Get', consuming large datasets for > visualization. I doubt I'll need Put or Exchange from the browser. > > Mark. > > -----Original Message----- > From: David Li <li.david...@gmail.com> > Sent: Saturday, August 15, 2020 5:53 PM > To: dev@arrow.apache.org > Subject: Re: Arrow Flight + Go, Arrow for Realtime > > I am curious what you accomplish with Arrow + Flight from the browser. > Right now, Flight is all gRPC-based, and browser compatibility is a bit > mixed. I expect the various transcoders/gRPC-Web can handle > GetFlightInfo/DoGet fine, though IIRC for DoGet, at least some of the > transcoders would have to buffer the entire stream before sending it to the > browser. DoPut/DoExchange seem harder/impossible to bridge right now due to > the bidirectional streaming. > > Best, > David > > On 8/14/20, m...@markfarnan.com <m...@markfarnan.com> wrote: > > Thanks Wes, > > > > I'll likely work on that once I get my head around Arrow in general > > and confirm will use for the project. > > > > Considerations for how to account for the streaming append problem to an > > otherwise immutable dataset is current concern. Still thinking through > > that. > > > > Regards > > > > Mark. > > > > -----Original Message----- > > From: Wes McKinney <wesmck...@gmail.com> > > Sent: Wednesday, August 12, 2020 3:59 PM > > To: dev <dev@arrow.apache.org> > > Subject: Re: Arrow Flight + Go, Arrow for Realtime > > > > There's a WIP patch for Flight support in Go > > > > https://github.com/apache/arrow/pull/6731 > > > > I hope to see someone taking up this work as first-class Flight > > support in Go would be very useful for building data services. > > > > On Wed, Aug 12, 2020 at 5:08 AM Adam Lippai <a...@rigo.sk> wrote: > >> > >> Arrow is mainly about batching data and leveraging all the > >> opportunities this gives. > >> This means you either have to buffer the data yourself and flush it > >> when a reasonable sized batch is complete or play with preallocating > >> Arrow structures This was discussed recently, you might be interested > >> in the thread: > >> https://www.mail-archive.com/dev@arrow.apache.org/msg19862.html > >> > >> Note: I'm not an Arrow developer, I'm just following the "streaming" > >> features of the Arrow lib, I'm interested in having a "rolling window" > >> API (like a fixed size FIFO queue). > >> > >> Best regards, > >> Adam Lippai > >> > >> On Wed, Aug 12, 2020 at 11:29 AM <m...@markfarnan.com> wrote: > >> > >> > I'm looking at using Arrow for a realtime IoT project which > >> > includes use cases both on server, and also for transferring /using > >> > in a Browser via WASM, and have a few questions. > >> > > >> > > >> > > >> > Language in use is Go. > >> > > >> > > >> > > >> > Is anyone working on implementing Arrow-Flight in Go ? > (According > >> > to > >> > the feature matrix, nothing ready yet, so wanted to check. > >> > > >> > > >> > > >> > Has anyone tried using Apache Arrow in Go WASM (Webassembly) ? if > >> > so, > >> > any issues ? > >> > > >> > > >> > > >> > Any pointers/documentation on using/extending Arrow for realtime > >> > streaming > >> > cases. (Specifically where a DataFrame is requested, but then it > needs > >> > to > >> > 'grow' as new data arrives, often at high speed). > >> > > >> > Not language specific, just trying to understand the right pattern > >> > for using Arrow for this, and couldn't' find much in the docs. > >> > > >> > > >> > > >> > Regards > >> > > >> > > >> > > >> > Mark. > >> > > >> > > > > > > >
