Re: updates on top of arrow
It's not clear if this will actually hit your use case or not. Specifically, low overhead means different things to different people. Also, my suggestion is not a database -- it is an analytics engine. The persistence part of the problem is off the table at this time, too. I wanted to mention it up front in case any of that is important to you. With all of the caveats out, let's get started. I work at Deephaven. We have released a community edition of a powerful tool that is targeted around real time updates. We use Arrow Flight for IPC. To subscribe to a changing stream of data, we use DoExchange which enables the client to inform the server which subset of the data it is interested in when it does not want all of the data (consider GUI use cases). The append-only pattern is the dip-your-feet-wet with updating data. I highly recommend this article, in which our CTO goes into detail about the history and evolution of our update model design. This article will give you a lot of insight into what we tried to build and is likely to inform you whether or not it might fit your needs. Maybe it will excite you. Link: https://deephaven.io/core/docs/conceptual/table-update-model/ I wrote an article describing how we arrived at gRPC and Arrow. It assumes at least a high level understanding of the update model, as extending this to work over Arrow Flight was really about finding the right application metadata which enables consistent [Arrow] wire format for payload. Link: https://deephaven.io/core/docs/conceptual/deephaven-core-api/ I'm not finding a short and sweet demo on our youtube channel, but maybe you will find a weekly demo or two to be revealing of whether or not this is a tool worth investing your time to learn more about. Link: https://www.youtube.com/channel/UCoaYOlkX555PSTTJz8ZaI_w/ If you have an hour or two to explore and research -- it's very low-effort to get our simplest examples running. Source can be found (including getting started instructions): https://github.com/deephaven/deephaven-core I'd love to answer your direct questions or help you make the most of your research time. Feel free to jump into our gitter and ask questions: https://gitter.im/deephaven/deephaven If you play around with it and don't like it -- I'd love to hear that feedback, too. Nate On Wed, Nov 17, 2021 at 8:01 AM Kirill Lykov wrote: > Hi, > > I'm trying to build something like an embedded data base system around > arrow with some properties which I will list below. > The question is if someone can point me to existing solutions (or attempts) > for such kind of problem. > I'm trying to develop a data layout such that > * typically reading is not for single value but for a range of keys/fields > * this is all in-memory > * it provides low overhead for reading and writing > * multithreaded reading supported, but updates are single threaded > * 99% of updates are primarily adding a new data point to the end of column > (a-la log) > * return data in arrow tables with minimal overhead (better not to copy > data from internal storage but do slicing whenever possible) > > On a high level I had in mind something like SSTables with level compaction > when the data is accessed for reading. But before doing this I wanted to > check if there is a solution for such a problem leveraging an arrow or if > there are all the bricks to build such a solution. Any input (including > papers) is welcomed. > > -- > Best regards, > Kirill Lykov > --
Re: [DISCUSS] next iteration of flatbuffer structures
I put the draft up here: https://github.com/apache/arrow/pull/11646 Thanks. On Mon, Nov 8, 2021 at 1:57 PM David Li wrote: > Hey Nate, > > Thanks for doing this! Would you be interested in putting that commit up > as a draft PR for discussion? I think we can discuss there. > > I'm not sure anyone is actively working on RLE or other encoding schemes > at the moment. > > -David > > On Mon, Nov 8, 2021, at 13:19, Nate Bauernfeind wrote: > > I've written up the ColumnBag proposal addressing items 1 and 2 on the > > list. I'm open to any and all feedback/suggestions. > > > > I'd be happy to add item 3 (binary metadata) to the proposed change set. > > Let me know if you want me to whip up the initial suggestion for that > > version (and whether or not to keep it separate from ColumnBag). > > > > Would RLE related efforts change the structure of RecordBatch or > ColumnBag > > (if accepted)? > > > > Here is the brief history-discussion around why ColumnBag: > > > https://docs.google.com/document/d/1jsmmqLTyJkU8fx0sUGIqd6yu72N4v9uHFsuGSgB_DfE/ > > > > Here is a brief commit doctoring up the flatbuffer to support this > version > > of the proposed change: > > https://github.com/nbauernfeind/arrow/tree/column_bag_demo_v1 > > > > I don't know if it's better to comment in the document or bring comments > > back to the list. If it ends up being document heavy, then I'll summarize > > the main points back on the list. > > > > I think I'll get started on a Java impl just to learn more even if it > ends > > up being extra work. > > > > Looking forward to your feedback, > > Nate > > > > On Mon, Aug 9, 2021 at 10:06 PM Micah Kornfield > > wrote: > > > > > I'm still interested in RLE related effort, but not sure about my > available > > > bandwidth (which is why I haven't made more of an effort there). > > > > > > On Tue, Aug 3, 2021 at 6:00 PM Wes McKinney > wrote: > > > > > > > Another Flatbuffers/Message.fbs project we should rekindle soon, in > > > > addition to the schema evolution/replacement question which has been > > > > raised with Flight, is that of sparse/compressed data (e.g. RLE). I > > > > have a vacation plus some travel coming up so won't be able to devote > > > > meaningful attention to this until the last part of August, but would > > > > like to help it move forward. > > > > > > > > > > > > On Tue, Jul 27, 2021 at 1:40 PM David Li > wrote: > > > > > > > > > > Hey Nate, > > > > > > > > > > For the first two points, semantically I'm tempted to think of it > more > > > > like the ability to send a "bag of columns" according to some schema > (and > > > > hence columns could have differing lengths or even be absent). This > could > > > > be a new structure alongside a record batch, which is semantically > like a > > > > "slice of a table" (and hence rectangular and complete), instead of > > > > exposing existing users of RecordBatch to rather different behavior. > > > > > > > > > > For #3, a different thread was discussing some of the points there > - it > > > > sounds like it may be possible to relax from map to > > > > map. > > > > > > > > > > -David > > > > > > > > > > On Mon, Jul 26, 2021, at 11:01, Nate Bauernfeind wrote: > > > > > > Wes suggested that maybe there are enough new ideas that it may > make > > > > sense > > > > > > to evolve-past the existing structures rather than to bolt-on new > > > > > > functionality. I would like to learn what requirements exist > should > > > new > > > > > > structures be adopted, and if applicable, would like to turn this > > > into > > > > a > > > > > > full POC proposal. > > > > > > > > > > > > These are the features that I feel are missing from the existing > > > > design: > > > > > > - the ability to notify that the columns are not consistent in > length > > > > (e.g. > > > > > > setting RecordBatch.length to -1; and give the arrow/flight user > the > > > > true > > > > > > FieldNode lengths). > > > > > > - the ability to skip top-level field nodes that have length 0 > at a > > > > small > > > > > &g
Re: [DISCUSS] next iteration of flatbuffer structures
I've written up the ColumnBag proposal addressing items 1 and 2 on the list. I'm open to any and all feedback/suggestions. I'd be happy to add item 3 (binary metadata) to the proposed change set. Let me know if you want me to whip up the initial suggestion for that version (and whether or not to keep it separate from ColumnBag). Would RLE related efforts change the structure of RecordBatch or ColumnBag (if accepted)? Here is the brief history-discussion around why ColumnBag: https://docs.google.com/document/d/1jsmmqLTyJkU8fx0sUGIqd6yu72N4v9uHFsuGSgB_DfE/ Here is a brief commit doctoring up the flatbuffer to support this version of the proposed change: https://github.com/nbauernfeind/arrow/tree/column_bag_demo_v1 I don't know if it's better to comment in the document or bring comments back to the list. If it ends up being document heavy, then I'll summarize the main points back on the list. I think I'll get started on a Java impl just to learn more even if it ends up being extra work. Looking forward to your feedback, Nate On Mon, Aug 9, 2021 at 10:06 PM Micah Kornfield wrote: > I'm still interested in RLE related effort, but not sure about my available > bandwidth (which is why I haven't made more of an effort there). > > On Tue, Aug 3, 2021 at 6:00 PM Wes McKinney wrote: > > > Another Flatbuffers/Message.fbs project we should rekindle soon, in > > addition to the schema evolution/replacement question which has been > > raised with Flight, is that of sparse/compressed data (e.g. RLE). I > > have a vacation plus some travel coming up so won't be able to devote > > meaningful attention to this until the last part of August, but would > > like to help it move forward. > > > > > > On Tue, Jul 27, 2021 at 1:40 PM David Li wrote: > > > > > > Hey Nate, > > > > > > For the first two points, semantically I'm tempted to think of it more > > like the ability to send a "bag of columns" according to some schema (and > > hence columns could have differing lengths or even be absent). This could > > be a new structure alongside a record batch, which is semantically like a > > "slice of a table" (and hence rectangular and complete), instead of > > exposing existing users of RecordBatch to rather different behavior. > > > > > > For #3, a different thread was discussing some of the points there - it > > sounds like it may be possible to relax from map to > > map. > > > > > > -David > > > > > > On Mon, Jul 26, 2021, at 11:01, Nate Bauernfeind wrote: > > > > Wes suggested that maybe there are enough new ideas that it may make > > sense > > > > to evolve-past the existing structures rather than to bolt-on new > > > > functionality. I would like to learn what requirements exist should > new > > > > structures be adopted, and if applicable, would like to turn this > into > > a > > > > full POC proposal. > > > > > > > > These are the features that I feel are missing from the existing > > design: > > > > - the ability to notify that the columns are not consistent in length > > (e.g. > > > > setting RecordBatch.length to -1; and give the arrow/flight user the > > true > > > > FieldNode lengths). > > > > - the ability to skip top-level field nodes that have length 0 at a > > small > > > > cost (such as in a bitset) > > > > - the ability to embed binary payload in the Message flatbuffer > wrapper > > > > (instead of String payload only) > > > > - the ability to concurrently use more than one schema (the most > > likely API > > > > will look like how one identifies a dictionary. ideally dictionaries > > could > > > > be shared across field nodes in a schema or across schemas in the > same > > > > flight) > > > > > > > > What other features, or improvements, could/should be considered? Any > > > > strong opinions against the ideas above? (Remember, that a goal of > > mine is > > > > to be able to send a RecordBatch of rows that were modified > intersected > > > > only by the field-nodes that have changed (including those with only > > inner > > > > node changes); thus the columns are a subset of the full schema and > > that > > > > the length of each node is independent of the other). > > > > > > > > On Fri, Jul 9, 2021 at 9:26 AM Wes McKinney > > wrote: > > > > > It sounds like we may want to discuss some potential evolutions of > > the > > > > > Arrow binary protocol (for example: new Message types). Certainly a > > > > > can of worms but rather than trying to bolt some new functionality > > > > > onto the existing structures, it might be better to support the new > > > > > use cases through some new structures which will be more clear cut > > > > > from a forward compatibility standpoint. > > > > > > > > Nate > > > > > > > > -- > > > > > > > --
Re: Arrow sync call September 15 at 12:00 US/Eastern, 16:00 UTC
Meeting notes for arrow-sync on 09/15/2021. Attendees: - Nate Bauernfeind - Nic Crane - Alenka Frim - Rok Mihevc - Niranda Perera There was no discussion this week; all attendees were here to lurk, listen, and be-a-fly-on-the-wall. See you in two weeks. -- On Wed, Sep 15, 2021 at 8:08 AM Ian Cook wrote: > Hi all, > > Our biweekly sync call is today at 12:00 noon Eastern time. > > Many (possibly all) of the folks at Voltron Data will be attending a > separate meeting at this same time today, so attendance might be > lighter than usual. Please carry on without us. We ask that someone > who does attend please take notes indicating who was present and what > was discussed. > > The Zoom meeting URL for this and other biweekly Arrow sync calls is: > https://zoom.us/j/87649033008?pwd=SitsRHluQStlREM0TjJVYkRibVZsUT09 > The meeting will start even if the host is not present. > > Alternatively, enter this information into the Zoom website or app to > join the call: > Meeting ID: 876 4903 3008 > Passcode: 958092 > > Thanks, > Ian > --
Re: HTTP traffic of Arrow Flight
HTTP (and HTTP/2) traffic is sent over TCP. You might need to be more specific, or possibly do some more research on your end Which arrow flight client are you using in your test? Java? C++? Which version? Can you provide a simple gRPC server/client example that shows up in WireShark as you expect it? Nate On Tue, Sep 7, 2021 at 10:40 AM Mohamed Abdelhakem < mohamed.abdelha...@incorta.com> wrote: > When I built a simple FlightServer and FlightClient, I noticed that the > traffic captured by WireShark is TCP, not HTTP/2 > MY question is how to configure Arrow Flight to use HTTP/2 protocol traffic > --
Re: WireFormat of Flight SchemaResult and Flight FlightInfo's Schema
David, do you mind assigning to me? I'll get a PR up within the next two
days. The JIRA username is 'nbauernfeind'.
On Mon, Jul 26, 2021 at 8:41 AM David Li wrote:
> I filed ARROW-13449 for this, thanks for the clarification.
>
> -David
>
> On Mon, Jul 26, 2021, at 10:25, Wes McKinney wrote:
> > IIRC, we did intend to encapsulate the Schema in an IPC message.
> > Updating the docs / comments in the Flight.proto is the right thing to
> > do.
> >
> > On Sun, Jul 25, 2021 at 12:47 PM Micah Kornfield
> wrote:
> > >
> > > I think if all reference implementations are doing this the same way we
> > > should update the docs and I don't think a vote is necessary.
> > >
> > > On Sun, Jul 25, 2021 at 10:27 AM David Li wrote:
> > >
> > > > Hey Nate,
> > > >
> > > > Good catch. I would say it was intentional to use an IPC message
> (while
> > > > the length is redundant, it also contains the metadata version and
> custom
> > > > metadata), and the comment is a little vague. I'm not sure if we
> need a
> > > > vote to update this since it is changing files in the format dir.
> > > >
> > > > I did check the Java implementation quickly and even in the initial
> > > > version, the schema is IPC-encapsulated[1].
> > > >
> > > > -David
> > > >
> > > > [1]:
> > > >
> https://github.com/apache/arrow/commit/b3cd616f6ce39e9a820b658ba2f2e2fd3b153555#diff-80e8aef2d6b475949641689fe8d686db2cc1d6703cafe8f6f0dc120b04f964b7R106
> > > >
> > > > On Fri, Jul 23, 2021, at 20:28, Nate Bauernfeind wrote:
> > > > > In flight.proto [1] it states that the encoded bytes are as
> described in
> > > > > the flatbuffer schema.
> > > > >
> > > > > ```
> > > > > /*
> > > > > * Wrap the result of a getSchema call
> > > > > */
> > > > > message SchemaResult {
> > > > > // schema of the dataset as described in Schema.fbs::Schema.
> > > > > bytes schema = 1;
> > > > > }
> > > > > ```
> > > > >
> > > > > However, both this schema and the schema on the flight info are
> actually
> > > > > encoded like an IPC file stream message.
> > > > >
> > > > > They start with a 4-byte IPC_CONTINUATION_TOKEN, followed by a
> 4-byte
> > > > > message size, followed by a Message wrapping a Schema. See [2] and
> [3]
> > > > for
> > > > > evidence in the Java client.
> > > > >
> > > > > Is this an accidental bug that has propagated to all client
> > > > implementations?
> > > > > Or was it intentional and I should submit a PR to update the
> comments?
> > > > >
> > > > > Thanks,
> > > > > Nate
> > > > > [1]
> https://github.com/apache/arrow/blob/master/format/Flight.proto#L195
> > > > > [2]
> > > > >
> > > >
> https://github.com/apache/arrow/blob/master/java/flight/flight-core/src/main/java/org/apache/arrow/flight/FlightInfo.java#L141
> > > > > [3]
> > > > >
> > > >
> https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java#L161
> > > > > --
> > > > >
> > > >
> >
>
--
[DISCUSS] next iteration of flatbuffer structures
Wes suggested that maybe there are enough new ideas that it may make sense to evolve-past the existing structures rather than to bolt-on new functionality. I would like to learn what requirements exist should new structures be adopted, and if applicable, would like to turn this into a full POC proposal. These are the features that I feel are missing from the existing design: - the ability to notify that the columns are not consistent in length (e.g. setting RecordBatch.length to -1; and give the arrow/flight user the true FieldNode lengths). - the ability to skip top-level field nodes that have length 0 at a small cost (such as in a bitset) - the ability to embed binary payload in the Message flatbuffer wrapper (instead of String payload only) - the ability to concurrently use more than one schema (the most likely API will look like how one identifies a dictionary. ideally dictionaries could be shared across field nodes in a schema or across schemas in the same flight) What other features, or improvements, could/should be considered? Any strong opinions against the ideas above? (Remember, that a goal of mine is to be able to send a RecordBatch of rows that were modified intersected only by the field-nodes that have changed (including those with only inner node changes); thus the columns are a subset of the full schema and that the length of each node is independent of the other). On Fri, Jul 9, 2021 at 9:26 AM Wes McKinney wrote: > It sounds like we may want to discuss some potential evolutions of the > Arrow binary protocol (for example: new Message types). Certainly a > can of worms but rather than trying to bolt some new functionality > onto the existing structures, it might be better to support the new > use cases through some new structures which will be more clear cut > from a forward compatibility standpoint. Nate --
WireFormat of Flight SchemaResult and Flight FlightInfo's Schema
In flight.proto [1] it states that the encoded bytes are as described in
the flatbuffer schema.
```
/*
* Wrap the result of a getSchema call
*/
message SchemaResult {
// schema of the dataset as described in Schema.fbs::Schema.
bytes schema = 1;
}
```
However, both this schema and the schema on the flight info are actually
encoded like an IPC file stream message.
They start with a 4-byte IPC_CONTINUATION_TOKEN, followed by a 4-byte
message size, followed by a Message wrapping a Schema. See [2] and [3] for
evidence in the Java client.
Is this an accidental bug that has propagated to all client implementations?
Or was it intentional and I should submit a PR to update the comments?
Thanks,
Nate
[1] https://github.com/apache/arrow/blob/master/format/Flight.proto#L195
[2]
https://github.com/apache/arrow/blob/master/java/flight/flight-core/src/main/java/org/apache/arrow/flight/FlightInfo.java#L141
[3]
https://github.com/apache/arrow/blob/master/java/vector/src/main/java/org/apache/arrow/vector/ipc/message/MessageSerializer.java#L161
--
Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams
> Flatbuffers does not support modifying structs > in any forwards or backwards compatible way > (only tables support evolution). Bah. I did not realize that. To reiterate the feature that would be ideal: I realize the specific feature I am missing is the ability to encode that a field (i.e. its FieldNode and accompanying Buffers in the RecordBatch) is empty/has-no-data in O(0) cost (yes; for free). Since RecordBatch is a table, might there be interest in adding a field that is a bitset (formatted as a byte vector) to indicate the subset of root FieldNodes that are included in the RecordBatch's field-node-list (noting that the buffer list most be appropriately filtered, too)? If the bitset is omitted/empty we can be forwards-compatible by assuming every field is included (seems fair; because why send a record batch without a payload?). This does bring down the cost from 16 bytes (field node) + 32 bytes (2x buffer node minimum; validity buffer node + payload buffer node) to 1 bit per field node -- that's a compression ratio of >= 384:1 -- although it's not free, it's a lot closer. Are there any other alternatives the Arrow community might consider? > This sounds a lot like what DenseUnion provides though? The use case is as follows: We send multiple record batches that aggregate / accumulate into a single logical update. The first set of record batches contain payload for rows that were added since the last logical update (this is a set of updates to accommodate that not every implementation supports 32-bit lengths). A field node for every column and every added row will be sent. For this half of the logical update the RecordBatch length matches the root FieldNode lengths. The second set of record batches contain payload for only rows, and fields, that were actually modified. If only one field changed, we send only that payload for that row. There is additional metadata that allows the client to understand which existing row is to be replaced by any given row for a given Field/Column. In this context, each root field node has a different mapping to minimize total payload size. We might be able to use DenseUnion but it certainly feels like folding the entire data-source into a dense union makes the design less useful and less accessible. I will spend some time sleeping on your suggestion, but I'm not immediately excited about it. At this moment, I suspect I will continue to lie and state that the RecordBatch's length is the max length across all root field node lengths (and be content that it's not ideal from a copy/allocation perspective). - On Wed, Jul 7, 2021 at 10:57 AM Micah Kornfield wrote: > > > > Might there be interest in adding a "field_id" to the FieldNode (which is > > encoded on the RecordBatch flatbuffer)? I see a simple forward-compatible > > upgrade (by either keying off of 0, or explicitly set the field default > to > > -1) which would allow the sender to "skip" fields that have 1) FieldNode > > length of zero, and 2) all Buffer's associated at that level (and further > > nested) are also equally empty (i.e. Buffer length is zero). > > > FieldNode is a struct in Message.fbs. Flatbuffers does not support > modifying structs in any forwards or backwards compatible way (only tables > support evolution). I think there was originally more metadata in > FieldNode and it was stripped down due to size concerns. > > I understand this concept slightly interferes with RecordBatch's `length` > > field, and that many implementations use that length to resize the > > root-level FieldNodes. The use-case I have in mind has different logical > > lengths per field node; current implementations require sending a > > RecordBatch length of the max length across all root level field nodes. I > > believe this requires a copy of data whenever a field node is too short; > I > > don't know if there is a decent solution to this slight inefficiency. I > am > > bringing it up because if "skipping a field node when it is empty" is a > > feature, then we may not want to allocate space for those nodes given > that > > the record batch length will likely be greater than zero. > > > Having conflicting RecordBatch and top-level field nodes is something that > I believe we have pushed back on in the past. This sounds a lot like > what DenseUnion provides though? > > On Wed, Jul 7, 2021 at 8:32 AM Nate Bauernfeind < > natebauernfe...@deephaven.io> wrote: > > > Deephaven and I are very supportive of "upgrading" the value half of the > kv > > pair to a byte vector. What is the best way to find out if there is > > sufficient interest? > > > > > > I've been stewing on the ideas here around schema evolution, and I > realize > > the specific feature
Re: Arrow sync call July 7 at 12:00 US/Eastern, 16:00 UTC
Is this still happening today? On Tue, Jul 6, 2021 at 11:07 AM Ian Cook wrote: > Hi all, > > Our biweekly sync call is tomorrow at > https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes > will be shared with the mailing list afterward. > > Ian > --
Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams
gt; clearly specified and edge cases should be considered, especially > if we > > > > > might want to 'backport' this into the stream format later. > > > > > > > > > > > > Yes, for dictionaries we just need to be careful to define semantics > and > > > > ensure implementations are validating them with regards to > dictionaries. > > > > There likely isn't any need to change current implementations though. > > > > > > > > On Mon, Jun 28, 2021 at 1:25 PM David Li > wrote: > > > > > > > > > Right, I had wanted to focus the discussion on Flight as I think > schema > > > > > evolution or multiplexing streams (more so the latter) is a > property of the > > > > > transport and not the stream format itself. If we are leaning > towards just > > > > > schema evolution then maybe it makes sense to discuss it for the > IPC stream > > > > > format and leverage that in Flight. I'd be interested in what > others think. > > > > > > > > > > Especially if we are looking at multiplexing streams - I would > wonder if > > > > > that's actually better served by making it easier to implement > using the > > > > > Flight implementation as it stands (by managing concurrent RPC > calls and/or > > > > > performing the union-of-structs encoding trick for you), instead > of having > > > > > to change the protocol. > > > > > > > > > > Nate: it may be worth starting a separate discussion about more > general > > > > > metadata in the IPC message. I'm not aware of why key-value > metadata was > > > > > chosen/if opaque bytes were considered in the past. Off the top of > my head > > > > > if it's for on-disk storage and fully application-defined it may > make sense > > > > > to store as a separate file alongside the Arrow file (indexed by > record > > > > > batch index) where you can take advantage of whatever format is > most > > > > > suitable. > > > > > > > > > > -David > > > > > > > > > > On Sun, Jun 27, 2021, at 07:50, Gosh Arzumanyan wrote: > > > > > > Hi guys, > > > > > > > > > > > > 1. Regarding IPC vs Flight: in fact my initial suggestion was to > add this > > > > > > feature starting from the IPC(I moved initial write up steps to > the > > > > > bottom > > > > > > of the doc). Afterwards David suggested focusing on Flight and > that's how > > > > > > we ended up with the protobufs change in the proposal. This > being said I > > > > > do > > > > > > think that the place where this should be impemented is a good > question > > > > > on > > > > > > its own. Maybe it makes sense to have this kind of a feature in > IPC and > > > > > > somehow use it in Flight, maybe not. > > > > > > 2. The point about dictionaries deserves a dedicated section in > the > > > > > > proposal. Nate and David brought it up and shared some insights. > I'll try > > > > > > to aggregate them and we can continue the discussion form there. > > > > > > > > > > > > Cheers, > > > > > > Gosh > > > > > > > > > > > > On Sat., 26 Jun. 2021, 17:26 Nate Bauernfeind, < > > > > > natebauernfe...@deephaven.io> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > makes it more difficult to bring schema evolution back > into the > > > > > > > > > > IPC Stream format (i.e. it would live only in flight) > > > > > > > > > > > > > > > > > > Gosh's proposal extends the flatbuffer structures not the > > > > > protobufs. > > > > > > > Can > > > > > > > > > you help me understand how difficult it would be to bring > the > > > > > > > `schema_id` > > > > > > > > > approach to the IPC stream format? > > > > > > > > > > > > > > > > I thought we were talking solely about the Flight Protobuf > > > > > definitions - > > > > > > > > not the Flatbuffers (and the Google doc at least only talks > about the > > > > > > > > Protobufs). > > > > > > > > > > > > > > > > > > > > > > I somehow missed that schema_id is being added to protobuf in > the > > > > > document. > > > > > > > It feels to me that the schema_id is a property that would > ideally only > > > > > > > apply to the RecordBatch. I better understand Micah's > dictionary > > > > > concerns, > > > > > > > now, too. > > > > > > > > > > > > > > > Side Question: Why isn't the IPC stream format a series of > the flight > > > > > > > > > protobufs? It's a real shame that there is no standard way > to > > > > > > > > > capture/replay a stream with app_metadata. (Obviously > ignoring the > > > > > > > > > annoyances around protobuf wrapping flatbuffers.) > > > > > > > > > > > > > > > > The IPC format was defined long before Flight, and Flight's > > > > > app_metadata > > > > > > > > was added after Flight's initial definition. Note an IPC > message does > > > > > > > have > > > > > > > > a provision for key-value metadata, though I think APIs for > that are > > > > > not > > > > > > > > fully exposed. (See ARROW-6940: > > > > > > > > https://issues.apache.org/jira/browse/ARROW-6940 and > despite my > > > > > comments > > > > > > > > there perhaps we need to unify or at least consider how > Flight's > > > > > > > > app_metadata relates to the IPC message custom_metadata. Also > > > > > perhaps see > > > > > > > > ARROW-1059.) > > > > > > > > > > > > > > > > > > > > > > KeyValue unfortunately is string to string. In flatbuffer > strings are > > > > > only > > > > > > > UTF-8 or 7-bit ASCII. The app_metadata on the other hand is > opaque > > > > > bytes. > > > > > > > The latter is a bit more useful. > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > > > > > --
Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams
> > > > makes it more difficult to bring schema evolution back into the > > > IPC Stream format (i.e. it would live only in flight) > > > > Gosh's proposal extends the flatbuffer structures not the protobufs. Can > > you help me understand how difficult it would be to bring the `schema_id` > > approach to the IPC stream format? > > I thought we were talking solely about the Flight Protobuf definitions - > not the Flatbuffers (and the Google doc at least only talks about the > Protobufs). > I somehow missed that schema_id is being added to protobuf in the document. It feels to me that the schema_id is a property that would ideally only apply to the RecordBatch. I better understand Micah's dictionary concerns, now, too. > Side Question: Why isn't the IPC stream format a series of the flight > > protobufs? It's a real shame that there is no standard way to > > capture/replay a stream with app_metadata. (Obviously ignoring the > > annoyances around protobuf wrapping flatbuffers.) > > The IPC format was defined long before Flight, and Flight's app_metadata > was added after Flight's initial definition. Note an IPC message does have > a provision for key-value metadata, though I think APIs for that are not > fully exposed. (See ARROW-6940: > https://issues.apache.org/jira/browse/ARROW-6940 and despite my comments > there perhaps we need to unify or at least consider how Flight's > app_metadata relates to the IPC message custom_metadata. Also perhaps see > ARROW-1059.) > KeyValue unfortunately is string to string. In flatbuffer strings are only UTF-8 or 7-bit ASCII. The app_metadata on the other hand is opaque bytes. The latter is a bit more useful. --
Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams
t; 1. It seems like renaming stream_id to schema_id and delegating >>>> "logical >>>> > stream" distinction to app_metadata mitigates the "multiplexing" point >>>> > while at the same time it gives enough flexibility to address both >>>> Nate's >>>> > and our use cases. >>>> >>>> >>>> I don't think this is the case. It seems that having no additional >>>> fields >>>> added and then sending a new schema when necessary combined with Union >>>> of >>>> Structs would solve most use-cases. The main downside could be >>>> potential >>>> performance implications if the schema is changing frequently. Gosh, >>>> could >>>> you address why this wouldn't be sufficient (either here or on the doc). >>>> >>>> Thanks, >>>> -Micah >>>> >>>> On Fri, Jun 25, 2021 at 5:30 AM Gosh Arzumanyan >>>> wrote: >>>> >>>> > Hi guys, >>>> > >>>> > Thanks for sharing your insights/concerns! I also left some comments >>>> based >>>> > on the discussion we had. Briefly: >>>> > >>>> > 1. It seems like renaming stream_id to schema_id and delegating >>>> "logical >>>> > stream" distinction to app_metadata mitigates the "multiplexing" point >>>> > while at the same time it gives enough flexibility to address both >>>> Nate's >>>> > and our use cases. >>>> > 2. To David's point about other transports: in fact currently we are >>>> using >>>> > other transports(aside from gRPC) so we don't wanna depend on only >>>> gRPC >>>> > features. >>>> > >>>> > Cheers, >>>> > Gosh >>>> > >>>> > On Wed, Jun 23, 2021 at 10:40 PM David Li >>>> wrote: >>>> > >>>> > > Thanks for chiming in - I've replied in the doc. Scoping it to just >>>> > schema >>>> > > evolution would be preferable, but I'm not sure if Gosh's usecase >>>> > requires >>>> > > more flexibility than that or not. >>>> > > >>>> > > Again, though, given that 1) gRPC recycles a connection, so repeated >>>> > calls >>>> > > aren't necessarily expensive and 2) encoding tricks like >>>> > union-of-structs, >>>> > > any solution needs to be weighed against those/we should make sure >>>> to >>>> > > document why they aren't sufficient. (For instance, 1) is hampered >>>> by the >>>> > > use of L7 load balancers and/or client-side load balancing policies >>>> in >>>> > gRPC >>>> > > and assumes statefulness which is undesirable in general. There's >>>> also >>>> > the >>>> > > eventual desire to have a transport besides gRPC someday.) >>>> > > >>>> > > -David >>>> > > >>>> > > On Wed, Jun 23, 2021, at 16:24, Nate Bauernfeind wrote: >>>> > > >>>> > > Thanks for writing this up! I added a few general comments, but >>>> have a >>>> > > question on the approach because it's not quite what I was >>>> expecting. >>>> > > >>>> > > I am slightly concerned that the proposal looks more like support >>>> for >>>> > > "multiplexing" IPC streams into a single RPC stream rather than >>>> support >>>> > for >>>> > > a changing Schema of an otherwise consistently logical stream. gRPC >>>> > already >>>> > > does a good job decoupling RPC streams from one another. I feel that >>>> > > throwing that idea into the IPC stream increases client-library >>>> > > implementation cost by quite a lot. >>>> > > >>>> > > Why is it not good enough to replace the Schema when we see a >>>> duplicate? >>>> > > This is undoubtedly less work across all client implementations. >>>> > > >>>> > > The benefit I see is that you might have two schemas that you swap >>>> > between >>>> > > frequently then you can indicate with a single integer. If that's >>>> what >>>> >
Re: [C++] Reducing branching in compute/kernels/vector_selection.cc
> Basically, it reset/set the borrow bit in eflag register based on the if
condition, and runs `outpos = outpos - (-1) - borrow_bit`.
That's clever, and I clearly didn't see that!
On Thu, Jun 24, 2021 at 8:57 PM Yibo Cai wrote:
>
>
> On 6/25/21 6:58 AM, Nate Bauernfeind wrote:
> > FYI, the bench was slightly broken; but the results stand.
> >
> >> benchmark::DoNotOptimize(output[rand()]);
> > Since rand() has a domain of 0 to MAX_INT it blows past the output array
> > (of length 4k). It segfaults in GCC; I'm not sure why the Clang benchmark
> > is happy with that.
> >
> > I modified [1] it to:
> >> benchmark::DoNotOptimize(output[rand() % N]);
> >
> > The benchmarks run:
> > The Clang11 -O3 speed up is 2.3x.
> > The GCC10.2 -O3 speed up is 2.6x.
> >
> > Interestingly, I added a second branching benchmark. The original
> branching
> > does this:
> > ```
> >if (bitmap[i/8] & (1 << (i%8))) {
> > output[outpos++] = input[i];
> >}
> >
> ```
> >
> > My additional branching benchmark pulls the assignment outside of the
> > branch:
> > ```
> >output[outpos] = input[i];
> >if (bitmap[i/8] & (1 << (i%8))) {
> > ++outpos;
> >}
> > ```
>
> From the disassembler, compiler optimizes this c code with below
> branch-less assembly code.
>
> 10.06% sar%cl,%edx
> 10.74% and$0x1,%edx
> 9.93% cmp$0x1,%edx
> sbb$0x,%esi
>
> Basically, it reset/set the borrow bit in eflag register based on the if
> condition, and runs `outpos = outpos - (-1) - borrow_bit`.
>
> >
> > The benchmarks run:
> > The GCC10.2 -O3 speed up compared to the original branching code is 2.3x.
> > (are you as surprised as me?)
> > The GCC10.2 -O3 speed up compared to the original non-branching code is
> > 0.9x. (yes; it is slightly slower)
> >
> > Point: reducing the footprint of a false branch prediction is as
> worthwhile
> > an investment when you can't simply get rid of the conditional.
> >
> > Nate
> > [1] https://quick-bench.com/q/kDFoF2pOuvPo9aVFufkVJMjWf-g
> >
> > On Thu, Jun 24, 2021 at 1:01 PM Niranda Perera >
> > wrote:
> >
> >> I created a JIRA for this. I will do the changes in select kernels and
> >> report back with benchmark results
> >> https://issues.apache.org/jira/browse/ARROW-13170
> >>
> >>
> >> On Thu, Jun 24, 2021 at 12:27 AM Yibo Cai wrote:
> >>
> >>> Did a quick test. For random bitmaps and my trivial test code, the
> >>> branch-less code is 3.5x faster than branch one.
> >>> https://quick-bench.com/q/UD22IIdMgKO9HU1PsPezj05Kkro
> >>>
> >>> On 6/23/21 11:21 PM, Wes McKinney wrote:
> >>>> One project I was interested in getting to but haven't had the time
> >>>> was introducing branch-free code into vector_selection.cc and reducing
> >>>> the use of if-statements to try to improve performance.
> >>>>
> >>>> One way to do this is to take code that looks like this:
> >>>>
> >>>> if (BitUtil::GetBit(filter_data_, filter_offset_ + in_position)) {
> >>>> BitUtil::SetBit(out_is_valid_, out_offset_ + out_position_);
> >>>> out_data_[out_position_++] = values_data_[in_position];
> >>>> }
> >>>> ++in_position;
> >>>>
> >>>> and change it to a branch-free version
> >>>>
> >>>> bool advance = BitUtil::GetBit(filter_data_, filter_offset_ +
> >>> in_position);
> >>>> BitUtil::SetBitTo(out_is_valid_, out_offset_ + out_position_,
> advance);
> >>>> out_data_[out_position_] = values_data_[in_position];
> >>>> out_position_ += advance; // may need static_cast here
> >>>> ++in_position;
> >>>>
> >>>> Since more people are working on kernels and computing now, I thought
> >>>> this might be an interesting project for someone to explore and see
> >>>> what improvements are possible (and what the differences between e.g.
> >>>> x86 and ARM architecture are like when it comes to reducing
> >>>> branching). Another thing to look at might be batch-at-a-time
> >>>> bitpacking in the output bitmap versus bit-at-a-time.
> >>>>
> >>>
> >>
> >>
> >> --
> >> Niranda Perera
> >> https://niranda.dev/
> >> @n1r44 <https://twitter.com/N1R44>
> >>
> >
> >
> > --
> >
>
--
Re: [C++] Reducing branching in compute/kernels/vector_selection.cc
FYI, the bench was slightly broken; but the results stand.
> benchmark::DoNotOptimize(output[rand()]);
Since rand() has a domain of 0 to MAX_INT it blows past the output array
(of length 4k). It segfaults in GCC; I'm not sure why the Clang benchmark
is happy with that.
I modified [1] it to:
> benchmark::DoNotOptimize(output[rand() % N]);
The benchmarks run:
The Clang11 -O3 speed up is 2.3x.
The GCC10.2 -O3 speed up is 2.6x.
Interestingly, I added a second branching benchmark. The original branching
does this:
```
if (bitmap[i/8] & (1 << (i%8))) {
output[outpos++] = input[i];
}
```
My additional branching benchmark pulls the assignment outside of the
branch:
```
output[outpos] = input[i];
if (bitmap[i/8] & (1 << (i%8))) {
++outpos;
}
```
The benchmarks run:
The GCC10.2 -O3 speed up compared to the original branching code is 2.3x.
(are you as surprised as me?)
The GCC10.2 -O3 speed up compared to the original non-branching code is
0.9x. (yes; it is slightly slower)
Point: reducing the footprint of a false branch prediction is as worthwhile
an investment when you can't simply get rid of the conditional.
Nate
[1] https://quick-bench.com/q/kDFoF2pOuvPo9aVFufkVJMjWf-g
On Thu, Jun 24, 2021 at 1:01 PM Niranda Perera
wrote:
> I created a JIRA for this. I will do the changes in select kernels and
> report back with benchmark results
> https://issues.apache.org/jira/browse/ARROW-13170
>
>
> On Thu, Jun 24, 2021 at 12:27 AM Yibo Cai wrote:
>
> > Did a quick test. For random bitmaps and my trivial test code, the
> > branch-less code is 3.5x faster than branch one.
> > https://quick-bench.com/q/UD22IIdMgKO9HU1PsPezj05Kkro
> >
> > On 6/23/21 11:21 PM, Wes McKinney wrote:
> > > One project I was interested in getting to but haven't had the time
> > > was introducing branch-free code into vector_selection.cc and reducing
> > > the use of if-statements to try to improve performance.
> > >
> > > One way to do this is to take code that looks like this:
> > >
> > > if (BitUtil::GetBit(filter_data_, filter_offset_ + in_position)) {
> > >BitUtil::SetBit(out_is_valid_, out_offset_ + out_position_);
> > >out_data_[out_position_++] = values_data_[in_position];
> > > }
> > > ++in_position;
> > >
> > > and change it to a branch-free version
> > >
> > > bool advance = BitUtil::GetBit(filter_data_, filter_offset_ +
> > in_position);
> > > BitUtil::SetBitTo(out_is_valid_, out_offset_ + out_position_, advance);
> > > out_data_[out_position_] = values_data_[in_position];
> > > out_position_ += advance; // may need static_cast here
> > > ++in_position;
> > >
> > > Since more people are working on kernels and computing now, I thought
> > > this might be an interesting project for someone to explore and see
> > > what improvements are possible (and what the differences between e.g.
> > > x86 and ARM architecture are like when it comes to reducing
> > > branching). Another thing to look at might be batch-at-a-time
> > > bitpacking in the output bitmap versus bit-at-a-time.
> > >
> >
>
>
> --
> Niranda Perera
> https://niranda.dev/
> @n1r44
>
--
Re: [STRAW POLL] (How) should Arrow define storage for "Instant"s
Option C. On Thu, Jun 24, 2021 at 1:53 PM Joris Peeters wrote: > C > > On Thu, Jun 24, 2021 at 8:39 PM Antoine Pitrou wrote: > > > > > Option C. > > > > > > Le 24/06/2021 à 21:24, Weston Pace a écrit : > > > > > > This proposal states that Arrow should define how to encode an Instant > > > into Arrow data. There are several ways this could happen, some which > > > change schema.fbs and some which do not. > > > --- > > > > > > For sample arguments (currently grouped as "for changing schema.fbs" > > > and "against changing schema.fbs") see [2]. For a detailed definition > > > of the terms LocalDateTime, ZonedDateTime, and Instant and a > > > discussion of their semantics see [3]. > > > > > > Options: > > > > > > A) Do nothing, don’t introduce the nuance of “instants” into Arrow > > > B) Do nothing, but update the comments in schema.fbs to acknowledge > > > the existence of the concept and explain that implementations are free > > > to decide if/how to support the type. > > > C) Define timestamp with timezone “UTC” as “instant”. > > > D) Add a first class instant type to schema.fbs > > > E) Add instant as a canonical extension type > > > > > > Note: This is just a straw poll and the results will not be binding in > > > any way but will help craft a future vote. For example, if the > > > plurality of votes goes to C but a majority of votes is spread across > > > A & B then some flavor of A/B would likely be pursued. > > > > > > Vote for as many options as you would like. > > > > > > I will summarize and send out the results in 72 hours. > > > > > > --
Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams
Thanks for writing this up! I added a few general comments, but have a question on the approach because it's not quite what I was expecting. I am slightly concerned that the proposal looks more like support for "multiplexing" IPC streams into a single RPC stream rather than support for a changing Schema of an otherwise consistently logical stream. gRPC already does a good job decoupling RPC streams from one another. I feel that throwing that idea into the IPC stream increases client-library implementation cost by quite a lot. Why is it not good enough to replace the Schema when we see a duplicate? This is undoubtedly less work across all client implementations. The benefit I see is that you might have two schemas that you swap between frequently then you can indicate with a single integer. If that's what you want to support I would rather think of them as `schema_id` instead of `stream_id` and not give this impression that multiplexing is a goal. As you have proposed, it seems that the "done writing for a stream" needs a callback notifying the user receiving the stream that a logical subset of the flight is complete. Alternatively, if they aren't independent streams (to the end-user), we could tell the Arrow layer that a particular schema is no longer needed without also needing to communicate further downstream. On Wed, Jun 23, 2021 at 1:39 PM David Li wrote: > Ah to be clear, the API is indeed inconsistent - DoExchange was added some > time later (and by its nature returning a FlightDataStream would not have > been possible, since it's meant to be able to interleave reading/writing). > But really, DoGet is indeed the odd one out in the C++ API and it may be > worth correcting. You could also perhaps imagine making a FlightDataStream > implementation that accepts a closure and provides it a fake writer, if the > API mismatch is hard to work with... > > That said: this has some benefits, e.g. for a Python service that returns > a Table, that means data can be fed into gRPC entirely in C++ without > having to bounce into Python for each chunk. > > Best, > David > > On Wed, Jun 23, 2021, at 15:33, Gosh Arzumanyan wrote: > > Hi David, > > > > Got you. In fact I was looking at this more from the point of view of > consistency of the API in terms of "inputs" and thought DoExchange is kind > of a DoGet+ so might make sense to have the same classes being utilized in > both places. But again, I might be missing something and I get the point > about breaking change. > > > > Cheers, > > Gosh > > > > On Wed, Jun 23, 2021 at 2:58 PM David Li wrote: > >> __ > >> It's mostly a quirk of implementation (and just for clarification, > they're all nearly identical on the format/protocol level). > >> > >> DoGet is conceptualized as your application returning a readable stream > of batches, instead of your application imperatively writing batches to the > client. (This is different than how Flight is implemented in Java.) You > would normally not implement FlightDataStream - you would return a > RecordBatchStream. > >> > >> DoGet could not have FlightMessageWriter as a return type as that > wouldn't make sense, but it could accept an instance of that as a parameter > instead, much like DoExchange. That would be a breaking change. > >> > >> Best, > >> David > >> > >> On Wed, Jun 23, 2021, at 08:47, Gosh Arzumanyan wrote: > >>> Hi David, > >>> > >>> Going through the ArrowFlight API: got confused a bit on DoGet and > >>> DoPut/DoExachange apis: the former one expects FlightDataStream which > talks > >>> in already serialized message terms while the latter to > >>> accept FlightMessageReader/Writer which expect the user to pass in > >>> RecordBatches etc. Is there any reason why the DoGet can't have > >>> FlightMessageWriter as a return type? > >>> > >>> Cheers, > >>> Gosh > >>> > >>> On Mon, Jun 21, 2021 at 9:47 PM Gosh Arzumanyan > wrote: > >>> > >>> > Thanks David! > >>> > > >>> > I also responded/added more suggestions/questions to the doc. I > think it > >>> > makes sense to have two sections: one purely protocol oriented and > second > >>> > API oriented(examples in c++ or in any other language should make > the idea > >>> > easier to digest). > >>> > > >>> > Thanks for the reference too! > >>> > > >>> > Cheers, > >>> > Gosh > >>> > > >>> > On Mon, Jun 21, 2021 at 4:41 PM David Li > wrote: > >>> > > >>> >> Thanks! I've left some initial comments/suggestions to expand it in > terms > >>> >> of the format definitions and not the C++ APIs. > >>> >> > >>> >> I'll also note something like this was proposed a long time ago - > there's > >>> >> not very much discussion about it there but for reference: > >>> >> > https://lists.apache.org/thread.html/0e5ba78c48cdd0e357f3a4a6d8affd31767c34376b62c001910823af%40%3Cdev.arrow.apache.org%3E > >>> >> (or see the thread '[Discuss][FlightRPC] Extensions to Flight: > >>> >> "DoBidirectional"' from 2019-2020). It might be good to address why > the > >>> >> proposed workaround there
Re: [ANNOUNCE] New Arrow PMC member: David M Li
Congratulations! Well earned! On Mon, Jun 21, 2021 at 4:20 PM Ian Cook wrote: > Congratulations, David! > > Ian > > > On Mon, Jun 21, 2021 at 6:19 PM Wes McKinney wrote: > > > > The Project Management Committee (PMC) for Apache Arrow has invited > > David M Li to become a PMC member and we are pleased to announce > > that David has accepted. > > > > Congratulations and welcome! >
Re: [C++] Async Arrow Flight
Google Cloud supports asynchronous grpc in C++ for parts of their API.
One such client-facing API method is this:
```
future> AsyncPublish(
google::cloud::CompletionQueue& cq,
std::unique_ptr context,
google::pubsub::v1::PublishRequest const& request) override {
return cq.MakeUnaryRpc(
[this](grpc::ClientContext* context,
google::pubsub::v1::PublishRequest const& request,
grpc::CompletionQueue* cq) {
return grpc_stub_->AsyncPublish(context, request, cq);
},
request, std::move(context));
}
```
Source:
https://github.com/googleapis/google-cloud-cpp/blob/0dfde771a42cff19ca3ef6e192473ca0646df497/google/cloud/pubsub/internal/publisher_stub.cc#L109-L120
The implementation integrates with gRPC's CompletionQueue interface:
https://github.com/googleapis/google-cloud-cpp/blob/main/google/cloud/completion_queue.h
The completion queue returns a future abiding to this API:
https://github.com/googleapis/google-cloud-cpp/blob/main/google/cloud/internal/future_base.h
It looks like a really nice solution as far as async gRPC; I wish gRPC had
such an implementation in the base library. What do you think of this?
Would this pattern appeal to you, David - or to anyone listening from the
larger Arrow community?
Thanks,
Nate
On Thu, Jun 3, 2021 at 4:50 PM David Li wrote:
> I see, thanks. That sounds reasonable, and indeed if you are subscribing
> to lots of data sources (instead of just trying to maximize throughput as I
> think most people have done before) async may be helpful. There are
> actually two features being described here though:
> 1. The C++ Flight client needs some way to wrap an existing gRPC client,
> much like it can in Java. This may be tricky in C++ (and will require
> applications to carefully manage gRPC versions) and will probably not be
> possible for Python.
> 2. The C++ Flight client should offer async APIs (perhaps callback-based
> APIs or perhaps completion-queue based or something else).
> https://issues.apache.org/jira/browse/ARROW-1009 may be relevant here.
> I think gRPC C++ has experimental callback-based async APIs now but it may
> be hard for us to actually take advantage of unless we can draw a hard line
> on minimum required gRPC version. The main question will probably be
> finding someone to do the work.
>
> Best,
> David
>
> On Thu, Jun 3, 2021, at 12:11, Nate Bauernfeind wrote:
> > In addition to Arrow Flight we have other gRPC APIs that work together
> as a
> > whole. For example, the API client establishes a session with the server.
> > Then the client tells the server to create a derivative data stream by
> > filtering/joining/computing/etc several source streams. The server will
> > keep this derivative stream available as long as the session exists. This
> > session is kept alive with a heartbeat. So a small part of this system
> > requires a regular (but not overwhelming) heartbeat. If this derivative
> > stream has any data source that is live (aka regularly updating) then the
> > derivative is also live. The API client would likely want to subscribe to
> > the derivative stream, but still needs to heartbeat. The subscription
> > (communicated via Flight's DoExchange) should be able to last forever.
> > Naturally, this API pattern is simply better suited to the async pattern.
> >
> > Ideally, we could re-use the same http2 connections, task queue and/or
> > thread-pool between the arrow flight api and the other grpc apis (talking
> > to the same endpoint).
> >
> > The gRPC callback pattern in Java is nice; it's a bit of a shame that
> gRPC
> > hasn't settled on a c++ callback pattern.
> >
> > These are the kinds of ideas that it enables:
> > - Subscribe to multiple ticking sources simultaneously.
> > - Can have multiple gRPC requests outstanding at the same time
> > (particularly useful if server is in a remote data center with high RTT).
> > - Can communicate with multiple remote hosts simultaneously.
> > - In general, enables the client to be event driven.
> >
> > Note that our clients tend to be light; they typically listen to derived
> > data and then act based on the information without further computation
> > locally.
> >
> > The gRPC c++ async library does indeed look a bit under-documented. There
> > are a few blog posts that highlight some of the surprises, but async
> > behavior is a requirement for what we're working on. (for async-cpp-grpc
> > surprises see:
> >
> https://www.gresearch.co.uk/article/lessons-learnt-from-writing-asynchronous-streaming-grpc-services-in-c/
> > ).
> >
> > Nate
> >
> > On Wed, Jun 2, 2021 at 8:44 PM David Li
Re: [C++] Async Arrow Flight
In addition to Arrow Flight we have other gRPC APIs that work together as a whole. For example, the API client establishes a session with the server. Then the client tells the server to create a derivative data stream by filtering/joining/computing/etc several source streams. The server will keep this derivative stream available as long as the session exists. This session is kept alive with a heartbeat. So a small part of this system requires a regular (but not overwhelming) heartbeat. If this derivative stream has any data source that is live (aka regularly updating) then the derivative is also live. The API client would likely want to subscribe to the derivative stream, but still needs to heartbeat. The subscription (communicated via Flight's DoExchange) should be able to last forever. Naturally, this API pattern is simply better suited to the async pattern. Ideally, we could re-use the same http2 connections, task queue and/or thread-pool between the arrow flight api and the other grpc apis (talking to the same endpoint). The gRPC callback pattern in Java is nice; it's a bit of a shame that gRPC hasn't settled on a c++ callback pattern. These are the kinds of ideas that it enables: - Subscribe to multiple ticking sources simultaneously. - Can have multiple gRPC requests outstanding at the same time (particularly useful if server is in a remote data center with high RTT). - Can communicate with multiple remote hosts simultaneously. - In general, enables the client to be event driven. Note that our clients tend to be light; they typically listen to derived data and then act based on the information without further computation locally. The gRPC c++ async library does indeed look a bit under-documented. There are a few blog posts that highlight some of the surprises, but async behavior is a requirement for what we're working on. (for async-cpp-grpc surprises see: https://www.gresearch.co.uk/article/lessons-learnt-from-writing-asynchronous-streaming-grpc-services-in-c/ ). Nate On Wed, Jun 2, 2021 at 8:44 PM David Li wrote: > Hey Nate, > > I think there's an open JIRA for something like this. I'd love to have > something that plays nicely with asyncio/trio in Python and is hopefully > more efficient. (I think it would also let us finally have per-message > timeouts instead of only a per-call deadline.) There are some challenges > though, e.g. we wouldn't expose gRPC's event loop directly so that we could > support other transports, but then that leaves more things to design. I > also recall the async C++ APIs being very underdocumented, I get the sense > that they aren't actually used except to improve some benchmarks. I'll note > for instance gRPC in Python, which offers async support, uses the "core" > APIs directly and doesn't use anything C++ offers. > > But long story short, if you're interested in this I think it would be a > useful addition. What sorts of things would it enable for you? > > -David > > On Wed, Jun 2, 2021, at 16:20, Nate Bauernfeind wrote: > > It seems to me that the c++ arrow flight implementation uses only the > > synchronous version of the gRPC API. gRPC supports asynchronous message > > delivery in C++ via a CompletionQueue that must be polled. Has there been > > any desire to standardize on a solution for asynchronous use cases, > perhaps > > delivered via a provided CompletionQueue? > > > > For a simple async grpc c++ example you can look here: > > > https://github.com/grpc/grpc/blob/master/examples/cpp/helloworld/greeter_async_client.cc > > > > Thanks, > > Nate > > > > -- > > --
[C++] Async Arrow Flight
It seems to me that the c++ arrow flight implementation uses only the synchronous version of the gRPC API. gRPC supports asynchronous message delivery in C++ via a CompletionQueue that must be polled. Has there been any desire to standardize on a solution for asynchronous use cases, perhaps delivered via a provided CompletionQueue? For a simple async grpc c++ example you can look here: https://github.com/grpc/grpc/blob/master/examples/cpp/helloworld/greeter_async_client.cc Thanks, Nate --
Re: [Flight Extension] Request for Comments
eeling for this concern. > > > > > > > > I think this is definitely something to measure. I wouldn't expect > the > > > > performance differential to be that large. > > > > > > > > Regarding the schema evolution idea: > > > > > What can I do to get started? Does it make sense to target the > > feature > > > as > > > > > a new field in the protobuf so that it can be used in contexts with > > > other > > > > > header metadata types? Do you have time to riff on the format that > > will > > > > > apply to the other contexts? I believe all I would need is a bitset > > > > > identifying which columns are included, but if enabling/disabling > > > features > > > > > is a nice-to-have then a bitset is going to be a bit weak. I can > > also, > > > for > > > > > now, cheat and send empty field nodes and empty buffers for those > > > columns > > > > > (but I am, already, slightly concerned with overhead). > > > > > > > > I think David might be able to give more guidance. My recollection > of > > > the > > > > library specifics are hazy, but I think we could potentially just > > > interpret > > > > a new schema arriving as indicating all record batches after that > > schema > > > > would follow the new schema. Would that work for your use case? > David > > > > would probably be able to give guidance on how feasible a change like > > > that > > > > would be. Typically, before we officially alter the specification we > > > want > > > > to see working implementation in Java and C++ that pass an > integration > > > > test. But I think we can figure out the specifics here if we can > > > > understand concrete requirements. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 8, 2021 at 6:42 PM Nate Bauernfeind < > > > > natebauernfe...@deephaven.io> wrote: > > > > > > > > > >note that FlightData already has a separate app_metadata field > > > > > > > > > > That is an interesting point; are there any conventions on how to > use > > > the > > > > > app_metadata compatibly without stepping on other ideas/projects > > doing > > > the > > > > > same? It would be convenient for the server to verify that the > client > > > is > > > > > making the request that the server interprets. Do projects use a > > magic > > > > > number prefix? Or possibly is there some sort of common header? I > > > suspect > > > > > that other projects may benefit from having the ability to publish > > > > > incremental updates, too. So, I'm just curious if there is any > > > pre-existing > > > > > domain-knowledge in this respect. > > > > > > > > > > Nate > > > > > > > > > > On Mon, Mar 8, 2021 at 1:55 PM David Li > wrote: > > > > > > > > > > > Hey - pretty much, I think. I'd just like to note that FlightData > > > already > > > > > > has a separate app_metadata field, for metadata on top of any > > > Arrow-level > > > > > > data, so you could ship the Barrage metadata alongside the first > > > record > > > > > > batch, without having to modify anything about the record batch > > > itself, > > > > > and > > > > > > without having to define a new metadata header at the Arrow > level - > > > > > > everything could be implemented on top of the existing > definitions. > > > > > > > > > > > > David > > > > > > > > > > > > On Sat, Mar 6, 2021, at 01:07, Nate Bauernfeind wrote: > > > > > > > Eww. I didn't specify why I had two sets of record batches. > > > Slightly > > > > > > > revised: > > > > > > > > > > > > > > Are you suggesting this pattern of messages per incremental > > update? > > > > > > > - FlightData with [the new] metadata header that includes > > > > > > > added/removed/modified information, the number of add record > > > batches, > > > > > and > > > > > > > the number of modified record
Re: Long title on github page
Suggestion: faster -> more efficiently "Apache Arrow is a cross-language development platform for in-memory data. It enables systems to process and transport data more efficiently." On Sun, May 16, 2021 at 11:35 AM Wes McKinney wrote: > Here's what there now: > > "Apache Arrow is a cross-language development platform for in-memory > data. It specifies a standardized language-independent columnar memory > format for flat and hierarchical data, organized for efficient > analytic operations on modern hardware. It also provides computational > libraries and zero-copy streaming messaging and interprocess > communication…" > > How about something shorter like > > "Apache Arrow is a cross-language development platform for in-memory > data. It enables systems to process and transport data faster." > > Suggestions / refinements from others welcome > > > On Sat, May 15, 2021 at 9:12 PM Dominik Moritz wrote: > > > > Super minor issue but could someone make the description on GitHub > shorter? > > > > > > > > GitHub puts the description into the title of the page and makes it hard > to find it in URL autocomplete. > > > --
Re: [Java] Source control of generated flatbuffers code
> I think checking in the java files is fine and probably better then relying > on a third party package. We should make sure there are instructions on > how to regenerate them along with the PR Micah, I just opened a pull-request to satisfy ARROW-12111. This is my first contribution to an apache project; please let me know if there is anything else that I need to do to get this past the finish line. https://github.com/apache/arrow/pull/10058 Thanks, Nate On Wed, Apr 14, 2021 at 11:45 PM Nate Bauernfeind < natebauernfe...@deephaven.io> wrote: > Hey Bob, > > Someone did publish a 1.12 version of the flatc maven plugin. I double > checked that the plugin and binaries look correct and legit.. but you know, > it's always shady to download some random executable from the internet and > run it. However, I have been using it to generate the arrow flatbuffer > files because I _really_ wanted some features that are in flatc 1.12's > runtime jar (there are performance improvements for array types in > particular). > > You can see them here: > https://search.maven.org/search?q=com.github.shinanca > The repository fork is here: https://github.com/shinanca/flatc > > On the bright side that developer appears to have published an x86_64 > windows binary which might satisfy one of your earlier complaints in the > thread. > > On the other hand, if everyone is comfortable checking in the flatc > generated files (obviously with the additional documentation on how to > re-generate should the fbs files change), it's a relatively small change to > replace the existing apache/arrow/java/format source. Based on the previous > discussion on this thread, it seems that the arrow dev team _is_ > comfortable with the check-in-the-generated-files approach. > > Although 4.0 is near the release phase, there are still a few blocking > issues that people are trying to fix (according to the arrow-sync call > earlier today). I don't mind jumping in and doing this; it appears that > there might be enough time for such a small change to make it into the > release if the work is performed and merged ASAP. > > I guess, I'm either looking for the "pull request is on the way" or the > "thumbs up - we definitely want this; I'll get the code review for you when > it's ready" style reply =D. > > On Wed, Apr 14, 2021 at 10:43 PM Bob Tinsman wrote: > >> I apologize for leaving this hanging, but it looks like 4.0 is leaving >> the station :( >> Yes, it makes sense to bump it to 1.12, but you can't do that in >> isolation, because the flatc binary which is fetched as a Maven dependency >> is only available for 1.9. I will get back onto this and finish it this >> week. >> >> FWIW, I was looking around and catalogued the various ways of generating >> flatbuffers for all the languages--you can look at it in my branch: >> https://github.com/bobtins/arrow/tree/check-in-gen-code/java/dev >> Let me know if any info is wrong or missing. >> The methods of generation are all over the map, and some have no script >> or build file, just doc. Would there be any value in making this more >> uniform? >> >> On 2021/04/14 16:36:47, Nate Bauernfeind >> wrote: >> > It would also be nice to upgrade that java flatbuffer version from 1.9 >> to >> > 1.12. Is anyone planning on doing this work (as listed in ARROW-12111)? >> > >> > If I did this work today, might it be possible to get it included in the >> > 4.0.0 release? >> > >> > On Fri, Mar 26, 2021 at 3:25 PM bobtins wrote: >> > >> > > OK, originally this was part of >> > > https://issues.apache.org/jira/browse/ARROW-12006 and I was going to >> just >> > > add some doc on flatc, but I will make this a new bug because it's a >> little >> > > bigger: https://issues.apache.org/jira/browse/ARROW-12111 >> > > >> > > On 2021/03/23 23:40:50, Micah Kornfield >> wrote: >> > > > > >> > > > > I have a concern, though. Four other languages (Java would be >> five) >> > > check >> > > > > in the generated flatbuffers code, and it appears (based on a >> quick >> > > scan of >> > > > > Git logs) that this is done manually. Is there a danger that the >> binary >> > > > > format could change, but some language might get forgotten, and >> thus be >> > > > > working with the old format? >> > > > >> > > > The format changes relatively slowly and any changes at this point >> should >> > > > be backwards compatible
Re: [Java] Source control of generated flatbuffers code
Hey Bob, Someone did publish a 1.12 version of the flatc maven plugin. I double checked that the plugin and binaries look correct and legit.. but you know, it's always shady to download some random executable from the internet and run it. However, I have been using it to generate the arrow flatbuffer files because I _really_ wanted some features that are in flatc 1.12's runtime jar (there are performance improvements for array types in particular). You can see them here: https://search.maven.org/search?q=com.github.shinanca The repository fork is here: https://github.com/shinanca/flatc On the bright side that developer appears to have published an x86_64 windows binary which might satisfy one of your earlier complaints in the thread. On the other hand, if everyone is comfortable checking in the flatc generated files (obviously with the additional documentation on how to re-generate should the fbs files change), it's a relatively small change to replace the existing apache/arrow/java/format source. Based on the previous discussion on this thread, it seems that the arrow dev team _is_ comfortable with the check-in-the-generated-files approach. Although 4.0 is near the release phase, there are still a few blocking issues that people are trying to fix (according to the arrow-sync call earlier today). I don't mind jumping in and doing this; it appears that there might be enough time for such a small change to make it into the release if the work is performed and merged ASAP. I guess, I'm either looking for the "pull request is on the way" or the "thumbs up - we definitely want this; I'll get the code review for you when it's ready" style reply =D. On Wed, Apr 14, 2021 at 10:43 PM Bob Tinsman wrote: > I apologize for leaving this hanging, but it looks like 4.0 is leaving the > station :( > Yes, it makes sense to bump it to 1.12, but you can't do that in > isolation, because the flatc binary which is fetched as a Maven dependency > is only available for 1.9. I will get back onto this and finish it this > week. > > FWIW, I was looking around and catalogued the various ways of generating > flatbuffers for all the languages--you can look at it in my branch: > https://github.com/bobtins/arrow/tree/check-in-gen-code/java/dev > Let me know if any info is wrong or missing. > The methods of generation are all over the map, and some have no script or > build file, just doc. Would there be any value in making this more uniform? > > On 2021/04/14 16:36:47, Nate Bauernfeind > wrote: > > It would also be nice to upgrade that java flatbuffer version from 1.9 to > > 1.12. Is anyone planning on doing this work (as listed in ARROW-12111)? > > > > If I did this work today, might it be possible to get it included in the > > 4.0.0 release? > > > > On Fri, Mar 26, 2021 at 3:25 PM bobtins wrote: > > > > > OK, originally this was part of > > > https://issues.apache.org/jira/browse/ARROW-12006 and I was going to > just > > > add some doc on flatc, but I will make this a new bug because it's a > little > > > bigger: https://issues.apache.org/jira/browse/ARROW-12111 > > > > > > On 2021/03/23 23:40:50, Micah Kornfield wrote: > > > > > > > > > > I have a concern, though. Four other languages (Java would be five) > > > check > > > > > in the generated flatbuffers code, and it appears (based on a quick > > > scan of > > > > > Git logs) that this is done manually. Is there a danger that the > binary > > > > > format could change, but some language might get forgotten, and > thus be > > > > > working with the old format? > > > > > > > > The format changes relatively slowly and any changes at this point > should > > > > be backwards compatible. > > > > > > > > > > > > > > > > > Or is there enough interop testing that the problem would get > caught > > > right > > > > > away? > > > > > > > > In most cases I would expect integration tests to catch these types > of > > > > error. > > > > > > > > On Tue, Mar 23, 2021 at 4:26 PM bobtins wrote: > > > > > > > > > I'm happy to check in the generated Java source. I would also > update > > > the > > > > > Java build info to reflect this change and document how to > regenerate > > > the > > > > > source as needed. > > > > > > > > > > I have a concern, though. Four other languages (Java would be five) > > > check > > > > > in the generated flatbuffers code, and it appears (based on a quick
Re: [Java] Source control of generated flatbuffers code
It would also be nice to upgrade that java flatbuffer version from 1.9 to 1.12. Is anyone planning on doing this work (as listed in ARROW-12111)? If I did this work today, might it be possible to get it included in the 4.0.0 release? On Fri, Mar 26, 2021 at 3:25 PM bobtins wrote: > OK, originally this was part of > https://issues.apache.org/jira/browse/ARROW-12006 and I was going to just > add some doc on flatc, but I will make this a new bug because it's a little > bigger: https://issues.apache.org/jira/browse/ARROW-12111 > > On 2021/03/23 23:40:50, Micah Kornfield wrote: > > > > > > I have a concern, though. Four other languages (Java would be five) > check > > > in the generated flatbuffers code, and it appears (based on a quick > scan of > > > Git logs) that this is done manually. Is there a danger that the binary > > > format could change, but some language might get forgotten, and thus be > > > working with the old format? > > > > The format changes relatively slowly and any changes at this point should > > be backwards compatible. > > > > > > > > > Or is there enough interop testing that the problem would get caught > right > > > away? > > > > In most cases I would expect integration tests to catch these types of > > error. > > > > On Tue, Mar 23, 2021 at 4:26 PM bobtins wrote: > > > > > I'm happy to check in the generated Java source. I would also update > the > > > Java build info to reflect this change and document how to regenerate > the > > > source as needed. > > > > > > I have a concern, though. Four other languages (Java would be five) > check > > > in the generated flatbuffers code, and it appears (based on a quick > scan of > > > Git logs) that this is done manually. Is there a danger that the binary > > > format could change, but some language might get forgotten, and thus be > > > working with the old format? Or is there enough interop testing that > the > > > problem would get caught right away? > > > > > > I'm new to the project and don't know how big of an issue this is in > > > practice. Thanks for any enlightenment. > > > > > > On 2021/03/23 07:39:16, Micah Kornfield wrote: > > > > I think checking in the java files is fine and probably better then > > > relying > > > > on a third party package. We should make sure there are > instructions on > > > > how to regenerate them along with the PR > > > > > > > > On Monday, March 22, 2021, Antoine Pitrou > wrote: > > > > > > > > > > > > > > Le 22/03/2021 à 20:17, bobtins a écrit : > > > > > > > > > >> TL;DR: The Java implementation doesn't have generated flatbuffers > code > > > > >> under source control, and the code generation depends on an > > > > >> unofficially-maintained Maven artifact. Other language > > > implementations do > > > > >> check in the generated code; would it make sense for this to be > done > > > for > > > > >> Java as well? > > > > >> > > > > >> I'm currently focusing on Java development; I started building on > > > Windows > > > > >> and got a failure under java/format, because I couldn't download > the > > > > >> flatbuffers compiler (flatc) to generate Java source. > > > > >> The artifact for the flatc binary is provided "unofficially" (not > by > > > the > > > > >> flatbuffers project), and there was no Windows version, so I had > to > > > jump > > > > >> through hoops to build it and proceed. > > > > >> > > > > > > > > > > While this does not answer the more general question of checking > in the > > > > > generated Flatbuffers code (which sounds like a good idea, but I'm > not > > > a > > > > > Java developer), note that you could workaround this by installing > the > > > > > Conda-provided flatbuffers package: > > > > > > > > > > $ conda install flatbuffers > > > > > > > > > > which should get you the `flatc` compiler, even on Windows. > > > > > (see https://docs.conda.io/projects/conda/en/latest/ for > installing > > > conda) > > > > > > > > > > You may also try other package managers such as Chocolatey: > > > > > > > > > > https://chocolatey.org/packages/flatc > > > > > > > > > > Regards > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > --
Re: [INFO_REQUEST][FLIGHT] - Dynamic schema changes in ArrowFlight streams
> possibly in coordination with the Deephaven/Barrage team, if they're also still interested Good opportunity for me to chime in =). I think we still have interest in this feature. On the other thread, it took a little cajoling, but I've come around to agree with the conclusions of taking a RecordBatch and splitting it up (a set of RecordBatches for added rows followed by a set of RecordBatches for modifications). In this case I think it's best not to evolve the schema between added row RecordBatches and modified row RecordBatches (sending empty buffer nodes and field nodes will be significantly cheaper). However, the schema evolution would be very useful for when the rpc client changes the set of columns that they are subscribed to (which is relatively rare compared to when the subscribed table itself ticks). That said, schema evolution is not yet particularly high in our queue. On Tue, Apr 13, 2021 at 9:12 AM David Li wrote: > Thanks for the details. I'll note a few things, but adding schema > evolution to Flight is reasonable, if you'd like to put together a > proposal for discussion (possibly in coordination with the > Deephaven/Barrage team, if they're also still interested). > > >3. Assume that there is a strong reason to query A1,..,AK together. > > While I don't know the details here, at least with Flight/gRPC, it's > not necessarily expensive to make several requests to the same server, > as gRPC will consolidate them into the same underlying network > connection. You could issue one GetFlightInfo request for all streams > at once, and get back a list of endpoints for each individual > subquery, which you could then issue separate DoGet requests for. > > There's a slight mismatch there in that GetFlightInfo returns a > FlightInfo, which assumes all endpoints have the same schema. But for > a specific application, you could ignore that field (nothing in Flight > checks that schema against the actual data). > > Of course, if said strong reason is that all the data is really > retrieved together despite being distinct datasets, then this would > complicate the server side implementation quite a bit. But it's one > option. > > > A potential way to address this(with the existing tools) could be having > a > > union schema of all fields across all entities(potentially prefixed with > > the field name just like in sql joins) and setting the values to NA which > > do not belong to an entity. > > I had a similar use case in the past, and it was suggested to use > Arrow's Union type which handles this directly. A Union of Struct > types essentially lets you have multiple distinct schemas all encoded > in the same overall table, with explicit information about which > schema is currently in use. But as you point out this isn't helpful if > you don't know all the schemas up front. > > Best, > David > > On 2021/04/13 11:21:20, Gosh Arzumanyan wrote: > > Hi David, > > > > Thanks for sharing the link! > > > > Here is how a potential use case might look like: > > > >1. Assume that we have a service S which accepts expressions in some > >language X. > >2. Assume that a typical query to this service requests entities A_1, > >A_2,..,A_K. Each of those entities generates a stream of record > batches. > >Record batches for a single A_I share the same schema, yet there is no > >guarantee that schemas are equal across all streams. > >3. Assume that there is a strong reason to query A1,..,AK together. > >4. Service generates record batches(concurrently), tags those(e.g. > with > >schema level metadata) and sends them over. > > > > A potential way to address this(with the existing tools) could be having > a > > union schema of all fields across all entities(potentially prefixed with > > the field name just like in sql joins) and setting the values to NA which > > do not belong to an entity. However this solution might not work in cases > > where we are not able to construct the unified schema before opening the > > stream(e.g. in case of changes in the schema for a specific entity upon > > realtime input feeding or an unpredictable generator expression). > > > > Cheers, > > Gosh > > > > > > On Mon., 12 Apr. 2021, 13:45 David Li, wrote: > > > > > Hi Gosh, > > > > > > There was indeed a discussion where schema evolution was proposed as a > > > solution for another use case: > > > > > > > https://lists.apache.org/thread.html/re800c63f0eb08022c8cd5e1b2236fd69a2e85afdc34daf6b75e3b7b3%40%3Cdev.arrow.apache.org%3E > > > > > > I am curious though, what is your use case here? > > > > > > Best, > > > David > > > > > > On 2021/04/12 10:49:00, Gosh Arzumanyan wrote: > > > > Hi guys, hope you are well! > > > > > > > > Judging from the Flight API > > > > < > > > > https://github.com/apache/arrow/blob/5b08205f7e864ed29f53ed3d836845fed62d5d4a/cpp/src/arrow/flight/types.h#L461 > > > > > > > > and > > > > from the documentation/examples out there, it seems like data schema > is > > > >
Re: No replacement dictionaries supported in pyarrow?
Actually, I slightly want to rephrase my claim. I see the footer is defined
as:
table Footer {
version: org.apache.arrow.flatbuf.MetadataVersion;
schema: org.apache.arrow.flatbuf.Schema;
dictionaries: [ Block ];
recordBatches: [ Block ];
/// User-defined metadata
custom_metadata: [ KeyValue ];
}
So, the footer does not contain the dictionary batch definitions, but
rather points to them. Since dictionary deltas are append-only, one can
build the dictionary up front and then support O(1) random access on the
record batches.
You could support dictionary replacement by having an "effective"
dictionary for contiguous blocks of record batches; which ends up being
O(log(n_k)) for the dictionary with id k that has n_k replacements.
However, the documentation claim, that you can use a dictionary and
dictionary key that has not yet been defined, isn't giving a lot of wiggle
room for alternatives.
On Fri, Mar 19, 2021 at 10:03 AM Nate Bauernfeind <
nate.bauernfe...@gmail.com> wrote:
> The dictionary is not allowed to change throughout the file; which is
> ultimately OP's request. This is because all of the dictionary definition
> is in the footer of the file; which was clearly done to support random
> access of record batches.
>
> To quote the documentation:
>
> > We define a “file format” supporting random access that is build with
> the stream format.
> >
> > [...]
> >
> > In the file format, there is no requirement that dictionary keys should
> be defined in a DictionaryBatch before they are used in a RecordBatch, as
> long as the keys are defined somewhere in the file. Further more, it is
> invalid to have more than one non-delta dictionary batch per dictionary ID
> (i.e. dictionary replacement is not supported). Delta dictionaries are
> applied in the order they appear in the file footer.
>
>
> On Fri, Mar 19, 2021 at 6:37 AM Wes McKinney wrote:
>
>> I am also under the impression that the file format is supposed to support
>> deltas, but not replacements. Is this not implemented in C++?
>>
>> On Thu, Mar 18, 2021 at 9:57 PM Nate Bauernfeind <
>> nate.bauernfe...@gmail.com>
>> wrote:
>>
>> > If dictionary replacements were supported, then the IPC file format
>> > couldn't guarantee random access reads.
>> >
>> > Personally, I would like to support a stream-based file format that is a
>> > series of the Flight protobufs. In my extension of arrow flight, by
>> > stuffing our state-based data into the app_metadata field on the
>> FlightData
>> > object, we can't write down a stream natively in the IPC based file
>> format
>> > (for testing, or sharing the reproduction of an error). In particular,
>> the
>> > IPC format is based around the flatbuffer payloads instead of the Flight
>> > protobuf payloads. It might be nice to support an additional type of IPC
>> > file for stateful streams. If interested, it would be easy to integrate
>> > with the existing code using a different magic field in the footer
>> (such as
>> > 'FLGHT1', instead of 'ARROW1'). In addition to the offsets and sizes of
>> > payloads, it might be nice to indicate the type of payload (RecordBatch
>> vs
>> > DictionaryBatch, etc). We wouldn't have O(1) random access, but I think
>> in
>> > the "replay of a stream" scenario, one probably isn't looking for random
>> > access anyways.
>> >
>> > On Thu, Mar 18, 2021 at 9:07 PM Micah Kornfield
>> > wrote:
>> >
>> > > Hmm, I noticed this "The IPC file format doesn't support dictionary
>> > > replacements or deltas." I was under the impression we aimed to
>> support
>> > > dictionary deltas in the file format. If not we should remove "Delta
>> > > dictionaries are applied in the order they appear in the file footer."
>> > from
>> > > the specification.
>> > >
>> > > On Thu, Mar 18, 2021 at 8:48 AM Antoine Pitrou
>> > wrote:
>> > >
>> > > >
>> > > > It's a bit more configurable, but basically yes. See the IPC write
>> > > > options:
>> > > >
>> > >
>> >
>> https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/options.h#L73
>> > > >
>> > > > Regards
>> > > >
>> > > > Antoine.
>> > > >
>> > > >
>> > > > Le 18/03/2021 à 16:37, Jacob Quinn a écrit :
>> > > > > Ah, interesting. So to make sure I understand correctly, the C++
>> > write
>> >
Re: No replacement dictionaries supported in pyarrow?
The dictionary is not allowed to change throughout the file; which is ultimately OP's request. This is because all of the dictionary definition is in the footer of the file; which was clearly done to support random access of record batches. To quote the documentation: > We define a “file format” supporting random access that is build with the stream format. > > [...] > > In the file format, there is no requirement that dictionary keys should be defined in a DictionaryBatch before they are used in a RecordBatch, as long as the keys are defined somewhere in the file. Further more, it is invalid to have more than one non-delta dictionary batch per dictionary ID (i.e. dictionary replacement is not supported). Delta dictionaries are applied in the order they appear in the file footer. On Fri, Mar 19, 2021 at 6:37 AM Wes McKinney wrote: > I am also under the impression that the file format is supposed to support > deltas, but not replacements. Is this not implemented in C++? > > On Thu, Mar 18, 2021 at 9:57 PM Nate Bauernfeind < > nate.bauernfe...@gmail.com> > wrote: > > > If dictionary replacements were supported, then the IPC file format > > couldn't guarantee random access reads. > > > > Personally, I would like to support a stream-based file format that is a > > series of the Flight protobufs. In my extension of arrow flight, by > > stuffing our state-based data into the app_metadata field on the > FlightData > > object, we can't write down a stream natively in the IPC based file > format > > (for testing, or sharing the reproduction of an error). In particular, > the > > IPC format is based around the flatbuffer payloads instead of the Flight > > protobuf payloads. It might be nice to support an additional type of IPC > > file for stateful streams. If interested, it would be easy to integrate > > with the existing code using a different magic field in the footer (such > as > > 'FLGHT1', instead of 'ARROW1'). In addition to the offsets and sizes of > > payloads, it might be nice to indicate the type of payload (RecordBatch > vs > > DictionaryBatch, etc). We wouldn't have O(1) random access, but I think > in > > the "replay of a stream" scenario, one probably isn't looking for random > > access anyways. > > > > On Thu, Mar 18, 2021 at 9:07 PM Micah Kornfield > > wrote: > > > > > Hmm, I noticed this "The IPC file format doesn't support dictionary > > > replacements or deltas." I was under the impression we aimed to support > > > dictionary deltas in the file format. If not we should remove "Delta > > > dictionaries are applied in the order they appear in the file footer." > > from > > > the specification. > > > > > > On Thu, Mar 18, 2021 at 8:48 AM Antoine Pitrou > > wrote: > > > > > > > > > > > It's a bit more configurable, but basically yes. See the IPC write > > > > options: > > > > > > > > > > https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/options.h#L73 > > > > > > > > Regards > > > > > > > > Antoine. > > > > > > > > > > > > Le 18/03/2021 à 16:37, Jacob Quinn a écrit : > > > > > Ah, interesting. So to make sure I understand correctly, the C++ > > write > > > > > implementation will scan all "batches" and unify all dictionary > > values > > > > > before writing out the schema + dictionary messages? But only when > > > > writing > > > > > the file format? In the streaming case, it would still write > > > > > replacement/delta dictionary messages as needed. > > > > > > > > > > -Jacob > > > > > > > > > > On Thu, Mar 18, 2021 at 9:10 AM Neal Richardson < > > > > neal.p.richard...@gmail.com> > > > > > wrote: > > > > > > > > > >> Somewhat related issue: > > > > https://issues.apache.org/jira/browse/ARROW-10406 > > > > >> > > > > >> On Wed, Mar 17, 2021 at 11:22 PM Micah Kornfield < > > > emkornfi...@gmail.com > > > > > > > > > >> wrote: > > > > >> > > > > >>> BTW, this nuance always felt a little strange to me, but would > have > > > > >>> required adding additional information to the file format, to > > > > >> disambiguate > > > > >>> when exactly a dictionary was intended to be replaced. > > > > >>&g
Re: No replacement dictionaries supported in pyarrow?
If dictionary replacements were supported, then the IPC file format couldn't guarantee random access reads. Personally, I would like to support a stream-based file format that is a series of the Flight protobufs. In my extension of arrow flight, by stuffing our state-based data into the app_metadata field on the FlightData object, we can't write down a stream natively in the IPC based file format (for testing, or sharing the reproduction of an error). In particular, the IPC format is based around the flatbuffer payloads instead of the Flight protobuf payloads. It might be nice to support an additional type of IPC file for stateful streams. If interested, it would be easy to integrate with the existing code using a different magic field in the footer (such as 'FLGHT1', instead of 'ARROW1'). In addition to the offsets and sizes of payloads, it might be nice to indicate the type of payload (RecordBatch vs DictionaryBatch, etc). We wouldn't have O(1) random access, but I think in the "replay of a stream" scenario, one probably isn't looking for random access anyways. On Thu, Mar 18, 2021 at 9:07 PM Micah Kornfield wrote: > Hmm, I noticed this "The IPC file format doesn't support dictionary > replacements or deltas." I was under the impression we aimed to support > dictionary deltas in the file format. If not we should remove "Delta > dictionaries are applied in the order they appear in the file footer." from > the specification. > > On Thu, Mar 18, 2021 at 8:48 AM Antoine Pitrou wrote: > > > > > It's a bit more configurable, but basically yes. See the IPC write > > options: > > > https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/options.h#L73 > > > > Regards > > > > Antoine. > > > > > > Le 18/03/2021 à 16:37, Jacob Quinn a écrit : > > > Ah, interesting. So to make sure I understand correctly, the C++ write > > > implementation will scan all "batches" and unify all dictionary values > > > before writing out the schema + dictionary messages? But only when > > writing > > > the file format? In the streaming case, it would still write > > > replacement/delta dictionary messages as needed. > > > > > > -Jacob > > > > > > On Thu, Mar 18, 2021 at 9:10 AM Neal Richardson < > > neal.p.richard...@gmail.com> > > > wrote: > > > > > >> Somewhat related issue: > > https://issues.apache.org/jira/browse/ARROW-10406 > > >> > > >> On Wed, Mar 17, 2021 at 11:22 PM Micah Kornfield < > emkornfi...@gmail.com > > > > > >> wrote: > > >> > > >>> BTW, this nuance always felt a little strange to me, but would have > > >>> required adding additional information to the file format, to > > >> disambiguate > > >>> when exactly a dictionary was intended to be replaced. > > >>> > > >>> On Wed, Mar 17, 2021 at 11:19 PM Micah Kornfield < > > emkornfi...@gmail.com> > > >>> wrote: > > >>> > > Hi Jacob, > > There is nuance. The file format does not support dictionary > > >>> replacement, > > the specification [1] why that is currently the case. Only the > > "stream > > format" supports replacement (i.e. no magic number, only schema > > >> followed > > >>> by > > one or more dictionary/record-batch messages). > > > > -Micah > > > > [1] > > https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format > > > > On Wed, Mar 17, 2021 at 11:04 PM Jacob Quinn < > quinn.jac...@gmail.com> > > wrote: > > > > > Had an issue come up here: > > > > > >> > https://github.com/JuliaData/Arrow.jl/issues/129#issuecomment-777350450 > > >>> . > > > From the implementation status page, it says C++ supports > > replacement > > > dictionaries and that python tracks the C++ implementation. Is this > > >>> just a > > > pyarrow issue where it specifically doesn't support replacement > > > dictionaries? Or it's not "hooked in" properly? > > > > > > -Jacob > > > > > > > >>> > > >> > > > > > >
Re: Is Zulip still the preferred chat application for Arrow?
> I also found out today that there is an official ASF slack with multiple Arrow channels, but this is only open to people who already have an apache.org email address (committers / PMC). FYI, non committers / PMC members can join the slack using this link: https://s.apache.org/slack-invite On Wed, Mar 10, 2021 at 11:04 AM Antoine Pitrou wrote: > > Hi Andy, > > Le 10/03/2021 à 19:00, Andy Grove a écrit : > > We had a discussion on the Arrow Rust Sync call about the best place to > > co-ordinate on work. For example, quick questions like "is anyone working > > on ARROW-12345? should I pick this up?". > > > > I know that Ursa Lab hosts Zulip and I have used that in the past for > these > > types of discussion. > > > > I also found out today that there is an official ASF slack with multiple > > Arrow channels, but this is only open to people who already have an > > apache.org email address (committers / PMC). > > I didn't know that the ASF had an official Slack. I suppose that's the > Apache way of favoring open source software. > > I find Slack uncomfortable and annoying to deal with, and I wouldn't go > there. > > Regards > > Antoine. > --
Re: [Flight Extension] Request for Comments
>note that FlightData already has a separate app_metadata field That is an interesting point; are there any conventions on how to use the app_metadata compatibly without stepping on other ideas/projects doing the same? It would be convenient for the server to verify that the client is making the request that the server interprets. Do projects use a magic number prefix? Or possibly is there some sort of common header? I suspect that other projects may benefit from having the ability to publish incremental updates, too. So, I'm just curious if there is any pre-existing domain-knowledge in this respect. Nate On Mon, Mar 8, 2021 at 1:55 PM David Li wrote: > Hey - pretty much, I think. I'd just like to note that FlightData already > has a separate app_metadata field, for metadata on top of any Arrow-level > data, so you could ship the Barrage metadata alongside the first record > batch, without having to modify anything about the record batch itself, and > without having to define a new metadata header at the Arrow level - > everything could be implemented on top of the existing definitions. > > David > > On Sat, Mar 6, 2021, at 01:07, Nate Bauernfeind wrote: > > Eww. I didn't specify why I had two sets of record batches. Slightly > > revised: > > > > Are you suggesting this pattern of messages per incremental update? > > - FlightData with [the new] metadata header that includes > > added/removed/modified information, the number of add record batches, and > > the number of modified record batches. Noting that there could be more > than > > one record batch per added or modified to enable serializing more than > > 2^31-1 rows in a single update. Also noting that it would have an empty > > body (similar to Schema). > > - A set of FlightData record batches using the normal RecordBatch > > flatbuffer for added rows. > > - A set of FlightData record batches also using the normal RecordBatch > > flatbuffer for modified rows. > > > > On Fri, Mar 5, 2021 at 11:00 PM Nate Bauernfeind < > > natebauernfe...@deephaven.io> wrote: > > > > > > It seems that atomic application could also be something controlled > in > > > metadata (i.e. this is batch 1 or X)? > > > > > > You know what? This is actually a nicer solution than I am giving it > > > credit for. I've been trying to think about how to handle the > > > Integer.MAX_VALUE limit that arrow strongly suggests to maintain > > > compatibility with Java, while still respecting the need to apply an > update > > > atomically. > > > > > > Alright, yeah, I'm game with this approach. > > > > > > > Right - presumably this could go in the Flight metadata instead of > > > having to be inlined into the batch's metadata. > > > > > > I'm not sure I follow. These fields (addedRows, addedRowsIncluded, > > > removedRows, modifiedRows, and modifiedRowsIncluded) apply only to a > > > specific atomic incremental update. For a given update these are the > > > indices for the rows that were added/removed/modified -- and therefore > > > cannot be part of the "global" Flight metadata. > > > > > > Are you suggesting this pattern of messages per incremental update? > > > - FlightData with [the new] metadata header that includes > > > added/removed/modified information, the number of add record batches, > and > > > the number of modified record batches. Noting that there could be more > than > > > one record batch per added or modified to enable serializing more than > > > 2^31-1 rows in a single update. Also noting that it would have an empty > > > body (similar to Schema). > > > - A set of FlightData record batches using the normal RecordBatch > > > flatbuffer. > > > - A set of FlightData record batches also using the normal RecordBatch > > > flatbuffer. > > > > > > My biggest concern with this approach is that small updates are likely > > > going to have significant overhead. Maybe it won't matter, but it is > the > > > first thing thought that jumps out. We do typically coalesce updates > > > somewhere between 50ms and 1s depending on the sensitivity of the > listener; > > > so maybe that's enough to eliminate my concern. I might just need to > get > > > data/statistics to get a better feeling for this concern. > > > > > > Regarding the schema evolution idea: > > > What can I do to get started? Does it make sense to target the feature > as > > > a new field in the protobuf so that it can be used in contexts with > other >
Re: [Flight Extension] Request for Comments
Eww. I didn't specify why I had two sets of record batches. Slightly revised: Are you suggesting this pattern of messages per incremental update? - FlightData with [the new] metadata header that includes added/removed/modified information, the number of add record batches, and the number of modified record batches. Noting that there could be more than one record batch per added or modified to enable serializing more than 2^31-1 rows in a single update. Also noting that it would have an empty body (similar to Schema). - A set of FlightData record batches using the normal RecordBatch flatbuffer for added rows. - A set of FlightData record batches also using the normal RecordBatch flatbuffer for modified rows. On Fri, Mar 5, 2021 at 11:00 PM Nate Bauernfeind < natebauernfe...@deephaven.io> wrote: > > It seems that atomic application could also be something controlled in > metadata (i.e. this is batch 1 or X)? > > You know what? This is actually a nicer solution than I am giving it > credit for. I've been trying to think about how to handle the > Integer.MAX_VALUE limit that arrow strongly suggests to maintain > compatibility with Java, while still respecting the need to apply an update > atomically. > > Alright, yeah, I'm game with this approach. > > > Right - presumably this could go in the Flight metadata instead of > having to be inlined into the batch's metadata. > > I'm not sure I follow. These fields (addedRows, addedRowsIncluded, > removedRows, modifiedRows, and modifiedRowsIncluded) apply only to a > specific atomic incremental update. For a given update these are the > indices for the rows that were added/removed/modified -- and therefore > cannot be part of the "global" Flight metadata. > > Are you suggesting this pattern of messages per incremental update? > - FlightData with [the new] metadata header that includes > added/removed/modified information, the number of add record batches, and > the number of modified record batches. Noting that there could be more than > one record batch per added or modified to enable serializing more than > 2^31-1 rows in a single update. Also noting that it would have an empty > body (similar to Schema). > - A set of FlightData record batches using the normal RecordBatch > flatbuffer. > - A set of FlightData record batches also using the normal RecordBatch > flatbuffer. > > My biggest concern with this approach is that small updates are likely > going to have significant overhead. Maybe it won't matter, but it is the > first thing thought that jumps out. We do typically coalesce updates > somewhere between 50ms and 1s depending on the sensitivity of the listener; > so maybe that's enough to eliminate my concern. I might just need to get > data/statistics to get a better feeling for this concern. > > Regarding the schema evolution idea: > What can I do to get started? Does it make sense to target the feature as > a new field in the protobuf so that it can be used in contexts with other > header metadata types? Do you have time to riff on the format that will > apply to the other contexts? I believe all I would need is a bitset > identifying which columns are included, but if enabling/disabling features > is a nice-to-have then a bitset is going to be a bit weak. I can also, for > now, cheat and send empty field nodes and empty buffers for those columns > (but I am, already, slightly concerned with overhead). > > So, based on the feedback so far, I should be able to boil down the way I > integrate with Arrow to, more or less, a pair of flatbuffers. I'm going to > start riffing on these changes and see where I end up. Feel free to jump up > and down if I misunderstood you. > > > On Fri, Mar 5, 2021 at 9:23 PM Micah Kornfield > wrote: > >> > >> > And then having two sets of buffers, is the same as having two record >> > batches, albeit you need both sets to be delivered together, as noted. >> >> >> It seems that atomic application could also be something controlled in >> metadata (i.e. this is batch 1 or X)? >> >> The schema evolution question is interesting, it could be useful in other >> contexts as well. (e.g. switching dictionary encoding on/off). >> >> -Micah >> >> >> On Fri, Mar 5, 2021 at 11:42 AM David Li wrote: >> >> > (responses inline) >> > >> > On Thu, Mar 4, 2021, at 17:26, Nate Bauernfeind wrote: >> > > Regarding the BarrageRecordBatch: >> > > >> > > I have been concatenating them; it’s one batch with two sets of arrow >> > > payloads. They don’t have separate metadata headers; the update is to >> be >> > > applied atomically. I have only studied the Java Arr
Re: [Flight Extension] Request for Comments
> It seems that atomic application could also be something controlled in metadata (i.e. this is batch 1 or X)? You know what? This is actually a nicer solution than I am giving it credit for. I've been trying to think about how to handle the Integer.MAX_VALUE limit that arrow strongly suggests to maintain compatibility with Java, while still respecting the need to apply an update atomically. Alright, yeah, I'm game with this approach. > Right - presumably this could go in the Flight metadata instead of having to be inlined into the batch's metadata. I'm not sure I follow. These fields (addedRows, addedRowsIncluded, removedRows, modifiedRows, and modifiedRowsIncluded) apply only to a specific atomic incremental update. For a given update these are the indices for the rows that were added/removed/modified -- and therefore cannot be part of the "global" Flight metadata. Are you suggesting this pattern of messages per incremental update? - FlightData with [the new] metadata header that includes added/removed/modified information, the number of add record batches, and the number of modified record batches. Noting that there could be more than one record batch per added or modified to enable serializing more than 2^31-1 rows in a single update. Also noting that it would have an empty body (similar to Schema). - A set of FlightData record batches using the normal RecordBatch flatbuffer. - A set of FlightData record batches also using the normal RecordBatch flatbuffer. My biggest concern with this approach is that small updates are likely going to have significant overhead. Maybe it won't matter, but it is the first thing thought that jumps out. We do typically coalesce updates somewhere between 50ms and 1s depending on the sensitivity of the listener; so maybe that's enough to eliminate my concern. I might just need to get data/statistics to get a better feeling for this concern. Regarding the schema evolution idea: What can I do to get started? Does it make sense to target the feature as a new field in the protobuf so that it can be used in contexts with other header metadata types? Do you have time to riff on the format that will apply to the other contexts? I believe all I would need is a bitset identifying which columns are included, but if enabling/disabling features is a nice-to-have then a bitset is going to be a bit weak. I can also, for now, cheat and send empty field nodes and empty buffers for those columns (but I am, already, slightly concerned with overhead). So, based on the feedback so far, I should be able to boil down the way I integrate with Arrow to, more or less, a pair of flatbuffers. I'm going to start riffing on these changes and see where I end up. Feel free to jump up and down if I misunderstood you. On Fri, Mar 5, 2021 at 9:23 PM Micah Kornfield wrote: > > > > And then having two sets of buffers, is the same as having two record > > batches, albeit you need both sets to be delivered together, as noted. > > > It seems that atomic application could also be something controlled in > metadata (i.e. this is batch 1 or X)? > > The schema evolution question is interesting, it could be useful in other > contexts as well. (e.g. switching dictionary encoding on/off). > > -Micah > > > On Fri, Mar 5, 2021 at 11:42 AM David Li wrote: > > > (responses inline) > > > > On Thu, Mar 4, 2021, at 17:26, Nate Bauernfeind wrote: > > > Regarding the BarrageRecordBatch: > > > > > > I have been concatenating them; it’s one batch with two sets of arrow > > > payloads. They don’t have separate metadata headers; the update is to > be > > > applied atomically. I have only studied the Java Arrow Flight > > > implementation, and I believe it is usable maybe with some minor > changes. > > > The piece of code in Flight that does the deserialization takes two > > > parallel lists/iterators, a `Buffer` list (these describe the length > of a > > > section of the body payload) and a `FieldNode` list (these describe num > > > rows and null_count). Each field node is 2-3 buffers depending on > schema > > > type. Buffers are allowed to have length of 0, to omit their payloads; > > > this, for example, is how you omit the validity buffer when null_count > is > > > zero. > > > > > > The proposed barrage payload keeps this structural pattern (list of > > buffer, > > > list of field node) with the following modifications: > > > - we only include field nodes / buffers for subscribed columns > > > - the first set of field nodes are for added rows; these may be omitted > > if > > > there are no added rows included in the update > > > - the second set of field nodes are for modified rows; we omit columns > > that &g
Re: [Flight Extension] Request for Comments
Regarding the BarrageRecordBatch: I have been concatenating them; it’s one batch with two sets of arrow payloads. They don’t have separate metadata headers; the update is to be applied atomically. I have only studied the Java Arrow Flight implementation, and I believe it is usable maybe with some minor changes. The piece of code in Flight that does the deserialization takes two parallel lists/iterators, a `Buffer` list (these describe the length of a section of the body payload) and a `FieldNode` list (these describe num rows and null_count). Each field node is 2-3 buffers depending on schema type. Buffers are allowed to have length of 0, to omit their payloads; this, for example, is how you omit the validity buffer when null_count is zero. The proposed barrage payload keeps this structural pattern (list of buffer, list of field node) with the following modifications: - we only include field nodes / buffers for subscribed columns - the first set of field nodes are for added rows; these may be omitted if there are no added rows included in the update - the second set of field nodes are for modified rows; we omit columns that have no modifications included in the update I believe the only thing that is missing is the ability to control the field types to be deserialized (like a third list/iterator parallel to field nodes and buffers). Note that the BarrageRecordBatch.addedRowsIncluded, BarrageFieldNode.addedRows, BarrageFieldNode.modifiedRows and BarrageFieldNode.includedRows (all part of the flatbuffer metadata) are intended to be used by code one layer of abstraction higher than that actual wire-format parser. The parser doesn't really need them except to know which columns to expect in the payload. Technically, we could encode the field nodes / buffers as empty, too (but why be wasteful if this information is already encoded?). Regarding Browser Flight Support: Was this company FactSet by chance? (I saw they are mentioned in the JS thread that recently was bumped on the dev list.) I looked at the ticket and wanted to comment how we are handling bi-directional streams for our web-ui. We use ArrowFlight's concept of Ticket to allow a client to create and identify temporary state (new tables / views / REPL sessions / etc). Any bidirectional stream we support also has a server-streaming only variant with the ability for the client to attach a Ticket to reference/identify that stream. The client may then send a message, out-of-band, to the Ticket. They are sequenced by the client (since gRPC doesn't guarantee ordered delivery) and delivered to the piece of code controlling that server-stream. It does require that the server be a bit stateful; but it works =). On Thu, Mar 4, 2021 at 6:58 AM David Li wrote: > Re: the multiple batches, that makes sense. In that case, depending on how > exactly the two record batches are laid out, I'd suggest considering a > Union of Struct columns (where a Struct is essentially interchangeable with > a record batch or table) - that would let you encode two distinct record > batches inside the same physical batch. Or if the two batches have > identical schemas, you could just concatenate them and include indices in > your metadata. > > As for browser Flight support - there's an existing ticket: > https://issues.apache.org/jira/browse/ARROW-9860 > > I was sure I had seen another organization talking about browser support > recently, but now I can't find them. I'll update here if I do figure it out. > > Best, > David > > On Wed, Mar 3, 2021, at 21:00, Nate Bauernfeind wrote: > > > if each payload has two batches with different purposes [...] > > > > The purposes of the payloads are slightly different, however they are > > intended to be applied atomically. If there are guarantees by the table > > operation generating the updates then those guarantees are only valid on > > each boundary of applying the update to your local state. In a sense, one > > is relatively useless without the other. Record batches fit well in > > map-reduce paradigms / algorithms, but what we have is stateful to > > enable/support incremental updates. For example, sorting a flight of data > > is best done map-reduce-style and requires one to re-sort the entire data > > set when it changes. Our approach focuses on producing incremental > updates > > which are used to manipulate your existing client state using a much > > smaller footprint (in both time and space). You can imagine, in the sort > > scenario, if you evaluate the table after adding rows but before > modifying > > existing rows your table won’t be sorted between the two updates. The > > client would then need to wait until it receives the pair of > RecordBatches > > anyways, so it seems more natural to deliver them together. > > > > > As a side note - is sa
Re: [Flight Extension] Request for Comments
t; > Best, > David > > On Wed, Mar 3, 2021, at 18:05, Nate Bauernfeind wrote: > > Thanks for the interest =). > > > > > However, if I understand right, you're sending data without a fixed > > schema [...] > > > > The dataset does have a known schema ahead of time, which is similar to > > Flight. However, as you point out, the subscription can change which > > columns it is interested in without re-acquiring data for columns it was > > already subscribed to. This is mostly for convenience. We use it > primarily > > to limit which columns are sent to our user interface until the user > > scrolls them into view. > > > > The enhancement of the RecordBatch here, aside from the additional > > metadata, is only in that the payload has two sets of RecordBatch > payloads. > > The first payload is for added rows, every added row must send data for > > each column subscribed; based on the subscribed columns this is otherwise > > fixed width (in the number of columns / buffers). The second payload is > for > > modified rows. Here we only send the columns that have rows that are > > modified. Aside from this difference, I have been aiming to be compatible > > enough to be able to reuse the payload parsing that is already written > for > > Arrow. > > > > > I don't quite see why it couldn't be carried as metadata on the side > of a > > record batch, instead of having to duplicate the record batch structure > > [...] > > > > Whoa, this is a good point. I have iterated on this a few times to get it > > closer to Arrow's setup and did not realize that 'BarrageData' is now > > officially identical to `FlightData`. This is an instance of being too > > close to the project and forgetting to step back once in a while. > > > > > Flight already has a bidirectional streaming endpoint, DoExchange, that > > allows arbitrary payloads (with mixed metadata/data or only one of the > > two), which seems like it should be able to cover the SubscriptionRequest > > endpoint. > > > > This is exactly the kind of feedback I'm looking for! I wasn't seeing the > > solution where the client-side stream doesn't actually need payload and > > that the subscription changes can be described with another flatbuffer > > metadata type. I like that. > > > > Thanks David! > > Nate > > > > On Wed, Mar 3, 2021 at 3:28 PM David Li wrote: > > > > > Hey Nate, > > > > > > Thanks for sharing this & for the detailed docs and writeup. I think > your > > > use case is interesting, but I'd like to clarify a few things. > > > > > > I would say Arrow Flight doesn't try to impose a particular model, but > I > > > agree that Barrage does things that aren't easily doable with Flight. > > > Flight does name concepts in a way that suggests how to apply it to > > > something that looks like a database, but you can mostly think of > Flight as > > > an efficient way to transfer Arrow data over the network upon which > you can > > > layer further semantics. > > > > > > However, if I understand right, you're sending data without a fixed > > > schema, in the sense that each BarrageRecordBatch may have only a > subset of > > > the columns declared up front, or may carry new columns? I think this > is > > > the main thing you can't easily do currently, as Flight (and Arrow IPC > in > > > general) assumes a fixed schema (and expects all columns in a batch to > have > > > the same length). > > > > > > Otherwise, the encoding for identifying rows and changes is > interesting, > > > but I don't quite see why it couldn't be carried as metadata on the > side of > > > a record batch, instead of having to duplicate the record batch > structure, > > > except for the aforementioned schema issue. And in that case it might > be > > > better to work out the schema evolution issue & any ergonomic issues > with > > > Flight's existing metadata fields/API that would prevent you from using > > > them, as that way you (and we!) don't have to fully duplicate one of > > > Arrow's format definitions. Similarly, Flight already has a > bidirectional > > > streaming endpoint, DoExchange, that allows arbitrary payloads (with > mixed > > > metadata/data or only one of the two), which seems like it should be > able > > > to cover the SubscriptionRequest endpoint. > > > > > > Best, > > > David > > > > > > On Wed, Mar 3, 2021, at
Re: [Flight Extension] Request for Comments
Thanks for the interest =). > However, if I understand right, you're sending data without a fixed schema [...] The dataset does have a known schema ahead of time, which is similar to Flight. However, as you point out, the subscription can change which columns it is interested in without re-acquiring data for columns it was already subscribed to. This is mostly for convenience. We use it primarily to limit which columns are sent to our user interface until the user scrolls them into view. The enhancement of the RecordBatch here, aside from the additional metadata, is only in that the payload has two sets of RecordBatch payloads. The first payload is for added rows, every added row must send data for each column subscribed; based on the subscribed columns this is otherwise fixed width (in the number of columns / buffers). The second payload is for modified rows. Here we only send the columns that have rows that are modified. Aside from this difference, I have been aiming to be compatible enough to be able to reuse the payload parsing that is already written for Arrow. > I don't quite see why it couldn't be carried as metadata on the side of a record batch, instead of having to duplicate the record batch structure [...] Whoa, this is a good point. I have iterated on this a few times to get it closer to Arrow's setup and did not realize that 'BarrageData' is now officially identical to `FlightData`. This is an instance of being too close to the project and forgetting to step back once in a while. > Flight already has a bidirectional streaming endpoint, DoExchange, that allows arbitrary payloads (with mixed metadata/data or only one of the two), which seems like it should be able to cover the SubscriptionRequest endpoint. This is exactly the kind of feedback I'm looking for! I wasn't seeing the solution where the client-side stream doesn't actually need payload and that the subscription changes can be described with another flatbuffer metadata type. I like that. Thanks David! Nate On Wed, Mar 3, 2021 at 3:28 PM David Li wrote: > Hey Nate, > > Thanks for sharing this & for the detailed docs and writeup. I think your > use case is interesting, but I'd like to clarify a few things. > > I would say Arrow Flight doesn't try to impose a particular model, but I > agree that Barrage does things that aren't easily doable with Flight. > Flight does name concepts in a way that suggests how to apply it to > something that looks like a database, but you can mostly think of Flight as > an efficient way to transfer Arrow data over the network upon which you can > layer further semantics. > > However, if I understand right, you're sending data without a fixed > schema, in the sense that each BarrageRecordBatch may have only a subset of > the columns declared up front, or may carry new columns? I think this is > the main thing you can't easily do currently, as Flight (and Arrow IPC in > general) assumes a fixed schema (and expects all columns in a batch to have > the same length). > > Otherwise, the encoding for identifying rows and changes is interesting, > but I don't quite see why it couldn't be carried as metadata on the side of > a record batch, instead of having to duplicate the record batch structure, > except for the aforementioned schema issue. And in that case it might be > better to work out the schema evolution issue & any ergonomic issues with > Flight's existing metadata fields/API that would prevent you from using > them, as that way you (and we!) don't have to fully duplicate one of > Arrow's format definitions. Similarly, Flight already has a bidirectional > streaming endpoint, DoExchange, that allows arbitrary payloads (with mixed > metadata/data or only one of the two), which seems like it should be able > to cover the SubscriptionRequest endpoint. > > Best, > David > > On Wed, Mar 3, 2021, at 16:08, Nate Bauernfeind wrote: > > Hello, > > > > My colleagues at Deephaven Data Labs and I have been addressing problems > at > > the intersection of data-driven applications, data science, and updating > > (/ticking) data for some years. > > > > Deephaven has a query engine that supports updating tabular data via a > > protocol that communicates precise changes about datasets, such as 1) > which > > rows were removed, 2) which rows were added, 3) which rows were modified > > (and for which columns). We are inspired by Arrow and would like to > adopt a > > version of this protocol that adheres to goals similar to Arrow and Arrow > > Flight. > > > > Out of the box, Arrow Flight is insufficient to represent such a stream > of > > changes. For example, because you cannot identify a particular row within > > an Arrow Flight, you cannot indicate which rows were removed or m
[Flight Extension] Request for Comments
Hello, My colleagues at Deephaven Data Labs and I have been addressing problems at the intersection of data-driven applications, data science, and updating (/ticking) data for some years. Deephaven has a query engine that supports updating tabular data via a protocol that communicates precise changes about datasets, such as 1) which rows were removed, 2) which rows were added, 3) which rows were modified (and for which columns). We are inspired by Arrow and would like to adopt a version of this protocol that adheres to goals similar to Arrow and Arrow Flight. Out of the box, Arrow Flight is insufficient to represent such a stream of changes. For example, because you cannot identify a particular row within an Arrow Flight, you cannot indicate which rows were removed or modified. The project integrates with Arrow Flight at the header-metadata level. We have preliminarily named the project Barrage as in a "barrage of arrows" which plays in the same "namespace" as a "flight of arrows." We built this as part of an initiative to modernize and open up our table IPC mechanisms. This is part of a larger open source effort which will become more visible in the next month or so once we've finished the work necessary to share our core software components, including a unified static and real time query engine complete with data visualization tools, a REPL experience, Jupyter integration, and more. I would like to find out: - if we have understood the primary goals of Arrow, and are honoring them as closely as possible - if there are other projects that might benefit from sharing this extension of Arrow Flight - if there are any gaps that are best addressed early on to maximize future compatibility A great place to digest the concepts that differ from Arrow Flight are here: https://deephaven.github.io/barrage/Concepts.html The proposed protocol can be perused here: https://github.com/deephaven/barrage Internally, we already have a java server and java client implemented as a working proof of concept for our use case. I really look forward to your feedback; thank you! Nate Bauernfeind Deephaven Data Labs - https://deephaven.io/ --
