> I believe several array implementations (e.g., numpy, R) are able to broadcast/recycle a length-1 array. Run-end-encoding is also an option that would make that broadcast explicit without expanding the scalar.
Some libraries behave this way, i.e. Polars, but others like Pandas and cuDF only broadcast up dimensions. I.E. scalars can be broadcast across columns or dataframes, columns can be broadcast across dataframes, but length 1 columns do not broadcast across columns where trying to add say a length 5 and length 1 column isn't valid but adding a length 5 column and a scalar is. Additionally, it differentiates between operations that are guaranteed to return a scalar, i.e. something like a reduction of `sum()` versus operations that can return a length 1 column depending on the data, i.e. `unique()`. > For UDFs: UDFs are a system-specific interface. Presumably, that interface can encode whether an Arrow array is meant to represent a column or scalar (or record batch or ...). Again, because Arrow doesn't define scalars (for now...) or UDFs, the UDF interface needs to layer its own semantics on top of Arrow. > > In other words, I don't think the C Data Interface was meant to be something where you can expect to _only_ pass the ArrowDeviceArray around and have it encode all the semantics for a particular system, right? The UDF example is something where the engine would pass an ArrowDeviceArray plus additional context. There's a growing trend in execution engines supporting UDFs of Arrow in and Arrow out, DuckDB, PySpark, DataFusion, etc. Many of them have different options of passing in RecordBatches vs Arrays where they currently rely on the Arrow library containers in order to differentiate them. Additionally, libcudf has some generic functions that currently use Arrow C++ containers ( https://docs.rapids.ai/api/cudf/stable/libcudf_docs/api_docs/interop_arrow/) for differentiating between RecordBatches, Arrays, and Scalars which could be moved to using the C Data Interfaces, Polars has similar ( https://docs.pola.rs/py-polars/html/reference/api/polars.from_arrow.html) that currently uses PyArrow containers, and you could imagine other DataFrame libraries having similar. Ultimately, there's a desire to be able to move Arrow data between different libraries, applications, frameworks, etc. and given Arrow implementations like C++, Rust, and Go have containers for RecordBatches, Arrays, and Scalars respectively, things have been built around and differentiated around the concepts. Maybe trying to differentiate this information at runtime isn't the correct path, but I believe there's a demonstrated desire for being able to differentiate things in a library agnostic way. On Tue, Apr 23, 2024 at 8:37 PM David Li <lidav...@apache.org> wrote: > For scalars: Arrow doesn't define scalars. They're an implementation > concept. (They may be a *useful* one, but if we want to define them more > generally, that's a separate discussion.) > > For UDFs: UDFs are a system-specific interface. Presumably, that interface > can encode whether an Arrow array is meant to represent a column or scalar > (or record batch or ...). Again, because Arrow doesn't define scalars (for > now...) or UDFs, the UDF interface needs to layer its own semantics on top > of Arrow. > > In other words, I don't think the C Data Interface was meant to be > something where you can expect to _only_ pass the ArrowDeviceArray around > and have it encode all the semantics for a particular system, right? The > UDF example is something where the engine would pass an ArrowDeviceArray > plus additional context. > > > since we can't determine which a given ArrowArray is on its own. In the > > libcudf situation, it came up with what happens if you pass a non-struct > > column to the from_arrow_device method which returns a cudf::table? > Should > > it error, or should it create a table with a single column? > > Presumably it should just error? I can see this being ambiguous if there > were an API that dynamically returned either a table or a column based on > the input shape (where before it would be less ambiguous since you'd > explicitly pass pa.RecordBatch or pa.Array, and now it would be ambiguous > since you only pass ArrowDeviceArray). But it doesn't sound like that's the > case? > > On Tue, Apr 23, 2024, at 11:15, Weston Pace wrote: > > I tend to agree with Dewey. Using run-end-encoding to represent a scalar > > is clever and would keep the c data interface more compact. Also, a > struct > > array is a superset of a record batch (assuming the metadata is kept in > the > > schema). Consumers should always be able to deserialize into a struct > > array and then downcast to a record batch if that is what they want to do > > (raising an error if there happen to be nulls). > > > >> Depending on the function in question, it could be valid to pass a > struct > >> column vs a record batch with different results. > > > > Are there any concrete examples where this is the case? The closest > > example I can think of is something like the `drop_nulls` function, > which, > > given a record batch, would choose to drop rows where any column is null > > and, given an array, only drops rows where the top-level struct is null. > > However, it might be clearer to just give the two functions different > names > > anyways. > > > > On Mon, Apr 22, 2024 at 1:01 PM Dewey Dunnington > > <de...@voltrondata.com.invalid> wrote: > > > >> Thank you for the background! > >> > >> I still wonder if these distinctions are the responsibility of the > >> ArrowSchema to communicate (although perhaps links to the specific > >> discussions would help highlight use-cases that I am not envisioning). > >> I think these distinctions are definitely important in the contexts > >> you mentioned; however, I am not sure that the FFI layer is going to > >> be helpful. > >> > >> > In the libcudf situation, it came up with what happens if you pass a > >> non-struct > >> > column to the from_arrow_device method which returns a cudf::table? > >> Should > >> > it error, or should it create a table with a single column? > >> > >> I suppose that I would have expected two functions (one to create a > >> table and one to create a column). As a consumer I can't envision a > >> situation where I would want to import an ArrowDeviceArray but where I > >> would want some piece of run-time information to decide what the > >> return type of the function would be? (With apologies if I am missing > >> a piece of the discussion). > >> > >> > If A and B have different lengths, this is invalid > >> > >> I believe several array implementations (e.g., numpy, R) are able to > >> broadcast/recycle a length-1 array. Run-end-encoding is also an option > >> that would make that broadcast explicit without expanding the scalar. > >> > >> > Depending on the function in question, it could be valid to pass a > >> struct column vs a record batch with different results. > >> > >> If this is an important distinction for an FFI signature of a UDF, > >> there would probably be a struct definition for the UDF where there > >> would be an opportunity to make this distinction (and perhaps others > >> that are relevant) without loading this concept onto the existing > >> structs. > >> > >> > If no flags are set, then the behavior shouldn't change > >> > from what it is now. If the ARROW_FLAG_RECORD_BATCH flag is set, then > it > >> > should error unless calling ImportRecordBatch. > >> > >> I am not sure I would have expected that (since a struct array has an > >> unambiguous interpretation as a record batch and as a user I've very > >> explicitly decided that I want one, since I'm using that function). > >> > >> In the other direction, I am not sure a producer would be able to set > >> these flags without breaking backwards compatibility with earlier > >> producers that did not set them (since earlier threads have suggested > >> that it is good practice to error when an unsupported flag is > >> encountered). > >> > >> On Sun, Apr 21, 2024 at 6:16 PM Matt Topol <zotthewiz...@gmail.com> > wrote: > >> > > >> > First, I forgot a flag in my examples. There should also be an > >> > ARROW_FLAG_SCALAR too! > >> > > >> > The motivation for this distinction came up from discussions during > >> adding > >> > support for ArrowDeviceArray to libcudf in order to better indicate > the > >> > difference between a cudf::table and a cudf::column which are handled > >> quite > >> > differently. This also relates to the fact that we currently need > >> external > >> > context like the explicit ImportArray() and ImportRecordBatch() > functions > >> > since we can't determine which a given ArrowArray is on its own. In > the > >> > libcudf situation, it came up with what happens if you pass a > non-struct > >> > column to the from_arrow_device method which returns a cudf::table? > >> Should > >> > it error, or should it create a table with a single column? > >> > > >> > The other motivation for this distinction is with UDFs in an engine > that > >> > uses the C data interface. When dealing with queries and engines, it > >> > becomes important to be able to distinguish between a record batch, a > >> > column and a scalar. For example, take the expression A + B: > >> > > >> > If A and B have different lengths, this is invalid..... unless one of > >> them > >> > is a Scalar. This is because Scalars are broadcastable, columns are > not. > >> > > >> > Depending on the function in question, it could be valid to pass a > struct > >> > column vs a record batch with different results. It also resolves some > >> > ambiguity for UDFs and processing. For instance, given a single > >> ArrowArray > >> > of length 1, which is a struct: Is that a Struct Column? A Record > Batch? > >> or > >> > is it a scalar? There's no way to know what the producer's intention > was > >> or > >> > the context without these flags or having to side-channel the > information > >> > somehow. > >> > > >> > > It seems like it may cause some ambiguous > >> > situations...should C++'s ImportArray() error, for example, if the > >> > schema has a ARROW_FLAG_RECORD_BATCH flag? > >> > > >> > I would argue yes. If no flags are set, then the behavior shouldn't > >> change > >> > from what it is now. If the ARROW_FLAG_RECORD_BATCH flag is set, then > it > >> > should error unless calling ImportRecordBatch. It allows the producer > to > >> > provide context as to the source and intention of the structure of the > >> data. > >> > > >> > --Matt > >> > > >> > On Fri, Apr 19, 2024 at 8:24 PM Dewey Dunnington > >> > <de...@voltrondata.com.invalid> wrote: > >> > > >> > > Thanks for bringing this up! > >> > > > >> > > Could you share the motivation where this distinction is important > in > >> > > the context of transfer across the C data interface? The "struct == > >> > > record batch" concept has always made sense to me because in R, a > >> > > data.frame can have a column that is also a data.frame and there is > no > >> > > distinction between the two. It seems like it may cause some > ambiguous > >> > > situations...should C++'s ImportArray() error, for example, if the > >> > > schema has a ARROW_FLAG_RECORD_BATCH flag? > >> > > > >> > > Cheers, > >> > > > >> > > -dewey > >> > > > >> > > On Fri, Apr 19, 2024 at 6:34 PM Matt Topol <zotthewiz...@gmail.com> > >> wrote: > >> > > > > >> > > > Hey everyone, > >> > > > > >> > > > With some of the other developments surrounding libraries adopting > >> the > >> > > > Arrow C Data interfaces, there's been a consistent question about > >> > > handling > >> > > > tables (record batch) vs columns vs scalars. > >> > > > > >> > > > Right now, a Record Batch is sent through the C interface as a > struct > >> > > > column whose children are the individual columns of the batch and > a > >> > > Scalar > >> > > > would be sent through as just an array of length 1. Applications > >> would > >> > > have > >> > > > to create their own contextual way of indicating whether the Array > >> being > >> > > > passed should be interpreted as just a single array/column or > should > >> be > >> > > > treated as a full table/record batch. > >> > > > > >> > > > Rather than introducing new members or otherwise complicating the > >> > > structs, > >> > > > I wanted to gauge how people felt about introducing new flags for > the > >> > > > ArrowSchema object. > >> > > > > >> > > > Right now, we only have 3 defined flags: > >> > > > > >> > > > ARROW_FLAG_DICTIONARY_ORDERED > >> > > > ARROW_FLAG_NULLABLE > >> > > > ARROW_FLAG_MAP_KEYS_SORTED > >> > > > > >> > > > The flags member of the struct is an int64, so we have another 61 > >> bits to > >> > > > play with! If no one has any strong objections, I wanted to > propose > >> > > adding > >> > > > at least 2 new flags: > >> > > > > >> > > > ARROW_FLAG_RECORD_BATCH > >> > > > ARROW_FLAG_SINGLE_COLUMN > >> > > > > >> > > > If neither flag is set, then it is contextual as to whether it > >> should be > >> > > > expected that the corresponding data is a table or a single > column. > >> If > >> > > > ARROW_FLAG_RECORD_BATCH is set, then the corresponding data MUST > be a > >> > > > struct array and should be interpreted as a record batch by any > >> consumers > >> > > > (erroring otherwise). If ARROW_FLAG_SINGLE_COLUMN is set, then the > >> > > > corresponding ArrowArray should be interpreted and utilized as a > >> single > >> > > > array/column regardless of its type. > >> > > > > >> > > > This provides a standardized way for producers of Arrow data to > >> indicate > >> > > in > >> > > > the schema to consumers how the data they produced should be used > >> (as a > >> > > > table or column) rather than forcing everyone to come up with > their > >> own > >> > > > contextualized way of handling things (extra arguments, > differently > >> named > >> > > > functions for RecordBatch / Array, etc.). > >> > > > > >> > > > If there's no objections to this, I'll take a pass at implementing > >> these > >> > > > flags in C++ and Go to put up a PR and make a Vote thread. I just > >> wanted > >> > > to > >> > > > see what others on the mailing list thought before I go ahead and > put > >> > > > effort into this. > >> > > > > >> > > > Thanks everyone! Take care! > >> > > > > >> > > > --Matt > >> > > > >> >
