The Python Substrait package[1] is on PyPi[2] and currently has python wrappers for the Substrait protobuf objects. I think this will be a great opportunity to identify helper features that users of this protocol would like to see. I'll be keeping an eye out as this develops, but also feel free to file feature requests in the project!
[1]https://github.com/substrait-io/substrait-python [2]https://pypi.org/project/substrait/ On Thu, Aug 31, 2023 at 10:05 PM Will Jones <will.jones...@gmail.com> wrote: > Hello Arrow devs, > > We discussed this further in the Arrow community call on 2023-08-30 [1], > and concluded we should create an entirely new protocol that uses Substrait > expressions. I have created an issue [2] to track this and will start a PR > soon. > > It does look like we might block this on creating a PyCapsule based > protocol for arrays, schemas, and streams. That is tracked here [3]. > Hopefully that isn't too ambitious :) > > Best, > > Will Jones > > > [1] > > https://docs.google.com/document/d/1xrji8fc6_24TVmKiHJB4ECX1Zy2sy2eRbBjpVJMnPmk/edit > [2] https://github.com/apache/arrow/issues/37504 > [3] https://github.com/apache/arrow/issues/35531 > > > On Tue, Aug 29, 2023 at 2:59 PM Ian Cook <ianmc...@apache.org> wrote: > > > An update about this: > > > > Weston's PR https://github.com/apache/arrow/pull/34834/ merged last > > week. This makes it possible to convert PyArrow expressions to/from > > Substrait expressions. > > > > As Fokko previously noted, the PR does not change the PyArrow Dataset > > interface at all. It simply enables a Substrait expression to be > > converted to a PyArrow expression, which can then be used to > > filter/project a Dataset. > > > > There is a basic example here demonstrating this: > > https://gist.github.com/ianmcook/f70fc185d29ae97bdf85ffe0378c68e0 > > > > We might now consider whether to build upon this to create a Dataset > > protocol that is independent of the PyArrow Expression implementation > > and that could interoperate across languages. > > > > Ian > > > > On Mon, Jul 3, 2023 at 5:48 PM Will Jones <will.jones...@gmail.com> > wrote: > > > > > > Hello, > > > > > > After thinking about it, I think I understand the approach David Li and > > Ian > > > are suggesting with respect to expressions. There will be some > arguments > > > that only PyArrow's own datasets support, but that aren't in the > generic > > > protocol. Passing > > > PyArrow expressions to the filters argument should be considered one of > > > those. DuckDB and others are currently passing them down, so they > aren't > > > yet using the protocol properly. But once we add support in the > protocol > > > for passing filters via Substrait expressions, we'll move DuckDB and > > others > > > over to be fully compliant with the protocol. > > > > > > It's a bit of an awkward temporary state for now, but so would having > > > PyArrow expressions in the protocol just to be deprecated in a few > > months. > > > One caveat is that we'll need to provide DuckDB and other consumers > with > > a > > > way to tell whether the dataset supports passing filters as Substrait > > > expression or PyArrow ones, since I doubt they'll want to lose support > > for > > > integrating with older PyArrow versions. > > > > > > I've removed filters from the protocol for now, with the intention of > > > bringing them back as soon as we can get Substrait support. I think we > > can > > > do this in the 14.0.0 release. > > > > > > Best, > > > > > > Will Jones > > > > > > > > > On Mon, Jul 3, 2023 at 7:45 AM Fokko Driesprong <fo...@apache.org> > > wrote: > > > > > > > Hey everyone, > > > > > > > > Chiming in here from the PyIceberg side. I would love to see the > > protocol > > > > as proposed in the PR. I did a small test > > > > < > > https://github.com/apache/arrow/pull/35568#pullrequestreview-1480259722 > >, > > > > and it seems to be quite straightforward to implement and it brings a > > lot > > > > of potential. Unsurprisingly, I leaning toward the first option: > > > > > > > > 1. We keep PyArrow expressions in the API initially, but once we have > > > > > Substrait-based alternatives we deprecate the PyArrow expression > > support. > > > > > This is what I intended with the current design, and I think it > > provides > > > > > the most obvious migration paths for existing producers and > > consumers. > > > > > > > > > > > > Let me give my vision on some of the concerns raised. > > > > > > > > Will, I see that you've already addressed this issue to some extent > in > > > > > your proposal. For example, you mention that we should initially > > > > > define this protocol to include only a minimal subset of the > Dataset > > > > > API. I agree, but I think there are some loose ends we should be > > > > > careful to tie up. I strongly agree with the comments made by > David, > > > > > Weston, and Dewey arguing that we should avoid any use of PyArrow > > > > > expressions in this API. Expressions are an implementation detail > of > > > > > PyArrow, not a part of the Arrow standard. It would be much safer > for > > > > > the initial version of this protocol to not define *any* > > > > > methods/arguments that take expressions. This will allow us to take > > > > > some more time to finish up the Substrait expression implementation > > > > > work that is underway [7][8], then introduce Substrait-based > > > > > expressions in a latter version of this protocol. This approach > will > > > > > better position this protocol to be implemented in other languages > > > > > besides Python. > > > > > > > > > > > > I'm confused here. Looking at GH-33985 > > > > <https://github.com/apache/arrow/pull/34834/files> I don't see any > new > > > > primitives being introduced for composing an expression. As I > > understand > > > > it, in PyArrow the expression as it exists today will continue to > > exist. In > > > > the case of inter-process communication, it goes to Substrait, and > > then it > > > > gets de-serialized in the native expression construct (In PyIceberg, > a > > > > BoundPredicate). I would say that the protocol and substrait are > > > > complementary. > > > > > > > > Another concern I have is that we have not fully explained why we > want > > > > > to use Dataset instead of RecordBatchReader [9] as the basis of > this > > > > > protocol. I would like to see an explanation of why > RecordBatchReader > > > > > is not sufficient for this. RecordBatchReader seems like another > > > > > possible way to represent "unmaterialized dataframes" and there are > > > > > some parallels between RecordBatch/RecordBatchReader and > > > > > Fragment/Dataset. We should help developers and users understand > why > > > > > Arrow needs both of these. > > > > > > > > > > > > Just to clarify, I think there are different use cases. For example, > > Lance > > > > provides its own readers, but PyIceberg does not have any intent to > > provide > > > > its own Parquet readers. Iceberg will generate the list of files that > > need > > > > to be read, and do the filtering/projection/deletes/etc. This would > > make > > > > the Dataset a better choice than the RecordBatchReader. > > > > > > > > That wouldn't remove the feature from DuckDB, would it? It would just > > mean > > > > > that we recognize that PyArrow expressions don't have well-defined > > > > > semantics that we are committing to at this time. As long as we > have > > > > > `**kwargs` everywhere, we can in the future introduce a > > > > > `substrait_filter_expression` or similar argument, while allowing > > current > > > > > implementors to handle `filter` if possible. (As a compromise, we > > could > > > > > reserve `filter` and existing arguments and note that PyArrow > > Expression > > > > > semantics are subject to change without notice?) > > > > > > > > > > > > I think we can even re-use the existing filter argument. The > signature > > > > would evolve from pc.Expression to Union[pc.Expression, > > > > pas.BoundExpressions]. In the case we get an expression, we'll > convert > > it > > > > to substrait. > > > > > > > > Concluding, I think we can do things in parallel, and I don't think > > they > > > > are conflicting. I'm happy to contribute to the PyArrow side to make > > this > > > > happen. > > > > > > > > Kind regards, > > > > Fokko > > > > > > > > Op wo 28 jun 2023 om 22:47 schreef Will Jones < > will.jones...@gmail.com > > >: > > > > > > > > > > > > > > > > That wouldn't remove the feature from DuckDB, would it? It would > > just > > > > > mean > > > > > > that we recognize that PyArrow expressions don't have > well-defined > > > > > > semantics that we are committing to at this time. > > > > > > > > > > > > > > > > That's a fair point, David. I would be fine excluding it from the > > > > protocol > > > > > initially, and keep the existing integrations in DuckDB, Polars, > and > > > > > Datafusion "secret" or "not officially supported" for the time > > being. At > > > > > the very least, documenting the pattern to get a Arrow C stream > will > > be a > > > > > step forward. > > > > > > > > > > Best, > > > > > > > > > > Will Jones > > > > > > > > > > On Wed, Jun 28, 2023 at 12:35 PM Jonathan Keane <jke...@gmail.com> > > > > wrote: > > > > > > > > > > > > I would understand this objection more if DuckDB hasn't been > > relying > > > > on > > > > > > > being able to pass PyArrow expressions for 18 months now [1]. > > Unless, > > > > > do > > > > > > we > > > > > > > just think this isn't widely used enough that we don't care? > > > > > > > > > > > > This isn't a pro or a con of specifically adopting the PyArrow > > > > expression > > > > > > semantics as is / with a warning about changing / not at all, but > > > > having > > > > > > some kind of standardization in this interface would be very > nice. > > This > > > > > > even came up while collaborating with the DuckDB folks that using > > some > > > > of > > > > > > the expression bits here (and in the R equivalents) was a little > > bit > > > > odd > > > > > > and having something like a proper API for that would have made > > that > > > > > > more natural (and likely that would have been used had it existed > > 18 > > > > > months > > > > > > ago :)) > > > > > > > > > > > > -Jon > > > > > > > > > > > > > > > > > > On Wed, Jun 28, 2023 at 1:17 PM David Li <lidav...@apache.org> > > wrote: > > > > > > > > > > > > > That wouldn't remove the feature from DuckDB, would it? It > would > > just > > > > > > mean > > > > > > > that we recognize that PyArrow expressions don't have > > well-defined > > > > > > > semantics that we are committing to at this time. As long as we > > have > > > > > > > `**kwargs` everywhere, we can in the future introduce a > > > > > > > `substrait_filter_expression` or similar argument, while > allowing > > > > > current > > > > > > > implementors to handle `filter` if possible. (As a compromise, > we > > > > could > > > > > > > reserve `filter` and existing arguments and note that PyArrow > > > > > Expression > > > > > > > semantics are subject to change without notice?) > > > > > > > > > > > > > > On Wed, Jun 28, 2023, at 13:38, Will Jones wrote: > > > > > > > > Hi Ian, > > > > > > > > > > > > > > > > > > > > > > > >> I favor option 2 out of concern that option 1 could create a > > > > > > > >> temptation for users of this protocol to depend on a feature > > that > > > > we > > > > > > > >> intend to deprecate. > > > > > > > >> > > > > > > > > > > > > > > > > I would understand this objection more if DuckDB hasn't been > > > > relying > > > > > on > > > > > > > > being able to pass PyArrow expressions for 18 months now [1]. > > > > Unless, > > > > > > do > > > > > > > we > > > > > > > > just think this isn't widely used enough that we don't care? > > > > > > > > > > > > > > > > Best, > > > > > > > > Will > > > > > > > > > > > > > > > > [1] https://duckdb.org/2021/12/03/duck-arrow.html > > > > > > > > > > > > > > > > On Tue, Jun 27, 2023 at 11:19 AM Ian Cook < > ianmc...@apache.org > > > > > > > > wrote: > > > > > > > > > > > > > > > >> > I think there's three routes we can go here: > > > > > > > >> > > > > > > > > >> > 1. We keep PyArrow expressions in the API initially, but > > once we > > > > > > have > > > > > > > >> > Substrait-based alternatives we deprecate the PyArrow > > expression > > > > > > > support. > > > > > > > >> > This is what I intended with the current design, and I > > think it > > > > > > > provides > > > > > > > >> > the most obvious migration paths for existing producers > and > > > > > > consumers. > > > > > > > >> > 2. We keep the overall dataset API, but don't introduce > the > > > > filter > > > > > > and > > > > > > > >> > projection arguments until we have Substrait support. I'm > > not > > > > sure > > > > > > > what > > > > > > > >> the > > > > > > > >> > migration path looks like for producers and consumers, > but I > > > > think > > > > > > > this > > > > > > > >> > just implicitly becomes the same as (1), but with worse > > > > > > documentation. > > > > > > > >> > 3. We write a protocol completely from scratch, that > > doesn't try > > > > > to > > > > > > > >> > describe the existing dataset API. Producers and consumers > > would > > > > > > then > > > > > > > >> > migrate to use the new protocol and deprecate their > existing > > > > > dataset > > > > > > > >> > integrations. We could introduce a dunder method in that > API > > > > (sort > > > > > > of > > > > > > > >> like > > > > > > > >> > __arrow_array__) that would make the migration seamless > > from the > > > > > > > end-user > > > > > > > >> > perspective. > > > > > > > >> > > > > > > > > >> > *Which do you all think is the best path forward?* > > > > > > > >> > > > > > > > >> I favor option 2 out of concern that option 1 could create a > > > > > > > >> temptation for users of this protocol to depend on a feature > > that > > > > we > > > > > > > >> intend to deprecate. I think option 2 also creates a > stronger > > > > > > > >> motivation to complete the Substrait expression integration > > work, > > > > > > > >> which is underway in > > https://github.com/apache/arrow/pull/34834. > > > > > > > >> > > > > > > > >> Ian > > > > > > > >> > > > > > > > >> > > > > > > > >> On Fri, Jun 23, 2023 at 1:25 PM Weston Pace < > > > > weston.p...@gmail.com> > > > > > > > wrote: > > > > > > > >> > > > > > > > > >> > > The trouble is that Dataset was not designed to serve > as a > > > > > > > >> > > general-purpose unmaterialized dataframe. For example, > the > > > > > PyArrow > > > > > > > >> > > Dataset constructor [5] exposes options for specifying a > > list > > > > of > > > > > > > >> > > source files and a partitioning scheme, which are > > irrelevant > > > > for > > > > > > > many > > > > > > > >> > > of the applications that Will anticipates. And some work > > is > > > > > needed > > > > > > > to > > > > > > > >> > > reconcile the methods of the PyArrow Dataset object [6] > > with > > > > the > > > > > > > >> > > methods of the Table object. Some methods like filter() > > are > > > > > > exposed > > > > > > > by > > > > > > > >> > > both and behave lazily on Datasets and eagerly on > Tables, > > as a > > > > > > user > > > > > > > >> > > might expect. But many other Table methods are not > > implemented > > > > > for > > > > > > > >> > > Dataset though they potentially could be, and it is > > unclear > > > > > where > > > > > > we > > > > > > > >> > > should draw the line between adding methods to Dataset > vs. > > > > > > > encouraging > > > > > > > >> > > new scanner implementations to expose options > controlling > > what > > > > > > lazy > > > > > > > >> > > operations should be performed as they see fit. > > > > > > > >> > > > > > > > > >> > In my mind there is a distinction between the "compute > > domain" > > > > > > (e.g. a > > > > > > > >> > pandas dataframe or something like ibis or SQL) and the > > "data > > > > > > domain" > > > > > > > >> (e.g. > > > > > > > >> > pyarrow datasets). I think, in a perfect world, you could > > push > > > > > any > > > > > > > and > > > > > > > >> all > > > > > > > >> > compute up and down the chain as far as possible. > However, > > in > > > > > > > practice, > > > > > > > >> I > > > > > > > >> > think there is a healthy set of tools and libraries that > say > > > > > "simple > > > > > > > >> column > > > > > > > >> > projection and filtering is good enough". I would argue > > that > > > > > there > > > > > > is > > > > > > > >> room > > > > > > > >> > for both APIs and while the temptation is always present > to > > > > "shove > > > > > > as > > > > > > > >> much > > > > > > > >> > compute as you can" I think pyarrow datasets seem to have > > found > > > > a > > > > > > > balance > > > > > > > >> > between the two that users like. > > > > > > > >> > > > > > > > > >> > So I would argue that this protocol may never become a > > > > > > general-purpose > > > > > > > >> > unmaterialized dataframe and that isn't necessarily a bad > > thing. > > > > > > > >> > > > > > > > > >> > > they are splittable and serializable, so that fragments > > can be > > > > > > > >> distributed > > > > > > > >> > > amongst processes / workers. > > > > > > > >> > > > > > > > > >> > Just to clarify, the proposal currently only requires the > > > > > fragments > > > > > > > to be > > > > > > > >> > serializable correct? > > > > > > > >> > > > > > > > > >> > On Fri, Jun 23, 2023 at 11:48 AM Will Jones < > > > > > > will.jones...@gmail.com> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > > >> > > Thanks Ian for your extensive feedback. > > > > > > > >> > > > > > > > > > >> > > I strongly agree with the comments made by David, > > > > > > > >> > > > Weston, and Dewey arguing that we should avoid any use > > of > > > > > > PyArrow > > > > > > > >> > > > expressions in this API. Expressions are an > > implementation > > > > > > detail > > > > > > > of > > > > > > > >> > > > PyArrow, not a part of the Arrow standard. It would be > > much > > > > > > safer > > > > > > > for > > > > > > > >> > > > the initial version of this protocol to not define > *any* > > > > > > > >> > > > methods/arguments that take expressions. > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > I would agree with this point, if we were starting from > > > > scratch. > > > > > > But > > > > > > > >> one of > > > > > > > >> > > my goals is for this protocol to be descriptive of the > > > > existing > > > > > > > dataset > > > > > > > >> > > integrations in the ecosystem, which all currently rely > on > > > > > PyArrow > > > > > > > >> > > expressions. For example, you'll notice in the PR that > > there > > > > are > > > > > > > unit > > > > > > > >> tests > > > > > > > >> > > to verify the current PyArrow Dataset classes conform to > > this > > > > > > > protocol, > > > > > > > >> > > without changes. > > > > > > > >> > > > > > > > > > >> > > I think there's three routes we can go here: > > > > > > > >> > > > > > > > > > >> > > 1. We keep PyArrow expressions in the API initially, but > > once > > > > we > > > > > > > have > > > > > > > >> > > Substrait-based alternatives we deprecate the PyArrow > > > > expression > > > > > > > >> support. > > > > > > > >> > > This is what I intended with the current design, and I > > think > > > > it > > > > > > > >> provides > > > > > > > >> > > the most obvious migration paths for existing producers > > and > > > > > > > consumers. > > > > > > > >> > > 2. We keep the overall dataset API, but don't introduce > > the > > > > > filter > > > > > > > and > > > > > > > >> > > projection arguments until we have Substrait support. > I'm > > not > > > > > sure > > > > > > > >> what the > > > > > > > >> > > migration path looks like for producers and consumers, > > but I > > > > > think > > > > > > > this > > > > > > > >> > > just implicitly becomes the same as (1), but with worse > > > > > > > documentation. > > > > > > > >> > > 3. We write a protocol completely from scratch, that > > doesn't > > > > try > > > > > > to > > > > > > > >> > > describe the existing dataset API. Producers and > consumers > > > > would > > > > > > > then > > > > > > > >> > > migrate to use the new protocol and deprecate their > > existing > > > > > > dataset > > > > > > > >> > > integrations. We could introduce a dunder method in that > > API > > > > > (sort > > > > > > > of > > > > > > > >> like > > > > > > > >> > > __arrow_array__) that would make the migration seamless > > from > > > > the > > > > > > > >> end-user > > > > > > > >> > > perspective. > > > > > > > >> > > > > > > > > > >> > > *Which do you all think is the best path forward?* > > > > > > > >> > > > > > > > > > >> > > Another concern I have is that we have not fully > > explained why > > > > > we > > > > > > > want > > > > > > > >> > > > to use Dataset instead of RecordBatchReader [9] as the > > basis > > > > > of > > > > > > > this > > > > > > > >> > > > protocol. I would like to see an explanation of why > > > > > > > RecordBatchReader > > > > > > > >> > > > is not sufficient for this. RecordBatchReader seems > like > > > > > another > > > > > > > >> > > > possible way to represent "unmaterialized dataframes" > > and > > > > > there > > > > > > > are > > > > > > > >> > > > some parallels between RecordBatch/RecordBatchReader > and > > > > > > > >> > > > Fragment/Dataset. > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > This is a good point. I can add a section describing the > > > > > > > differences. > > > > > > > >> The > > > > > > > >> > > main ones I can think of are that: (1) Datasets are > > > > "pruneable": > > > > > > one > > > > > > > >> can > > > > > > > >> > > select a subset of columns and apply a filter on rows to > > avoid > > > > > IO > > > > > > > and > > > > > > > >> (2) > > > > > > > >> > > they are splittable and serializable, so that fragments > > can be > > > > > > > >> distributed > > > > > > > >> > > amongst processes / workers. > > > > > > > >> > > > > > > > > > >> > > Best, > > > > > > > >> > > > > > > > > > >> > > Will Jones > > > > > > > >> > > > > > > > > > >> > > On Fri, Jun 23, 2023 at 10:48 AM Ian Cook < > > > > ianmc...@apache.org> > > > > > > > wrote: > > > > > > > >> > > > > > > > > > >> > > > Thanks Will for this proposal! > > > > > > > >> > > > > > > > > > > >> > > > For anyone familiar with PyArrow, this idea has a > clear > > > > > > intuitive > > > > > > > >> > > > logic to it. It provides an expedient solution to the > > > > current > > > > > > > lack of > > > > > > > >> > > > a practical means for interchanging "unmaterialized > > > > > dataframes" > > > > > > > >> > > > between different Python libraries. > > > > > > > >> > > > > > > > > > > >> > > > To elaborate on that: If you look at how people use > the > > > > Arrow > > > > > > > Dataset > > > > > > > >> > > > API—which is implemented in the Arrow C++ library [1] > > and > > > > has > > > > > > > >> bindings > > > > > > > >> > > > not just for Python [2] but also for Java [3] and R > > > > [4]—you'll > > > > > > see > > > > > > > >> > > > that Dataset is often used simply as a "virtual" > > variant of > > > > > > > Table. It > > > > > > > >> > > > is used in cases when the data is larger than memory > or > > when > > > > > it > > > > > > is > > > > > > > >> > > > desirable to defer reading (materializing) the data > into > > > > > memory. > > > > > > > >> > > > > > > > > > > >> > > > So we can think of a Table as a materialized dataframe > > and a > > > > > > > Dataset > > > > > > > >> > > > as an unmaterialized dataframe. That aspect of Dataset > > is I > > > > > > think > > > > > > > >> what > > > > > > > >> > > > makes it most attractive as a protocol for enabling > > > > > > > interoperability: > > > > > > > >> > > > it allows libraries to easily "speak Arrow" in cases > > where > > > > > > > >> > > > materializing the full data in memory upfront is > > impossible > > > > or > > > > > > > >> > > > undesirable. > > > > > > > >> > > > > > > > > > > >> > > > The trouble is that Dataset was not designed to serve > > as a > > > > > > > >> > > > general-purpose unmaterialized dataframe. For example, > > the > > > > > > PyArrow > > > > > > > >> > > > Dataset constructor [5] exposes options for > specifying a > > > > list > > > > > of > > > > > > > >> > > > source files and a partitioning scheme, which are > > irrelevant > > > > > for > > > > > > > many > > > > > > > >> > > > of the applications that Will anticipates. And some > > work is > > > > > > > needed to > > > > > > > >> > > > reconcile the methods of the PyArrow Dataset object > [6] > > with > > > > > the > > > > > > > >> > > > methods of the Table object. Some methods like > filter() > > are > > > > > > > exposed > > > > > > > >> by > > > > > > > >> > > > both and behave lazily on Datasets and eagerly on > > Tables, > > > > as a > > > > > > > user > > > > > > > >> > > > might expect. But many other Table methods are not > > > > implemented > > > > > > for > > > > > > > >> > > > Dataset though they potentially could be, and it is > > unclear > > > > > > where > > > > > > > we > > > > > > > >> > > > should draw the line between adding methods to Dataset > > vs. > > > > > > > >> encouraging > > > > > > > >> > > > new scanner implementations to expose options > > controlling > > > > what > > > > > > > lazy > > > > > > > >> > > > operations should be performed as they see fit. > > > > > > > >> > > > > > > > > > > >> > > > Will, I see that you've already addressed this issue > to > > some > > > > > > > extent > > > > > > > >> in > > > > > > > >> > > > your proposal. For example, you mention that we should > > > > > initially > > > > > > > >> > > > define this protocol to include only a minimal subset > > of the > > > > > > > Dataset > > > > > > > >> > > > API. I agree, but I think there are some loose ends we > > > > should > > > > > be > > > > > > > >> > > > careful to tie up. I strongly agree with the comments > > made > > > > by > > > > > > > David, > > > > > > > >> > > > Weston, and Dewey arguing that we should avoid any use > > of > > > > > > PyArrow > > > > > > > >> > > > expressions in this API. Expressions are an > > implementation > > > > > > detail > > > > > > > of > > > > > > > >> > > > PyArrow, not a part of the Arrow standard. It would be > > much > > > > > > safer > > > > > > > for > > > > > > > >> > > > the initial version of this protocol to not define > *any* > > > > > > > >> > > > methods/arguments that take expressions. This will > > allow us > > > > to > > > > > > > take > > > > > > > >> > > > some more time to finish up the Substrait expression > > > > > > > implementation > > > > > > > >> > > > work that is underway [7][8], then introduce > > Substrait-based > > > > > > > >> > > > expressions in a latter version of this protocol. This > > > > > approach > > > > > > > will > > > > > > > >> > > > better position this protocol to be implemented in > other > > > > > > languages > > > > > > > >> > > > besides Python. > > > > > > > >> > > > > > > > > > > >> > > > Another concern I have is that we have not fully > > explained > > > > why > > > > > > we > > > > > > > >> want > > > > > > > >> > > > to use Dataset instead of RecordBatchReader [9] as the > > basis > > > > > of > > > > > > > this > > > > > > > >> > > > protocol. I would like to see an explanation of why > > > > > > > RecordBatchReader > > > > > > > >> > > > is not sufficient for this. RecordBatchReader seems > like > > > > > another > > > > > > > >> > > > possible way to represent "unmaterialized dataframes" > > and > > > > > there > > > > > > > are > > > > > > > >> > > > some parallels between RecordBatch/RecordBatchReader > and > > > > > > > >> > > > Fragment/Dataset. We should help developers and users > > > > > understand > > > > > > > why > > > > > > > >> > > > Arrow needs both of these. > > > > > > > >> > > > > > > > > > > >> > > > Thanks Will for your thoughtful prose explanations > about > > > > this > > > > > > > >> proposed > > > > > > > >> > > > API. After we arrive at a decision about this, I think > > we > > > > > should > > > > > > > >> > > > reproduce some of these explanations in docs, blog > > posts, > > > > > > cookbook > > > > > > > >> > > > recipes, etc. because there is some important nuance > > here > > > > that > > > > > > > will > > > > > > > >> be > > > > > > > >> > > > important for integrators of this API to understand. > > > > > > > >> > > > > > > > > > > >> > > > Ian > > > > > > > >> > > > > > > > > > > >> > > > [1] > https://arrow.apache.org/docs/cpp/api/dataset.html > > > > > > > >> > > > [2] https://arrow.apache.org/docs/python/dataset.html > > > > > > > >> > > > [3] https://arrow.apache.org/docs/java/dataset.html > > > > > > > >> > > > [4] > > https://arrow.apache.org/docs/r/articles/dataset.html > > > > > > > >> > > > [5] > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://arrow.apache.org/docs/python/generated/pyarrow.dataset.dataset.html#pyarrow.dataset.dataset > > > > > > > >> > > > [6] > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://arrow.apache.org/docs/python/generated/pyarrow.dataset.Dataset.html > > > > > > > >> > > > [7] https://github.com/apache/arrow/issues/33985 > > > > > > > >> > > > [8] https://github.com/apache/arrow/issues/34252 > > > > > > > >> > > > [9] > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatchReader.html > > > > > > > >> > > > > > > > > > > >> > > > On Wed, Jun 21, 2023 at 2:09 PM Will Jones < > > > > > > > will.jones...@gmail.com> > > > > > > > >> > > > wrote: > > > > > > > >> > > > > > > > > > > > >> > > > > Hello Arrow devs, > > > > > > > >> > > > > > > > > > > > >> > > > > I have drafted a PR defining an experimental > protocol > > > > which > > > > > > > would > > > > > > > >> allow > > > > > > > >> > > > > third-party libraries to imitate the PyArrow Dataset > > API > > > > > [5]. > > > > > > > This > > > > > > > >> > > > protocol > > > > > > > >> > > > > is intended to endorse an integration pattern that > is > > > > > starting > > > > > > > to > > > > > > > >> be > > > > > > > >> > > used > > > > > > > >> > > > > in the Python ecosystem, where some libraries are > > > > providing > > > > > > > their > > > > > > > >> own > > > > > > > >> > > > > scanners with this API, while query engines are > > accepting > > > > > > these > > > > > > > as > > > > > > > >> > > > > duck-typed objects. > > > > > > > >> > > > > > > > > > > > >> > > > > To give some background: back at the end of 2021, we > > > > > > > collaborated > > > > > > > >> with > > > > > > > >> > > > > DuckDB to be able to read datasets (an Arrow C++ > > concept), > > > > > > > >> supporting > > > > > > > >> > > > > column selection and filter pushdown. This was > > > > accomplished > > > > > by > > > > > > > >> having > > > > > > > >> > > > > DuckDB manipulating Python (or R) objects to get a > > > > > > > >> RecordBatchReader > > > > > > > >> > > and > > > > > > > >> > > > > then exporting over the C Stream Interface. > > > > > > > >> > > > > > > > > > > > >> > > > > Since then, DataFusion [2] and Polars have both made > > > > similar > > > > > > > >> > > > > implementations for their Python bindings, allowing > > them > > > > to > > > > > > > consume > > > > > > > >> > > > PyArrow > > > > > > > >> > > > > datasets. This has created an implicit protocol, > > whereby > > > > > > > arbitrary > > > > > > > >> > > > compute > > > > > > > >> > > > > engines can push down queries into the PyArrow > dataset > > > > > > scanner. > > > > > > > >> > > > > > > > > > > > >> > > > > Now, libraries supporting table formats including > > Delta > > > > > Lake, > > > > > > > >> Lance, > > > > > > > >> > > and > > > > > > > >> > > > > Iceberg are looking to be able to support these > > engines, > > > > > while > > > > > > > >> bringing > > > > > > > >> > > > > their own scanners and metadata handling > > implementations. > > > > > One > > > > > > > >> possible > > > > > > > >> > > > > route is allowing them to imitate the PyArrow > datasets > > > > API. > > > > > > > >> > > > > > > > > > > > >> > > > > Bringing these use cases together, I'd like to > > propose an > > > > > > > >> experimental > > > > > > > >> > > > > protocol, made out of the minimal subset of the > > PyArrow > > > > > > Dataset > > > > > > > API > > > > > > > >> > > > > necessary to facilitate this kind of integration. > This > > > > would > > > > > > > allow > > > > > > > >> any > > > > > > > >> > > > > library to produce a scanner implementation and that > > > > > arbitrary > > > > > > > >> query > > > > > > > >> > > > > engines could call into. I've drafted a PR [3] and > > there > > > > is > > > > > > some > > > > > > > >> > > > background > > > > > > > >> > > > > research available in a google doc [4]. > > > > > > > >> > > > > > > > > > > > >> > > > > I've already gotten some good feedback on both, and > > would > > > > > > > welcome > > > > > > > >> more. > > > > > > > >> > > > > > > > > > > > >> > > > > One last point: I'd like for this to be a first step > > > > rather > > > > > > > than a > > > > > > > >> > > > > comprehensive API. This PR focuses on making > explicit > > a > > > > > > protocol > > > > > > > >> that > > > > > > > >> > > is > > > > > > > >> > > > > already in use in the ecosystem, but without much > > concrete > > > > > > > >> definition. > > > > > > > >> > > > Once > > > > > > > >> > > > > this is established, we can use our experience from > > this > > > > > > > protocol > > > > > > > >> to > > > > > > > >> > > > design > > > > > > > >> > > > > something more permanent that takes advantage of > newer > > > > > > > innovations > > > > > > > >> in > > > > > > > >> > > the > > > > > > > >> > > > > Arrow ecosystem (such as the PyCapsule for C Data > > > > Interface > > > > > or > > > > > > > >> > > > > Substrait for passing expressions / scan plans). I > am > > > > > tracking > > > > > > > such > > > > > > > >> > > > future > > > > > > > >> > > > > improvements in [5]. > > > > > > > >> > > > > > > > > > > > >> > > > > Best, > > > > > > > >> > > > > > > > > > > > >> > > > > Will Jones > > > > > > > >> > > > > > > > > > > > >> > > > > [1] https://duckdb.org/2021/12/03/duck-arrow.html > > > > > > > >> > > > > [2] > > > > > https://github.com/apache/arrow-datafusion-python/pull/9 > > > > > > > >> > > > > [3] https://github.com/apache/arrow/pull/35568 > > > > > > > >> > > > > [4] > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1r56nt5Un2E7yPrZO9YPknBN4EDtptpx-tqOZReHvq1U/edit?pli=1 > > > > > > > >> > > > > [5] > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1-uVkSZeaBtOALVbqMOPeyV3s2UND7Wl-IGEZ-P-gMXQ/edit > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >
