Yeah, the other option is to include it in the common.sql package since they are related. But I am okay with the common.dataframe, too.
On Mon, 24 Jun 2024 at 20:04, Jarek Potiuk <ja...@potiuk.com> wrote: > Hello here, > > At Pycon US earlier this year I had a number of interesting conversations > and one of the - very interesting - conversations I had was with the Ibis > team and I thought maybe we should consider releasing "common.dataframe" > provider for Airflow - following up after "common.sql" and "common.io". > > Ibis is gaining a lot of popularity recently and it might be at > more-or-less the same "place" as fsspec when Bolke added "common.io". Plus > if airflow adds it as a community provider, it might also bring Ibis' > popularity up. > > In short - Ibis is a "Portable Python dataframe library". It becomes more > and more popular and it not only serves 20+ dataframe backends with the > same, portable API, but also allows to mix SQL with dataframes and few more > things. Some time ago there were some ideas that we could add "SQLAlchemy" > as an additional "common" interface in "common.sql" - but actually it seems > that Ibis provides a much better abstraction that unifies SQL and Dataframe > approach nicely - way better suited for the "data science" world of > Airflow. > > You can see very nice overview "why Ibis" here: > https://ibis-project.org/why > - and I think it would be pretty natural thing to add on top of > "common.sql" and "common.io" - following "Airflow As a Platform" mantra. > > WDYT? > > J. >
