Re: [DISCUSS] Implement common.dataframe (or similar) provider based on Ibis

And another - far more important reason (and reason why we have common.sql) - we could VERY LIKELY (maybe Maciej and Kacper could comment on that) - we could have equivalent of column-level lineage implemented once for all the engines - by adding "common.dataframe" open-lineage information. On Tue

Re: [DISCUSS] Implement common.dataframe (or similar) provider based on Ibis

That is a very good question and I forgot to mention it. The main reason is the same as in common.io - we could make it work with our standard "Hook/Connection" framework so that you could get authentication information from Airflow Connections, plugging in the Secrets/ DB Connection information.

[Meeting Notes] Airflow 3.0 Dev call - 20 June 2024

Hey all, Apologies for the delay! I have updated our meeting notes document to summarize the discussion from our 20th June dev call for Airflow 3.0. Link: https://cwiki.apache.org/confluence/x/8ApeEg#Airflow3Devcall:MeetingNotes-20June2024 To all those who attended, can you please double-check

Re: [DISCUSS] Implement common.dataframe (or similar) provider based on Ibis

There might be a good case for "why ibis", but why should airflow wrap ibis? Why do we need a common dataframe library? Is ibis not "that" already? On Mon, Jun 24, 2024 at 3:31 PM Kaxil Naik wrote: > Yeah, the other option is to include it in the common.sql package since > they are related

Re: [DISCUSS] Implement common.dataframe (or similar) provider based on Ibis

Yeah, the other option is to include it in the common.sql package since they are related. But I am okay with the common.dataframe, too. On Mon, 24 Jun 2024 at 20:04, Jarek Potiuk wrote: > Hello here, > > At Pycon US earlier this year I had a number of interesting conversations > and one of the

[DISCUSS] Implement common.dataframe (or similar) provider based on Ibis

Hello here, At Pycon US earlier this year I had a number of interesting conversations and one of the - very interesting - conversations I had was with the Ibis team and I thought maybe we should consider releasing "common.dataframe" provider for Airflow - following up after "common.sql" and "commo

Re: System Test Dashboards - Phase Two??

> The information in our database is similar to the structure of the AWS > providers json file > https://aws-mwaa.github.io/open-source/system-tests/dashboard.json + a field > for logs. > We also have an extra field that specifies the commit-id against which the > CI was run, > which I belie

Re: [VOTE] Airflow Providers prepared on June 22, 2024

+1 (non-binding) Tested my set of changes in the Databricks provider #40013 , #39771 and #39941 . All works fine. Thank you for the release efforts. Best reg

Re: [VOTE] Airflow Providers prepared on June 22, 2024

+1 (non-binding) Checked a few dags, looks good -- Regards, Aritra Basu On Mon, Jun 24, 2024, 5:23 PM Ephraim Anierobi wrote: > +1(binding). Checked files, reproducibility, signatures, licenses, and > checksums > > On Mon, 24 Jun 2024 at 12:49, Hussein Awala wrote: > > > +1 (binding): checked s

Re: System Test Dashboards - Phase Two??

Hi, +1 to this idea. I think standardizing the format of the presented test run results makes sense. I also agree that we don't necessarily need to enforce it in any hard way. However, given that we have dashboards of these three major providers, we could consider enforcing the presence of *some*

Re: [VOTE] Airflow Providers prepared on June 22, 2024

+1(binding). Checked files, reproducibility, signatures, licenses, and checksums On Mon, 24 Jun 2024 at 12:49, Hussein Awala wrote: > +1 (binding): checked signatures, checksums, licences, and sources. All > looks good. > > On Mon, Jun 24, 2024 at 1:47 PM Pankaj Singh > wrote: > > > + 1 (non-bi

Re: [VOTE] Airflow Providers prepared on June 22, 2024

+1 (binding): checked signatures, checksums, licences, and sources. All looks good. On Mon, Jun 24, 2024 at 1:47 PM Pankaj Singh wrote: > + 1 (non-binding) > > My changes look good to me. > > On Mon, Jun 24, 2024 at 3:27 PM Wei Lee wrote: > > > + 1 non-binding. Verified my change and tested the

Re: [VOTE] Airflow Providers prepared on June 22, 2024

+ 1 (non-binding) My changes look good to me. On Mon, Jun 24, 2024 at 3:27 PM Wei Lee wrote: > + 1 non-binding. Verified my change and tested the RCs with some example > DAGs. > > Best, > Wei > > > On Jun 24, 2024, at 12:38 PM, Rahul Vats wrote: > > > > +1 non-binding. Verified running some ex

Re: [VOTE] Airflow Providers prepared on June 22, 2024

+ 1 non-binding. Verified my change and tested the RCs with some example DAGs. Best, Wei > On Jun 24, 2024, at 12:38 PM, Rahul Vats wrote: > > +1 non-binding. Verified running some example dags. LGTM! > > Regards, > Rahul Vats > 9953794332 > > > On Mon, 24 Jun 2024 at 09:41, Amogh Desai wro