Inconsistent class names for AWS integrations

Hi all. While I was working on AIRFLOW-5803, I found there are inconsistencies between class names for each AWS operators/hooks. For example, Athena hook is named as 'AWSAthenaHook', but Dynamodb hook is named as 'AwsDynamoDBHook', and S3 hook is just 'S3Hook' (So, AIRFLOW-5803 is trying to re

Re: Tomorrow is our first SIG-Knative meeting

Looking forward !! Thanks Daniel On Tue, Oct 29, 2019 at 9:40 PM Daniel Imberman wrote: > Hello fellow Airflowers! > > Wanted to send a quick reminder that our first airflow SIG-knative meeting > will take place tomorrow morning at 11AM EST! We'll be going to go over > the > proposed architectu

Tomorrow is our first SIG-Knative meeting

Hello fellow Airflowers! Wanted to send a quick reminder that our first airflow SIG-knative meeting will take place tomorrow morning at 11AM EST! We'll be going to go over the proposed architecture, current progress, and potential opportunities for collaboration on the upcoming KnativeExecutor

Re: Donating code to add Common Workflow Language import to Airflow

I have not looked at the details of the code yet, it I would like to understand few things: 1) is the CWL package more of a converter of CWL to Python DAG files (that can then be scheduled as usual) or whether it is running alongside of the scheduler and schedules tasks and operators separately us

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

The namespace feature looks promising and from your tests, it looks like it would work well from Airflow 2.0 and onwards. I will look at it in-depth and see if I have more suggestions or opinion on it On Tue, Oct 29, 2019 at 3:32 PM Jarek Potiuk wrote: > TL;DR; We did some testing about namespa

Re: [PROPOSE] Ease future migration path to 2.0 by provider's operators/hook backporting to 1.10.*

TL;DR; We did some testing about namespaces and packaging (and potential backporting options for 1.10.* python3 Airflows) and we think it's best to use namespaces quickly and use different package name "airflow-integrations" for all non-fundamental integrations. Unless we missed some tricks, we ca

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

I agree we can merge providers and services (especially that they are not "cloud providers" any more :) >From the discussion above I think the specific proposals: - *fundamentals* - those are all the operators/hooks/sensors that are the "Core" of Airflow (base, dbapi) and allow you to run b

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Also, ansible has something similar: https://github.com/ansible/ansible/tree/devel/lib/ansible/modules Generally, I have been inspired by how Terraform and Ansible have implemented it and can serve as an inspiration to us. On Tue, Oct 29, 2019 at 12:51 PM Ash Berlin-Taylor wrote: > Also provide

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Also providers and SAAS could be merged (taking inspiration from Terraform here: https://www.terraform.io/docs/providers/index.html - ignore the menu on the left, that is just for Docs layout, which we could do too -- Docs grouping doesn't ha

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Thanks Jarek for clearing that up. Personally I would omit the Apache one. We should not step into the fallacy as before with not being sure if it was in contrib or not. I would even consider merging software and protocols, as it not entirely clear what a protocol is or not. In the end, everything

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Yep. We should definitely discuss the split! For me these are the criteria: - fundamentals - those are all the operators/hooks/sensors that are the "Core" of Airflow (base, dbapi) and allow you to run basic examples, implements basic logic of Airflow (subdags, branch etc.) + generic

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

1. Sounds good to me 2. Also fine 3. We should have some consensus here. E.g. I’m not sure what groups “fundamentals” and “software” are meant to be :-) While we’re at it: we should really move the BaseOperator out of models. The BaseOperator has no representation in the DB and should b

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

After some consideration and seeing the actual move in practice I wanted to propose 3rd amendment ;) to the AIP-21. I have a few observations from seeing the discussions and observing the actual moving process. I have the following proposals: *1) Between-providers transfer operators should be kept

Re: Apache Airflow 1.10.6 released!

Thank Ash! On Tue, 29 Oct 2019 at 09:56, Felix Uellendall wrote: > Thanks Ash :) > > Sent from ProtonMail Mobile > > On Tue, Oct 29, 2019 at 07:32, Driesprong, Fokko > wrote: > > > Awesome, thanks Ash for doing the release! > > > > Cheers, Fokko > > > > Op di 29 okt. 2019 om 05:09 schreef Aizha

Re: Apache Airflow 1.10.6 released!

Thanks Ash :) Sent from ProtonMail Mobile On Tue, Oct 29, 2019 at 07:32, Driesprong, Fokko wrote: > Awesome, thanks Ash for doing the release! > > Cheers, Fokko > > Op di 29 okt. 2019 om 05:09 schreef Aizhamal Nurmamat kyzy < > aizha...@apache.org>: > >> Thanks Ash and everyone who was involved