Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

I agree we can merge providers and services (especially that they are not "cloud providers" any more :) >From the discussion above I think the specific proposals: - *fundamentals* - those are all the operators/hooks/sensors that are the "Core" of Airflow (base, dbapi) and allow you to run b

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Also, ansible has something similar: https://github.com/ansible/ansible/tree/devel/lib/ansible/modules Generally, I have been inspired by how Terraform and Ansible have implemented it and can serve as an inspiration to us. On Tue, Oct 29, 2019 at 12:51 PM Ash Berlin-Taylor wrote: > Also provide

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Also providers and SAAS could be merged (taking inspiration from Terraform here: https://www.terraform.io/docs/providers/index.html - ignore the menu on the left, that is just for Docs layout, which we could do too -- Docs grouping doesn't ha

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Thanks Jarek for clearing that up. Personally I would omit the Apache one. We should not step into the fallacy as before with not being sure if it was in contrib or not. I would even consider merging software and protocols, as it not entirely clear what a protocol is or not. In the end, everything

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Yep. We should definitely discuss the split! For me these are the criteria: - fundamentals - those are all the operators/hooks/sensors that are the "Core" of Airflow (base, dbapi) and allow you to run basic examples, implements basic logic of Airflow (subdags, branch etc.) + generic

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

1. Sounds good to me 2. Also fine 3. We should have some consensus here. E.g. I’m not sure what groups “fundamentals” and “software” are meant to be :-) While we’re at it: we should really move the BaseOperator out of models. The BaseOperator has no representation in the DB and should b

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

After some consideration and seeing the actual move in practice I wanted to propose 3rd amendment ;) to the AIP-21. I have a few observations from seeing the discussions and observing the actual moving process. I have the following proposals: *1) Between-providers transfer operators should be kept

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Hello everyone. I updated AIP-21 and updated examples. Point D. of AIP-21 is now as follows: *D. * Group operators/sensors/hooks in *airflow/providers/*/operators(sensors, hooks). Each provider can define its own internal structure of that package. For example in case of "google" provider the

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Yeah. I think the important point is that the latest doc changes by Kamil index all available operators and hooks nicely and make them easy to find. That also includes (as of today) automated CI checking if new operators and hooks added are added to the documentation : https://github.com/apache/ai

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

It's not obvious to me why an S3ToMsSQLOperator in the aws package is "silly". Why do you say it made sense to create a MsSqlFromS3Operator? Basically all of these operators could be thought of as "move data from A to B" or "move data to B from A". I think what feels natural to each individual wi

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

One case popped up for us recently, where it made sense to make a MsSql *From*S3Operator . I think using "source" makes sense in general, but in this case calling this a S3ToMsSqlOperator and putting it under AWS seems silly, even though you could say s3 is "source" here. I think in most of these

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

It looks like we have general consensus about putting transfer operators into "source provider" package. That's great for me as well. Since I will be updating AIP-21 to reflect the "google" vs. "gcp" case, I will also update it to add this decision. If no-one objects (Lazy Consensus

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

On Mon, Sep 23, 2019 at 7:42 PM Chris Palmer wrote: > > On Mon, Sep 23, 2019 at 1:22 PM Kamil Breguła > wrote: > > > On Mon, Sep 23, 2019 at 7:04 PM Chris Palmer wrote: > > > > > > Is there a reason why we can't use symlinks to have copies of the files > > > show up in both subpackages? So that

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

On Mon, Sep 23, 2019 at 1:22 PM Kamil Breguła wrote: > On Mon, Sep 23, 2019 at 7:04 PM Chris Palmer wrote: > > > > Is there a reason why we can't use symlinks to have copies of the files > > show up in both subpackages? So that `gcs_to_s3.py` would be under both > > `aws/operators/` and `gcp/ope

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

On Mon, Sep 23, 2019 at 7:04 PM Chris Palmer wrote: > > Is there a reason why we can't use symlinks to have copies of the files > show up in both subpackages? So that `gcs_to_s3.py` would be under both > `aws/operators/` and `gcp/operators`. I could imagine there may be > technical reasons why thi

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Is there a reason why we can't use symlinks to have copies of the files show up in both subpackages? So that `gcs_to_s3.py` would be under both `aws/operators/` and `gcp/operators`. I could imagine there may be technical reasons why this is a bad idea, but just thought I would ask. If that is not

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

I'm not really in favor of the cross_transfer package. It sounds really technical, and if you're new to the project, I would not know what to expect in this package. We had something related in the past with Kubernetes, we solved like this: https://github.com/apache/airflow/blob/master/airflow/con

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

I also think that transfer operators should be put in origin package. Maybe it is also worth to consider to make import available in “destination” for example by import? This would make it easier for user to find a right operator. T. On Sat, 21 Sep 2019 at 22:04, Felix Uellendall wrote: > In

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

In my opinion the source of the transfer operation is what matters - without knowing the source you don‘t need to know where it can be transferred to. So I prefer to put those „cross transfer“ operators to its source. For example: GoogleApiToS3 -> gcp Best Regards, Felix Sent from ProtonMail M

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

I like putting them with source. On Sat, Sep 21, 2019, 12:57 PM Philippe Gagnon wrote: > Another way to resolve this could be to define a convention such that a > "cross transfer" operator should belong to either the source or destination > cloud provider's package. > > Personally though I do no

Re: AIP-21 (Move operators to Core) - "cross_transfer" packages

Another way to resolve this could be to define a convention such that a "cross transfer" operator should belong to either the source or destination cloud provider's package. Personally though I do not see any technical issues with having a "cross_transfer" package, but I don't find the name to rol

AIP-21 (Move operators to Core) - "cross_transfer" packages

I have a question: Should we put all transfer operators between into separate "cross_transfer" package ? *Context:* We had one unresolved point when we decided about AIP-21 - where to put transfer operators between service providers. In the middle of implementing it, it turned out that we need to