I'm not really in favor of the cross_transfer package. It sounds really technical, and if you're new to the project, I would not know what to expect in this package.
We had something related in the past with Kubernetes, we solved like this: https://github.com/apache/airflow/blob/master/airflow/contrib/example_dags/example_kubernetes_operator.py#L66-L69 But this isn't really nice of course. When doing the initdb the examples shouldn't be loaded in my opinion, this would solve the whole issue. WDYT? Cheers, Fokko Op za 21 sep. 2019 om 22:53 schreef Tomasz Urbaszek < [email protected]>: > I also think that transfer operators should be put in origin package. Maybe > it is also worth to consider to make import available in “destination” for > example by import? This would make it easier for user to find a right > operator. > > T. > > > On Sat, 21 Sep 2019 at 22:04, Felix Uellendall <[email protected]> > wrote: > > > In my opinion the source of the transfer operation is what matters - > > without knowing the source you don‘t need to know where it can be > > transferred to. > > > > So I prefer to put those „cross transfer“ operators to its source. For > > example: GoogleApiToS3 -> gcp > > > > Best Regards, > > Felix > > > > Sent from ProtonMail Mobile > > > > On Sat, Sep 21, 2019 at 21:52, Jarek Potiuk <[email protected]> > > wrote: > > > > > I have a question: Should we put all transfer operators between into > > > separate "cross_transfer" package ? > > > > > > *Context:* > > > > > > We had one unresolved point when we decided about AIP-21 - where to put > > > transfer operators between service providers. In the middle of > > implementing > > > it, it turned out that we need to make some decisions as it has some > > > undesirable side effects if we just move the transfer operators to core > > > without any structure. Detailed discussion in this PR: > > > https://github.com/apache/airflow/pull/6147 > > > > > > We can solve it easily by choosing "cross_transfer" package for all > > > transfer operators that are crossing "service provider" boundary. > > > > > > This way we will have "gcp" (or maybe even "alphabet" soon), "aws", > > "azure" > > > etc. and "cross_transfer" for all the S3->GCP, AWS->S3 etc. > > > > > > What do you think? Anyone strongly against this? Or maybe we can follow > > > lazy consensus rule for this? Or maybe someone can come up with a > better > > > name :) ? > > > > > > J. > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > -- > > Tomasz Urbaszek > Polidea <https://www.polidea.com/> | Junior Software Engineer > > M: +48 505 628 493 <+48505628493> > E: [email protected] <[email protected]> > > Unique Tech > Check out our projects! <https://www.polidea.com/our-work> >
