1 looks nice and clean Op za 30 mei 2020 om 11:37 schreef Jarek Potiuk <jarek.pot...@polidea.com>
> That's a completely new thing/idea. > > We might have separate vote for that. Kamil - can you please lead the vote > and renaming if it happens? > > I am for the change. I think it would make sense to separate them. > > I wonder what others think. > > J. > > > On Sat, May 30, 2020 at 11:11 AM Kamil Breguła <kamil.breg...@polidea.com> > wrote: > > > I would like to move transfer operators to the new package - transfer > > airflow.providers.*.transfer.* > > airflow.providers.*.operator.* > > airflow.providers.*.sensor.* > > > > Transfer operators are very often separated in the documentation and it > > would be helpful if this division was included in the code. > > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html > > > > > > > > > > > > > > > > On Fri, May 29, 2020, 21:27 Xinbin Huang <bin.huan...@gmail.com> wrote: > > > > > I also vote [1]: S3ToHiveOperator > > > > > > Since there is no such concept on Airflow about transfer operators, we > > > better not introduce extra complexity with a potential generic > `Transfer` > > > entity. > > > > > > Here are some personal thoughts: > > > - there are only `import *.operators.*` and `*.sensors.*` in the > > codebase > > > - `Transfer` is just a subclass of `.operators` > > > - the implementation of a transfer is rather similar to an operator - > do > > > something in the `.execute` as opposed to returning True/False in the > > > `.poke` > > > > > > Another thing worth mentioning is that I saw that from different > places > > > there was a misconception that there are 3 types of operators: > Operators, > > > Sensors, and Transfers. > > > So whatever the final decision is, we need to communicate this more > > clearly > > > in the doc > > https://airflow.apache.org/docs/stable/concepts.html#operators > > > > > > On Fri, May 29, 2020 at 10:43 AM Daniel Standish <dpstand...@gmail.com > > > > > wrote: > > > > > > > I also vote [1]: S3ToHiveOperator > > > > Transfer is redundant. > > > > And why not 3 --- as of now there is no such distinction and I am not > > > > convinced it is justified. As of now, this is just another operator. > > > > Probably most operators do some kind of "transferring" of data and > > trying > > > > to decide what is transfer and what is not at this time would be > rather > > > > squishy. > > > > > > > > On Fri, May 29, 2020 at 9:11 AM Samuel Tu <samue...@gmail.com> > wrote: > > > > > > > > > I am in favor of [2] as some operators can handle operations such > as > > > > > branching. > > > > > > > > > > Thanks > > > > > Sam > > > > > > > > > > > On May 29, 2020, at 9:00 AM, Jarek Potiuk < > > jarek.pot...@polidea.com> > > > > > wrote: > > > > > > > > > > > > Hello everyone > > > > > > > > > > > > We had a discussion in Slack which turned out that we have yet > > > another > > > > > > opportunity to name the operators/hooks a bit more consistently. > > > Seems > > > > > that > > > > > > we did not have any rule on how to name Transfer operators and we > > > have > > > > > > different conventions already. > > > > > > > > > > > > Explanation those are the two examples of conventions we have for > > > > > transfer > > > > > > operators: > > > > > > > > > > > > [1] *S3ToHiveOperator* > > > > > > [2] *S3ToHiveTransferOperator* > > > > > > [3] *S3ToHiveTransfer* > > > > > > [4] We do not care about consistency > > > > > > > > > > > > Some initial comments that I gathered from the discussions:: > > > > > > > > > > > > - Why [1] and not [2,3]: "To" and "Transfer" seem a bit redundant > > > > > > - Why [2] and not [1,3]: Longest, but most descriptive. It's easy > > to > > > > see > > > > > > that it's an Operator, but you get the Transfer purpose as well. > > > > > sometimes > > > > > > when we use Acronyms (S3ToGCS :) ) it's hard to distinguish "To" > > from > > > > the > > > > > > acronyms. > > > > > > - Why [1, 2] and not [3]: All Operators (but not Sensor) end with > > > > > "Operator" > > > > > > - Why [3]: and not [1,2]: To introduce distinction: "Sensor", > > > > "Operator", > > > > > > so maybe "Transfer" should be another "entity" and in the future, > > we > > > > > might > > > > > > implement a more generic Transfer approach > > > > > > > > > > > > I will let the discussion run till the end of today and cast a > > formal > > > > > vote > > > > > > afterwards > > > > > > > > > > > > I do not yet cancel Backport RC3, because I am not sure if this > is > > > > > > something we might want to do - maybe after discussion we decide > we > > > > leave > > > > > > the status quo. > > > > > > > > > > > > Discussion in slack here: > > > > > > > > > > > > > > > > > > > > > https://apache-airflow.slack.com/archives/CCPRP7943/p1590746507407100?thread_ts=1590742848.402600&cid=CCPRP7943 > > > > > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Jarek Potiuk > > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
