I also vote [1]: S3ToHiveOperator Since there is no such concept on Airflow about transfer operators, we better not introduce extra complexity with a potential generic `Transfer` entity.
Here are some personal thoughts: - there are only `import *.operators.*` and `*.sensors.*` in the codebase - `Transfer` is just a subclass of `.operators` - the implementation of a transfer is rather similar to an operator - do something in the `.execute` as opposed to returning True/False in the `.poke` Another thing worth mentioning is that I saw that from different places there was a misconception that there are 3 types of operators: Operators, Sensors, and Transfers. So whatever the final decision is, we need to communicate this more clearly in the doc https://airflow.apache.org/docs/stable/concepts.html#operators On Fri, May 29, 2020 at 10:43 AM Daniel Standish <dpstand...@gmail.com> wrote: > I also vote [1]: S3ToHiveOperator > Transfer is redundant. > And why not 3 --- as of now there is no such distinction and I am not > convinced it is justified. As of now, this is just another operator. > Probably most operators do some kind of "transferring" of data and trying > to decide what is transfer and what is not at this time would be rather > squishy. > > On Fri, May 29, 2020 at 9:11 AM Samuel Tu <samue...@gmail.com> wrote: > > > I am in favor of [2] as some operators can handle operations such as > > branching. > > > > Thanks > > Sam > > > > > On May 29, 2020, at 9:00 AM, Jarek Potiuk <jarek.pot...@polidea.com> > > wrote: > > > > > > Hello everyone > > > > > > We had a discussion in Slack which turned out that we have yet another > > > opportunity to name the operators/hooks a bit more consistently. Seems > > that > > > we did not have any rule on how to name Transfer operators and we have > > > different conventions already. > > > > > > Explanation those are the two examples of conventions we have for > > transfer > > > operators: > > > > > > [1] *S3ToHiveOperator* > > > [2] *S3ToHiveTransferOperator* > > > [3] *S3ToHiveTransfer* > > > [4] We do not care about consistency > > > > > > Some initial comments that I gathered from the discussions:: > > > > > > - Why [1] and not [2,3]: "To" and "Transfer" seem a bit redundant > > > - Why [2] and not [1,3]: Longest, but most descriptive. It's easy to > see > > > that it's an Operator, but you get the Transfer purpose as well. > > sometimes > > > when we use Acronyms (S3ToGCS :) ) it's hard to distinguish "To" from > the > > > acronyms. > > > - Why [1, 2] and not [3]: All Operators (but not Sensor) end with > > "Operator" > > > - Why [3]: and not [1,2]: To introduce distinction: "Sensor", > "Operator", > > > so maybe "Transfer" should be another "entity" and in the future, we > > might > > > implement a more generic Transfer approach > > > > > > I will let the discussion run till the end of today and cast a formal > > vote > > > afterwards > > > > > > I do not yet cancel Backport RC3, because I am not sure if this is > > > something we might want to do - maybe after discussion we decide we > leave > > > the status quo. > > > > > > Discussion in slack here: > > > > > > https://apache-airflow.slack.com/archives/CCPRP7943/p1590746507407100?thread_ts=1590742848.402600&cid=CCPRP7943 > > > > > > > > > J. > > > > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > >
