#1 for me, to me the “To” in the name already implies a transfer between two systems.
Bas > On 29 May 2020, at 21:27, Xinbin Huang <bin.huan...@gmail.com> wrote: > > I also vote [1]: S3ToHiveOperator > > Since there is no such concept on Airflow about transfer operators, we > better not introduce extra complexity with a potential generic `Transfer` > entity. > > Here are some personal thoughts: > - there are only `import *.operators.*` and `*.sensors.*` in the codebase > - `Transfer` is just a subclass of `.operators` > - the implementation of a transfer is rather similar to an operator - do > something in the `.execute` as opposed to returning True/False in the > `.poke` > > Another thing worth mentioning is that I saw that from different places > there was a misconception that there are 3 types of operators: Operators, > Sensors, and Transfers. > So whatever the final decision is, we need to communicate this more clearly > in the doc https://airflow.apache.org/docs/stable/concepts.html#operators > > On Fri, May 29, 2020 at 10:43 AM Daniel Standish <dpstand...@gmail.com> > wrote: > >> I also vote [1]: S3ToHiveOperator >> Transfer is redundant. >> And why not 3 --- as of now there is no such distinction and I am not >> convinced it is justified. As of now, this is just another operator. >> Probably most operators do some kind of "transferring" of data and trying >> to decide what is transfer and what is not at this time would be rather >> squishy. >> >> On Fri, May 29, 2020 at 9:11 AM Samuel Tu <samue...@gmail.com> wrote: >> >>> I am in favor of [2] as some operators can handle operations such as >>> branching. >>> >>> Thanks >>> Sam >>> >>>> On May 29, 2020, at 9:00 AM, Jarek Potiuk <jarek.pot...@polidea.com> >>> wrote: >>>> >>>> Hello everyone >>>> >>>> We had a discussion in Slack which turned out that we have yet another >>>> opportunity to name the operators/hooks a bit more consistently. Seems >>> that >>>> we did not have any rule on how to name Transfer operators and we have >>>> different conventions already. >>>> >>>> Explanation those are the two examples of conventions we have for >>> transfer >>>> operators: >>>> >>>> [1] *S3ToHiveOperator* >>>> [2] *S3ToHiveTransferOperator* >>>> [3] *S3ToHiveTransfer* >>>> [4] We do not care about consistency >>>> >>>> Some initial comments that I gathered from the discussions:: >>>> >>>> - Why [1] and not [2,3]: "To" and "Transfer" seem a bit redundant >>>> - Why [2] and not [1,3]: Longest, but most descriptive. It's easy to >> see >>>> that it's an Operator, but you get the Transfer purpose as well. >>> sometimes >>>> when we use Acronyms (S3ToGCS :) ) it's hard to distinguish "To" from >> the >>>> acronyms. >>>> - Why [1, 2] and not [3]: All Operators (but not Sensor) end with >>> "Operator" >>>> - Why [3]: and not [1,2]: To introduce distinction: "Sensor", >> "Operator", >>>> so maybe "Transfer" should be another "entity" and in the future, we >>> might >>>> implement a more generic Transfer approach >>>> >>>> I will let the discussion run till the end of today and cast a formal >>> vote >>>> afterwards >>>> >>>> I do not yet cancel Backport RC3, because I am not sure if this is >>>> something we might want to do - maybe after discussion we decide we >> leave >>>> the status quo. >>>> >>>> Discussion in slack here: >>>> >>> >> https://apache-airflow.slack.com/archives/CCPRP7943/p1590746507407100?thread_ts=1590742848.402600&cid=CCPRP7943 >>>> >>>> >>>> J. >>>> >>>> >>>> -- >>>> >>>> Jarek Potiuk >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>> >>>> M: +48 660 796 129 <+48660796129> >>>> [image: Polidea] <https://www.polidea.com/> >>> >>
