Ah https://airflow.apache.org/docs/1.10.10/_api/index.html#operators -- I haven't looked at that content closely in a long while.
On May 30 2020, at 5:40 pm, Ash Berlin-Taylor <a...@apache.org> wrote: > Oh really, can you point at any talks? I've not seen them referrred to > as a first class concept before. > > (Sensors are also different because they can be rescheduled - > TransferOperators aren't currently in any way more special to Airflow > than any other operator.) > > -a > > On May 30 2020, at 5:31 pm, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > >> Well. Sensors are based on BaseSensorOperator which is based on >> BaseOperator. So they are technically also operators (just >> grand-children). In the future maybe we will have BaseTransferOperator >> as well (as per our discussion in Slack). I think this might make >> sense to have transfer at the same level as operator and sensor. >> >> In many talks we already speak about those (Sensor/Transfer/Operators) >> as different kinds of beasts so having packages at the same level >> makes more sense IMHO.. >> >> J. >> >> >> . >> >> >> On Sat, May 30, 2020 at 5:54 PM Ash Berlin-Taylor <a...@apache.org> wrote: >>> >>> I understand what you mean, and some separation could make sense, but >>> transfer operators are still operators (unlike sensors which are a >>> different interface/base class). >>> >>> How would people feel about airflow.providers.*. operators.transfer.X? >>> >>> On 30 May 2020 15:36:42 BST, Felix Uellendall >>> <felue...@pm.me.INVALID> wrote: >>> >I really like the idea, too. It makes the users search for >>> transfers a >>> >lot easier. >>> > >>> >Best Regards, >>> >Felix >>> > >>> >Sent from ProtonMail Mobile >>> > >>> >On Sat, May 30, 2020 at 15:59, Jarek Potiuk <jarek.pot...@polidea.com> >>> >wrote: >>> > >>> >> I definitely like the idea - if we can discussion and voting on it >>> >before >>> >> we prepare backport package final (???) RC candidate, that would be >>> >great. >>> >> >>> >> On Sat, May 30, 2020 at 3:31 PM Tomasz Urbaszek >>> ><turbas...@apache.org> >>> >> wrote: >>> >> >>> >>> Oh, makes even more sense. I like the idea. >>> >>> >>> >>> T. >>> >>> >>> >>> On Sat, May 30, 2020 at 3:14 PM Kamil Breguła >>> ><kamil.breg...@polidea.com> >>> >>> wrote: >>> >>> >>> >>> > I don't want to create package >>> >>> > airflow.providers.transfer >>> >>> > but package >>> >>> > airflow.providers.*.transfer >>> >>> > All classes will be in the same provider package but in a >>> >different >>> >>> nested >>> >>> > package. You will need to import packages from the transfer >>> >package, not >>> >>> > the operator package. >>> >>> > >>> >>> > >>> >>> > On Sat, May 30, 2020, 15:05 Tomasz Urbaszek <turbas...@apache.org> >>> >>> wrote: >>> >>> > >>> >>> > > I am in favor of this change. However, do I correctly understand >>> >that >>> >>> > > providers.transfer will need many other providers (GCS=gcp, >>> >S3=aws, >>> >>> > > etc)? >>> >>> > > >>> >>> > > T. >>> >>> > > >>> >>> > > >>> >>> > > On Sat, May 30, 2020 at 2:28 PM Kamil Breguła < >>> >>> kamil.breg...@polidea.com >>> >>> > > >>> >>> > > wrote: >>> >>> > > > >>> >>> > > > Hello, >>> >>> > > > >>> >>> > > > We currently have transfer operators and other operators in >>> >one >>> >>> > > > package - operators. >>> >>> > > > This causes some packages to be large and will grow all the >>> >time. For >>> >>> > > > example: airflow.providers.google.cloud.operators have 52 >>> >modules >>> >>> > > > For this reason, the idea arose to move these operators to the >>> >new >>> >>> > > package. >>> >>> > > > As a result, we will have the following provider package >>> >structure. >>> >>> > > > >>> >>> > > > airflow.providers.*.example_dags >>> >>> > > > airflow.providers.*.hooks >>> >>> > > > airflow.providers.*.operators >>> >>> > > > airflow.providers.*.secrets >>> >>> > > > airflow.providers.*.sensors >>> >>> > > > airflow.providers.*.transfer >>> >>> > > > airflow.providers.*.utils >>> >>> > > > >>> >>> > > > Such a division already exists in the documentation. We >>> >distinguish >>> >>> > > > operators, transfer operators, and sensors. >>> >>> > > > >>> >https://airflow.readthedocs.io/en/latest/_api/index.html#operators >>> >>> > > > >>> >>> >>> >https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html >>> >>> > > > In my opinion, this change has many advantages: >>> >>> > > > * the documentation and the user habits would more suit the >>> >structure >>> >>> > > > of the code, >>> >>> > > > * would facilitate browsing of operators, >>> >>> > > > * would allow more comfortable documentation generation based >>> >on the >>> >>> > > > directory structure. It cannot be determined now whether >>> >>> > > > cloud_speech_to_text is transfer operators or a regular >>> >operator >>> >>> based >>> >>> > > > on the module name. >>> >>> > > > >>> >>> > > > The only minus is the need to change users' habits. Users will >>> >have >>> >>> to >>> >>> > > > start looking for operators in the new package, but if they >>> >find this >>> >>> > > > package, they will find the module faster. >>> >>> > > > >>> >>> > > > Best regards, >>> >>> > > > Kamil >>> >>> > > >>> >>> > >>> >>> >>> >> >>> >> -- >>> >> >>> >> Jarek Potiuk >>> >> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >> >>> >> M: +48 660 796 129 <+48660796129> >>> >> [image: Polidea] <https://www.polidea.com/> >> >> >> >> -- >> >> Jarek Potiuk >> Polidea | Principal Software Engineer >> >> M: +48 660 796 129 >>
