Oh really, can you point at any talks? I've not seen them referrred to as a first class concept before.
(Sensors are also different because they can be rescheduled - TransferOperators aren't currently in any way more special to Airflow than any other operator.) -a On May 30 2020, at 5:31 pm, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Well. Sensors are based on BaseSensorOperator which is based on > BaseOperator. So they are technically also operators (just > grand-children). In the future maybe we will have BaseTransferOperator > as well (as per our discussion in Slack). I think this might make > sense to have transfer at the same level as operator and sensor. > > In many talks we already speak about those (Sensor/Transfer/Operators) > as different kinds of beasts so having packages at the same level > makes more sense IMHO.. > > J. > > > . > > > On Sat, May 30, 2020 at 5:54 PM Ash Berlin-Taylor <a...@apache.org> wrote: >> >> I understand what you mean, and some separation could make sense, but >> transfer operators are still operators (unlike sensors which are a >> different interface/base class). >> >> How would people feel about airflow.providers.*. operators.transfer.X? >> >> On 30 May 2020 15:36:42 BST, Felix Uellendall >> <felue...@pm.me.INVALID> wrote: >> >I really like the idea, too. It makes the users search for transfers a >> >lot easier. >> > >> >Best Regards, >> >Felix >> > >> >Sent from ProtonMail Mobile >> > >> >On Sat, May 30, 2020 at 15:59, Jarek Potiuk <jarek.pot...@polidea.com> >> >wrote: >> > >> >> I definitely like the idea - if we can discussion and voting on it >> >before >> >> we prepare backport package final (???) RC candidate, that would be >> >great. >> >> >> >> On Sat, May 30, 2020 at 3:31 PM Tomasz Urbaszek >> ><turbas...@apache.org> >> >> wrote: >> >> >> >>> Oh, makes even more sense. I like the idea. >> >>> >> >>> T. >> >>> >> >>> On Sat, May 30, 2020 at 3:14 PM Kamil Breguła >> ><kamil.breg...@polidea.com> >> >>> wrote: >> >>> >> >>> > I don't want to create package >> >>> > airflow.providers.transfer >> >>> > but package >> >>> > airflow.providers.*.transfer >> >>> > All classes will be in the same provider package but in a >> >different >> >>> nested >> >>> > package. You will need to import packages from the transfer >> >package, not >> >>> > the operator package. >> >>> > >> >>> > >> >>> > On Sat, May 30, 2020, 15:05 Tomasz Urbaszek <turbas...@apache.org> >> >>> wrote: >> >>> > >> >>> > > I am in favor of this change. However, do I correctly understand >> >that >> >>> > > providers.transfer will need many other providers (GCS=gcp, >> >S3=aws, >> >>> > > etc)? >> >>> > > >> >>> > > T. >> >>> > > >> >>> > > >> >>> > > On Sat, May 30, 2020 at 2:28 PM Kamil Breguła < >> >>> kamil.breg...@polidea.com >> >>> > > >> >>> > > wrote: >> >>> > > > >> >>> > > > Hello, >> >>> > > > >> >>> > > > We currently have transfer operators and other operators in >> >one >> >>> > > > package - operators. >> >>> > > > This causes some packages to be large and will grow all the >> >time. For >> >>> > > > example: airflow.providers.google.cloud.operators have 52 >> >modules >> >>> > > > For this reason, the idea arose to move these operators to the >> >new >> >>> > > package. >> >>> > > > As a result, we will have the following provider package >> >structure. >> >>> > > > >> >>> > > > airflow.providers.*.example_dags >> >>> > > > airflow.providers.*.hooks >> >>> > > > airflow.providers.*.operators >> >>> > > > airflow.providers.*.secrets >> >>> > > > airflow.providers.*.sensors >> >>> > > > airflow.providers.*.transfer >> >>> > > > airflow.providers.*.utils >> >>> > > > >> >>> > > > Such a division already exists in the documentation. We >> >distinguish >> >>> > > > operators, transfer operators, and sensors. >> >>> > > > >> >https://airflow.readthedocs.io/en/latest/_api/index.html#operators >> >>> > > > >> >>> >> >https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html >> >>> > > > In my opinion, this change has many advantages: >> >>> > > > * the documentation and the user habits would more suit the >> >structure >> >>> > > > of the code, >> >>> > > > * would facilitate browsing of operators, >> >>> > > > * would allow more comfortable documentation generation based >> >on the >> >>> > > > directory structure. It cannot be determined now whether >> >>> > > > cloud_speech_to_text is transfer operators or a regular >> >operator >> >>> based >> >>> > > > on the module name. >> >>> > > > >> >>> > > > The only minus is the need to change users' habits. Users will >> >have >> >>> to >> >>> > > > start looking for operators in the new package, but if they >> >find this >> >>> > > > package, they will find the module faster. >> >>> > > > >> >>> > > > Best regards, >> >>> > > > Kamil >> >>> > > >> >>> > >> >>> >> >> >> >> -- >> >> >> >> Jarek Potiuk >> >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> >> >> M: +48 660 796 129 <+48660796129> >> >> [image: Polidea] <https://www.polidea.com/> > > > > -- > > Jarek Potiuk > Polidea | Principal Software Engineer > > M: +48 660 796 129 >
