I am in favor of the idea and have a BaseTransferOperator similar to BaseSensorOperator. There was a PR at some point to implement a common interface for this, will check it once I am back to Laptoo.
Although this should not block the release of Backport Packages and can happen in the near future. Once we have that Base Operator, we can definitely have providers.*.transfer. Regards, Kaxil On Sat, May 30, 2020, 17:31 Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Well. Sensors are based on BaseSensorOperator which is based on > BaseOperator. So they are technically also operators (just > grand-children). In the future maybe we will have BaseTransferOperator > as well (as per our discussion in Slack). I think this might make > sense to have transfer at the same level as operator and sensor. > > In many talks we already speak about those (Sensor/Transfer/Operators) > as different kinds of beasts so having packages at the same level > makes more sense IMHO.. > > J. > > > . > > > On Sat, May 30, 2020 at 5:54 PM Ash Berlin-Taylor <a...@apache.org> wrote: > > > > I understand what you mean, and some separation could make sense, but > transfer operators are still operators (unlike sensors which are a > different interface/base class). > > > > How would people feel about airflow.providers.*. operators.transfer.X? > > > > On 30 May 2020 15:36:42 BST, Felix Uellendall <felue...@pm.me.INVALID> > wrote: > > >I really like the idea, too. It makes the users search for transfers a > > >lot easier. > > > > > >Best Regards, > > >Felix > > > > > >Sent from ProtonMail Mobile > > > > > >On Sat, May 30, 2020 at 15:59, Jarek Potiuk <jarek.pot...@polidea.com> > > >wrote: > > > > > >> I definitely like the idea - if we can discussion and voting on it > > >before > > >> we prepare backport package final (???) RC candidate, that would be > > >great. > > >> > > >> On Sat, May 30, 2020 at 3:31 PM Tomasz Urbaszek > > ><turbas...@apache.org> > > >> wrote: > > >> > > >>> Oh, makes even more sense. I like the idea. > > >>> > > >>> T. > > >>> > > >>> On Sat, May 30, 2020 at 3:14 PM Kamil Breguła > > ><kamil.breg...@polidea.com> > > >>> wrote: > > >>> > > >>> > I don't want to create package > > >>> > airflow.providers.transfer > > >>> > but package > > >>> > airflow.providers.*.transfer > > >>> > All classes will be in the same provider package but in a > > >different > > >>> nested > > >>> > package. You will need to import packages from the transfer > > >package, not > > >>> > the operator package. > > >>> > > > >>> > > > >>> > On Sat, May 30, 2020, 15:05 Tomasz Urbaszek <turbas...@apache.org> > > >>> wrote: > > >>> > > > >>> > > I am in favor of this change. However, do I correctly understand > > >that > > >>> > > providers.transfer will need many other providers (GCS=gcp, > > >S3=aws, > > >>> > > etc)? > > >>> > > > > >>> > > T. > > >>> > > > > >>> > > > > >>> > > On Sat, May 30, 2020 at 2:28 PM Kamil Breguła < > > >>> kamil.breg...@polidea.com > > >>> > > > > >>> > > wrote: > > >>> > > > > > >>> > > > Hello, > > >>> > > > > > >>> > > > We currently have transfer operators and other operators in > > >one > > >>> > > > package - operators. > > >>> > > > This causes some packages to be large and will grow all the > > >time. For > > >>> > > > example: airflow.providers.google.cloud.operators have 52 > > >modules > > >>> > > > For this reason, the idea arose to move these operators to the > > >new > > >>> > > package. > > >>> > > > As a result, we will have the following provider package > > >structure. > > >>> > > > > > >>> > > > airflow.providers.*.example_dags > > >>> > > > airflow.providers.*.hooks > > >>> > > > airflow.providers.*.operators > > >>> > > > airflow.providers.*.secrets > > >>> > > > airflow.providers.*.sensors > > >>> > > > airflow.providers.*.transfer > > >>> > > > airflow.providers.*.utils > > >>> > > > > > >>> > > > Such a division already exists in the documentation. We > > >distinguish > > >>> > > > operators, transfer operators, and sensors. > > >>> > > > > > >https://airflow.readthedocs.io/en/latest/_api/index.html#operators > > >>> > > > > > >>> > > >https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html > > >>> > > > In my opinion, this change has many advantages: > > >>> > > > * the documentation and the user habits would more suit the > > >structure > > >>> > > > of the code, > > >>> > > > * would facilitate browsing of operators, > > >>> > > > * would allow more comfortable documentation generation based > > >on the > > >>> > > > directory structure. It cannot be determined now whether > > >>> > > > cloud_speech_to_text is transfer operators or a regular > > >operator > > >>> based > > >>> > > > on the module name. > > >>> > > > > > >>> > > > The only minus is the need to change users' habits. Users will > > >have > > >>> to > > >>> > > > start looking for operators in the new package, but if they > > >find this > > >>> > > > package, they will find the module faster. > > >>> > > > > > >>> > > > Best regards, > > >>> > > > Kamil > > >>> > > > > >>> > > > >>> > > >> > > >> -- > > >> > > >> Jarek Potiuk > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >> > > >> M: +48 660 796 129 <+48660796129> > > >> [image: Polidea] <https://www.polidea.com/> > > > > -- > > Jarek Potiuk > Polidea | Principal Software Engineer > > M: +48 660 796 129 >
