I really like the idea, too. It makes the users search for transfers a lot easier.
Best Regards, Felix Sent from ProtonMail Mobile On Sat, May 30, 2020 at 15:59, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > I definitely like the idea - if we can discussion and voting on it before > we prepare backport package final (???) RC candidate, that would be great. > > On Sat, May 30, 2020 at 3:31 PM Tomasz Urbaszek <turbas...@apache.org> > wrote: > >> Oh, makes even more sense. I like the idea. >> >> T. >> >> On Sat, May 30, 2020 at 3:14 PM Kamil Breguła <kamil.breg...@polidea.com> >> wrote: >> >> > I don't want to create package >> > airflow.providers.transfer >> > but package >> > airflow.providers.*.transfer >> > All classes will be in the same provider package but in a different >> nested >> > package. You will need to import packages from the transfer package, not >> > the operator package. >> > >> > >> > On Sat, May 30, 2020, 15:05 Tomasz Urbaszek <turbas...@apache.org> >> wrote: >> > >> > > I am in favor of this change. However, do I correctly understand that >> > > providers.transfer will need many other providers (GCS=gcp, S3=aws, >> > > etc)? >> > > >> > > T. >> > > >> > > >> > > On Sat, May 30, 2020 at 2:28 PM Kamil Breguła < >> kamil.breg...@polidea.com >> > > >> > > wrote: >> > > > >> > > > Hello, >> > > > >> > > > We currently have transfer operators and other operators in one >> > > > package - operators. >> > > > This causes some packages to be large and will grow all the time. For >> > > > example: airflow.providers.google.cloud.operators have 52 modules >> > > > For this reason, the idea arose to move these operators to the new >> > > package. >> > > > As a result, we will have the following provider package structure. >> > > > >> > > > airflow.providers.*.example_dags >> > > > airflow.providers.*.hooks >> > > > airflow.providers.*.operators >> > > > airflow.providers.*.secrets >> > > > airflow.providers.*.sensors >> > > > airflow.providers.*.transfer >> > > > airflow.providers.*.utils >> > > > >> > > > Such a division already exists in the documentation. We distinguish >> > > > operators, transfer operators, and sensors. >> > > > https://airflow.readthedocs.io/en/latest/_api/index.html#operators >> > > > >> https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html >> > > > In my opinion, this change has many advantages: >> > > > * the documentation and the user habits would more suit the structure >> > > > of the code, >> > > > * would facilitate browsing of operators, >> > > > * would allow more comfortable documentation generation based on the >> > > > directory structure. It cannot be determined now whether >> > > > cloud_speech_to_text is transfer operators or a regular operator >> based >> > > > on the module name. >> > > > >> > > > The only minus is the need to change users' habits. Users will have >> to >> > > > start looking for operators in the new package, but if they find this >> > > > package, they will find the module faster. >> > > > >> > > > Best regards, >> > > > Kamil >> > > >> > >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/>
