I definitely like the idea - if we can discussion and voting on it before we prepare backport package final (???) RC candidate, that would be great.
On Sat, May 30, 2020 at 3:31 PM Tomasz Urbaszek <turbas...@apache.org> wrote: > Oh, makes even more sense. I like the idea. > > T. > > On Sat, May 30, 2020 at 3:14 PM Kamil Breguła <kamil.breg...@polidea.com> > wrote: > > > I don't want to create package > > airflow.providers.transfer > > but package > > airflow.providers.*.transfer > > All classes will be in the same provider package but in a different > nested > > package. You will need to import packages from the transfer package, not > > the operator package. > > > > > > On Sat, May 30, 2020, 15:05 Tomasz Urbaszek <turbas...@apache.org> > wrote: > > > > > I am in favor of this change. However, do I correctly understand that > > > providers.transfer will need many other providers (GCS=gcp, S3=aws, > > > etc)? > > > > > > T. > > > > > > > > > On Sat, May 30, 2020 at 2:28 PM Kamil Breguła < > kamil.breg...@polidea.com > > > > > > wrote: > > > > > > > > Hello, > > > > > > > > We currently have transfer operators and other operators in one > > > > package - operators. > > > > This causes some packages to be large and will grow all the time. For > > > > example: airflow.providers.google.cloud.operators have 52 modules > > > > For this reason, the idea arose to move these operators to the new > > > package. > > > > As a result, we will have the following provider package structure. > > > > > > > > airflow.providers.*.example_dags > > > > airflow.providers.*.hooks > > > > airflow.providers.*.operators > > > > airflow.providers.*.secrets > > > > airflow.providers.*.sensors > > > > airflow.providers.*.transfer > > > > airflow.providers.*.utils > > > > > > > > Such a division already exists in the documentation. We distinguish > > > > operators, transfer operators, and sensors. > > > > https://airflow.readthedocs.io/en/latest/_api/index.html#operators > > > > > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html > > > > In my opinion, this change has many advantages: > > > > * the documentation and the user habits would more suit the structure > > > > of the code, > > > > * would facilitate browsing of operators, > > > > * would allow more comfortable documentation generation based on the > > > > directory structure. It cannot be determined now whether > > > > cloud_speech_to_text is transfer operators or a regular operator > based > > > > on the module name. > > > > > > > > The only minus is the need to change users' habits. Users will have > to > > > > start looking for operators in the new package, but if they find this > > > > package, they will find the module faster. > > > > > > > > Best regards, > > > > Kamil > > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
