I don't want to create package airflow.providers.transfer but package airflow.providers.*.transfer All classes will be in the same provider package but in a different nested package. You will need to import packages from the transfer package, not the operator package.
On Sat, May 30, 2020, 15:05 Tomasz Urbaszek <turbas...@apache.org> wrote: > I am in favor of this change. However, do I correctly understand that > providers.transfer will need many other providers (GCS=gcp, S3=aws, > etc)? > > T. > > > On Sat, May 30, 2020 at 2:28 PM Kamil Breguła <kamil.breg...@polidea.com> > wrote: > > > > Hello, > > > > We currently have transfer operators and other operators in one > > package - operators. > > This causes some packages to be large and will grow all the time. For > > example: airflow.providers.google.cloud.operators have 52 modules > > For this reason, the idea arose to move these operators to the new > package. > > As a result, we will have the following provider package structure. > > > > airflow.providers.*.example_dags > > airflow.providers.*.hooks > > airflow.providers.*.operators > > airflow.providers.*.secrets > > airflow.providers.*.sensors > > airflow.providers.*.transfer > > airflow.providers.*.utils > > > > Such a division already exists in the documentation. We distinguish > > operators, transfer operators, and sensors. > > https://airflow.readthedocs.io/en/latest/_api/index.html#operators > > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html > > In my opinion, this change has many advantages: > > * the documentation and the user habits would more suit the structure > > of the code, > > * would facilitate browsing of operators, > > * would allow more comfortable documentation generation based on the > > directory structure. It cannot be determined now whether > > cloud_speech_to_text is transfer operators or a regular operator based > > on the module name. > > > > The only minus is the need to change users' habits. Users will have to > > start looking for operators in the new package, but if they find this > > package, they will find the module faster. > > > > Best regards, > > Kamil >
