Side comment: In the near future (AIP-26 ??) we can also adopt the structure reflected in the current documentation restructuring by Kamil
https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html# - Fundamentals <https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#fundamentals> - ASF: Apache Software Foundation <https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#asf-apache-software-foundation> - Service integrations <https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#service-integrations> - Software integrations <https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#software-integrations> - Protocol integrations <https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#protocol-integrations> Providers: - Azure: Microsoft Azure <https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#azure-microsoft-azure> - AWS: Amazon Web Services <https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#aws-amazon-web-services> - GCP: Google Cloud Platform <https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#gcp-google-cloud-platform> This will make it much cleaner on the airflow.* package level. J. On Fri, Oct 4, 2019 at 9:05 PM Driesprong, Fokko <[email protected]> wrote: > I agree with Jarek's suggestion to add another module. Personally I don't > like having too many on the airflow._ level, there are quite a few already. > > Also, I would also go for providers. Mostly because we have hook*s* and > operator*s*. When there are a lot of {operators,hooks,sensors} it makes > sense to package these in a separate module. I could imagine that other > companies that provide managed services > < > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#service-integrations > > > might > evolve to a separate package, but maybe we should come up with some > criteria. > > Cheers, Fokko > > Op vr 4 okt. 2019 om 19:00 schreef Jarek Potiuk <[email protected] > >: > > > Not that I want to open Pandora's box :). But I think the longer name - > the > > worse. Provider is quite nice and short. > > > > I agree with plural 'providers' (as we have hooks and operators). So for > > consistency it should be indeed plural. > > > > J. > > > > pt., 4 paź 2019, 18:48 użytkownik Chris Palmer <[email protected]> > > napisał: > > > > > My question about Oracle/MySql wasn't a serious one, but I forget > > sometimes > > > that sarcasm doesn't come across well on email. > > > > > > I guess my objection is that I don't think that 'provider' adds > anything > > of > > > value. I'm not convinced that there needs to be a level between > 'airflow' > > > and 'google' but if going that route I would advocate for at least the > > > plural form 'providers' or the more descriptive 'cloud_providers'. > > > > > > Chris > > > > > > On Fri, Oct 4, 2019 at 11:57 AM Kamil Breguła < > [email protected] > > > > > > wrote: > > > > > > > Hello, > > > > > > > > We think that only cloud providers should be separated from others, > > > > because these services are integrated with each other. Very often, > > > > when you use one cloud provider, you use many services of a given > > > > provider. Using a single provider solution provides a uniform way of > > > > authorization, etc. A large number of problems and mechanisms are > > > > common to one provider. You can see the amount of integration from > > > > Microsoft, Azure, Google on the reference list. > > > > > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html > > > > In this list, cloud reference providers have been placed in separate > > > > tables because they have a very large number of services. If Oracle > > > > will have many integrations, it is worth emphasizing this fact and > > > > moving these integrations to a separate place, so that it is easier > to > > > > find them. and use. Keeping all possible files in one place makes it > > > > very difficult to use them. > > > > > > > > Best regards, > > > > > > > > On Fri, Oct 4, 2019 at 5:32 PM Chris Palmer <[email protected]> > > wrote: > > > > > > > > > > This seems unnecessary to me. > > > > > > > > > > Is everything going to be under some 'provider' or just certain > sets > > of > > > > > operators, and if so what differentiates when something should be > > > under a > > > > > provider or not? For example, are the mysql operators going to go > > under > > > > > 'provider/oracle/'? > > > > > > > > > > Chris > > > > > > > > > > On Fri, Oct 4, 2019 at 9:21 AM Jarek Potiuk < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Agree with Ash. > > > > > > > > > > > > After doing the gcp move and seeing the result we agreed that > > > > 'provider' is > > > > > > better as additional prefix. > > > > > > > > > > > > If no-one objects (Lazy Consensus > > > > > > <https://community.apache.org/committers/lazyConsensus.html>) > till > > > > Monday > > > > > > 3.20 CEST, we will update AIP-21 and move the gcp operators to > > > > > > *provider/google/[gcp,gsuite]*. > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
