Ok. Happy to move it back then :). No problem with that. According to rules of AIP-21 it should actually be: "*from airflow.providers.kubernetes.operators.pod import KubernetesPodOperator*" (Case 2A. (drop _operator in module name) + Case 5B. (keep Operator in class name). We can have more than just a Pod operator for Kubernetes (KubernetesPod, KubernetesVolume, KubernetesIstio. and many more) so keeping KubernetesPod in class name and having separate module for pod operator(s?) makes sense IMHO.
It's similar to *from airflow.providers.google.cloud.operators.pubsub import PubSubTopicCreateOperator* for example. Re - remote log storage - indeed. That should be part of AIP- 8. J, On Mon, Nov 11, 2019 at 6:36 PM Ash Berlin-Taylor <a...@apache.org> wrote: > +1 for Python and Bash being in the stock install -- they are just _so_ > commonly used that I think it makes sense to keep them in the base install. > (and the virtualenv module is not an onerous dep, not caused us any > problems. Yet). > > Kubeneretes is also a slighlty funny one since the deps for that will be > in "core" anyway thanks to the Kube executor, but I think it probably makes > sense to have `from airflow.providers.kubernetes.operators import > KubernetesOperator`. Is that the pattern we are going with for the > "one-level" providers, or will it be `from > airflow.providers.kubernetes.operators.pod_operator import > KubernetesOperator`? > > Possibly more an AIP-8 question: with moving Azure Blob/S3/GCS to separate > packages we might have to look at how we enable remote log storage. > > -a > > > > On 11 Nov 2019, at 15:53, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > > > > On Mon, Nov 11, 2019 at 4:22 PM Kamil Breguła <kamil.breg...@polidea.com > <mailto:kamil.breg...@polidea.com>> > > wrote: > > > >> One more question. Are you sure you want to move Python and Bash from > >> core? These are the elements that are installed in every environment > >> because they are required by Airflow, so moving them to a separate > >> installed package is pointless in my opinion. > >> > >> I have no problem with moving them to "fundamentals", but I am not sure > if > > they are really required ? I looked through the code and other than few > > examples and tests, they are not really "required". Maybe that's enough > to > > keep them in fundamentals, > > Also Python operator has some dependencies - virtualenv - which is only > > required for this operator so maybe it's worth to keep it separate from > > "fundamentals". > > > > > >> On Mon, Nov 11, 2019 at 3:07 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > >>> > >>> I am fine with this list +1 > >>> > >>> On Mon, Nov 11, 2019 at 1:27 PM Jarek Potiuk <jarek.pot...@polidea.com > > > >>> wrote: > >>> > >>>> I am all for it Kamil! > >>>> > >>>> Super happy to treat Apache projects in the same way as "proprietary" > >>>> providers :). Anyone else has some other comments ? > >>>> > >>>> J. > >>>> > >>>> On Mon, Nov 11, 2019 at 2:17 PM Kamil Breguła < > >> kamil.breg...@polidea.com> > >>>> wrote: > >>>> > >>>>> I looked at this list and I'm only worried about two operators. > >>>>> > >>>>> airflow.contrib.operators.vertica_to_hive > >>>>> airflow.contrib.operators.s3_to_hive > >>>>> > >>>>> If we want the operators to be grouped according to destination, then > >>>>> this operator should be in apache package. It is the members of the > >>>>> Apache community who will care most about this operator being of high > >>>>> quality. Apache can be treated equally with other large cloud > >>>>> providers, such as GCP, AWS. I can imagine that a new Apache product > >>>>> will appear and it will want to promote the same way as products of > >>>>> cloud providers are promoted. By creating a large number of > >>>>> integrations that allow you to copy data to its operating range. > >>>>> There's another cases - building a strong Apache community. As a > >>>>> member of the Apache community, we should promote Apache products to > >>>>> ensure that the development of the community is correct, and > >> therefore > >>>>> also for integration into our products with other products. > >>>>> > >>>>> On Mon, Nov 11, 2019 at 12:28 AM Jarek Potiuk < > >> jarek.pot...@polidea.com> > >>>>> wrote: > >>>>>> > >>>>>> Just to select the "packages" for this update. Anyone has > >> objections > >>>> for > >>>>>> this structure (details including transfer operators in > >>>>>> > >>>>>> https://docs.google.com/spreadsheets/d/17zA5t2JVxnDdg5Cs1Cg_ > >>>>>> Mb1GXvGctmesfg2L089QSOk/edit#gid=0? > >>>>>> > >>>>>> *Fundamentals (no change)* > >>>>>> > >>>>>> > >>>>>> > >>>>>> providers > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> google > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> cloud > >>>>>> > >>>>>> > >>>>>> > >>>>>> gsuite > >>>>>> > >>>>>> > >>>>>> > >>>>>> marketing_platform > >>>>>> > >>>>>> > >>>>>> amazon > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> aws > >>>>>> > >>>>>> > >>>>>> microsoft > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> azure > >>>>>> > >>>>>> > >>>>>> apache > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> cassandra > >>>>>> > >>>>>> > >>>>>> > >>>>>> druid > >>>>>> > >>>>>> > >>>>>> > >>>>>> hadoop > >>>>>> > >>>>>> > >>>>>> > >>>>>> hive > >>>>>> > >>>>>> > >>>>>> > >>>>>> pig > >>>>>> > >>>>>> > >>>>>> > >>>>>> pinot > >>>>>> > >>>>>> > >>>>>> > >>>>>> spark > >>>>>> > >>>>>> > >>>>>> > >>>>>> sqoop > >>>>>> > >>>>>> > >>>>>> mysql > >>>>>> > >>>>>> > >>>>>> > >>>>>> jira > >>>>>> > >>>>>> > >>>>>> > >>>>>> databricks > >>>>>> > >>>>>> > >>>>>> > >>>>>> datadog > >>>>>> > >>>>>> > >>>>>> > >>>>>> dingding > >>>>>> > >>>>>> > >>>>>> > >>>>>> discord > >>>>>> > >>>>>> > >>>>>> > >>>>>> cloudant > >>>>>> > >>>>>> > >>>>>> > >>>>>> jenkins > >>>>>> > >>>>>> > >>>>>> > >>>>>> opsgenie > >>>>>> > >>>>>> > >>>>>> > >>>>>> qubole > >>>>>> > >>>>>> > >>>>>> > >>>>>> salesforce > >>>>>> > >>>>>> > >>>>>> > >>>>>> segment > >>>>>> > >>>>>> > >>>>>> > >>>>>> slack > >>>>>> > >>>>>> > >>>>>> > >>>>>> snowflake > >>>>>> > >>>>>> > >>>>>> > >>>>>> vertica > >>>>>> > >>>>>> > >>>>>> > >>>>>> zendesk > >>>>>> > >>>>>> > >>>>>> > >>>>>> celery > >>>>>> > >>>>>> > >>>>>> > >>>>>> docker > >>>>>> > >>>>>> > >>>>>> > >>>>>> bash > >>>>>> > >>>>>> > >>>>>> > >>>>>> kubernetes > >>>>>> > >>>>>> > >>>>>> > >>>>>> mssql > >>>>>> > >>>>>> > >>>>>> > >>>>>> mongodb > >>>>>> > >>>>>> > >>>>>> > >>>>>> mysql > >>>>>> > >>>>>> > >>>>>> > >>>>>> openfaas > >>>>>> > >>>>>> > >>>>>> > >>>>>> oracle > >>>>>> > >>>>>> > >>>>>> > >>>>>> papermill > >>>>>> > >>>>>> > >>>>>> > >>>>>> postgres > >>>>>> > >>>>>> > >>>>>> > >>>>>> presto > >>>>>> > >>>>>> > >>>>>> > >>>>>> python > >>>>>> > >>>>>> > >>>>>> > >>>>>> redis > >>>>>> > >>>>>> > >>>>>> > >>>>>> samba > >>>>>> > >>>>>> > >>>>>> > >>>>>> sqlite > >>>>>> > >>>>>> > >>>>>> > >>>>>> imap > >>>>>> > >>>>>> > >>>>>> > >>>>>> ssh > >>>>>> > >>>>>> > >>>>>> > >>>>>> filesystem > >>>>>> > >>>>>> > >>>>>> > >>>>>> sftp > >>>>>> > >>>>>> > >>>>>> > >>>>>> ftp > >>>>>> > >>>>>> > >>>>>> > >>>>>> http > >>>>>> > >>>>>> > >>>>>> > >>>>>> grpc > >>>>>> > >>>>>> > >>>>>> > >>>>>> smtp > >>>>>> > >>>>>> > >>>>>> > >>>>>> jdbc > >>>>>> > >>>>>> > >>>>>> > >>>>>> winrm > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Fri, Nov 8, 2019 at 5:47 PM Jarek Potiuk < > >> jarek.pot...@polidea.com> > >>>>>> wrote: > >>>>>> > >>>>>>> Let me then cancel this vote and I will restart it next week. > >>>>>>> > >>>>>>> Yeah. It's a bit like re-opening the Pandora's box but now that > >> we > >>>> know > >>>>>>> that we can do it, and we are unblocked in moving to google > >> (which is > >>>>> now > >>>>>>> the biggest move in-progress), we can spend more time on getting > >>>>> better > >>>>>>> (and more final) consensus. > >>>>>>> I decided to go through the list from the docs (once again Kamil > >> - > >>>>> great > >>>>>>> that you did it) and prepared this spreadsheet showing the > >>>> structure. I > >>>>>>> went through ALL the operators and put them in the right place > >> where > >>>>> our > >>>>>>> current rules place them. > >>>>>>> > >>>>>>> After this exercise, I think that makes sense: > >>>>>>> - put all the stuff except fundamentals in *"providers"* > >> (everything > >>>>>>> in "providers" will be potentially backportable). > >>>>>>> - grouping apache projects under *"apache"* - similar to > >>>>>>> google/amazon/microsoft (different kind of ownership but still > >> it is > >>>> an > >>>>>>> ownership) > >>>>>>> - for the rest I think what we can do is really to put the > >> operators > >>>> in > >>>>>>> folders per "service/company" (without sub-packages). That > >> includes > >>>>>>> sftp/ssh/ftp etc (should we group [ftp and sftp] or [ssh and > >> sftp] > >>>> ??). > >>>>>>> there is no "ownership" there and no reason to group them. That > >> will > >>>>> put > >>>>>>> "operators/hooks/sensors" at different levels in the directory > >> tree > >>>>> but we > >>>>>>> already have that for fundamentals and I am not too worried about > >>>>> that. We > >>>>>>> do not have to have everything at the same level. > >>>>>>> - I put transfer operators according to the rule where "to" side > >> is > >>>>> more > >>>>>>> important unless the other side is a public protocol (so sftp -> > >> gcs > >>>>> and > >>>>>>> gcs -> sftp both go to google/gcp). I did not have any doubt > >> where to > >>>>> put > >>>>>>> which transfer operator, so this is a good sign: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://docs.google.com/spreadsheets/d/17zA5t2JVxnDdg5Cs1Cg_Mb1GXvGctmesfg2L089QSOk/edit#gid=0 > >>>>>>> > >>>>>>> Can you please take a look and express your opinions here so > >> that we > >>>>> can > >>>>>>> have final voting next week (for those who are not yet tired > >> with the > >>>>>>> discussion ;)). > >>>>>>> > >>>>>>> J. > >>>>>>> > >>>>>>> On Fri, Nov 8, 2019 at 4:38 PM Kaxil Naik <kaxiln...@gmail.com> > >>>> wrote: > >>>>>>> > >>>>>>>> Yes, that makes sense. > >>>>>>>> > >>>>>>>> On Fri, Nov 8, 2019 at 3:22 PM Kamil Breguła < > >>>>> kamil.breg...@polidea.com> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> In the case of Hadoop, it is published by Apache, so it can > >> be in > >>>>> the > >>>>>>>>> apache directory. This will mimic the grouping presented in > >> the > >>>>>>>>> documentation. > >>>>>>>>> > >>>>>>>> > >>>>> > >>>> > >> > https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#software-operators-and-hooks > >>>>>>>>> > >>>>>>>>> On Fri, Nov 8, 2019 at 3:47 PM Kaxil Naik < > >> kaxiln...@gmail.com> > >>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> I think we should keep the vote open at least until mid next > >>>> week > >>>>> to > >>>>>>>> have > >>>>>>>>>> more thought and inputs on this one. > >>>>>>>>>> > >>>>>>>>>> In general, I am happy with the approach but > >> operators/hooks and > >>>>>>>> sensors > >>>>>>>>>> shouldn't be a provider. "hadoop" can be its provider and > >> hdfs > >>>>> can be > >>>>>>>> a > >>>>>>>>>> part of it. > >>>>>>>>>> > >>>>>>>>>> providers/ > >>>>>>>>>> google > >>>>>>>>>> cloud > >>>>>>>>>> operators > >>>>>>>>>> hooks > >>>>>>>>>> sensors > >>>>>>>>>> gsuite > >>>>>>>>>> operators > >>>>>>>>>> ... > >>>>>>>>>> amazon > >>>>>>>>>> aws > >>>>>>>>>> operators > >>>>>>>>>> ... > >>>>>>>>>> microsoft > >>>>>>>>>> azure > >>>>>>>>>> operators > >>>>>>>>>> ... > >>>>>>>>>> hadoop > >>>>>>>>>> hdfs > >>>>>>>>>> operators > >>>>>>>>>> ... > >>>>>>>>>> > >>>>>>>>>> We can also define what is a "provider" so we know what to > >> add > >>>> in > >>>>> it > >>>>>>>> in > >>>>>>>>> the > >>>>>>>>>> future. SSH/FTP/SFTP belongs to the same family group. Do we > >>>> want > >>>>> to > >>>>>>>> have > >>>>>>>>>> separate providers for each one of them ??? > >>>>>>>>>> > >>>>>>>>>> Regards, > >>>>>>>>>> Kaxil > >>>>>>>>>> > >>>>>>>>>> On Fri, Nov 8, 2019 at 9:08 AM Jarek Potiuk < > >>>>> jarek.pot...@polidea.com > >>>>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> I really like to make everything a provider. That's a > >> great > >>>>> idea ! > >>>>>>>>> This way > >>>>>>>>>>> everything "backportable" will have to be in "providers" > >>>>> package. > >>>>>>>>> Really > >>>>>>>>>>> nice and clean separation (and less mess in "airflow"). > >> And we > >>>>> will > >>>>>>>> not > >>>>>>>>>>> have to have any artificial grouping (we can still group > >> them > >>>>> at the > >>>>>>>>>>> documentation level). > >>>>>>>>>>> > >>>>>>>>>>> We do not need backport in name. And I think it's more of > >>>>> technical > >>>>>>>>> detail > >>>>>>>>>>> on naming the package which we can work out while > >> reviewing > >>>> PRs > >>>>> and > >>>>>>>> we > >>>>>>>>> can > >>>>>>>>>>> agree final naming of the released packaged on PMC level > >> (PMCs > >>>>> will > >>>>>>>>> have to > >>>>>>>>>>> vote on releasing those). > >>>>>>>>>>> > >>>>>>>>>>> The thinking is that it's intention is really to be only > >>>>> backported > >>>>>>>> to > >>>>>>>>> 1.10 > >>>>>>>>>>> - we are not going (yet) to use the packages in Airflow > >> 2.*. > >>>> so > >>>>> I > >>>>>>>>> thought > >>>>>>>>>>> by naming them backport we can express that intent more > >>>> clearly. > >>>>>>>>>>> > >>>>>>>>>>> So let me clarify the structure of folders we are going to > >>>> have > >>>>> if > >>>>>>>> we > >>>>>>>>>>> follow it (i just added some examples) including the > >> already > >>>>> agreed > >>>>>>>>> changes > >>>>>>>>>>> from AIP-21: > >>>>>>>>>>> > >>>>>>>>>>> providers/ > >>>>>>>>>>> google > >>>>>>>>>>> cloud > >>>>>>>>>>> operators > >>>>>>>>>>> hooks > >>>>>>>>>>> sensors > >>>>>>>>>>> gsuite > >>>>>>>>>>> operators > >>>>>>>>>>> ... > >>>>>>>>>>> amazon > >>>>>>>>>>> aws > >>>>>>>>>>> operators > >>>>>>>>>>> ... > >>>>>>>>>>> microsoft > >>>>>>>>>>> azure > >>>>>>>>>>> operators > >>>>>>>>>>> ... > >>>>>>>>>>> operators > >>>>>>>>>>> sqlite.py > >>>>>>>>>>> oracle.py > >>>>>>>>>>> docker.py > >>>>>>>>>>> hooks > >>>>>>>>>>> hdfs.py > >>>>>>>>>>> sqlite.py > >>>>>>>>>>> sensors > >>>>>>>>>>> http.py > >>>>>>>>>>> sql.py > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> J. > >>>>>>>>>>> > >>>>>>>>>>> On Fri, Nov 8, 2019 at 9:43 AM Ash Berlin-Taylor < > >>>>> a...@apache.org> > >>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Do we need to include `-backport,`? What was the > >> thinking > >>>>> behind > >>>>>>>>> that? > >>>>>>>>>>>> > >>>>>>>>>>>> I think software and protocol should be merged. I would > >> also > >>>>> say > >>>>>>>>>>>> _everything_ is a provider, so > >>>>> airflow.providers.ssh.SSHOperator > >>>>>>>> for > >>>>>>>>>>>> instance is what I would prefer > >>>>>>>>>>>> > >>>>>>>>>>>> -a > >>>>>>>>>>>> > >>>>>>>>>>>> On 8 November 2019 08:32:42 GMT, Jarek Potiuk < > >>>>>>>>> jarek.pot...@polidea.com> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>> One more day to go. I would love to see some opinions > >> on > >>>> this > >>>>>>>> AIP-21 > >>>>>>>>>>>>> update > >>>>>>>>>>>>> :). > >>>>>>>>>>>>> > >>>>>>>>>>>>> Executive summary: > >>>>>>>>>>>>> > >>>>>>>>>>>>> * we will be moving a number of integrations to > >>>> sub-packages > >>>>> of > >>>>>>>>>>>>> airflow. > >>>>>>>>>>>>> * they will be backportable to 1.10.*. There will be > >>>>>>>>>>>>> 'apache-airflow-[package]-backport' pypi installable > >> with > >>>>> python > >>>>>>>> 3 > >>>>>>>>> that > >>>>>>>>>>>>> will make Airflow 2.0 operators/hooks etc. available > >> with > >>>>> 1.10* > >>>>>>>>>>>>> operators. > >>>>>>>>>>>>> * the current proposal for sub-packages is > >>>>>>>>>>>>> "protocols/software/providers/" > >>>>>>>>>>>>> (but if you think merging protocols and software makes > >>>> sense > >>>>> - > >>>>>>>>> please > >>>>>>>>>>>>> express your opinion > >>>>>>>>>>>>> * we are not moving "fundamental" operators/hooks etc.. > >>>>>>>>>>>>> * Airflow 2.0 is still going to be installed as a > >> single > >>>>> package > >>>>>>>>> with > >>>>>>>>>>>>> all > >>>>>>>>>>>>> operators (so we are not yet implementing AIP-8) > >>>>>>>>>>>>> > >>>>>>>>>>>>> J. > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Wed, Nov 6, 2019 at 10:07 AM Jarek Potiuk < > >>>>>>>>> jarek.pot...@polidea.com> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> I think all this cases are valid but maybe I was not > >>>>>>>> super-clear. > >>>>>>>>>>>>> It's > >>>>>>>>>>>>>> only the transfer operators that we need to decide > >> where > >>>> to > >>>>>>>> put - > >>>>>>>>> not > >>>>>>>>>>>>>> hooks. > >>>>>>>>>>>>>> Usually the complexity of communication with > >> particular > >>>>>>>> storages > >>>>>>>>> is > >>>>>>>>>>>>> (or at > >>>>>>>>>>>>>> least should be) in the Hooks rather than Operators. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Operators should be just thin wrappers over the > >> logic in > >>>>> the > >>>>>>>>> hooks. > >>>>>>>>>>>>>> Hooks are going to stay where they belong - S3 Hooks > >> in > >>>>> amazon, > >>>>>>>>> GCS > >>>>>>>>>>>>> Hooks > >>>>>>>>>>>>>> in google.cloud, GoogleSheet Hooks in google.gsuite. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Since we actually have mono-repo - this will be no > >>>> problem > >>>>>>>> (and no > >>>>>>>>>>>>> cross > >>>>>>>>>>>>>> dependencies problem) to have S3 -> GCS operator in > >>>>> google and > >>>>>>>>> use > >>>>>>>>>>>>> hooks > >>>>>>>>>>>>>> from both google/amazon. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I hope this alleviates your concern Daniel ? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> J. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> What about GoogleSheetsToS3? GoogleSheetsToGCS? > >> These > >>>>> you > >>>>>>>> would > >>>>>>>>>>>>> put in > >>>>>>>>>>>>>>> the target, i.e. the storage? But > >> GoogleSheetsToSftp > >>>>> would > >>>>>>>> be in > >>>>>>>>>>>>> google > >>>>>>>>>>>>>>> sheets operators file? The complexity, and the > >> shared > >>>>> code, > >>>>>>>> are > >>>>>>>>> in > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>> gsheet component -- not into the storage > >> destination. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Tue, Nov 5, 2019 at 5:46 PM Jarek Potiuk > >>>>>>>>>>>>> <jarek.pot...@polidea.com> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hello Airflow Community, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> The email calls for a vote to update AIP-21 > >> Changes in > >>>>>>>> import > >>>>>>>>>>>>> paths > >>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> with > >>>>>>>>>>>>>>>> the changes described below. The vote will last > >> till > >>>>>>>> Saturday > >>>>>>>>> 8th > >>>>>>>>>>>>> 2am > >>>>>>>>>>>>>>> CEST > >>>>>>>>>>>>>>>> (72 hours). Committers have a binding vote but > >>>> everyone > >>>>> from > >>>>>>>>> the > >>>>>>>>>>>>>>> community > >>>>>>>>>>>>>>>> is encouraged to cast an advisory vote. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> *Summary*: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> The proposal is to update AIP-21 to move all > >> non-core > >>>>>>>>>>>>>>>> operators/hooks/sensor (and related files) to > >>>>> sub-packages > >>>>>>>>> within > >>>>>>>>>>>>>>> airflow > >>>>>>>>>>>>>>>> (protocols/software/providers) or > >>>> (software/providers). > >>>>>>>>>>>>>>>> I am also happy to merge protocols+software, so > >> if you > >>>>> have > >>>>>>>> a > >>>>>>>>>>>>> strong > >>>>>>>>>>>>>>>> opinion on it - please state it with your vote > >> and we > >>>>> can > >>>>>>>>> decide > >>>>>>>>>>>>> based > >>>>>>>>>>>>>>> on > >>>>>>>>>>>>>>>> majority. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Those packages will be separately released > >>>>> (schedule/process > >>>>>>>>> TBD) > >>>>>>>>>>>>> and > >>>>>>>>>>>>>>> will > >>>>>>>>>>>>>>>> be backportable to 1.10.* airflow series, so that > >>>> users > >>>>> can > >>>>>>>>>>>>> install it > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>> start using new Airflow2.0 operators in their > >> Python 3 > >>>>>>>> Airflow > >>>>>>>>>>>>> 1.10 > >>>>>>>>>>>>>>>> environments (only Python 3.5+ is supported). > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> We will proceed with migrating the providers > >> package > >>>> to > >>>>>>>> already > >>>>>>>>>>>>> agreed > >>>>>>>>>>>>>>>> paths without waiting for the final vote > >> (following > >>>>> current > >>>>>>>>>>>>> version of > >>>>>>>>>>>>>>>> AIP-21). Since we have working POC - we know the > >>>> agreed > >>>>>>>> paths > >>>>>>>>> will > >>>>>>>>>>>>> work > >>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>> us. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> *Previous discussions: * > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> - > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>> > >>>> > >> > https://lists.apache.org/thread.html/b07a93c9114e3d3c55d4ee514955bac79bc012c7a00db627c6b4c55f@%3Cdev.airflow.apache.org%3E > >>>>>>>>>>>>>>>> - > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>> > >>>> > >> > https://lists.apache.org/thread.html/e25ddc546e367a4af3e594fecbd4431959bd5a89045e748e4206e7ff@%3Cdev.airflow.apache.org%3E > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> *More Details*: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 1) Information that we are going in the direction > >> of > >>>>> AIP-8 > >>>>>>>> but > >>>>>>>>> not > >>>>>>>>>>>>> yet > >>>>>>>>>>>>>>>> reaching it - focusing on separating out > >> backportable > >>>>>>>> packages > >>>>>>>>>>>>>>> installable > >>>>>>>>>>>>>>>> in Airflow releases 1.10.* . Airflow 2.0 will > >> still be > >>>>>>>>> installed > >>>>>>>>>>>>> as a > >>>>>>>>>>>>>>> whole > >>>>>>>>>>>>>>>> and all the source will be kept in one repo, but > >> we > >>>> now > >>>>>>>> have a > >>>>>>>>> way > >>>>>>>>>>>>> to > >>>>>>>>>>>>>>> build > >>>>>>>>>>>>>>>> backportable packages for groups of operators. POC > >>>>> available > >>>>>>>>> here: > >>>>>>>>>>>>>>>> https://github.com/apache/airflow/pull/6507 > >> (based on > >>>>> Ash's > >>>>>>>>>>>>>>>> https://github.com/ashb/airflow-submodule-test) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 2) We move all integrations to new packages > >> (keeping > >>>>>>>> deprecated > >>>>>>>>>>>>> import > >>>>>>>>>>>>>>>> aliases in the old places). The following split > >>>>> (according > >>>>>>>> to > >>>>>>>>>>>>>>> "stewardship" > >>>>>>>>>>>>>>>> over the integrations): > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> - *fundamentals* - core of ariflow - they are > >>>> really > >>>>>>>> part of > >>>>>>>>>>>>> Apache > >>>>>>>>>>>>>>>> Airflow. Stewards - core Airflow team. Not > >>>>>>>>>>>>> backportable/separated > >>>>>>>>>>>>>>> out. > >>>>>>>>>>>>>>>> - *protocols* - are not owned by anyone, they > >> are > >>>>> public > >>>>>>>> and > >>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> implementation is fully "open". There are no > >>>>> particular > >>>>>>>>>>>>> stewards (no > >>>>>>>>>>>>>>>> need). > >>>>>>>>>>>>>>>> Users of particular protocols should mainly > >>>> maintain > >>>>>>>> those > >>>>>>>>> and > >>>>>>>>>>>>> add > >>>>>>>>>>>>>>>> support > >>>>>>>>>>>>>>>> for different versions of the protocols. > >>>>>>>>>>>>>>>> - *software* - both API and software are > >> controlled > >>>>> by > >>>>>>>>> someone > >>>>>>>>>>>>>>> outside > >>>>>>>>>>>>>>>> of Airflow (commercial or open-source > >> project), but > >>>>> the > >>>>>>>>>>>>> deployment of > >>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>> software is "owned" by the user installing > >> Airflow. > >>>>> The > >>>>>>>>>>>>> "stewardship" > >>>>>>>>>>>>>>>> might > >>>>>>>>>>>>>>>> be also the users but the controlling party > >> (Oracle > >>>>> for > >>>>>>>>>>>>> example) > >>>>>>>>>>>>>>> might > >>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>> interested in maintaining those operators as > >> well. > >>>>>>>>>>>>>>>> - *providers* - API/software/deployments are > >> fully > >>>>>>>>> controlled > >>>>>>>>>>>>> by a > >>>>>>>>>>>>>>> 3rd > >>>>>>>>>>>>>>>> party. Here most likely "provider" will be > >>>>> interested in > >>>>>>>>>>>>> maintaining > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> operators (and for example like Google - > >> provide > >>>>>>>> integration > >>>>>>>>>>>>>>> guidelines > >>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>> > >>>> > >> > https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?usp=drive_web&ouid=112320280470690058978 > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>> their hooks/operators/sensors) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 3) Between-providers transfer operators should be > >> kept > >>>>> at > >>>>>>>> the > >>>>>>>>>>>>> "target" > >>>>>>>>>>>>>>>> rather than "source" > >>>>>>>>>>>>>>>> For example S3 -> GCS should be in "google" > >> provider, > >>>>> but > >>>>>>>>> GCS-> S3 > >>>>>>>>>>>>>>> should > >>>>>>>>>>>>>>>> be in "amazon". > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 4) One-side provider transfer operators should be > >> kept > >>>>> at > >>>>>>>> the > >>>>>>>>>>>>> "provider" > >>>>>>>>>>>>>>>> regardless if they are target or source. > >>>>>>>>>>>>>>>> For example GCS-> SFTP or SFTP -> GCS should be in > >>>>> "google" > >>>>>>>>>>>>> provider. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> 5) If in doubt we will discuss individual cases > >>>>> separately. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> J. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Jarek Potiuk > >>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal > >>>> Software > >>>>>>>>> Engineer > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> > >>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -- > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Jarek Potiuk > >>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal > >> Software > >>>>>>>> Engineer > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> > >>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> > >>>>>>>>>>>>> Jarek Potiuk > >>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal > >> Software > >>>>> Engineer > >>>>>>>>>>>>> > >>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> > >>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> > >>>>>>>>>>> Jarek Potiuk > >>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software > >>>>> Engineer > >>>>>>>>>>> > >>>>>>>>>>> M: +48 660 796 129 <+48660796129> > >>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> > >>>>>>> Jarek Potiuk > >>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>>>>> > >>>>>>> M: +48 660 796 129 <+48660796129> > >>>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> -- > >>>>>> > >>>>>> Jarek Potiuk > >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>>>> > >>>>>> M: +48 660 796 129 <+48660796129> > >>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>> > >>>> > >>>> > >>>> -- > >>>> > >>>> Jarek Potiuk > >>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>> > >>>> M: +48 660 796 129 <+48660796129> > >>>> [image: Polidea] <https://www.polidea.com/> > >>>> > >> > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/ <https://www.polidea.com/>> | > Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/ <https://www.polidea.com/>> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
