One wrinkle with case 3+4 option D is inter-provider operators. Mainly it's storage I think e.g. XToS3Operator or XToGCSOperator where the X is a service some different provider.
Maybe the rule should be to locate the operator according to the first provider referenced. So e.g. s3_to_gcs_transfer_operator.py would go to aws. On Tue, Jul 30, 2019 at 6:21 AM Kamil Breguła <kamil.breg...@polidea.com> wrote: > Yes. All changes will be backwards compatible. In the case of using > the old path, a message containing a proposal for change will be > reported to the user. > > I prepared an example of how to change the name of a class in a case > with the use of a native solution. > Source code: > https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename > Preview from IDE: https://imgur.com/a/45NxS5W > > On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <a...@apache.org> wrote: > > > > Just cos I'm not sure it's _explicitly_ stated, but all of the moves > will have a deprecation of the old name right? > > > > 3+4 case D gets my vote too. > > > > -a > > > > > On 30 Jul 2019, at 09:58, Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > > > > > > I went ahead and updated the page (just to speed it up) as I think it > > > really makes sense to join those two cases (and I do not see any > drawbacks > > > - I think the options we have cover all possible approaches) and we can > > > always go back if we need to. > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal > > > > > > My vote is *D*. > > > > > > - airflow/contrib/operators/gcp_bigtable_operator.py → airflow/gcp > > > /operators/bigtable_operator.py > > > - airflow/contrib/operators/ssh_operator.py -> > > > airflow/operators/ssh_operator.py > > > > > > I updated the page with my vote. > > > I propose that we vote till Friday on that one (all the rest is already > > > agreed). > > > > > > J. > > > > > > > > > On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <jarek.pot...@polidea.com > > > > > wrote: > > > > > >> I think almost everyone voted and we have almost perfect consensus. > We all > > >> agree amongst other on moving all operators out of contrib (Great!). > > >> > > >> The only doubts are for *Case 3* (Cloud provider prefix) and *Case 4* > > >> (Using Namespaces). > > >> I think there was actually an overlap between those two. Also Ash's > > >> comment/veto on that is quite clear. > > >> So I have final (I hope!) proposal to join those two cases and have > one > > >> voting (till Friday) only on that. > > >> > > >> I will update the doc if you all agree with me that makes more sense > as > > >> single case to vote on: > > >> > > >> *[Case 3 + Case 4]: Grouping Cloud Provider operators.* > > >> > > >> *Option A*. Keep operators/sensors/hooks in airflow/operators(sensors, > > >> hooks) and keep/add prefixes in file names. > > >> > > >> - > > >> *airflow/contrib/operators/sns_publish_operator.py becomes > > >> airflow/operators/**aws_sns_publish_operator.py* > > >> > > >> > > >> - *airflow/contrib/operators/dataproc_operator.py* > > >> becomes *airflow/operators/gcp_dataproc_operator.py* > > >> > > >> > > >> - > > >> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/* > > >> *hooks/gcp_bigtable_hook.py* > > >> - > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > > >> *operators/ssh_operator.py* > > >> > > >> > > >> > > >> *Option B*. Group operators/sensors/hooks in > airflow/operators(sensors, > > >> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to > > >> airflow/operators(sensors/hooks), Drop the prefix. > > >> > > >> - > > >> *airflow/contrib/operators/sns_publish_operator.py > > >> becomes airflow/operators/**aws/sns_publish_operator.py* > > >> > > >> > > >> - *airflow/contrib/operators/dataproc_operator.py* > > >> becomes *airflow/operators/gcp/dataproc_operator.py* > > >> > > >> > > >> - > > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes > *airflow/* > > >> *operators/gcp/bigtable_operator.py* > > >> - > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > > >> *operators/ssh_operator.py* > > >> > > >> > > >> *Option C*. Group operators/sensors/hooks in > airflow/operators(sensors, > > >> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to > > >> airflow/operators(sensors/hooks), Keep the prefix. > > >> > > >> - > > >> *airflow/contrib/operators/sns_publish_operator.py > > >> becomes airflow/operators/**aws/aws_sns_publish_operator.py* > > >> > > >> > > >> - *airflow/contrib/operators/dataproc_operator.py* > > >> becomes *airflow/operators/gcp/gcp_dataproc_operator.py* > > >> > > >> > > >> - > > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes > *airflow/* > > >> *operators/gcp/gcp_bigtable_operator.py* > > >> - > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > > >> *operators/ssh_operator.py* > > >> > > >> > > >> *Option D.* Group operators/sensors/hooks in > *airflow/<PROVIDER>*/operators(sensors, > > >> hooks). Non-cloud provider ones are moved to > > >> airflow/operators(sensors/hooks). Drop the prefix. > > >> > > >> - > > >> *airflow/contrib/operators/sns_publish_operator.py > > >> becomes airflow/aws/operators/**sns_publish_operator.py* > > >> > > >> > > >> - *airflow/contrib/operators/dataproc_operator.py* > > >> becomes *airflow/gcp/operators/dataproc_operator.py* > > >> > > >> > > >> - > > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes > *airflow/* > > >> *gcp/operators/bigtable_operator.py* > > >> - > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > > >> *operators/ssh_operator.py* > > >> > > >> > > >> *Option E.* Group operators/sensors/hooks in > *airflow/<PROVIDER>*/operators(sensors, > > >> hooks). Non-cloud provider ones are moved to > > >> airflow/operators(sensors/hooks).* Keep the prefix*. > > >> > > >> - > > >> *airflow/contrib/operators/sns_publish_operator.py > > >> becomes airflow/aws/operators/aws_**sns_publish_operator.py* > > >> > > >> > > >> - *airflow/contrib/operators/dataproc_operator.py* > > >> becomes *airflow/gcp/operators/gcp_dataproc_operator.py* > > >> > > >> > > >> - > > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes > *airflow/* > > >> *gcp/operators/gcp_bigtable_operator.py* > > >> - > > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > > >> *operators/ssh_operator.py* > > >> > > >> > > >> Shall I proceed with updating the page ? > > >> > > >> > > >> J. > > >> > > >> > > >> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <kaxiln...@gmail.com> > wrote: > > >> > > >>> Thanks Jarek, adding my vote. > > >>> > > >>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <a...@apache.org> > wrote: > > >>> > > >>>> Thanks Jarek, much clearer. I have voted there too. > > >>>> > > >>>> -ash > > >>>> > > >>>> > > >>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko <fo...@driesprong.frl> > > >>>> wrote: > > >>>>> > > >>>>> Thanks Jarek, the Wiki works much better for me. I've added my vote > > >>> there > > >>>>> as well. > > >>>>> > > >>>>> You've convinced me on the second case. I've changed my vote on > Case 2 > > >>>> from > > >>>>> A to B :-) > > >>>>> > > >>>>> Maybe we should leave the vote open a bit longer since it involves > > >>> quite > > >>>>> some reading, and I would like to get some proper feedback and > > >>> opinions > > >>>> on > > >>>>> this. Plus I only see two votes right now. If we don't think this > > >>>> through, > > >>>>> it might be that we're having this vote again in the near future > :-) > > >>>>> > > >>>>> Cheers, Fokko > > >>>>> > > >>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk < > > >>>> jarek.pot...@polidea.com>: > > >>>>> > > >>>>>> I updated the AIP-21 proposal in the way that should answer all > the > > >>>>>> concerns in this thread. > > >>>>>> > > >>>>>> NOTE! There is an action for all of those who are interested: > Please > > >>>>>> fill-in your voting in the voting table till Tuesday 30th of July > 6pm > > >>>> CEST > > >>>>>> (5pm BST, 9 am PST). > > >>>>>> > > >>>>>> I created a summary of the proposal > > >>>>>> < > > >>>>>> > > >>>> > > >>> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal > > >>>>>>> > > >>>>>> in table form - which contains also examples for each case. > > >>>>>> > > >>>>>> I added a voting table > > >>>>>> < > > >>>>>> > > >>>> > > >>> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting > > >>>>>>> > > >>>>>> where you should add your votes: > > >>>>>> > > >>>>>> I propose this voting mechanism: > > >>>>>> > > >>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5 pm > > >>> BST, > > >>>> 9am > > >>>>>> PST) . > > >>>>>> > > >>>>>> After the choice, the final consistent set of choices will be > > >>> announced > > >>>>>> (taking into account majority of binding vote, also including > > >>> potential > > >>>>>> vetos and consistency between the choices. Non-binding votes will > be > > >>>> taken > > >>>>>> into account in case there is a draw. The final set of choices > will > > >>> be > > >>>>>> announced at devlist thread > > >>>>>> < > > >>>>>> > > >>>> > > >>> > https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E > > >>>>>>> > > >>>>>> after > > >>>>>> the voting completes. > > >>>>>> > > >>>>>> Unless there is a veto raised to the final proposal, the final > > >>> proposal > > >>>> is > > >>>>>> accepted by Lazy Consensus > > >>>>>> <https://community.apache.org/committers/lazyConsensus.html> on > > >>>> *Friday* > > >>>>>> 02 > > >>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST). > > >>>>>> > > >>>>>> Please let me know if what I proposed can be further improved to > > >>> avoid > > >>>>>> chaos. > > >>>>>> > > >>>>>> J. > > >>>>>> > > >>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk < > > >>> jarek.pot...@polidea.com > > >>>>> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> Ok. I see that indeed voting could have been organised a bit > better > > >>> - > > >>>> dev > > >>>>>>> list does not seem to cope well with such a complex case :). > > >>>>>>> > > >>>>>>> As Kamil noticed - we already have a formal AIP-21 > > >>>>>>> < > > >>>>>> > > >>>> > > >>> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths > > >>>>>>> > > >>>>>>> in the Wiki - which I should have mentioned before. I have not > much > > >>>> time > > >>>>>>> today - but tomorrow I will try to summarise the options - based > on > > >>>> some > > >>>>>>> real code examples (to make it easier to digest). I think the > best > > >>>>>> approach > > >>>>>>> will be to make a voting matrix in the AIP-21 Wiki page where > people > > >>>> will > > >>>>>>> be able to state their preferences for each case (by adding a > row to > > >>>> the > > >>>>>>> table). I think that might provide much better clarity and a > single > > >>>> place > > >>>>>>> which will show the current aggregated state of voting. > > >>>>>>> > > >>>>>>> However - as Ash also stated - some of the options are > > >>> inter-dependent > > >>>> so > > >>>>>>> at the end we will have to adjust the results in case they are > not > > >>>>>>> consistent - and make final voting on aggregated set of cases - > > >>> but I > > >>>>>>> think in this case following "lasy consensus" - i,e. if noone > > >>> objects > > >>>> to > > >>>>>>> final set of cases, we will proceed with it.. > > >>>>>>> > > >>>>>>> Do you think that will work better ? > > >>>>>>> > > >>>>>>> J. > > >>>>>>> > > >>>>>>> Principal Software Engineer > > >>>>>>> Phone: +48660796129 > > >>>>>>> > > >>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła < > > >>>>>>> kamil.breg...@polidea.com> napisał: > > >>>>>>> > > >>>>>>>> Document is also available as AIP on wiki: > > >>>>>>>> > > >>>>>>>> > > >>>>>> > > >>>> > > >>> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths > > >>>>>>>> > > >>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko > > >>> <fo...@driesprong.frl > > >>>>> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> I agree with all, this is a bit chaotic. For the sake of > clarity, > > >>> I > > >>>>>>>> would > > >>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP. Create a > > >>>>>> structural > > >>>>>>>>> code improvement AIP and then and add a matrix of users/cases > > >>> where > > >>>>>>>>> everybody can add his vote per case. This gives also the > > >>> opportunity > > >>>>>> for > > >>>>>>>>> changing your vote on a specific case. WDYT? > > >>>>>>>>> > > >>>>>>>>> Cheers, Fokko > > >>>>>>>>> > > >>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong < > cix...@gmail.com>: > > >>>>>>>>> > > >>>>>>>>>> I agree with Ash. It is clear to vote on each case rather than > > >>> all > > >>>>>>>>>> together. > > >>>>>>>>>> > > >>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor < > > >>> a...@apache.org> > > >>>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> A number of proposals overlap or add on to each other, so I > > >>> think > > >>>>>>>> (?) a > > >>>>>>>>>>> single vote makes sense. > > >>>>>>>>>>> > > >>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly what we > are > > >>>>>>>> voting > > >>>>>>>>> on. > > >>>>>>>>>>> > > >>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik < > kaxiln...@gmail.com> > > >>>>>>>> wrote: > > >>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases > together, > > >>>>>> how > > >>>>>>>>>>>> about > > >>>>>>>>>>>> separate vote emails for all the cases? Each email can have > the > > >>>>>>>>>>>> description > > >>>>>>>>>>>> copied over from the doc. > > >>>>>>>>>>>> > > >>>>>>>>>>>> It would probably be a bit easier to track a decision in the > > >>>>>> future > > >>>>>>>> as > > >>>>>>>>>>>> well. > > >>>>>>>>>>>> > > >>>>>>>>>>>> What do you guys think? @Jarek Potiuk < > > >>> jarek.pot...@polidea.com> > > >>>>>>>>>>>> @Driesprong, > > >>>>>>>>>>>> Fokko <fo...@driesprong.frl> > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik < > > >>> kaxiln...@gmail.com> > > >>>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A * > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A * > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C* > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> *Case #4* Separate namespace for resources : > > >>>>>>>>>>>>> *Solution A* > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> *Case #5* *Operator : *Solution A* > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> *Case #7* : *Solution A* > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish > > >>>>>>>>>>>> <dpstand...@gmail.com> > > >>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>> > > >>>>>>>>>>>>>> I have tried to add some clarification to Jarek's summary, > > >>> but > > >>>>>> I > > >>>>>>>> am > > >>>>>>>>>>>> a > > >>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully > Jarek > > >>>>>> can > > >>>>>>>>>>>> further > > >>>>>>>>>>>>>> clarify. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6 are > all > > >>>>>>>> either > > >>>>>>>>>>>>>> proposal > > >>>>>>>>>>>>>> *rejections* or otherwise have no effect, so attention > can be > > >>>>>>>>>>>> focused on > > >>>>>>>>>>>>>> cases 1,2,3 and 7. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> *Motion is to adopt the following:* > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib > > >>>>>>>>>>>>>> All objects will be moved out of contrib folder and into > > >>>>>>>> respective > > >>>>>>>>>>>>>> non-contrib folder. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from > modules > > >>>>>>>>>>>>>> Example: > > >>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes > > >>>>>>>>>>>>>> airflow.operators.foo.FooOperator > > >>>>>>>>>>>>>> Example: > > >>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes > > >>>>>>>> airflow.hooks.foo.FooHook > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes > > >>>>>>>>>>>>>> *I am not sure exactly what the proposal is.* > > >>>>>>>>>>>>>> Example case is > > >>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py > > >>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume it is > > >>> now > > >>>>>>>> named > > >>>>>>>>>>>> like > > >>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py > > >>>>>>>>>>>>>> So what is proposed for this case? > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Case 4 - Solution B - *no change* > > >>>>>>>>>>>>>> This proposal was about namespaces but the motion is to > > >>> reject > > >>>>>>>> the > > >>>>>>>>>>>>>> proposal. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Case 5 - Solution B - *no change* > > >>>>>>>>>>>>>> The proposal was to remove class type suffix e.g. remove > the > > >>>>>>>>>>>> Operator in > > >>>>>>>>>>>>>> FooOperator, but the motion is to reject this proposal -- > > >>> i.e. > > >>>>>>>>>>>> motion is > > >>>>>>>>>>>>>> no > > >>>>>>>>>>>>>> change on this case, i.e. we resolve to keep "Operator" > (and > > >>>>>>>>>>>> "Sensor") > > >>>>>>>>>>>>>> suffixes on class names > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Case 6 - *No change* > > >>>>>>>>>>>>>> This case concerns object naming inconsistencies, but > motion > > >>>>>> does > > >>>>>>>>>>>> not > > >>>>>>>>>>>>>> enact > > >>>>>>>>>>>>>> any specific proposal. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for > > >>>>>> deprecation > > >>>>>>>>>>>> warnings > > >>>>>>>>>>>>>> (instead of zope) > > >>>>>>>>>>>>>> Example: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> # in old_package/old_mod.py > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> import warnings > > >>>>>>>>>>>>>> from new_package.new_mod import * > > >>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to > > >>>>>>>> new_package.new_mod. > > >>>>>>>>>>>>>> Import > > >>>>>>>>>>>>>> of " > > >>>>>>>>>>>>>> "old_package.old_mod will become unsupported in version > 2", > > >>>>>>>>>>>>>> DeprecationWarning, 2) > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> Reference implementation here: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>> > > >>>> > > >>> > https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1 > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3. > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> I am very happy that this motion now gets rid of contrib. > > >>>>>> Thank > > >>>>>>>> you > > >>>>>>>>>>>>>> Sergio > > >>>>>>>>>>>>>> for turning the tide with your effective argumentation ;) > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor < > > >>>>>>>> a...@apache.org> > > >>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Could someone summarise what the proposed name changes > are, > > >>>>>>>> here, > > >>>>>>>>>>>> in > > >>>>>>>>>>>>>> this > > >>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving partial > > >>>>>>>> examples > > >>>>>>>>>>>> is 1) > > >>>>>>>>>>>>>> a > > >>>>>>>>>>>>>>> bit difficult to quickly work out what we are voting on, > and > > >>>>>> 2) > > >>>>>>>>>>>> using a > > >>>>>>>>>>>>>>> google doc isn't great for a longer term record. > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> Thanks, > > >>>>>>>>>>>>>>> Ash > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk > > >>>>>>>>>>>> <jarek.pot...@polidea.com> > > >>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from 0 :) > . I > > >>>>>>>> would > > >>>>>>>>>>>> like > > >>>>>>>>>>>>>> to > > >>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three +1 > > >>>>>>>>>>>> (binding) > > >>>>>>>>>>>>>> votes > > >>>>>>>>>>>>>>> are > > >>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications to the > > >>>>>>>>>>>> previous > > >>>>>>>>>>>>>>> proposal. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Here is the proposal (details here > > >>>>>>>>>>>>>>>> < > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>> > > >>>> > > >>> > https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> ) > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,* > > >>>>>>>>>>>>>>>> - *Case 2: A: Airflow.operators.foo_operator.FooOperator > > >>>>>>>> could > > >>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator* > > >>>>>>>>>>>>>>>> - *Case 3: C: > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator > > >>>>>>>>>>>>>> could > > >>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator* > > >>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction* > > >>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on > > >>>>>>>> class > > >>>>>>>>>>>>>> names* > > >>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case > > >>>>>>>> (and > > >>>>>>>>>>>> my team > > >>>>>>>>>>>>>>> can > > >>>>>>>>>>>>>>>> do the job of GCP-related operators)* > > >>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE > > >>>>>>>>>>>> integration)* > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> This is my binding (+1) vote. > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk < > > >>>>>>>>>>>>>> jarek.pot...@polidea.com> > > >>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Hey Felix, > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one as > well > > >>>>>> (I > > >>>>>>>>>>>> use IDE > > >>>>>>>>>>>>>>> quite > > >>>>>>>>>>>>>>>>> often ;) ) > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> J. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko > > >>>>>>>>>>>>>> <fo...@driesprong.frl > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> wrote: > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I > wasn't > > >>>>>>>> able > > >>>>>>>>>>>> to > > >>>>>>>>>>>>>>> respond > > >>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops > throughout > > >>>>>>>> Europe > > >>>>>>>>>>>> :-) > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from contrib > > >>>>>> into > > >>>>>>>> a > > >>>>>>>>>>>> single > > >>>>>>>>>>>>>>>>>> package. > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference > would be > > >>>>>>>> to > > >>>>>>>>>>>> get > > >>>>>>>>>>>>>> rid of > > >>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or not > is a > > >>>>>>>>>>>> tricky > > >>>>>>>>>>>>>>> question, > > >>>>>>>>>>>>>>>>>> I think this is already reflected in the document > since > > >>>>>>>>>>>> "maturity" > > >>>>>>>>>>>>>> is > > >>>>>>>>>>>>>>> in > > >>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the > operator > > >>>>>>>> to > > >>>>>>>>>>>> the > > >>>>>>>>>>>>>> core > > >>>>>>>>>>>>>>> or > > >>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to > > >>>>>> incubating > > >>>>>>>>>>>> will > > >>>>>>>>>>>>>> confuse > > >>>>>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>> user even more. For people who aren't as known with > > >>>>>>>> Airflow it > > >>>>>>>>>>>> isn't > > >>>>>>>>>>>>>>> that > > >>>>>>>>>>>>>>>>>> transparent which operator lives where, especially if > you > > >>>>>>>>>>>> don't use > > >>>>>>>>>>>>>> an > > >>>>>>>>>>>>>>> IDE > > >>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is worth > > >>>>>>>> changing > > >>>>>>>>>>>> the > > >>>>>>>>>>>>>>> public > > >>>>>>>>>>>>>>>>>> API just for a different name. > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with this. > It > > >>>>>>>> will > > >>>>>>>>>>>> make > > >>>>>>>>>>>>>>> easier > > >>>>>>>>>>>>>>>>> as we will not have to define other rules as those > that we > > >>>>>>>>>>>> already > > >>>>>>>>>>>>>> have. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's say a > > >>>>>> +0) > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> I like to search on Github with t, and then search for > > >>>>>>>>>>>> BashOperator. > > >>>>>>>>>>>>>>> After > > >>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash. > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from your > > >>>>>> mail > > >>>>>>>> you > > >>>>>>>>>>>>>> state: - > > >>>>>>>>>>>>>>>>>> *Case 2: B: Airflow.operators.foo_operator.FooOperator > > >>>>>>>> could > > >>>>>>>>>>>> become > > >>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B > solution is > > >>>>>>>>>>>> keeping it > > >>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B: > > >>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would stay > > >>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator* > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a > little > > >>>>>>>>>>>> confusion > > >>>>>>>>>>>>>>> here. > > >>>>>>>>>>>>>>>>> This case is not for changing BashOperator into Bash > but > > >>>>>> for > > >>>>>>>>>>>> changing > > >>>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>> *module* name not the *class* name. So > > >>>>>>>>>>>> bash_operator.BashOperator -> > > >>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for > > >>>>>>>> BashOperator > > >>>>>>>>>>>> in this > > >>>>>>>>>>>>>>> case. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when it > gets > > >>>>>>>> its > > >>>>>>>>>>>> own > > >>>>>>>>>>>>>>> package > > >>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are > companies > > >>>>>>>> like > > >>>>>>>>>>>>>>> Databricks > > >>>>>>>>>>>>>>>>>> and Qubole which might also end up with their own > package > > >>>>>>>>>>>> because > > >>>>>>>>>>>>>> their > > >>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving this > is a > > >>>>>>>> bit > > >>>>>>>>>>>>>> painful > > >>>>>>>>>>>>>>>>>> because we need to deprecate the old path and move > > >>>>>>>> everything > > >>>>>>>>>>>> which > > >>>>>>>>>>>>>>>>>> introduces noise into version control etc. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's not as > > >>>>>>>>>>>> difficult as > > >>>>>>>>>>>>>> it > > >>>>>>>>>>>>>>>>> seems to group operators in packages and it has > numerous > > >>>>>>>>>>>> advantages. > > >>>>>>>>>>>>>> The > > >>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in the > > >>>>>>>>>>>> __init__.py of > > >>>>>>>>>>>>>> the > > >>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git will > > >>>>>>>> actually > > >>>>>>>>>>>>>> realise > > >>>>>>>>>>>>>>>>> that the file has been moved and you can follow that. > > >>>>>> Maybe > > >>>>>>>> not > > >>>>>>>>>>>> as > > >>>>>>>>>>>>>> super > > >>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's not a > > >>>>>> rocket > > >>>>>>>>>>>> science > > >>>>>>>>>>>>>>> either. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> *Case 7:* Python native solution > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Yes. > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> --- > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Apart from all, great work! > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Cheers, Fokko > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall > > >>>>>>>>>>>>>>>>>> <felue...@pm.me.invalid > > >>>>>>>>>>>>>>>>>>> : > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed change of > > >>>>>> these > > >>>>>>>>>>>> cases. > > >>>>>>>>>>>>>> So I > > >>>>>>>>>>>>>>>>>> go > > >>>>>>>>>>>>>>>>>>> with +1 (non-binding). > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and your > > >>>>>>>> intense > > >>>>>>>>>>>> work > > >>>>>>>>>>>>>>>>>> towards > > >>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even > > >>>>>> creating > > >>>>>>>>>>>> this > > >>>>>>>>>>>>>>>>>> document. I > > >>>>>>>>>>>>>>>>>>> like how the code is getting more and more consistent > > >>>>>> and > > >>>>>>>>>>>> clean :) > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> Kind regards, > > >>>>>>>>>>>>>>>>>>> Felix > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> -------- Original Message -------- > > >>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote: > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Hello everyone, > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in > import > > >>>>>>>> paths. > > >>>>>>>>>>>> It's > > >>>>>>>>>>>>>>> been > > >>>>>>>>>>>>>>>>>>>> discussed in > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>> > > >>>> > > >>> > https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July 30th > 6pm > > >>>>>>>> CEST), > > >>>>>>>>>>>> and > > >>>>>>>>>>>>>> at > > >>>>>>>>>>>>>>>>>> least > > >>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document from > > >>>>>> Kamil > > >>>>>>>>>>>>>>>>>>>> < > > >>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>> > > >>>> > > >>> > https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit# > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> The proposed solution is: > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that are > > >>>>>> "well" > > >>>>>>>>>>>> tested > > >>>>>>>>>>>>>> and > > >>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.* > > >>>>>>>>>>>>>>>>>>>> - *Case 2: B: > > >>>>>> Airflow.operators.foo_operator.FooOperator > > >>>>>>>>>>>> could > > >>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator* > > >>>>>>>>>>>>>>>>>>>> - *Case 3: C: > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator > > >>>>>>>>>>>>>>> could > > >>>>>>>>>>>>>>>>>>>> become > airflow.gcp.operators.bigtable.BigTableOperator* > > >>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction* > > >>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") > suffixes > > >>>>>> on > > >>>>>>>>>>>> class > > >>>>>>>>>>>>>> names* > > >>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on > case-by-case > > >>>>>>>> (and > > >>>>>>>>>>>> my > > >>>>>>>>>>>>>> team > > >>>>>>>>>>>>>>>>>> can > > >>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)* > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Best regards, > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> J. > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> Jarek Potiuk > > >>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal > > >>>>>> Software > > >>>>>>>>>>>> Engineer > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129) > > >>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)> > > >>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > > >>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> Jarek Potiuk > > >>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal > Software > > >>>>>>>>>>>> Engineer > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> > > >>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> -- > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> Jarek Potiuk > > >>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software > > >>>>>>>>> Engineer > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> > > >>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> -- > > >>>>>>>>>>>>> *Kaxil Naik* > > >>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer* > > >>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache > > >>>>>> Spark > > >>>>>>>> & > > >>>>>>>>>>>> Neo4j > > >>>>>>>>>>>>> Developer > > >>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> -- > > >>>>>>>>>>>> *Kaxil Naik* > > >>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer* > > >>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache > > >>> Spark > > >>>>>> & > > >>>>>>>>>>>> Neo4j > > >>>>>>>>>>>> Developer > > >>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>>> -- > > >>>>>> > > >>>>>> Jarek Potiuk > > >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >>>>>> > > >>>>>> M: +48 660 796 129 <+48660796129> > > >>>>>> [image: Polidea] <https://www.polidea.com/> > > >>>>>> > > >>>> > > >>>> > > >>> > > >>> -- > > >>> *Kaxil Naik* > > >>> *Big Data Consultant | DevOps Data Engineer* > > >>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & > Neo4j > > >>> Developer > > >>> *LinkedIn*: https://www.linkedin.com/in/kaxil > > >>> > > >> > > >> > > >> -- > > >> > > >> Jarek Potiuk > > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > > >> > > >> M: +48 660 796 129 <+48660796129> > > >> [image: Polidea] <https://www.polidea.com/> > > >> > > >> > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > >
