Yes. All changes will be backwards compatible. In the case of using the old path, a message containing a proposal for change will be reported to the user.
I prepared an example of how to change the name of a class in a case with the use of a native solution. Source code: https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename Preview from IDE: https://imgur.com/a/45NxS5W On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <[email protected]> wrote: > > Just cos I'm not sure it's _explicitly_ stated, but all of the moves will > have a deprecation of the old name right? > > 3+4 case D gets my vote too. > > -a > > > On 30 Jul 2019, at 09:58, Jarek Potiuk <[email protected]> wrote: > > > > I went ahead and updated the page (just to speed it up) as I think it > > really makes sense to join those two cases (and I do not see any drawbacks > > - I think the options we have cover all possible approaches) and we can > > always go back if we need to. > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal > > > > My vote is *D*. > > > > - airflow/contrib/operators/gcp_bigtable_operator.py → airflow/gcp > > /operators/bigtable_operator.py > > - airflow/contrib/operators/ssh_operator.py -> > > airflow/operators/ssh_operator.py > > > > I updated the page with my vote. > > I propose that we vote till Friday on that one (all the rest is already > > agreed). > > > > J. > > > > > > On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <[email protected]> > > wrote: > > > >> I think almost everyone voted and we have almost perfect consensus. We all > >> agree amongst other on moving all operators out of contrib (Great!). > >> > >> The only doubts are for *Case 3* (Cloud provider prefix) and *Case 4* > >> (Using Namespaces). > >> I think there was actually an overlap between those two. Also Ash's > >> comment/veto on that is quite clear. > >> So I have final (I hope!) proposal to join those two cases and have one > >> voting (till Friday) only on that. > >> > >> I will update the doc if you all agree with me that makes more sense as > >> single case to vote on: > >> > >> *[Case 3 + Case 4]: Grouping Cloud Provider operators.* > >> > >> *Option A*. Keep operators/sensors/hooks in airflow/operators(sensors, > >> hooks) and keep/add prefixes in file names. > >> > >> - > >> *airflow/contrib/operators/sns_publish_operator.py becomes > >> airflow/operators/**aws_sns_publish_operator.py* > >> > >> > >> - *airflow/contrib/operators/dataproc_operator.py* > >> becomes *airflow/operators/gcp_dataproc_operator.py* > >> > >> > >> - > >> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/* > >> *hooks/gcp_bigtable_hook.py* > >> - > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > >> *operators/ssh_operator.py* > >> > >> > >> > >> *Option B*. Group operators/sensors/hooks in airflow/operators(sensors, > >> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to > >> airflow/operators(sensors/hooks), Drop the prefix. > >> > >> - > >> *airflow/contrib/operators/sns_publish_operator.py > >> becomes airflow/operators/**aws/sns_publish_operator.py* > >> > >> > >> - *airflow/contrib/operators/dataproc_operator.py* > >> becomes *airflow/operators/gcp/dataproc_operator.py* > >> > >> > >> - > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/* > >> *operators/gcp/bigtable_operator.py* > >> - > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > >> *operators/ssh_operator.py* > >> > >> > >> *Option C*. Group operators/sensors/hooks in airflow/operators(sensors, > >> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to > >> airflow/operators(sensors/hooks), Keep the prefix. > >> > >> - > >> *airflow/contrib/operators/sns_publish_operator.py > >> becomes airflow/operators/**aws/aws_sns_publish_operator.py* > >> > >> > >> - *airflow/contrib/operators/dataproc_operator.py* > >> becomes *airflow/operators/gcp/gcp_dataproc_operator.py* > >> > >> > >> - > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/* > >> *operators/gcp/gcp_bigtable_operator.py* > >> - > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > >> *operators/ssh_operator.py* > >> > >> > >> *Option D.* Group operators/sensors/hooks in > >> *airflow/<PROVIDER>*/operators(sensors, > >> hooks). Non-cloud provider ones are moved to > >> airflow/operators(sensors/hooks). Drop the prefix. > >> > >> - > >> *airflow/contrib/operators/sns_publish_operator.py > >> becomes airflow/aws/operators/**sns_publish_operator.py* > >> > >> > >> - *airflow/contrib/operators/dataproc_operator.py* > >> becomes *airflow/gcp/operators/dataproc_operator.py* > >> > >> > >> - > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/* > >> *gcp/operators/bigtable_operator.py* > >> - > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > >> *operators/ssh_operator.py* > >> > >> > >> *Option E.* Group operators/sensors/hooks in > >> *airflow/<PROVIDER>*/operators(sensors, > >> hooks). Non-cloud provider ones are moved to > >> airflow/operators(sensors/hooks).* Keep the prefix*. > >> > >> - > >> *airflow/contrib/operators/sns_publish_operator.py > >> becomes airflow/aws/operators/aws_**sns_publish_operator.py* > >> > >> > >> - *airflow/contrib/operators/dataproc_operator.py* > >> becomes *airflow/gcp/operators/gcp_dataproc_operator.py* > >> > >> > >> - > >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/* > >> *gcp/operators/gcp_bigtable_operator.py* > >> - > >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > >> *operators/ssh_operator.py* > >> > >> > >> Shall I proceed with updating the page ? > >> > >> > >> J. > >> > >> > >> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <[email protected]> wrote: > >> > >>> Thanks Jarek, adding my vote. > >>> > >>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <[email protected]> wrote: > >>> > >>>> Thanks Jarek, much clearer. I have voted there too. > >>>> > >>>> -ash > >>>> > >>>> > >>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko <[email protected]> > >>>> wrote: > >>>>> > >>>>> Thanks Jarek, the Wiki works much better for me. I've added my vote > >>> there > >>>>> as well. > >>>>> > >>>>> You've convinced me on the second case. I've changed my vote on Case 2 > >>>> from > >>>>> A to B :-) > >>>>> > >>>>> Maybe we should leave the vote open a bit longer since it involves > >>> quite > >>>>> some reading, and I would like to get some proper feedback and > >>> opinions > >>>> on > >>>>> this. Plus I only see two votes right now. If we don't think this > >>>> through, > >>>>> it might be that we're having this vote again in the near future :-) > >>>>> > >>>>> Cheers, Fokko > >>>>> > >>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk < > >>>> [email protected]>: > >>>>> > >>>>>> I updated the AIP-21 proposal in the way that should answer all the > >>>>>> concerns in this thread. > >>>>>> > >>>>>> NOTE! There is an action for all of those who are interested: Please > >>>>>> fill-in your voting in the voting table till Tuesday 30th of July 6pm > >>>> CEST > >>>>>> (5pm BST, 9 am PST). > >>>>>> > >>>>>> I created a summary of the proposal > >>>>>> < > >>>>>> > >>>> > >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal > >>>>>>> > >>>>>> in table form - which contains also examples for each case. > >>>>>> > >>>>>> I added a voting table > >>>>>> < > >>>>>> > >>>> > >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting > >>>>>>> > >>>>>> where you should add your votes: > >>>>>> > >>>>>> I propose this voting mechanism: > >>>>>> > >>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5 pm > >>> BST, > >>>> 9am > >>>>>> PST) . > >>>>>> > >>>>>> After the choice, the final consistent set of choices will be > >>> announced > >>>>>> (taking into account majority of binding vote, also including > >>> potential > >>>>>> vetos and consistency between the choices. Non-binding votes will be > >>>> taken > >>>>>> into account in case there is a draw. The final set of choices will > >>> be > >>>>>> announced at devlist thread > >>>>>> < > >>>>>> > >>>> > >>> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E > >>>>>>> > >>>>>> after > >>>>>> the voting completes. > >>>>>> > >>>>>> Unless there is a veto raised to the final proposal, the final > >>> proposal > >>>> is > >>>>>> accepted by Lazy Consensus > >>>>>> <https://community.apache.org/committers/lazyConsensus.html> on > >>>> *Friday* > >>>>>> 02 > >>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST). > >>>>>> > >>>>>> Please let me know if what I proposed can be further improved to > >>> avoid > >>>>>> chaos. > >>>>>> > >>>>>> J. > >>>>>> > >>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk < > >>> [email protected] > >>>>> > >>>>>> wrote: > >>>>>> > >>>>>>> Ok. I see that indeed voting could have been organised a bit better > >>> - > >>>> dev > >>>>>>> list does not seem to cope well with such a complex case :). > >>>>>>> > >>>>>>> As Kamil noticed - we already have a formal AIP-21 > >>>>>>> < > >>>>>> > >>>> > >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths > >>>>>>> > >>>>>>> in the Wiki - which I should have mentioned before. I have not much > >>>> time > >>>>>>> today - but tomorrow I will try to summarise the options - based on > >>>> some > >>>>>>> real code examples (to make it easier to digest). I think the best > >>>>>> approach > >>>>>>> will be to make a voting matrix in the AIP-21 Wiki page where people > >>>> will > >>>>>>> be able to state their preferences for each case (by adding a row to > >>>> the > >>>>>>> table). I think that might provide much better clarity and a single > >>>> place > >>>>>>> which will show the current aggregated state of voting. > >>>>>>> > >>>>>>> However - as Ash also stated - some of the options are > >>> inter-dependent > >>>> so > >>>>>>> at the end we will have to adjust the results in case they are not > >>>>>>> consistent - and make final voting on aggregated set of cases - > >>> but I > >>>>>>> think in this case following "lasy consensus" - i,e. if noone > >>> objects > >>>> to > >>>>>>> final set of cases, we will proceed with it.. > >>>>>>> > >>>>>>> Do you think that will work better ? > >>>>>>> > >>>>>>> J. > >>>>>>> > >>>>>>> Principal Software Engineer > >>>>>>> Phone: +48660796129 > >>>>>>> > >>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła < > >>>>>>> [email protected]> napisał: > >>>>>>> > >>>>>>>> Document is also available as AIP on wiki: > >>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths > >>>>>>>> > >>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko > >>> <[email protected] > >>>>> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> I agree with all, this is a bit chaotic. For the sake of clarity, > >>> I > >>>>>>>> would > >>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP. Create a > >>>>>> structural > >>>>>>>>> code improvement AIP and then and add a matrix of users/cases > >>> where > >>>>>>>>> everybody can add his vote per case. This gives also the > >>> opportunity > >>>>>> for > >>>>>>>>> changing your vote on a specific case. WDYT? > >>>>>>>>> > >>>>>>>>> Cheers, Fokko > >>>>>>>>> > >>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <[email protected]>: > >>>>>>>>> > >>>>>>>>>> I agree with Ash. It is clear to vote on each case rather than > >>> all > >>>>>>>>>> together. > >>>>>>>>>> > >>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor < > >>> [email protected]> > >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> A number of proposals overlap or add on to each other, so I > >>> think > >>>>>>>> (?) a > >>>>>>>>>>> single vote makes sense. > >>>>>>>>>>> > >>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly what we are > >>>>>>>> voting > >>>>>>>>> on. > >>>>>>>>>>> > >>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <[email protected]> > >>>>>>>> wrote: > >>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases together, > >>>>>> how > >>>>>>>>>>>> about > >>>>>>>>>>>> separate vote emails for all the cases? Each email can have the > >>>>>>>>>>>> description > >>>>>>>>>>>> copied over from the doc. > >>>>>>>>>>>> > >>>>>>>>>>>> It would probably be a bit easier to track a decision in the > >>>>>> future > >>>>>>>> as > >>>>>>>>>>>> well. > >>>>>>>>>>>> > >>>>>>>>>>>> What do you guys think? @Jarek Potiuk < > >>> [email protected]> > >>>>>>>>>>>> @Driesprong, > >>>>>>>>>>>> Fokko <[email protected]> > >>>>>>>>>>>> > >>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik < > >>> [email protected]> > >>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A * > >>>>>>>>>>>>> > >>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A * > >>>>>>>>>>>>> > >>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C* > >>>>>>>>>>>>> > >>>>>>>>>>>>> *Case #4* Separate namespace for resources : > >>>>>>>>>>>>> *Solution A* > >>>>>>>>>>>>> > >>>>>>>>>>>>> *Case #5* *Operator : *Solution A* > >>>>>>>>>>>>> > >>>>>>>>>>>>> *Case #7* : *Solution A* > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish > >>>>>>>>>>>> <[email protected]> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> I have tried to add some clarification to Jarek's summary, > >>> but > >>>>>> I > >>>>>>>> am > >>>>>>>>>>>> a > >>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully Jarek > >>>>>> can > >>>>>>>>>>>> further > >>>>>>>>>>>>>> clarify. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6 are all > >>>>>>>> either > >>>>>>>>>>>>>> proposal > >>>>>>>>>>>>>> *rejections* or otherwise have no effect, so attention can be > >>>>>>>>>>>> focused on > >>>>>>>>>>>>>> cases 1,2,3 and 7. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> *Motion is to adopt the following:* > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib > >>>>>>>>>>>>>> All objects will be moved out of contrib folder and into > >>>>>>>> respective > >>>>>>>>>>>>>> non-contrib folder. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from modules > >>>>>>>>>>>>>> Example: > >>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes > >>>>>>>>>>>>>> airflow.operators.foo.FooOperator > >>>>>>>>>>>>>> Example: > >>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes > >>>>>>>> airflow.hooks.foo.FooHook > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes > >>>>>>>>>>>>>> *I am not sure exactly what the proposal is.* > >>>>>>>>>>>>>> Example case is > >>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py > >>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume it is > >>> now > >>>>>>>> named > >>>>>>>>>>>> like > >>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py > >>>>>>>>>>>>>> So what is proposed for this case? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Case 4 - Solution B - *no change* > >>>>>>>>>>>>>> This proposal was about namespaces but the motion is to > >>> reject > >>>>>>>> the > >>>>>>>>>>>>>> proposal. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Case 5 - Solution B - *no change* > >>>>>>>>>>>>>> The proposal was to remove class type suffix e.g. remove the > >>>>>>>>>>>> Operator in > >>>>>>>>>>>>>> FooOperator, but the motion is to reject this proposal -- > >>> i.e. > >>>>>>>>>>>> motion is > >>>>>>>>>>>>>> no > >>>>>>>>>>>>>> change on this case, i.e. we resolve to keep "Operator" (and > >>>>>>>>>>>> "Sensor") > >>>>>>>>>>>>>> suffixes on class names > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Case 6 - *No change* > >>>>>>>>>>>>>> This case concerns object naming inconsistencies, but motion > >>>>>> does > >>>>>>>>>>>> not > >>>>>>>>>>>>>> enact > >>>>>>>>>>>>>> any specific proposal. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for > >>>>>> deprecation > >>>>>>>>>>>> warnings > >>>>>>>>>>>>>> (instead of zope) > >>>>>>>>>>>>>> Example: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> # in old_package/old_mod.py > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> import warnings > >>>>>>>>>>>>>> from new_package.new_mod import * > >>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to > >>>>>>>> new_package.new_mod. > >>>>>>>>>>>>>> Import > >>>>>>>>>>>>>> of " > >>>>>>>>>>>>>> "old_package.old_mod will become unsupported in version 2", > >>>>>>>>>>>>>> DeprecationWarning, 2) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Reference implementation here: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I am very happy that this motion now gets rid of contrib. > >>>>>> Thank > >>>>>>>> you > >>>>>>>>>>>>>> Sergio > >>>>>>>>>>>>>> for turning the tide with your effective argumentation ;) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor < > >>>>>>>> [email protected]> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Could someone summarise what the proposed name changes are, > >>>>>>>> here, > >>>>>>>>>>>> in > >>>>>>>>>>>>>> this > >>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving partial > >>>>>>>> examples > >>>>>>>>>>>> is 1) > >>>>>>>>>>>>>> a > >>>>>>>>>>>>>>> bit difficult to quickly work out what we are voting on, and > >>>>>> 2) > >>>>>>>>>>>> using a > >>>>>>>>>>>>>>> google doc isn't great for a longer term record. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>> Ash > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk > >>>>>>>>>>>> <[email protected]> > >>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from 0 :) . I > >>>>>>>> would > >>>>>>>>>>>> like > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three +1 > >>>>>>>>>>>> (binding) > >>>>>>>>>>>>>> votes > >>>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications to the > >>>>>>>>>>>> previous > >>>>>>>>>>>>>>> proposal. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Here is the proposal (details here > >>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> ) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,* > >>>>>>>>>>>>>>>> - *Case 2: A: Airflow.operators.foo_operator.FooOperator > >>>>>>>> could > >>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator* > >>>>>>>>>>>>>>>> - *Case 3: C: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator > >>>>>>>>>>>>>> could > >>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator* > >>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction* > >>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on > >>>>>>>> class > >>>>>>>>>>>>>> names* > >>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case > >>>>>>>> (and > >>>>>>>>>>>> my team > >>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>> do the job of GCP-related operators)* > >>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE > >>>>>>>>>>>> integration)* > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> This is my binding (+1) vote. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk < > >>>>>>>>>>>>>> [email protected]> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hey Felix, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one as well > >>>>>> (I > >>>>>>>>>>>> use IDE > >>>>>>>>>>>>>>> quite > >>>>>>>>>>>>>>>>> often ;) ) > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> J. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko > >>>>>>>>>>>>>> <[email protected] > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I wasn't > >>>>>>>> able > >>>>>>>>>>>> to > >>>>>>>>>>>>>>> respond > >>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops throughout > >>>>>>>> Europe > >>>>>>>>>>>> :-) > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from contrib > >>>>>> into > >>>>>>>> a > >>>>>>>>>>>> single > >>>>>>>>>>>>>>>>>> package. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference would be > >>>>>>>> to > >>>>>>>>>>>> get > >>>>>>>>>>>>>> rid of > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or not is a > >>>>>>>>>>>> tricky > >>>>>>>>>>>>>>> question, > >>>>>>>>>>>>>>>>>> I think this is already reflected in the document since > >>>>>>>>>>>> "maturity" > >>>>>>>>>>>>>> is > >>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the operator > >>>>>>>> to > >>>>>>>>>>>> the > >>>>>>>>>>>>>> core > >>>>>>>>>>>>>>> or > >>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to > >>>>>> incubating > >>>>>>>>>>>> will > >>>>>>>>>>>>>> confuse > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> user even more. For people who aren't as known with > >>>>>>>> Airflow it > >>>>>>>>>>>> isn't > >>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>>>> transparent which operator lives where, especially if you > >>>>>>>>>>>> don't use > >>>>>>>>>>>>>> an > >>>>>>>>>>>>>>> IDE > >>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is worth > >>>>>>>> changing > >>>>>>>>>>>> the > >>>>>>>>>>>>>>> public > >>>>>>>>>>>>>>>>>> API just for a different name. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with this. It > >>>>>>>> will > >>>>>>>>>>>> make > >>>>>>>>>>>>>>> easier > >>>>>>>>>>>>>>>>> as we will not have to define other rules as those that we > >>>>>>>>>>>> already > >>>>>>>>>>>>>> have. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's say a > >>>>>> +0) > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I like to search on Github with t, and then search for > >>>>>>>>>>>> BashOperator. > >>>>>>>>>>>>>>> After > >>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from your > >>>>>> mail > >>>>>>>> you > >>>>>>>>>>>>>> state: - > >>>>>>>>>>>>>>>>>> *Case 2: B: Airflow.operators.foo_operator.FooOperator > >>>>>>>> could > >>>>>>>>>>>> become > >>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B solution is > >>>>>>>>>>>> keeping it > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B: > >>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would stay > >>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator* > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a little > >>>>>>>>>>>> confusion > >>>>>>>>>>>>>>> here. > >>>>>>>>>>>>>>>>> This case is not for changing BashOperator into Bash but > >>>>>> for > >>>>>>>>>>>> changing > >>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> *module* name not the *class* name. So > >>>>>>>>>>>> bash_operator.BashOperator -> > >>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for > >>>>>>>> BashOperator > >>>>>>>>>>>> in this > >>>>>>>>>>>>>>> case. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when it gets > >>>>>>>> its > >>>>>>>>>>>> own > >>>>>>>>>>>>>>> package > >>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are companies > >>>>>>>> like > >>>>>>>>>>>>>>> Databricks > >>>>>>>>>>>>>>>>>> and Qubole which might also end up with their own package > >>>>>>>>>>>> because > >>>>>>>>>>>>>> their > >>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving this is a > >>>>>>>> bit > >>>>>>>>>>>>>> painful > >>>>>>>>>>>>>>>>>> because we need to deprecate the old path and move > >>>>>>>> everything > >>>>>>>>>>>> which > >>>>>>>>>>>>>>>>>> introduces noise into version control etc. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's not as > >>>>>>>>>>>> difficult as > >>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>> seems to group operators in packages and it has numerous > >>>>>>>>>>>> advantages. > >>>>>>>>>>>>>> The > >>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in the > >>>>>>>>>>>> __init__.py of > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git will > >>>>>>>> actually > >>>>>>>>>>>>>> realise > >>>>>>>>>>>>>>>>> that the file has been moved and you can follow that. > >>>>>> Maybe > >>>>>>>> not > >>>>>>>>>>>> as > >>>>>>>>>>>>>> super > >>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's not a > >>>>>> rocket > >>>>>>>>>>>> science > >>>>>>>>>>>>>>> either. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> *Case 7:* Python native solution > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Yes. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> --- > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Apart from all, great work! > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Cheers, Fokko > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall > >>>>>>>>>>>>>>>>>> <[email protected] > >>>>>>>>>>>>>>>>>>> : > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed change of > >>>>>> these > >>>>>>>>>>>> cases. > >>>>>>>>>>>>>> So I > >>>>>>>>>>>>>>>>>> go > >>>>>>>>>>>>>>>>>>> with +1 (non-binding). > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and your > >>>>>>>> intense > >>>>>>>>>>>> work > >>>>>>>>>>>>>>>>>> towards > >>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even > >>>>>> creating > >>>>>>>>>>>> this > >>>>>>>>>>>>>>>>>> document. I > >>>>>>>>>>>>>>>>>>> like how the code is getting more and more consistent > >>>>>> and > >>>>>>>>>>>> clean :) > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Kind regards, > >>>>>>>>>>>>>>>>>>> Felix > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> -------- Original Message -------- > >>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hello everyone, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in import > >>>>>>>> paths. > >>>>>>>>>>>> It's > >>>>>>>>>>>>>>> been > >>>>>>>>>>>>>>>>>>>> discussed in > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July 30th 6pm > >>>>>>>> CEST), > >>>>>>>>>>>> and > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>>>>>> least > >>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document from > >>>>>> Kamil > >>>>>>>>>>>>>>>>>>>> < > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit# > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> The proposed solution is: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that are > >>>>>> "well" > >>>>>>>>>>>> tested > >>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.* > >>>>>>>>>>>>>>>>>>>> - *Case 2: B: > >>>>>> Airflow.operators.foo_operator.FooOperator > >>>>>>>>>>>> could > >>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator* > >>>>>>>>>>>>>>>>>>>> - *Case 3: C: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>> > >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator > >>>>>>>>>>>>>>> could > >>>>>>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator* > >>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction* > >>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes > >>>>>> on > >>>>>>>>>>>> class > >>>>>>>>>>>>>> names* > >>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case > >>>>>>>> (and > >>>>>>>>>>>> my > >>>>>>>>>>>>>> team > >>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)* > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Best regards, > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> J. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Jarek Potiuk > >>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal > >>>>>> Software > >>>>>>>>>>>> Engineer > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129) > >>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)> > >>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Jarek Potiuk > >>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software > >>>>>>>>>>>> Engineer > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> > >>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Jarek Potiuk > >>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software > >>>>>>>>> Engineer > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> > >>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> *Kaxil Naik* > >>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer* > >>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache > >>>>>> Spark > >>>>>>>> & > >>>>>>>>>>>> Neo4j > >>>>>>>>>>>>> Developer > >>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> -- > >>>>>>>>>>>> *Kaxil Naik* > >>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer* > >>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache > >>> Spark > >>>>>> & > >>>>>>>>>>>> Neo4j > >>>>>>>>>>>> Developer > >>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>>> -- > >>>>>> > >>>>>> Jarek Potiuk > >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>>>>> > >>>>>> M: +48 660 796 129 <+48660796129> > >>>>>> [image: Polidea] <https://www.polidea.com/> > >>>>>> > >>>> > >>>> > >>> > >>> -- > >>> *Kaxil Naik* > >>> *Big Data Consultant | DevOps Data Engineer* > >>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j > >>> Developer > >>> *LinkedIn*: https://www.linkedin.com/in/kaxil > >>> > >> > >> > >> -- > >> > >> Jarek Potiuk > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > >> M: +48 660 796 129 <+48660796129> > >> [image: Polidea] <https://www.polidea.com/> > >> > >> > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> >
