Just cos I'm not sure it's _explicitly_ stated, but all of the moves will have a deprecation of the old name right?
3+4 case D gets my vote too. -a > On 30 Jul 2019, at 09:58, Jarek Potiuk <[email protected]> wrote: > > I went ahead and updated the page (just to speed it up) as I think it > really makes sense to join those two cases (and I do not see any drawbacks > - I think the options we have cover all possible approaches) and we can > always go back if we need to. > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal > > My vote is *D*. > > - airflow/contrib/operators/gcp_bigtable_operator.py → airflow/gcp > /operators/bigtable_operator.py > - airflow/contrib/operators/ssh_operator.py -> > airflow/operators/ssh_operator.py > > I updated the page with my vote. > I propose that we vote till Friday on that one (all the rest is already > agreed). > > J. > > > On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk <[email protected]> > wrote: > >> I think almost everyone voted and we have almost perfect consensus. We all >> agree amongst other on moving all operators out of contrib (Great!). >> >> The only doubts are for *Case 3* (Cloud provider prefix) and *Case 4* >> (Using Namespaces). >> I think there was actually an overlap between those two. Also Ash's >> comment/veto on that is quite clear. >> So I have final (I hope!) proposal to join those two cases and have one >> voting (till Friday) only on that. >> >> I will update the doc if you all agree with me that makes more sense as >> single case to vote on: >> >> *[Case 3 + Case 4]: Grouping Cloud Provider operators.* >> >> *Option A*. Keep operators/sensors/hooks in airflow/operators(sensors, >> hooks) and keep/add prefixes in file names. >> >> - >> *airflow/contrib/operators/sns_publish_operator.py becomes >> airflow/operators/**aws_sns_publish_operator.py* >> >> >> - *airflow/contrib/operators/dataproc_operator.py* >> becomes *airflow/operators/gcp_dataproc_operator.py* >> >> >> - >> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/* >> *hooks/gcp_bigtable_hook.py* >> - >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* >> *operators/ssh_operator.py* >> >> >> >> *Option B*. Group operators/sensors/hooks in airflow/operators(sensors, >> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to >> airflow/operators(sensors/hooks), Drop the prefix. >> >> - >> *airflow/contrib/operators/sns_publish_operator.py >> becomes airflow/operators/**aws/sns_publish_operator.py* >> >> >> - *airflow/contrib/operators/dataproc_operator.py* >> becomes *airflow/operators/gcp/dataproc_operator.py* >> >> >> - >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/* >> *operators/gcp/bigtable_operator.py* >> - >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* >> *operators/ssh_operator.py* >> >> >> *Option C*. Group operators/sensors/hooks in airflow/operators(sensors, >> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to >> airflow/operators(sensors/hooks), Keep the prefix. >> >> - >> *airflow/contrib/operators/sns_publish_operator.py >> becomes airflow/operators/**aws/aws_sns_publish_operator.py* >> >> >> - *airflow/contrib/operators/dataproc_operator.py* >> becomes *airflow/operators/gcp/gcp_dataproc_operator.py* >> >> >> - >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/* >> *operators/gcp/gcp_bigtable_operator.py* >> - >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* >> *operators/ssh_operator.py* >> >> >> *Option D.* Group operators/sensors/hooks in >> *airflow/<PROVIDER>*/operators(sensors, >> hooks). Non-cloud provider ones are moved to >> airflow/operators(sensors/hooks). Drop the prefix. >> >> - >> *airflow/contrib/operators/sns_publish_operator.py >> becomes airflow/aws/operators/**sns_publish_operator.py* >> >> >> - *airflow/contrib/operators/dataproc_operator.py* >> becomes *airflow/gcp/operators/dataproc_operator.py* >> >> >> - >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/* >> *gcp/operators/bigtable_operator.py* >> - >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* >> *operators/ssh_operator.py* >> >> >> *Option E.* Group operators/sensors/hooks in >> *airflow/<PROVIDER>*/operators(sensors, >> hooks). Non-cloud provider ones are moved to >> airflow/operators(sensors/hooks).* Keep the prefix*. >> >> - >> *airflow/contrib/operators/sns_publish_operator.py >> becomes airflow/aws/operators/aws_**sns_publish_operator.py* >> >> >> - *airflow/contrib/operators/dataproc_operator.py* >> becomes *airflow/gcp/operators/gcp_dataproc_operator.py* >> >> >> - >> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/* >> *gcp/operators/gcp_bigtable_operator.py* >> - >> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* >> *operators/ssh_operator.py* >> >> >> Shall I proceed with updating the page ? >> >> >> J. >> >> >> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <[email protected]> wrote: >> >>> Thanks Jarek, adding my vote. >>> >>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor <[email protected]> wrote: >>> >>>> Thanks Jarek, much clearer. I have voted there too. >>>> >>>> -ash >>>> >>>> >>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko <[email protected]> >>>> wrote: >>>>> >>>>> Thanks Jarek, the Wiki works much better for me. I've added my vote >>> there >>>>> as well. >>>>> >>>>> You've convinced me on the second case. I've changed my vote on Case 2 >>>> from >>>>> A to B :-) >>>>> >>>>> Maybe we should leave the vote open a bit longer since it involves >>> quite >>>>> some reading, and I would like to get some proper feedback and >>> opinions >>>> on >>>>> this. Plus I only see two votes right now. If we don't think this >>>> through, >>>>> it might be that we're having this vote again in the near future :-) >>>>> >>>>> Cheers, Fokko >>>>> >>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk < >>>> [email protected]>: >>>>> >>>>>> I updated the AIP-21 proposal in the way that should answer all the >>>>>> concerns in this thread. >>>>>> >>>>>> NOTE! There is an action for all of those who are interested: Please >>>>>> fill-in your voting in the voting table till Tuesday 30th of July 6pm >>>> CEST >>>>>> (5pm BST, 9 am PST). >>>>>> >>>>>> I created a summary of the proposal >>>>>> < >>>>>> >>>> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal >>>>>>> >>>>>> in table form - which contains also examples for each case. >>>>>> >>>>>> I added a voting table >>>>>> < >>>>>> >>>> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting >>>>>>> >>>>>> where you should add your votes: >>>>>> >>>>>> I propose this voting mechanism: >>>>>> >>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5 pm >>> BST, >>>> 9am >>>>>> PST) . >>>>>> >>>>>> After the choice, the final consistent set of choices will be >>> announced >>>>>> (taking into account majority of binding vote, also including >>> potential >>>>>> vetos and consistency between the choices. Non-binding votes will be >>>> taken >>>>>> into account in case there is a draw. The final set of choices will >>> be >>>>>> announced at devlist thread >>>>>> < >>>>>> >>>> >>> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E >>>>>>> >>>>>> after >>>>>> the voting completes. >>>>>> >>>>>> Unless there is a veto raised to the final proposal, the final >>> proposal >>>> is >>>>>> accepted by Lazy Consensus >>>>>> <https://community.apache.org/committers/lazyConsensus.html> on >>>> *Friday* >>>>>> 02 >>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST). >>>>>> >>>>>> Please let me know if what I proposed can be further improved to >>> avoid >>>>>> chaos. >>>>>> >>>>>> J. >>>>>> >>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk < >>> [email protected] >>>>> >>>>>> wrote: >>>>>> >>>>>>> Ok. I see that indeed voting could have been organised a bit better >>> - >>>> dev >>>>>>> list does not seem to cope well with such a complex case :). >>>>>>> >>>>>>> As Kamil noticed - we already have a formal AIP-21 >>>>>>> < >>>>>> >>>> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths >>>>>>> >>>>>>> in the Wiki - which I should have mentioned before. I have not much >>>> time >>>>>>> today - but tomorrow I will try to summarise the options - based on >>>> some >>>>>>> real code examples (to make it easier to digest). I think the best >>>>>> approach >>>>>>> will be to make a voting matrix in the AIP-21 Wiki page where people >>>> will >>>>>>> be able to state their preferences for each case (by adding a row to >>>> the >>>>>>> table). I think that might provide much better clarity and a single >>>> place >>>>>>> which will show the current aggregated state of voting. >>>>>>> >>>>>>> However - as Ash also stated - some of the options are >>> inter-dependent >>>> so >>>>>>> at the end we will have to adjust the results in case they are not >>>>>>> consistent - and make final voting on aggregated set of cases - >>> but I >>>>>>> think in this case following "lasy consensus" - i,e. if noone >>> objects >>>> to >>>>>>> final set of cases, we will proceed with it.. >>>>>>> >>>>>>> Do you think that will work better ? >>>>>>> >>>>>>> J. >>>>>>> >>>>>>> Principal Software Engineer >>>>>>> Phone: +48660796129 >>>>>>> >>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła < >>>>>>> [email protected]> napisał: >>>>>>> >>>>>>>> Document is also available as AIP on wiki: >>>>>>>> >>>>>>>> >>>>>> >>>> >>> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths >>>>>>>> >>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko >>> <[email protected] >>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I agree with all, this is a bit chaotic. For the sake of clarity, >>> I >>>>>>>> would >>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP. Create a >>>>>> structural >>>>>>>>> code improvement AIP and then and add a matrix of users/cases >>> where >>>>>>>>> everybody can add his vote per case. This gives also the >>> opportunity >>>>>> for >>>>>>>>> changing your vote on a specific case. WDYT? >>>>>>>>> >>>>>>>>> Cheers, Fokko >>>>>>>>> >>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <[email protected]>: >>>>>>>>> >>>>>>>>>> I agree with Ash. It is clear to vote on each case rather than >>> all >>>>>>>>>> together. >>>>>>>>>> >>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor < >>> [email protected]> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> A number of proposals overlap or add on to each other, so I >>> think >>>>>>>> (?) a >>>>>>>>>>> single vote makes sense. >>>>>>>>>>> >>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly what we are >>>>>>>> voting >>>>>>>>> on. >>>>>>>>>>> >>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik <[email protected]> >>>>>>>> wrote: >>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases together, >>>>>> how >>>>>>>>>>>> about >>>>>>>>>>>> separate vote emails for all the cases? Each email can have the >>>>>>>>>>>> description >>>>>>>>>>>> copied over from the doc. >>>>>>>>>>>> >>>>>>>>>>>> It would probably be a bit easier to track a decision in the >>>>>> future >>>>>>>> as >>>>>>>>>>>> well. >>>>>>>>>>>> >>>>>>>>>>>> What do you guys think? @Jarek Potiuk < >>> [email protected]> >>>>>>>>>>>> @Driesprong, >>>>>>>>>>>> Fokko <[email protected]> >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik < >>> [email protected]> >>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A * >>>>>>>>>>>>> >>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A * >>>>>>>>>>>>> >>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C* >>>>>>>>>>>>> >>>>>>>>>>>>> *Case #4* Separate namespace for resources : >>>>>>>>>>>>> *Solution A* >>>>>>>>>>>>> >>>>>>>>>>>>> *Case #5* *Operator : *Solution A* >>>>>>>>>>>>> >>>>>>>>>>>>> *Case #7* : *Solution A* >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish >>>>>>>>>>>> <[email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I have tried to add some clarification to Jarek's summary, >>> but >>>>>> I >>>>>>>> am >>>>>>>>>>>> a >>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully Jarek >>>>>> can >>>>>>>>>>>> further >>>>>>>>>>>>>> clarify. >>>>>>>>>>>>>> >>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6 are all >>>>>>>> either >>>>>>>>>>>>>> proposal >>>>>>>>>>>>>> *rejections* or otherwise have no effect, so attention can be >>>>>>>>>>>> focused on >>>>>>>>>>>>>> cases 1,2,3 and 7. >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Motion is to adopt the following:* >>>>>>>>>>>>>> >>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib >>>>>>>>>>>>>> All objects will be moved out of contrib folder and into >>>>>>>> respective >>>>>>>>>>>>>> non-contrib folder. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from modules >>>>>>>>>>>>>> Example: >>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes >>>>>>>>>>>>>> airflow.operators.foo.FooOperator >>>>>>>>>>>>>> Example: >>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes >>>>>>>> airflow.hooks.foo.FooHook >>>>>>>>>>>>>> >>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes >>>>>>>>>>>>>> *I am not sure exactly what the proposal is.* >>>>>>>>>>>>>> Example case is >>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py >>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume it is >>> now >>>>>>>> named >>>>>>>>>>>> like >>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py >>>>>>>>>>>>>> So what is proposed for this case? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Case 4 - Solution B - *no change* >>>>>>>>>>>>>> This proposal was about namespaces but the motion is to >>> reject >>>>>>>> the >>>>>>>>>>>>>> proposal. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Case 5 - Solution B - *no change* >>>>>>>>>>>>>> The proposal was to remove class type suffix e.g. remove the >>>>>>>>>>>> Operator in >>>>>>>>>>>>>> FooOperator, but the motion is to reject this proposal -- >>> i.e. >>>>>>>>>>>> motion is >>>>>>>>>>>>>> no >>>>>>>>>>>>>> change on this case, i.e. we resolve to keep "Operator" (and >>>>>>>>>>>> "Sensor") >>>>>>>>>>>>>> suffixes on class names >>>>>>>>>>>>>> >>>>>>>>>>>>>> Case 6 - *No change* >>>>>>>>>>>>>> This case concerns object naming inconsistencies, but motion >>>>>> does >>>>>>>>>>>> not >>>>>>>>>>>>>> enact >>>>>>>>>>>>>> any specific proposal. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for >>>>>> deprecation >>>>>>>>>>>> warnings >>>>>>>>>>>>>> (instead of zope) >>>>>>>>>>>>>> Example: >>>>>>>>>>>>>> >>>>>>>>>>>>>> # in old_package/old_mod.py >>>>>>>>>>>>>> >>>>>>>>>>>>>> import warnings >>>>>>>>>>>>>> from new_package.new_mod import * >>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to >>>>>>>> new_package.new_mod. >>>>>>>>>>>>>> Import >>>>>>>>>>>>>> of " >>>>>>>>>>>>>> "old_package.old_mod will become unsupported in version 2", >>>>>>>>>>>>>> DeprecationWarning, 2) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Reference implementation here: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>> >>> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am very happy that this motion now gets rid of contrib. >>>>>> Thank >>>>>>>> you >>>>>>>>>>>>>> Sergio >>>>>>>>>>>>>> for turning the tide with your effective argumentation ;) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor < >>>>>>>> [email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Could someone summarise what the proposed name changes are, >>>>>>>> here, >>>>>>>>>>>> in >>>>>>>>>>>>>> this >>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving partial >>>>>>>> examples >>>>>>>>>>>> is 1) >>>>>>>>>>>>>> a >>>>>>>>>>>>>>> bit difficult to quickly work out what we are voting on, and >>>>>> 2) >>>>>>>>>>>> using a >>>>>>>>>>>>>>> google doc isn't great for a longer term record. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Ash >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk >>>>>>>>>>>> <[email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from 0 :) . I >>>>>>>> would >>>>>>>>>>>> like >>>>>>>>>>>>>> to >>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three +1 >>>>>>>>>>>> (binding) >>>>>>>>>>>>>> votes >>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications to the >>>>>>>>>>>> previous >>>>>>>>>>>>>>> proposal. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Here is the proposal (details here >>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>> >>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,* >>>>>>>>>>>>>>>> - *Case 2: A: Airflow.operators.foo_operator.FooOperator >>>>>>>> could >>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator* >>>>>>>>>>>>>>>> - *Case 3: C: >>>>>>>>>>>>>>>> >>>>>>>>>>>> >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator >>>>>>>>>>>>>> could >>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator* >>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction* >>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on >>>>>>>> class >>>>>>>>>>>>>> names* >>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case >>>>>>>> (and >>>>>>>>>>>> my team >>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>> do the job of GCP-related operators)* >>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE >>>>>>>>>>>> integration)* >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This is my binding (+1) vote. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk < >>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hey Felix, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one as well >>>>>> (I >>>>>>>>>>>> use IDE >>>>>>>>>>>>>>> quite >>>>>>>>>>>>>>>>> often ;) ) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> J. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko >>>>>>>>>>>>>> <[email protected] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I wasn't >>>>>>>> able >>>>>>>>>>>> to >>>>>>>>>>>>>>> respond >>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops throughout >>>>>>>> Europe >>>>>>>>>>>> :-) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from contrib >>>>>> into >>>>>>>> a >>>>>>>>>>>> single >>>>>>>>>>>>>>>>>> package. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference would be >>>>>>>> to >>>>>>>>>>>> get >>>>>>>>>>>>>> rid of >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or not is a >>>>>>>>>>>> tricky >>>>>>>>>>>>>>> question, >>>>>>>>>>>>>>>>>> I think this is already reflected in the document since >>>>>>>>>>>> "maturity" >>>>>>>>>>>>>> is >>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the operator >>>>>>>> to >>>>>>>>>>>> the >>>>>>>>>>>>>> core >>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to >>>>>> incubating >>>>>>>>>>>> will >>>>>>>>>>>>>> confuse >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> user even more. For people who aren't as known with >>>>>>>> Airflow it >>>>>>>>>>>> isn't >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>> transparent which operator lives where, especially if you >>>>>>>>>>>> don't use >>>>>>>>>>>>>> an >>>>>>>>>>>>>>> IDE >>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is worth >>>>>>>> changing >>>>>>>>>>>> the >>>>>>>>>>>>>>> public >>>>>>>>>>>>>>>>>> API just for a different name. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with this. It >>>>>>>> will >>>>>>>>>>>> make >>>>>>>>>>>>>>> easier >>>>>>>>>>>>>>>>> as we will not have to define other rules as those that we >>>>>>>>>>>> already >>>>>>>>>>>>>> have. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's say a >>>>>> +0) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I like to search on Github with t, and then search for >>>>>>>>>>>> BashOperator. >>>>>>>>>>>>>>> After >>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from your >>>>>> mail >>>>>>>> you >>>>>>>>>>>>>> state: - >>>>>>>>>>>>>>>>>> *Case 2: B: Airflow.operators.foo_operator.FooOperator >>>>>>>> could >>>>>>>>>>>> become >>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B solution is >>>>>>>>>>>> keeping it >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B: >>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would stay >>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator* >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a little >>>>>>>>>>>> confusion >>>>>>>>>>>>>>> here. >>>>>>>>>>>>>>>>> This case is not for changing BashOperator into Bash but >>>>>> for >>>>>>>>>>>> changing >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> *module* name not the *class* name. So >>>>>>>>>>>> bash_operator.BashOperator -> >>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for >>>>>>>> BashOperator >>>>>>>>>>>> in this >>>>>>>>>>>>>>> case. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when it gets >>>>>>>> its >>>>>>>>>>>> own >>>>>>>>>>>>>>> package >>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are companies >>>>>>>> like >>>>>>>>>>>>>>> Databricks >>>>>>>>>>>>>>>>>> and Qubole which might also end up with their own package >>>>>>>>>>>> because >>>>>>>>>>>>>> their >>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving this is a >>>>>>>> bit >>>>>>>>>>>>>> painful >>>>>>>>>>>>>>>>>> because we need to deprecate the old path and move >>>>>>>> everything >>>>>>>>>>>> which >>>>>>>>>>>>>>>>>> introduces noise into version control etc. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's not as >>>>>>>>>>>> difficult as >>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>> seems to group operators in packages and it has numerous >>>>>>>>>>>> advantages. >>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in the >>>>>>>>>>>> __init__.py of >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git will >>>>>>>> actually >>>>>>>>>>>>>> realise >>>>>>>>>>>>>>>>> that the file has been moved and you can follow that. >>>>>> Maybe >>>>>>>> not >>>>>>>>>>>> as >>>>>>>>>>>>>> super >>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's not a >>>>>> rocket >>>>>>>>>>>> science >>>>>>>>>>>>>>> either. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> *Case 7:* Python native solution >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yes. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Apart from all, great work! >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Cheers, Fokko >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall >>>>>>>>>>>>>>>>>> <[email protected] >>>>>>>>>>>>>>>>>>> : >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed change of >>>>>> these >>>>>>>>>>>> cases. >>>>>>>>>>>>>> So I >>>>>>>>>>>>>>>>>> go >>>>>>>>>>>>>>>>>>> with +1 (non-binding). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and your >>>>>>>> intense >>>>>>>>>>>> work >>>>>>>>>>>>>>>>>> towards >>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even >>>>>> creating >>>>>>>>>>>> this >>>>>>>>>>>>>>>>>> document. I >>>>>>>>>>>>>>>>>>> like how the code is getting more and more consistent >>>>>> and >>>>>>>>>>>> clean :) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>>>>>>>> Felix >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -------- Original Message -------- >>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hello everyone, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in import >>>>>>>> paths. >>>>>>>>>>>> It's >>>>>>>>>>>>>>> been >>>>>>>>>>>>>>>>>>>> discussed in >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>> >>> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July 30th 6pm >>>>>>>> CEST), >>>>>>>>>>>> and >>>>>>>>>>>>>> at >>>>>>>>>>>>>>>>>> least >>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document from >>>>>> Kamil >>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>> >>> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit# >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> The proposed solution is: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that are >>>>>> "well" >>>>>>>>>>>> tested >>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.* >>>>>>>>>>>>>>>>>>>> - *Case 2: B: >>>>>> Airflow.operators.foo_operator.FooOperator >>>>>>>>>>>> could >>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator* >>>>>>>>>>>>>>>>>>>> - *Case 3: C: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>> >>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator >>>>>>>>>>>>>>> could >>>>>>>>>>>>>>>>>>>> become airflow.gcp.operators.bigtable.BigTableOperator* >>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction* >>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes >>>>>> on >>>>>>>>>>>> class >>>>>>>>>>>>>> names* >>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on case-by-case >>>>>>>> (and >>>>>>>>>>>> my >>>>>>>>>>>>>> team >>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)* >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> J. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Jarek Potiuk >>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal >>>>>> Software >>>>>>>>>>>> Engineer >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129) >>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)> >>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Jarek Potiuk >>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software >>>>>>>>>>>> Engineer >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> >>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Jarek Potiuk >>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software >>>>>>>>> Engineer >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> >>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> *Kaxil Naik* >>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer* >>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache >>>>>> Spark >>>>>>>> & >>>>>>>>>>>> Neo4j >>>>>>>>>>>>> Developer >>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> *Kaxil Naik* >>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer* >>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache >>> Spark >>>>>> & >>>>>>>>>>>> Neo4j >>>>>>>>>>>> Developer >>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Jarek Potiuk >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>>>>> >>>>>> M: +48 660 796 129 <+48660796129> >>>>>> [image: Polidea] <https://www.polidea.com/> >>>>>> >>>> >>>> >>> >>> -- >>> *Kaxil Naik* >>> *Big Data Consultant | DevOps Data Engineer* >>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j >>> Developer >>> *LinkedIn*: https://www.linkedin.com/in/kaxil >>> >> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/>
