I updated the AIP-21 proposal in the way that should answer all the concerns in this thread.
NOTE! There is an action for all of those who are interested: Please fill-in your voting in the voting table till Tuesday 30th of July 6pm CEST (5pm BST, 9 am PST). I created a summary of the proposal <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal> in table form - which contains also examples for each case. I added a voting table <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting> where you should add your votes: I propose this voting mechanism: Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5 pm BST, 9am PST) . After the choice, the final consistent set of choices will be announced (taking into account majority of binding vote, also including potential vetos and consistency between the choices. Non-binding votes will be taken into account in case there is a draw. The final set of choices will be announced at devlist thread <https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E> after the voting completes. Unless there is a veto raised to the final proposal, the final proposal is accepted by Lazy Consensus <https://community.apache.org/committers/lazyConsensus.html> on *Friday* 02 Aug 2019 at 6pm CEST (5 pm BST, 9am PST). Please let me know if what I proposed can be further improved to avoid chaos. J. On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk <[email protected]> wrote: > Ok. I see that indeed voting could have been organised a bit better - dev > list does not seem to cope well with such a complex case :). > > As Kamil noticed - we already have a formal AIP-21 > <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths> > in the Wiki - which I should have mentioned before. I have not much time > today - but tomorrow I will try to summarise the options - based on some > real code examples (to make it easier to digest). I think the best approach > will be to make a voting matrix in the AIP-21 Wiki page where people will > be able to state their preferences for each case (by adding a row to the > table). I think that might provide much better clarity and a single place > which will show the current aggregated state of voting. > > However - as Ash also stated - some of the options are inter-dependent so > at the end we will have to adjust the results in case they are not > consistent - and make final voting on aggregated set of cases - but I > think in this case following "lasy consensus" - i,e. if noone objects to > final set of cases, we will proceed with it.. > > Do you think that will work better ? > > J. > > Principal Software Engineer > Phone: +48660796129 > > sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła < > [email protected]> napisał: > >> Document is also available as AIP on wiki: >> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths >> >> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko <[email protected]> >> wrote: >> >> > I agree with all, this is a bit chaotic. For the sake of clarity, I >> would >> > suggest moving the Google Doc to the Wiki as an AIP. Create a structural >> > code improvement AIP and then and add a matrix of users/cases where >> > everybody can add his vote per case. This gives also the opportunity for >> > changing your vote on a specific case. WDYT? >> > >> > Cheers, Fokko >> > >> > Op vr 26 jul. 2019 om 22:28 schreef Chen Tong <[email protected]>: >> > >> > > I agree with Ash. It is clear to vote on each case rather than all >> > > together. >> > > >> > > On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor <[email protected]> >> > wrote: >> > > >> > >> A number of proposals overlap or add on to each other, so I think >> (?) a >> > >> single vote makes sense. >> > >> >> > >> To be clear, my vote is -1 until it's clear exactly what we are >> voting >> > on. >> > >> >> > >> On 26 July 2019 20:39:56 BST, Kaxil Naik <[email protected]> >> wrote: >> > >> >I agree with Ash and instead of voting on all 7 Cases together, how >> > >> >about >> > >> >separate vote emails for all the cases? Each email can have the >> > >> >description >> > >> >copied over from the doc. >> > >> > >> > >> >It would probably be a bit easier to track a decision in the future >> as >> > >> >well. >> > >> > >> > >> >What do you guys think? @Jarek Potiuk <[email protected]> >> > >> >@Driesprong, >> > >> >Fokko <[email protected]> >> > >> > >> > >> >On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik <[email protected]> >> > wrote: >> > >> > >> > >> >> *Case #1* airflow.contrib.{resources} : *Solution A * >> > >> >> >> > >> >> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A * >> > >> >> >> > >> >> *Case #3* {aws/azure/gcp}_* : *Solution C* >> > >> >> >> > >> >> *Case #4* Separate namespace for resources : >> > >> >> *Solution A* >> > >> >> >> > >> >> *Case #5* *Operator : *Solution A* >> > >> >> >> > >> >> *Case #7* : *Solution A* >> > >> >> >> > >> >> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish >> > >> ><[email protected]> >> > >> >> wrote: >> > >> >> >> > >> >>> I have tried to add some clarification to Jarek's summary, but I >> am >> > >> >a >> > >> >>> little fuzzy on exact proposal for case 3 so hopefully Jarek can >> > >> >further >> > >> >>> clarify. >> > >> >>> >> > >> >>> According to my reading, in this motion cases 4,5,6 are all >> either >> > >> >>> proposal >> > >> >>> *rejections* or otherwise have no effect, so attention can be >> > >> >focused on >> > >> >>> cases 1,2,3 and 7. >> > >> >>> >> > >> >>> *Motion is to adopt the following:* >> > >> >>> >> > >> >>> Case 1 - Solution A - Get rid of contrib >> > >> >>> All objects will be moved out of contrib folder and into >> respective >> > >> >>> non-contrib folder. >> > >> >>> >> > >> >>> Case 2 - Solution A - Remove object type suffix from modules >> > >> >>> Example: >> > >> >>> Airflow.operators.foo_operator.FooOperator becomes >> > >> >>> airflow.operators.foo.FooOperator >> > >> >>> Example: >> > >> >>> Airflow.hooks.foo_hook.FooOperator becomes >> airflow.hooks.foo.FooHook >> > >> >>> >> > >> >>> Case 3 - conventions for cloud provider etc prefixes >> > >> >>> *I am not sure exactly what the proposal is.* >> > >> >>> Example case is >> airflow/contrib/operators/gcp_bigtable_operator.py >> > >> >>> Since motion adopts case 1 solution A, we can assume it is now >> named >> > >> >like >> > >> >>> so: airflow/operators/gcp_bigtable_operator.py >> > >> >>> So what is proposed for this case? >> > >> >>> >> > >> >>> Case 4 - Solution B - *no change* >> > >> >>> This proposal was about namespaces but the motion is to reject >> the >> > >> >>> proposal. >> > >> >>> >> > >> >>> Case 5 - Solution B - *no change* >> > >> >>> The proposal was to remove class type suffix e.g. remove the >> > >> >Operator in >> > >> >>> FooOperator, but the motion is to reject this proposal -- i.e. >> > >> >motion is >> > >> >>> no >> > >> >>> change on this case, i.e. we resolve to keep "Operator" (and >> > >> >"Sensor") >> > >> >>> suffixes on class names >> > >> >>> >> > >> >>> Case 6 - *No change* >> > >> >>> This case concerns object naming inconsistencies, but motion does >> > >> >not >> > >> >>> enact >> > >> >>> any specific proposal. >> > >> >>> >> > >> >>> Case 7 - Solution A - use warnings.warn template for deprecation >> > >> >warnings >> > >> >>> (instead of zope) >> > >> >>> Example: >> > >> >>> >> > >> >>> # in old_package/old_mod.py >> > >> >>> >> > >> >>> import warnings >> > >> >>> from new_package.new_mod import * >> > >> >>> warnings.warn("old_package.old_mod has moved to >> new_package.new_mod. >> > >> >>> Import >> > >> >>> of " >> > >> >>> "old_package.old_mod will become unsupported in version 2", >> > >> >>> DeprecationWarning, 2) >> > >> >>> >> > >> >>> Reference implementation here: >> > >> >>> >> > >> >>> >> > >> > >> > >> >> > >> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1 >> > >> >>> >> > >> >>> >> > >> >>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3. >> > >> >>> >> > >> >>> I am very happy that this motion now gets rid of contrib. Thank >> you >> > >> >>> Sergio >> > >> >>> for turning the tide with your effective argumentation ;) >> > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor < >> [email protected]> >> > >> >wrote: >> > >> >>> >> > >> >>> > Could someone summarise what the proposed name changes are, >> here, >> > >> >in >> > >> >>> this >> > >> >>> > thread? Pointing at a google doc and only giving partial >> examples >> > >> >is 1) >> > >> >>> a >> > >> >>> > bit difficult to quickly work out what we are voting on, and 2) >> > >> >using a >> > >> >>> > google doc isn't great for a longer term record. >> > >> >>> > >> > >> >>> > Thanks, >> > >> >>> > Ash >> > >> >>> > >> > >> >>> > > On 26 Jul 2019, at 09:14, Jarek Potiuk >> > >> ><[email protected]> >> > >> >>> wrote: >> > >> >>> > > >> > >> >>> > > I would like to recast the vote. Let's start from 0 :) . I >> would >> > >> >like >> > >> >>> to >> > >> >>> > > keep the July 30th 6pm CEST date, and at least three +1 >> > >> >(binding) >> > >> >>> votes >> > >> >>> > are >> > >> >>> > > needed to pass. I marked in red the modifications to the >> > >> >previous >> > >> >>> > proposal. >> > >> >>> > > >> > >> >>> > > Here is the proposal (details here >> > >> >>> > > < >> > >> >>> > >> > >> >>> >> > >> > >> > >> >> > >> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo >> > >> >>> > > >> > >> >>> > > ) >> > >> >>> > > >> > >> >>> > > - *Case 1: A: Get rid of contrib,* >> > >> >>> > > - *Case 2: A: Airflow.operators.foo_operator.FooOperator >> could >> > >> >>> > > become airflow.operators.foo.FooOperator* >> > >> >>> > > - *Case 3: C: >> > >> >>> > > >> > >> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator >> > >> >>> could >> > >> >>> > > become airflow.gcp.operators.bigtable.BigTableOperator* >> > >> >>> > > - *Case 4: B: no namespace introduction* >> > >> >>> > > - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on >> class >> > >> >>> names* >> > >> >>> > > - *Case 6: We will treat isolated cases on case-by-case >> (and >> > >> >my team >> > >> >>> > can >> > >> >>> > > do the job of GCP-related operators)* >> > >> >>> > > - *Case 7: Native python solution (with better IDE >> > >> >integration)* >> > >> >>> > > >> > >> >>> > > This is my binding (+1) vote. >> > >> >>> > > >> > >> >>> > > >> > >> >>> > > >> > >> >>> > > On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk < >> > >> >>> [email protected]> >> > >> >>> > > wrote: >> > >> >>> > > >> > >> >>> > >> Hey Felix, >> > >> >>> > >> >> > >> >>> > >> For 7* -> I am in favour of trying the native one as well (I >> > >> >use IDE >> > >> >>> > quite >> > >> >>> > >> often ;) ) >> > >> >>> > >> >> > >> >>> > >> J. >> > >> >>> > >> >> > >> >>> > >> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko >> > >> >>> <[email protected] >> > >> >>> > > >> > >> >>> > >> wrote: >> > >> >>> > >> >> > >> >>> > >>> Thanks Kamil for putting the document together. I wasn't >> able >> > >> >to >> > >> >>> > respond >> > >> >>> > >>> earlier since I was giving Airflow workshops throughout >> Europe >> > >> >:-) >> > >> >>> > >>> >> > >> >>> > >>> *Case 1: *Solution A → Remove everything from contrib into >> a >> > >> >single >> > >> >>> > >>> package. >> > >> >>> > >>> >> > >> >>> > >>> To make it easier to the end-user, my preference would be >> to >> > >> >get >> > >> >>> rid of >> > >> >>> > >>> the >> > >> >>> > >>> contrib package altogether. What's in contrib or not is a >> > >> >tricky >> > >> >>> > question, >> > >> >>> > >>> I think this is already reflected in the document since >> > >> >"maturity" >> > >> >>> is >> > >> >>> > in >> > >> >>> > >>> quotes. What would be the moment to transfer the operator >> to >> > >> >the >> > >> >>> core >> > >> >>> > or >> > >> >>> > >>> vice versa? I'm afraid that renaming contrib to incubating >> > >> >will >> > >> >>> confuse >> > >> >>> > >>> the >> > >> >>> > >>> user even more. For people who aren't as known with >> Airflow it >> > >> >isn't >> > >> >>> > that >> > >> >>> > >>> transparent which operator lives where, especially if you >> > >> >don't use >> > >> >>> an >> > >> >>> > IDE >> > >> >>> > >>> such as Ash ;) Besides that I don't think it is worth >> changing >> > >> >the >> > >> >>> > public >> > >> >>> > >>> API just for a different name. >> > >> >>> > >>> >> > >> >>> > >> >> > >> >>> > >> Ok. I am convinced. I will re-cast the vote with this. It >> will >> > >> >make >> > >> >>> > easier >> > >> >>> > >> as we will not have to define other rules as those that we >> > >> >already >> > >> >>> have. >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >>> *Case 2:* Slight preference to Solution B (let's say a +0) >> > >> >>> > >>> >> > >> >>> > >>> I like to search on Github with t, and then search for >> > >> >BashOperator. >> > >> >>> > After >> > >> >>> > >>> the change, you should search for Operator/Bash. >> > >> >>> > >>> >> > >> >>> > >>> Jarek, I'm confused with your vote on this, from your mail >> you >> > >> >>> state: - >> > >> >>> > >>> *Case 2: B: Airflow.operators.foo_operator.FooOperator >> could >> > >> >become >> > >> >>> > >>> airflow.operators.foo.FooOperator*. But the B solution is >> > >> >keeping it >> > >> >>> > the >> > >> >>> > >>> same, so that would mean - *Case 2: B: >> > >> >>> > >>> Airflow.operators.foo_operator.FooOperator *would stay >> > >> >>> > >>> airflow.operators.foo_operator*.FooOperator* >> > >> >>> > >>> >> > >> >>> > >>> My mistake. It was of course A. I think there is a little >> > >> >confusion >> > >> >>> > here. >> > >> >>> > >> This case is not for changing BashOperator into Bash but for >> > >> >changing >> > >> >>> > the >> > >> >>> > >> *module* name not the *class* name. So >> > >> >bash_operator.BashOperator -> >> > >> >>> > >> bash.BashOperator :) . you will still search for >> BashOperator >> > >> >in this >> > >> >>> > case. >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >>> *Case 3:* I'm leaning towards B >> > >> >>> > >>> >> > >> >>> > >>> Mostly because it isn't as trivial to decide when it gets >> its >> > >> >own >> > >> >>> > package >> > >> >>> > >>> or not. Besides the cloud providers, there are companies >> like >> > >> >>> > Databricks >> > >> >>> > >>> and Qubole which might also end up with their own package >> > >> >because >> > >> >>> their >> > >> >>> > >>> integration is getting richer over time. Moving this is a >> bit >> > >> >>> painful >> > >> >>> > >>> because we need to deprecate the old path and move >> everything >> > >> >which >> > >> >>> > >>> introduces noise into version control etc. >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> I'd say, we should go for C. As I proposed. It's not as >> > >> >difficult as >> > >> >>> it >> > >> >>> > >> seems to group operators in packages and it has numerous >> > >> >advantages. >> > >> >>> The >> > >> >>> > >> nice thing about deprecations - we can do them in the >> > >> >__init__.py of >> > >> >>> the >> > >> >>> > >> packages which causes the noise fairly small (Git will >> actually >> > >> >>> realise >> > >> >>> > >> that the file has been moved and you can follow that. Maybe >> not >> > >> >as >> > >> >>> super >> > >> >>> > >> easy to merge changes later to 1.10.4 but it's not a rocket >> > >> >science >> > >> >>> > either. >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >>> *Case 7:* Python native solution >> > >> >>> > >>> >> > >> >>> > >> >> > >> >>> > >> Yes. >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >>> >> > >> >>> > >>> --- >> > >> >>> > >>> >> > >> >>> > >>> Apart from all, great work! >> > >> >>> > >>> >> > >> >>> > >>> Cheers, Fokko >> > >> >>> > >>> >> > >> >>> > >>> >> > >> >>> > >>> Op di 23 jul. 2019 om 21:27 schreef Felix Uellendall >> > >> >>> > >>> <[email protected] >> > >> >>> > >>>> : >> > >> >>> > >>> >> > >> >>> > >>>> I have no strong "No" against any proposed change of these >> > >> >cases. >> > >> >>> So I >> > >> >>> > >>> go >> > >> >>> > >>>> with +1 (non-binding). >> > >> >>> > >>>> >> > >> >>> > >>>> P.S. Thanks Jarek for bringing this up again and your >> intense >> > >> >work >> > >> >>> > >>> towards >> > >> >>> > >>>> airflow currently :) and thanks to Kamil for even creating >> > >> >this >> > >> >>> > >>> document. I >> > >> >>> > >>>> like how the code is getting more and more consistent and >> > >> >clean :) >> > >> >>> > >>>> >> > >> >>> > >>>> Kind regards, >> > >> >>> > >>>> Felix >> > >> >>> > >>>> >> > >> >>> > >>>> Sent from ProtonMail mobile >> > >> >>> > >>>> >> > >> >>> > >>>> -------- Original Message -------- >> > >> >>> > >>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote: >> > >> >>> > >>>> >> > >> >>> > >>>>> Hello everyone, >> > >> >>> > >>>>> >> > >> >>> > >>>>> This email is calling a vote on the changes in import >> paths. >> > >> >It's >> > >> >>> > been >> > >> >>> > >>>>> discussed in >> > >> >>> > >>>>> >> > >> >>> > >>>> >> > >> >>> > >>> >> > >> >>> > >> > >> >>> >> > >> > >> > >> >> > >> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E >> > >> >>> > >>>>> >> > >> >>> > >>>>> The vote will last for at least 1 week (July 30th 6pm >> CEST), >> > >> >and >> > >> >>> at >> > >> >>> > >>> least >> > >> >>> > >>>>> three +1 (binding) votes have been cast. >> > >> >>> > >>>>> >> > >> >>> > >>>>> The proposal to vote is based on the document from Kamil >> > >> >>> > >>>>> < >> > >> >>> > >>>> >> > >> >>> > >>> >> > >> >>> > >> > >> >>> >> > >> > >> > >> >> > >> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit# >> > >> >>> > >>>>> >> > >> >>> > >>>>> >> > >> >>> > >>>>> The proposed solution is: >> > >> >>> > >>>>> >> > >> >>> > >>>>> - *Case 1: B: Contrib vs not: we move all that are "well" >> > >> >tested >> > >> >>> and >> > >> >>> > >>>>> rename contrib to "incubating" or similar.* >> > >> >>> > >>>>> - *Case 2: B: Airflow.operators.foo_operator.FooOperator >> > >> >could >> > >> >>> > >>>>> become airflow.operators.foo.FooOperator* >> > >> >>> > >>>>> - *Case 3: C: >> > >> >>> > >>>>> >> > >> >airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator >> > >> >>> > could >> > >> >>> > >>>>> become airflow.gcp.operators.bigtable.BigTableOperator* >> > >> >>> > >>>>> - *Case 4: B: no namespace introduction* >> > >> >>> > >>>>> - *Case 5: B: Keep "Operator" (and "Sensor") suffixes on >> > >> >class >> > >> >>> names* >> > >> >>> > >>>>> - *Case 6: We will treat isolated cases on case-by-case >> (and >> > >> >my >> > >> >>> team >> > >> >>> > >>> can >> > >> >>> > >>>>> do the job of GCP-related operators)* >> > >> >>> > >>>>> >> > >> >>> > >>>>> This is my (binding) +1 vote. >> > >> >>> > >>>>> >> > >> >>> > >>>>> Best regards, >> > >> >>> > >>>>> >> > >> >>> > >>>>> J. >> > >> >>> > >>>>> >> > >> >>> > >>>>> -- >> > >> >>> > >>>>> >> > >> >>> > >>>>> Jarek Potiuk >> > >> >>> > >>>>> Polidea <https://www.polidea.com/> | Principal Software >> > >> >Engineer >> > >> >>> > >>>>> >> > >> >>> > >>>>> M: [+48 660 796 129](tel:+48660796129) >> > >> >>> > >>> <[+48660796129](tel:+48660796129)> >> > >> >>> > >>>>> [image: Polidea] <https://www.polidea.com/> >> > >> >>> > >>> >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > >> -- >> > >> >>> > >> >> > >> >>> > >> Jarek Potiuk >> > >> >>> > >> Polidea <https://www.polidea.com/> | Principal Software >> > >> >Engineer >> > >> >>> > >> >> > >> >>> > >> M: +48 660 796 129 <+48660796129> >> > >> >>> > >> [image: Polidea] <https://www.polidea.com/> >> > >> >>> > >> >> > >> >>> > >> >> > >> >>> > > >> > >> >>> > > -- >> > >> >>> > > >> > >> >>> > > Jarek Potiuk >> > >> >>> > > Polidea <https://www.polidea.com/> | Principal Software >> > Engineer >> > >> >>> > > >> > >> >>> > > M: +48 660 796 129 <+48660796129> >> > >> >>> > > [image: Polidea] <https://www.polidea.com/> >> > >> >>> > >> > >> >>> > >> > >> >>> >> > >> >> >> > >> >> >> > >> >> -- >> > >> >> *Kaxil Naik* >> > >> >> *Big Data Consultant | DevOps Data Engineer* >> > >> >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark >> & >> > >> >Neo4j >> > >> >> Developer >> > >> >> *LinkedIn*: https://www.linkedin.com/in/kaxil >> > >> >> >> > >> > >> > >> > >> > >> >-- >> > >> >*Kaxil Naik* >> > >> >*Big Data Consultant | DevOps Data Engineer* >> > >> >*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & >> > >> >Neo4j >> > >> >Developer >> > >> >*LinkedIn*: https://www.linkedin.com/in/kaxil >> > >> >> > > >> > >> > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
