Seems that after this months of discussion we got an unanimous voting result. I summarised it in the AIP-21 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths> and changed its status to "Accepted". Thanks for all your feedback! We will start soon moving GCP operators following those rules and we will resolve any initial problems with those - then we might provide some additional learnings and see if we can easily (probably semi-automatically) move all other operators.
Case 1 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#1airflow.contrib.{resources}> What to do with the "contrib" folder Case 2 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#2git*_{operator/sensor}{/s}.py> Drop modules *_operator suffix Case 3 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#3{aws/azure/gcp}_*> + Case 4 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#4Separatenamespaceforresources> Grouping Cloud Providers operators/sensors/hooks Case 5 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#5*Operator> *Operator *Sensor *Hook in class name Case 6 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#6Otherisolatedcases> Isolated cases Case 7 <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#7> Deprecation method *A.* Move everything "contrib" → "airflow" *A.* Drop the suffix. Example: - *airflow.operator.gcp_bigtable_operator.py* becomes *airflow.operator.gcp_bigtable.py <http://airflow.operator.gcp_bigtable.py>*. *D. * Group operators/sensors/hooks in *airflow/<PROVIDER>*/operators(sensors, hooks). Non-cloud provider ones are moved to airflow/operators(sensors/hooks). *Drop the prefix.* - *airflow/contrib/operators/sns_publish_operator.py becomes airflow/aws/operators/**sns_publish_operator.py* - *airflow/contrib/operators/dataproc_operator.py* becomes *airflow/gcp/operators/dataproc_operator.py* - *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/* *gcp/operators/bigtable_operator.py* - *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* *operators/ssh_operator.py* *B.* No change - keep Operator/Sensor suffix in class name. *A.* Make individual decisions of renames for operators that do not follow common conventions used for other operators. Consistency trumps compatibility. Examples: *DataProcHadoopOperator* renamed to: *DataprocHadoopOperator* *A.* Native python method (with better IDE support and more flexible but a bit more verbose) J. On Wed, Jul 31, 2019 at 10:30 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Yep. Agree with Ash on it. There are a number of 'action' operators > specific for cloud providers and these should be our target. The transfer > ones require another AIP (A lot of that already discussed in AIP-8 > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303 > ). > > J. > > Principal Software Engineer > Phone: +48660796129 > > śr., 31 lip 2019, 10:01 użytkownik Ash Berlin-Taylor <a...@apache.org> > napisał: > >> This is a good idea for now. I'm also not overly concerned about these >> few non-cloud examples - FTPtoS3Operator can stay where it is and doesn't >> have to move under 'aws.' to my mind. >> >> Longer term I'd like to go back to making the "transfer/copy/transform" >> operators "composable" so that we can have a single DataCopyOperator, and >> give it a source and destination, and it uses hooks to do the work. This is >> not a small undertaking though to do well and efficiently though. >> >> -ash >> >> > On 30 Jul 2019, at 20:54, Tomasz Urbaszek <tomasz.urbas...@polidea.com> >> wrote: >> > >> > Maybe we can put all those AtoB operators under one name like >> “transfer”, >> > then it would be easier to look for such operator? >> > >> > Best, >> > Tomek >> > >> > On Tue, 30 Jul 2019 at 21:39, Chen Tong <cix...@gmail.com> wrote: >> > >> >> Daniel mentioned a good point. Such composed operator may also involves >> >> both cloud and non-cloud provider saying FTPtoS3Operator. Should it in >> AWS >> >> folder? >> >> >> >> Also, saying in the future, another cloud provider is growing large >> enough. >> >> Will we rename all plugins related to this provider? What's the >> criteria we >> >> say it's big enough? And option D gives an impression like these tools >> may >> >> be maintained and supported by the Cloud provider. I am not sure if it >> will >> >> be a truth or not. >> >> >> >> Best, >> >> >> >> >> >> On Tue, Jul 30, 2019 at 11:18 AM Daniel Standish <dpstand...@gmail.com >> > >> >> wrote: >> >> >> >>> One wrinkle with case 3+4 option D is inter-provider operators. >> Mainly >> >>> it's storage I think e.g. XToS3Operator or XToGCSOperator where the X >> is >> >> a >> >>> service some different provider. >> >>> >> >>> Maybe the rule should be to locate the operator according to the first >> >>> provider referenced. So e.g. s3_to_gcs_transfer_operator.py would go >> to >> >>> aws. >> >>> >> >>> >> >>> On Tue, Jul 30, 2019 at 6:21 AM Kamil Breguła < >> kamil.breg...@polidea.com >> >>> >> >>> wrote: >> >>> >> >>>> Yes. All changes will be backwards compatible. In the case of using >> >>>> the old path, a message containing a proposal for change will be >> >>>> reported to the user. >> >>>> >> >>>> I prepared an example of how to change the name of a class in a case >> >>>> with the use of a native solution. >> >>>> Source code: >> >>>> >> >> >> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename >> >>>> Preview from IDE: https://imgur.com/a/45NxS5W >> >>>> >> >>>> On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <a...@apache.org> >> >>> wrote: >> >>>>> >> >>>>> Just cos I'm not sure it's _explicitly_ stated, but all of the moves >> >>>> will have a deprecation of the old name right? >> >>>>> >> >>>>> 3+4 case D gets my vote too. >> >>>>> >> >>>>> -a >> >>>>> >> >>>>>> On 30 Jul 2019, at 09:58, Jarek Potiuk <jarek.pot...@polidea.com> >> >>>> wrote: >> >>>>>> >> >>>>>> I went ahead and updated the page (just to speed it up) as I think >> >> it >> >>>>>> really makes sense to join those two cases (and I do not see any >> >>>> drawbacks >> >>>>>> - I think the options we have cover all possible approaches) and we >> >>> can >> >>>>>> always go back if we need to. >> >>>>>> >> >>>>>> >> >>>> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal >> >>>>>> >> >>>>>> My vote is *D*. >> >>>>>> >> >>>>>> - airflow/contrib/operators/gcp_bigtable_operator.py → >> >>> airflow/gcp >> >>>>>> /operators/bigtable_operator.py >> >>>>>> - airflow/contrib/operators/ssh_operator.py -> >> >>>>>> airflow/operators/ssh_operator.py >> >>>>>> >> >>>>>> I updated the page with my vote. >> >>>>>> I propose that we vote till Friday on that one (all the rest is >> >>> already >> >>>>>> agreed). >> >>>>>> >> >>>>>> J. >> >>>>>> >> >>>>>> >> >>>>>> On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk < >> >>> jarek.pot...@polidea.com >> >>>>> >> >>>>>> wrote: >> >>>>>> >> >>>>>>> I think almost everyone voted and we have almost perfect >> >> consensus. >> >>>> We all >> >>>>>>> agree amongst other on moving all operators out of contrib >> >> (Great!). >> >>>>>>> >> >>>>>>> The only doubts are for *Case 3* (Cloud provider prefix) and *Case >> >>> 4* >> >>>>>>> (Using Namespaces). >> >>>>>>> I think there was actually an overlap between those two. Also >> >> Ash's >> >>>>>>> comment/veto on that is quite clear. >> >>>>>>> So I have final (I hope!) proposal to join those two cases and >> >> have >> >>>> one >> >>>>>>> voting (till Friday) only on that. >> >>>>>>> >> >>>>>>> I will update the doc if you all agree with me that makes more >> >> sense >> >>>> as >> >>>>>>> single case to vote on: >> >>>>>>> >> >>>>>>> *[Case 3 + Case 4]: Grouping Cloud Provider operators.* >> >>>>>>> >> >>>>>>> *Option A*. Keep operators/sensors/hooks in >> >>> airflow/operators(sensors, >> >>>>>>> hooks) and keep/add prefixes in file names. >> >>>>>>> >> >>>>>>> - >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py becomes >> >>>>>>> airflow/operators/**aws_sns_publish_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py* >> >>>>>>> becomes *airflow/operators/gcp_dataproc_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> - >> >>>>>>> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/* >> >>>>>>> *hooks/gcp_bigtable_hook.py* >> >>>>>>> - >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* >> >>>>>>> *operators/ssh_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> *Option B*. Group operators/sensors/hooks in >> >>>> airflow/operators(sensors, >> >>>>>>> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to >> >>>>>>> airflow/operators(sensors/hooks), Drop the prefix. >> >>>>>>> >> >>>>>>> - >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py >> >>>>>>> becomes airflow/operators/**aws/sns_publish_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py* >> >>>>>>> becomes *airflow/operators/gcp/dataproc_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> - >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes >> >>>> *airflow/* >> >>>>>>> *operators/gcp/bigtable_operator.py* >> >>>>>>> - >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* >> >>>>>>> *operators/ssh_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> *Option C*. Group operators/sensors/hooks in >> >>>> airflow/operators(sensors, >> >>>>>>> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to >> >>>>>>> airflow/operators(sensors/hooks), Keep the prefix. >> >>>>>>> >> >>>>>>> - >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py >> >>>>>>> becomes airflow/operators/**aws/aws_sns_publish_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py* >> >>>>>>> becomes *airflow/operators/gcp/gcp_dataproc_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> - >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes >> >>>> *airflow/* >> >>>>>>> *operators/gcp/gcp_bigtable_operator.py* >> >>>>>>> - >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* >> >>>>>>> *operators/ssh_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> *Option D.* Group operators/sensors/hooks in >> >>>> *airflow/<PROVIDER>*/operators(sensors, >> >>>>>>> hooks). Non-cloud provider ones are moved to >> >>>>>>> airflow/operators(sensors/hooks). Drop the prefix. >> >>>>>>> >> >>>>>>> - >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py >> >>>>>>> becomes airflow/aws/operators/**sns_publish_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py* >> >>>>>>> becomes *airflow/gcp/operators/dataproc_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> - >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes >> >>>> *airflow/* >> >>>>>>> *gcp/operators/bigtable_operator.py* >> >>>>>>> - >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* >> >>>>>>> *operators/ssh_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> *Option E.* Group operators/sensors/hooks in >> >>>> *airflow/<PROVIDER>*/operators(sensors, >> >>>>>>> hooks). Non-cloud provider ones are moved to >> >>>>>>> airflow/operators(sensors/hooks).* Keep the prefix*. >> >>>>>>> >> >>>>>>> - >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py >> >>>>>>> becomes airflow/aws/operators/aws_**sns_publish_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py* >> >>>>>>> becomes *airflow/gcp/operators/gcp_dataproc_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> - >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes >> >>>> *airflow/* >> >>>>>>> *gcp/operators/gcp_bigtable_operator.py* >> >>>>>>> - >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* >> >>>>>>> *operators/ssh_operator.py* >> >>>>>>> >> >>>>>>> >> >>>>>>> Shall I proceed with updating the page ? >> >>>>>>> >> >>>>>>> >> >>>>>>> J. >> >>>>>>> >> >>>>>>> >> >>>>>>> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik <kaxiln...@gmail.com> >> >>>> wrote: >> >>>>>>> >> >>>>>>>> Thanks Jarek, adding my vote. >> >>>>>>>> >> >>>>>>>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor < >> >> a...@apache.org> >> >>>> wrote: >> >>>>>>>> >> >>>>>>>>> Thanks Jarek, much clearer. I have voted there too. >> >>>>>>>>> >> >>>>>>>>> -ash >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko >> >> <fo...@driesprong.frl >> >>>> >> >>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>> Thanks Jarek, the Wiki works much better for me. I've added my >> >>> vote >> >>>>>>>> there >> >>>>>>>>>> as well. >> >>>>>>>>>> >> >>>>>>>>>> You've convinced me on the second case. I've changed my vote on >> >>>> Case 2 >> >>>>>>>>> from >> >>>>>>>>>> A to B :-) >> >>>>>>>>>> >> >>>>>>>>>> Maybe we should leave the vote open a bit longer since it >> >>> involves >> >>>>>>>> quite >> >>>>>>>>>> some reading, and I would like to get some proper feedback and >> >>>>>>>> opinions >> >>>>>>>>> on >> >>>>>>>>>> this. Plus I only see two votes right now. If we don't think >> >> this >> >>>>>>>>> through, >> >>>>>>>>>> it might be that we're having this vote again in the near >> >> future >> >>>> :-) >> >>>>>>>>>> >> >>>>>>>>>> Cheers, Fokko >> >>>>>>>>>> >> >>>>>>>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk < >> >>>>>>>>> jarek.pot...@polidea.com>: >> >>>>>>>>>> >> >>>>>>>>>>> I updated the AIP-21 proposal in the way that should answer >> >> all >> >>>> the >> >>>>>>>>>>> concerns in this thread. >> >>>>>>>>>>> >> >>>>>>>>>>> NOTE! There is an action for all of those who are interested: >> >>>> Please >> >>>>>>>>>>> fill-in your voting in the voting table till Tuesday 30th of >> >>> July >> >>>> 6pm >> >>>>>>>>> CEST >> >>>>>>>>>>> (5pm BST, 9 am PST). >> >>>>>>>>>>> >> >>>>>>>>>>> I created a summary of the proposal >> >>>>>>>>>>> < >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal >> >>>>>>>>>>>> >> >>>>>>>>>>> in table form - which contains also examples for each case. >> >>>>>>>>>>> >> >>>>>>>>>>> I added a voting table >> >>>>>>>>>>> < >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting >> >>>>>>>>>>>> >> >>>>>>>>>>> where you should add your votes: >> >>>>>>>>>>> >> >>>>>>>>>>> I propose this voting mechanism: >> >>>>>>>>>>> >> >>>>>>>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST (5 >> >>> pm >> >>>>>>>> BST, >> >>>>>>>>> 9am >> >>>>>>>>>>> PST) . >> >>>>>>>>>>> >> >>>>>>>>>>> After the choice, the final consistent set of choices will be >> >>>>>>>> announced >> >>>>>>>>>>> (taking into account majority of binding vote, also including >> >>>>>>>> potential >> >>>>>>>>>>> vetos and consistency between the choices. Non-binding votes >> >>> will >> >>>> be >> >>>>>>>>> taken >> >>>>>>>>>>> into account in case there is a draw. The final set of choices >> >>>> will >> >>>>>>>> be >> >>>>>>>>>>> announced at devlist thread >> >>>>>>>>>>> < >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>> >> >>> >> >> >> https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E >> >>>>>>>>>>>> >> >>>>>>>>>>> after >> >>>>>>>>>>> the voting completes. >> >>>>>>>>>>> >> >>>>>>>>>>> Unless there is a veto raised to the final proposal, the final >> >>>>>>>> proposal >> >>>>>>>>> is >> >>>>>>>>>>> accepted by Lazy Consensus >> >>>>>>>>>>> <https://community.apache.org/committers/lazyConsensus.html> >> >>> on >> >>>>>>>>> *Friday* >> >>>>>>>>>>> 02 >> >>>>>>>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST). >> >>>>>>>>>>> >> >>>>>>>>>>> Please let me know if what I proposed can be further improved >> >> to >> >>>>>>>> avoid >> >>>>>>>>>>> chaos. >> >>>>>>>>>>> >> >>>>>>>>>>> J. >> >>>>>>>>>>> >> >>>>>>>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk < >> >>>>>>>> jarek.pot...@polidea.com >> >>>>>>>>>> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>>> Ok. I see that indeed voting could have been organised a bit >> >>>> better >> >>>>>>>> - >> >>>>>>>>> dev >> >>>>>>>>>>>> list does not seem to cope well with such a complex case :). >> >>>>>>>>>>>> >> >>>>>>>>>>>> As Kamil noticed - we already have a formal AIP-21 >> >>>>>>>>>>>> < >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths >> >>>>>>>>>>>> >> >>>>>>>>>>>> in the Wiki - which I should have mentioned before. I have >> >> not >> >>>> much >> >>>>>>>>> time >> >>>>>>>>>>>> today - but tomorrow I will try to summarise the options - >> >>> based >> >>>> on >> >>>>>>>>> some >> >>>>>>>>>>>> real code examples (to make it easier to digest). I think the >> >>>> best >> >>>>>>>>>>> approach >> >>>>>>>>>>>> will be to make a voting matrix in the AIP-21 Wiki page where >> >>>> people >> >>>>>>>>> will >> >>>>>>>>>>>> be able to state their preferences for each case (by adding a >> >>>> row to >> >>>>>>>>> the >> >>>>>>>>>>>> table). I think that might provide much better clarity and a >> >>>> single >> >>>>>>>>> place >> >>>>>>>>>>>> which will show the current aggregated state of voting. >> >>>>>>>>>>>> >> >>>>>>>>>>>> However - as Ash also stated - some of the options are >> >>>>>>>> inter-dependent >> >>>>>>>>> so >> >>>>>>>>>>>> at the end we will have to adjust the results in case they >> >> are >> >>>> not >> >>>>>>>>>>>> consistent - and make final voting on aggregated set of >> >> cases >> >>> - >> >>>>>>>> but I >> >>>>>>>>>>>> think in this case following "lasy consensus" - i,e. if noone >> >>>>>>>> objects >> >>>>>>>>> to >> >>>>>>>>>>>> final set of cases, we will proceed with it.. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Do you think that will work better ? >> >>>>>>>>>>>> >> >>>>>>>>>>>> J. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Principal Software Engineer >> >>>>>>>>>>>> Phone: +48660796129 >> >>>>>>>>>>>> >> >>>>>>>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła < >> >>>>>>>>>>>> kamil.breg...@polidea.com> napisał: >> >>>>>>>>>>>> >> >>>>>>>>>>>>> Document is also available as AIP on wiki: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>> >> >>> >> >> >> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko >> >>>>>>>> <fo...@driesprong.frl >> >>>>>>>>>> >> >>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>> >> >>>>>>>>>>>>>> I agree with all, this is a bit chaotic. For the sake of >> >>>> clarity, >> >>>>>>>> I >> >>>>>>>>>>>>> would >> >>>>>>>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP. >> >> Create a >> >>>>>>>>>>> structural >> >>>>>>>>>>>>>> code improvement AIP and then and add a matrix of >> >> users/cases >> >>>>>>>> where >> >>>>>>>>>>>>>> everybody can add his vote per case. This gives also the >> >>>>>>>> opportunity >> >>>>>>>>>>> for >> >>>>>>>>>>>>>> changing your vote on a specific case. WDYT? >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Cheers, Fokko >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong < >> >>>> cix...@gmail.com>: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> I agree with Ash. It is clear to vote on each case rather >> >>> than >> >>>>>>>> all >> >>>>>>>>>>>>>>> together. >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor < >> >>>>>>>> a...@apache.org> >> >>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> A number of proposals overlap or add on to each other, >> >> so I >> >>>>>>>> think >> >>>>>>>>>>>>> (?) a >> >>>>>>>>>>>>>>>> single vote makes sense. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly what >> >> we >> >>>> are >> >>>>>>>>>>>>> voting >> >>>>>>>>>>>>>> on. >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik < >> >>>> kaxiln...@gmail.com> >> >>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases >> >>>> together, >> >>>>>>>>>>> how >> >>>>>>>>>>>>>>>>> about >> >>>>>>>>>>>>>>>>> separate vote emails for all the cases? Each email can >> >>> have >> >>>> the >> >>>>>>>>>>>>>>>>> description >> >>>>>>>>>>>>>>>>> copied over from the doc. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> It would probably be a bit easier to track a decision in >> >>> the >> >>>>>>>>>>> future >> >>>>>>>>>>>>> as >> >>>>>>>>>>>>>>>>> well. >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> What do you guys think? @Jarek Potiuk < >> >>>>>>>> jarek.pot...@polidea.com> >> >>>>>>>>>>>>>>>>> @Driesprong, >> >>>>>>>>>>>>>>>>> Fokko <fo...@driesprong.frl> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik < >> >>>>>>>> kaxiln...@gmail.com> >> >>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A * >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution A >> >> * >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C* >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> *Case #4* Separate namespace for resources : >> >>>>>>>>>>>>>>>>>> *Solution A* >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> *Case #5* *Operator : *Solution A* >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> *Case #7* : *Solution A* >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish >> >>>>>>>>>>>>>>>>> <dpstand...@gmail.com> >> >>>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> I have tried to add some clarification to Jarek's >> >>> summary, >> >>>>>>>> but >> >>>>>>>>>>> I >> >>>>>>>>>>>>> am >> >>>>>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so hopefully >> >>>> Jarek >> >>>>>>>>>>> can >> >>>>>>>>>>>>>>>>> further >> >>>>>>>>>>>>>>>>>>> clarify. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6 >> >> are >> >>>> all >> >>>>>>>>>>>>> either >> >>>>>>>>>>>>>>>>>>> proposal >> >>>>>>>>>>>>>>>>>>> *rejections* or otherwise have no effect, so attention >> >>>> can be >> >>>>>>>>>>>>>>>>> focused on >> >>>>>>>>>>>>>>>>>>> cases 1,2,3 and 7. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> *Motion is to adopt the following:* >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib >> >>>>>>>>>>>>>>>>>>> All objects will be moved out of contrib folder and >> >> into >> >>>>>>>>>>>>> respective >> >>>>>>>>>>>>>>>>>>> non-contrib folder. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from >> >>>> modules >> >>>>>>>>>>>>>>>>>>> Example: >> >>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes >> >>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator >> >>>>>>>>>>>>>>>>>>> Example: >> >>>>>>>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes >> >>>>>>>>>>>>> airflow.hooks.foo.FooHook >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes >> >>>>>>>>>>>>>>>>>>> *I am not sure exactly what the proposal is.* >> >>>>>>>>>>>>>>>>>>> Example case is >> >>>>>>>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py >> >>>>>>>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume >> >> it >> >>> is >> >>>>>>>> now >> >>>>>>>>>>>>> named >> >>>>>>>>>>>>>>>>> like >> >>>>>>>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py >> >>>>>>>>>>>>>>>>>>> So what is proposed for this case? >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Case 4 - Solution B - *no change* >> >>>>>>>>>>>>>>>>>>> This proposal was about namespaces but the motion is >> >> to >> >>>>>>>> reject >> >>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>> proposal. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Case 5 - Solution B - *no change* >> >>>>>>>>>>>>>>>>>>> The proposal was to remove class type suffix e.g. >> >> remove >> >>>> the >> >>>>>>>>>>>>>>>>> Operator in >> >>>>>>>>>>>>>>>>>>> FooOperator, but the motion is to reject this proposal >> >>> -- >> >>>>>>>> i.e. >> >>>>>>>>>>>>>>>>> motion is >> >>>>>>>>>>>>>>>>>>> no >> >>>>>>>>>>>>>>>>>>> change on this case, i.e. we resolve to keep >> >> "Operator" >> >>>> (and >> >>>>>>>>>>>>>>>>> "Sensor") >> >>>>>>>>>>>>>>>>>>> suffixes on class names >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Case 6 - *No change* >> >>>>>>>>>>>>>>>>>>> This case concerns object naming inconsistencies, but >> >>>> motion >> >>>>>>>>>>> does >> >>>>>>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>>>> enact >> >>>>>>>>>>>>>>>>>>> any specific proposal. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for >> >>>>>>>>>>> deprecation >> >>>>>>>>>>>>>>>>> warnings >> >>>>>>>>>>>>>>>>>>> (instead of zope) >> >>>>>>>>>>>>>>>>>>> Example: >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> # in old_package/old_mod.py >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> import warnings >> >>>>>>>>>>>>>>>>>>> from new_package.new_mod import * >> >>>>>>>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to >> >>>>>>>>>>>>> new_package.new_mod. >> >>>>>>>>>>>>>>>>>>> Import >> >>>>>>>>>>>>>>>>>>> of " >> >>>>>>>>>>>>>>>>>>> "old_package.old_mod will become unsupported in >> >> version >> >>>> 2", >> >>>>>>>>>>>>>>>>>>> DeprecationWarning, 2) >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> Reference implementation here: >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>> >> >>> >> >> >> https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1 >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3. >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> I am very happy that this motion now gets rid of >> >>> contrib. >> >>>>>>>>>>> Thank >> >>>>>>>>>>>>> you >> >>>>>>>>>>>>>>>>>>> Sergio >> >>>>>>>>>>>>>>>>>>> for turning the tide with your effective argumentation >> >>> ;) >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor < >> >>>>>>>>>>>>> a...@apache.org> >> >>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Could someone summarise what the proposed name >> >> changes >> >>>> are, >> >>>>>>>>>>>>> here, >> >>>>>>>>>>>>>>>>> in >> >>>>>>>>>>>>>>>>>>> this >> >>>>>>>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving >> >>> partial >> >>>>>>>>>>>>> examples >> >>>>>>>>>>>>>>>>> is 1) >> >>>>>>>>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>>>>>> bit difficult to quickly work out what we are voting >> >>> on, >> >>>> and >> >>>>>>>>>>> 2) >> >>>>>>>>>>>>>>>>> using a >> >>>>>>>>>>>>>>>>>>>> google doc isn't great for a longer term record. >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> Thanks, >> >>>>>>>>>>>>>>>>>>>> Ash >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk >> >>>>>>>>>>>>>>>>> <jarek.pot...@polidea.com> >> >>>>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from 0 >> >> :) >> >>>> . I >> >>>>>>>>>>>>> would >> >>>>>>>>>>>>>>>>> like >> >>>>>>>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least three >> >>> +1 >> >>>>>>>>>>>>>>>>> (binding) >> >>>>>>>>>>>>>>>>>>> votes >> >>>>>>>>>>>>>>>>>>>> are >> >>>>>>>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications to >> >>> the >> >>>>>>>>>>>>>>>>> previous >> >>>>>>>>>>>>>>>>>>>> proposal. >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> Here is the proposal (details here >> >>>>>>>>>>>>>>>>>>>>> < >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>> >> >>> >> >> >> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> ) >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,* >> >>>>>>>>>>>>>>>>>>>>> - *Case 2: A: >> >>> Airflow.operators.foo_operator.FooOperator >> >>>>>>>>>>>>> could >> >>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator* >> >>>>>>>>>>>>>>>>>>>>> - *Case 3: C: >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator >> >>>>>>>>>>>>>>>>>>> could >> >>>>>>>>>>>>>>>>>>>>> become >> >>> airflow.gcp.operators.bigtable.BigTableOperator* >> >>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction* >> >>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") >> >> suffixes >> >>> on >> >>>>>>>>>>>>> class >> >>>>>>>>>>>>>>>>>>> names* >> >>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on >> >>> case-by-case >> >>>>>>>>>>>>> (and >> >>>>>>>>>>>>>>>>> my team >> >>>>>>>>>>>>>>>>>>>> can >> >>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)* >> >>>>>>>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE >> >>>>>>>>>>>>>>>>> integration)* >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> This is my binding (+1) vote. >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk < >> >>>>>>>>>>>>>>>>>>> jarek.pot...@polidea.com> >> >>>>>>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Hey Felix, >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one >> >> as >> >>>> well >> >>>>>>>>>>> (I >> >>>>>>>>>>>>>>>>> use IDE >> >>>>>>>>>>>>>>>>>>>> quite >> >>>>>>>>>>>>>>>>>>>>>> often ;) ) >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> J. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko >> >>>>>>>>>>>>>>>>>>> <fo...@driesprong.frl >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. I >> >>>> wasn't >> >>>>>>>>>>>>> able >> >>>>>>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>>>>> respond >> >>>>>>>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops >> >>>> throughout >> >>>>>>>>>>>>> Europe >> >>>>>>>>>>>>>>>>> :-) >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from >> >>> contrib >> >>>>>>>>>>> into >> >>>>>>>>>>>>> a >> >>>>>>>>>>>>>>>>> single >> >>>>>>>>>>>>>>>>>>>>>>> package. >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference >> >>>> would be >> >>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>> get >> >>>>>>>>>>>>>>>>>>> rid of >> >>>>>>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or >> >> not >> >>>> is a >> >>>>>>>>>>>>>>>>> tricky >> >>>>>>>>>>>>>>>>>>>> question, >> >>>>>>>>>>>>>>>>>>>>>>> I think this is already reflected in the document >> >>>> since >> >>>>>>>>>>>>>>>>> "maturity" >> >>>>>>>>>>>>>>>>>>> is >> >>>>>>>>>>>>>>>>>>>> in >> >>>>>>>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the >> >>>> operator >> >>>>>>>>>>>>> to >> >>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>> core >> >>>>>>>>>>>>>>>>>>>> or >> >>>>>>>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to >> >>>>>>>>>>> incubating >> >>>>>>>>>>>>>>>>> will >> >>>>>>>>>>>>>>>>>>> confuse >> >>>>>>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>>> user even more. For people who aren't as known >> >> with >> >>>>>>>>>>>>> Airflow it >> >>>>>>>>>>>>>>>>> isn't >> >>>>>>>>>>>>>>>>>>>> that >> >>>>>>>>>>>>>>>>>>>>>>> transparent which operator lives where, especially >> >>> if >> >>>> you >> >>>>>>>>>>>>>>>>> don't use >> >>>>>>>>>>>>>>>>>>> an >> >>>>>>>>>>>>>>>>>>>> IDE >> >>>>>>>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is >> >>> worth >> >>>>>>>>>>>>> changing >> >>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>> public >> >>>>>>>>>>>>>>>>>>>>>>> API just for a different name. >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with >> >>> this. >> >>>> It >> >>>>>>>>>>>>> will >> >>>>>>>>>>>>>>>>> make >> >>>>>>>>>>>>>>>>>>>> easier >> >>>>>>>>>>>>>>>>>>>>>> as we will not have to define other rules as those >> >>>> that we >> >>>>>>>>>>>>>>>>> already >> >>>>>>>>>>>>>>>>>>> have. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's >> >>> say a >> >>>>>>>>>>> +0) >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> I like to search on Github with t, and then search >> >>> for >> >>>>>>>>>>>>>>>>> BashOperator. >> >>>>>>>>>>>>>>>>>>>> After >> >>>>>>>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash. >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from >> >>> your >> >>>>>>>>>>> mail >> >>>>>>>>>>>>> you >> >>>>>>>>>>>>>>>>>>> state: - >> >>>>>>>>>>>>>>>>>>>>>>> *Case 2: B: >> >>> Airflow.operators.foo_operator.FooOperator >> >>>>>>>>>>>>> could >> >>>>>>>>>>>>>>>>> become >> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B >> >>>> solution is >> >>>>>>>>>>>>>>>>> keeping it >> >>>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B: >> >>>>>>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator *would >> >>> stay >> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator* >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there is a >> >>>> little >> >>>>>>>>>>>>>>>>> confusion >> >>>>>>>>>>>>>>>>>>>> here. >> >>>>>>>>>>>>>>>>>>>>>> This case is not for changing BashOperator into >> >> Bash >> >>>> but >> >>>>>>>>>>> for >> >>>>>>>>>>>>>>>>> changing >> >>>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>> *module* name not the *class* name. So >> >>>>>>>>>>>>>>>>> bash_operator.BashOperator -> >> >>>>>>>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for >> >>>>>>>>>>>>> BashOperator >> >>>>>>>>>>>>>>>>> in this >> >>>>>>>>>>>>>>>>>>>> case. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide when >> >> it >> >>>> gets >> >>>>>>>>>>>>> its >> >>>>>>>>>>>>>>>>> own >> >>>>>>>>>>>>>>>>>>>> package >> >>>>>>>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are >> >>>> companies >> >>>>>>>>>>>>> like >> >>>>>>>>>>>>>>>>>>>> Databricks >> >>>>>>>>>>>>>>>>>>>>>>> and Qubole which might also end up with their own >> >>>> package >> >>>>>>>>>>>>>>>>> because >> >>>>>>>>>>>>>>>>>>> their >> >>>>>>>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving >> >> this >> >>>> is a >> >>>>>>>>>>>>> bit >> >>>>>>>>>>>>>>>>>>> painful >> >>>>>>>>>>>>>>>>>>>>>>> because we need to deprecate the old path and move >> >>>>>>>>>>>>> everything >> >>>>>>>>>>>>>>>>> which >> >>>>>>>>>>>>>>>>>>>>>>> introduces noise into version control etc. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's >> >> not >> >>> as >> >>>>>>>>>>>>>>>>> difficult as >> >>>>>>>>>>>>>>>>>>> it >> >>>>>>>>>>>>>>>>>>>>>> seems to group operators in packages and it has >> >>>> numerous >> >>>>>>>>>>>>>>>>> advantages. >> >>>>>>>>>>>>>>>>>>> The >> >>>>>>>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in >> >> the >> >>>>>>>>>>>>>>>>> __init__.py of >> >>>>>>>>>>>>>>>>>>> the >> >>>>>>>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git >> >>> will >> >>>>>>>>>>>>> actually >> >>>>>>>>>>>>>>>>>>> realise >> >>>>>>>>>>>>>>>>>>>>>> that the file has been moved and you can follow >> >> that. >> >>>>>>>>>>> Maybe >> >>>>>>>>>>>>> not >> >>>>>>>>>>>>>>>>> as >> >>>>>>>>>>>>>>>>>>> super >> >>>>>>>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's >> >> not a >> >>>>>>>>>>> rocket >> >>>>>>>>>>>>>>>>> science >> >>>>>>>>>>>>>>>>>>>> either. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> *Case 7:* Python native solution >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Yes. >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> --- >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> Apart from all, great work! >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix >> >> Uellendall >> >>>>>>>>>>>>>>>>>>>>>>> <felue...@pm.me.invalid >> >>>>>>>>>>>>>>>>>>>>>>>> : >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed change >> >>> of >> >>>>>>>>>>> these >> >>>>>>>>>>>>>>>>> cases. >> >>>>>>>>>>>>>>>>>>> So I >> >>>>>>>>>>>>>>>>>>>>>>> go >> >>>>>>>>>>>>>>>>>>>>>>>> with +1 (non-binding). >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again and >> >>> your >> >>>>>>>>>>>>> intense >> >>>>>>>>>>>>>>>>> work >> >>>>>>>>>>>>>>>>>>>>>>> towards >> >>>>>>>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for even >> >>>>>>>>>>> creating >> >>>>>>>>>>>>>>>>> this >> >>>>>>>>>>>>>>>>>>>>>>> document. I >> >>>>>>>>>>>>>>>>>>>>>>>> like how the code is getting more and more >> >>> consistent >> >>>>>>>>>>> and >> >>>>>>>>>>>>>>>>> clean :) >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> Kind regards, >> >>>>>>>>>>>>>>>>>>>>>>>> Felix >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> -------- Original Message -------- >> >>>>>>>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote: >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone, >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in >> >>>> import >> >>>>>>>>>>>>> paths. >> >>>>>>>>>>>>>>>>> It's >> >>>>>>>>>>>>>>>>>>>> been >> >>>>>>>>>>>>>>>>>>>>>>>>> discussed in >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>> >> >>> >> >> >> https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July >> >> 30th >> >>>> 6pm >> >>>>>>>>>>>>> CEST), >> >>>>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>>>> at >> >>>>>>>>>>>>>>>>>>>>>>> least >> >>>>>>>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast. >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document >> >> from >> >>>>>>>>>>> Kamil >> >>>>>>>>>>>>>>>>>>>>>>>>> < >> >>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>> >> >>> >> >> >> https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit# >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> The proposed solution is: >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that >> >> are >> >>>>>>>>>>> "well" >> >>>>>>>>>>>>>>>>> tested >> >>>>>>>>>>>>>>>>>>> and >> >>>>>>>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.* >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 2: B: >> >>>>>>>>>>> Airflow.operators.foo_operator.FooOperator >> >>>>>>>>>>>>>>>>> could >> >>>>>>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator* >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 3: C: >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>> airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator >> >>>>>>>>>>>>>>>>>>>> could >> >>>>>>>>>>>>>>>>>>>>>>>>> become >> >>>> airflow.gcp.operators.bigtable.BigTableOperator* >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction* >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") >> >>>> suffixes >> >>>>>>>>>>> on >> >>>>>>>>>>>>>>>>> class >> >>>>>>>>>>>>>>>>>>> names* >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on >> >>>> case-by-case >> >>>>>>>>>>>>> (and >> >>>>>>>>>>>>>>>>> my >> >>>>>>>>>>>>>>>>>>> team >> >>>>>>>>>>>>>>>>>>>>>>> can >> >>>>>>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)* >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote. >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> Best regards, >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> J. >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> -- >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk >> >>>>>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal >> >>>>>>>>>>> Software >> >>>>>>>>>>>>>>>>> Engineer >> >>>>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129) >> >>>>>>>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)> >> >>>>>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> >> >>>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> -- >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk >> >>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal >> >>>> Software >> >>>>>>>>>>>>>>>>> Engineer >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> >> >>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> -- >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> Jarek Potiuk >> >>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal >> >>> Software >> >>>>>>>>>>>>>> Engineer >> >>>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> >> >>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>>> -- >> >>>>>>>>>>>>>>>>>> *Kaxil Naik* >> >>>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer* >> >>>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* >> >>> Apache >> >>>>>>>>>>> Spark >> >>>>>>>>>>>>> & >> >>>>>>>>>>>>>>>>> Neo4j >> >>>>>>>>>>>>>>>>>> Developer >> >>>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil >> >>>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>>> -- >> >>>>>>>>>>>>>>>>> *Kaxil Naik* >> >>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer* >> >>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* >> >>> Apache >> >>>>>>>> Spark >> >>>>>>>>>>> & >> >>>>>>>>>>>>>>>>> Neo4j >> >>>>>>>>>>>>>>>>> Developer >> >>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil >> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> -- >> >>>>>>>>>>> >> >>>>>>>>>>> Jarek Potiuk >> >>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software >> >>> Engineer >> >>>>>>>>>>> >> >>>>>>>>>>> M: +48 660 796 129 <+48660796129> >> >>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> >> >>>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> -- >> >>>>>>>> *Kaxil Naik* >> >>>>>>>> *Big Data Consultant | DevOps Data Engineer* >> >>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache >> >> Spark & >> >>>> Neo4j >> >>>>>>>> Developer >> >>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil >> >>>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> -- >> >>>>>>> >> >>>>>>> Jarek Potiuk >> >>>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >>>>>>> >> >>>>>>> M: +48 660 796 129 <+48660796129> >> >>>>>>> [image: Polidea] <https://www.polidea.com/> >> >>>>>>> >> >>>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> >> >>>>>> Jarek Potiuk >> >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >>>>>> >> >>>>>> M: +48 660 796 129 <+48660796129> >> >>>>>> [image: Polidea] <https://www.polidea.com/> >> >>>>> >> >>>> >> >>> >> >> >> >> -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
