Hi Jarek, Just wondering how the vote is unanimous. On case 3 I've voted for B. I still believe introducing namespaces is hard to maintain, as I explained earlier: Mostly because it isn't as trivial to decide when it gets its own package or not. Besides the cloud providers, there are companies like Databricks and Qubole which might also end up with their own package because their integration is getting richer over time. Moving this is a bit painful because we need to deprecate the old path and move everything which introduces noise into version control etc.
Adding an additional option to the list, does not invalidate the older options, right? Cheers, Fokko Op ma 5 aug. 2019 om 15:43 schreef Jarek Potiuk <jarek.pot...@polidea.com>: > Seems that after this months of discussion we got an unanimous voting > result. I summarised it in the AIP-21 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths > > > and > changed its status to "Accepted". Thanks for all your feedback! We will > start soon moving GCP operators following those rules and we will resolve > any initial problems with those - then we might provide some additional > learnings and see if we can easily (probably semi-automatically) move all > other operators. > > Case 1 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#1airflow.contrib.{resources} > <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%231airflow.contrib.%7Bresources%7D> > > > > What to do with the "contrib" folder > > Case 2 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#2git*_{operator/sensor}{/s}.py > <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%232git*_%7Boperator/sensor%7D%7B/s%7D.py> > > > > Drop modules *_operator suffix > > Case 3 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#3{aws/azure/gcp}_* > <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%233%7Baws/azure/gcp%7D_*> > > > + Case 4 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#4Separatenamespaceforresources > <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%234Separatenamespaceforresources> > > > > Grouping Cloud Providers operators/sensors/hooks > > Case 5 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#5*Operator > <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%235*Operator> > > > > *Operator *Sensor *Hook in class name > > Case 6 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#6Otherisolatedcases > <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%236Otherisolatedcases> > > > > Isolated cases > > Case 7 > < > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case#7 > <https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Case%237> > > > > Deprecation method > > *A.* Move everything "contrib" → "airflow" > > *A.* Drop the suffix. > > Example: > > - *airflow.operator.gcp_bigtable_operator.py* > becomes *airflow.operator.gcp_bigtable.py > <http://airflow.operator.gcp_bigtable.py>*. > > *D. * Group operators/sensors/hooks in > *airflow/<PROVIDER>*/operators(sensors, > hooks). Non-cloud provider ones are moved to > airflow/operators(sensors/hooks). > *Drop the prefix.* > > - > *airflow/contrib/operators/sns_publish_operator.py > becomes airflow/aws/operators/**sns_publish_operator.py* > > > - *airflow/contrib/operators/dataproc_operator.py* > becomes *airflow/gcp/operators/dataproc_operator.py* > > > > - > *airflow/contrib/operators/gcp_bigtable_operator.py *becomes *airflow/* > *gcp/operators/bigtable_operator.py* > > > > - > *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > *operators/ssh_operator.py* > > *B.* No change - keep Operator/Sensor suffix in class name. > > *A.* Make individual decisions of renames for operators that do not follow > common conventions used for other operators. > > Consistency trumps compatibility. > > Examples: > > *DataProcHadoopOperator* > > renamed to: > > *DataprocHadoopOperator* > *A.* Native python method (with better IDE support and more flexible but a > bit more verbose) > > J. > > On Wed, Jul 31, 2019 at 10:30 AM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > > > Yep. Agree with Ash on it. There are a number of 'action' operators > > specific for cloud providers and these should be our target. The transfer > > ones require another AIP (A lot of that already discussed in AIP-8 > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303 > > ). > > > > J. > > > > Principal Software Engineer > > Phone: +48660796129 > > > > śr., 31 lip 2019, 10:01 użytkownik Ash Berlin-Taylor <a...@apache.org> > > napisał: > > > >> This is a good idea for now. I'm also not overly concerned about these > >> few non-cloud examples - FTPtoS3Operator can stay where it is and > doesn't > >> have to move under 'aws.' to my mind. > >> > >> Longer term I'd like to go back to making the "transfer/copy/transform" > >> operators "composable" so that we can have a single DataCopyOperator, > and > >> give it a source and destination, and it uses hooks to do the work. > This is > >> not a small undertaking though to do well and efficiently though. > >> > >> -ash > >> > >> > On 30 Jul 2019, at 20:54, Tomasz Urbaszek < > tomasz.urbas...@polidea.com> > >> wrote: > >> > > >> > Maybe we can put all those AtoB operators under one name like > >> “transfer”, > >> > then it would be easier to look for such operator? > >> > > >> > Best, > >> > Tomek > >> > > >> > On Tue, 30 Jul 2019 at 21:39, Chen Tong <cix...@gmail.com> wrote: > >> > > >> >> Daniel mentioned a good point. Such composed operator may also > involves > >> >> both cloud and non-cloud provider saying FTPtoS3Operator. Should it > in > >> AWS > >> >> folder? > >> >> > >> >> Also, saying in the future, another cloud provider is growing large > >> enough. > >> >> Will we rename all plugins related to this provider? What's the > >> criteria we > >> >> say it's big enough? And option D gives an impression like these > tools > >> may > >> >> be maintained and supported by the Cloud provider. I am not sure if > it > >> will > >> >> be a truth or not. > >> >> > >> >> Best, > >> >> > >> >> > >> >> On Tue, Jul 30, 2019 at 11:18 AM Daniel Standish < > dpstand...@gmail.com > >> > > >> >> wrote: > >> >> > >> >>> One wrinkle with case 3+4 option D is inter-provider operators. > >> Mainly > >> >>> it's storage I think e.g. XToS3Operator or XToGCSOperator where the > X > >> is > >> >> a > >> >>> service some different provider. > >> >>> > >> >>> Maybe the rule should be to locate the operator according to the > first > >> >>> provider referenced. So e.g. s3_to_gcs_transfer_operator.py would > go > >> to > >> >>> aws. > >> >>> > >> >>> > >> >>> On Tue, Jul 30, 2019 at 6:21 AM Kamil Breguła < > >> kamil.breg...@polidea.com > >> >>> > >> >>> wrote: > >> >>> > >> >>>> Yes. All changes will be backwards compatible. In the case of using > >> >>>> the old path, a message containing a proposal for change will be > >> >>>> reported to the user. > >> >>>> > >> >>>> I prepared an example of how to change the name of a class in a > case > >> >>>> with the use of a native solution. > >> >>>> Source code: > >> >>>> > >> >> > >> > https://github.com/mik-laj/airflow-deprecation-sample/tree/master/rename > >> >>>> Preview from IDE: https://imgur.com/a/45NxS5W > >> >>>> > >> >>>> On Tue, Jul 30, 2019 at 2:28 PM Ash Berlin-Taylor <a...@apache.org> > >> >>> wrote: > >> >>>>> > >> >>>>> Just cos I'm not sure it's _explicitly_ stated, but all of the > moves > >> >>>> will have a deprecation of the old name right? > >> >>>>> > >> >>>>> 3+4 case D gets my vote too. > >> >>>>> > >> >>>>> -a > >> >>>>> > >> >>>>>> On 30 Jul 2019, at 09:58, Jarek Potiuk <jarek.pot...@polidea.com > > > >> >>>> wrote: > >> >>>>>> > >> >>>>>> I went ahead and updated the page (just to speed it up) as I > think > >> >> it > >> >>>>>> really makes sense to join those two cases (and I do not see any > >> >>>> drawbacks > >> >>>>>> - I think the options we have cover all possible approaches) and > we > >> >>> can > >> >>>>>> always go back if we need to. > >> >>>>>> > >> >>>>>> > >> >>>> > >> >>> > >> >> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal > >> >>>>>> > >> >>>>>> My vote is *D*. > >> >>>>>> > >> >>>>>> - airflow/contrib/operators/gcp_bigtable_operator.py → > >> >>> airflow/gcp > >> >>>>>> /operators/bigtable_operator.py > >> >>>>>> - airflow/contrib/operators/ssh_operator.py -> > >> >>>>>> airflow/operators/ssh_operator.py > >> >>>>>> > >> >>>>>> I updated the page with my vote. > >> >>>>>> I propose that we vote till Friday on that one (all the rest is > >> >>> already > >> >>>>>> agreed). > >> >>>>>> > >> >>>>>> J. > >> >>>>>> > >> >>>>>> > >> >>>>>> On Tue, Jul 30, 2019 at 9:42 AM Jarek Potiuk < > >> >>> jarek.pot...@polidea.com > >> >>>>> > >> >>>>>> wrote: > >> >>>>>> > >> >>>>>>> I think almost everyone voted and we have almost perfect > >> >> consensus. > >> >>>> We all > >> >>>>>>> agree amongst other on moving all operators out of contrib > >> >> (Great!). > >> >>>>>>> > >> >>>>>>> The only doubts are for *Case 3* (Cloud provider prefix) and > *Case > >> >>> 4* > >> >>>>>>> (Using Namespaces). > >> >>>>>>> I think there was actually an overlap between those two. Also > >> >> Ash's > >> >>>>>>> comment/veto on that is quite clear. > >> >>>>>>> So I have final (I hope!) proposal to join those two cases and > >> >> have > >> >>>> one > >> >>>>>>> voting (till Friday) only on that. > >> >>>>>>> > >> >>>>>>> I will update the doc if you all agree with me that makes more > >> >> sense > >> >>>> as > >> >>>>>>> single case to vote on: > >> >>>>>>> > >> >>>>>>> *[Case 3 + Case 4]: Grouping Cloud Provider operators.* > >> >>>>>>> > >> >>>>>>> *Option A*. Keep operators/sensors/hooks in > >> >>> airflow/operators(sensors, > >> >>>>>>> hooks) and keep/add prefixes in file names. > >> >>>>>>> > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py becomes > >> >>>>>>> airflow/operators/**aws_sns_publish_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py* > >> >>>>>>> becomes *airflow/operators/gcp_dataproc_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/hooks/gcp_bigtable_hook.py *becomes *airflow/* > >> >>>>>>> *hooks/gcp_bigtable_hook.py* > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > >> >>>>>>> *operators/ssh_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> *Option B*. Group operators/sensors/hooks in > >> >>>> airflow/operators(sensors, > >> >>>>>>> hooks)/*<PROVIDER>.* Non-cloud provider ones are moved to > >> >>>>>>> airflow/operators(sensors/hooks), Drop the prefix. > >> >>>>>>> > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py > >> >>>>>>> becomes airflow/operators/**aws/sns_publish_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py* > >> >>>>>>> becomes *airflow/operators/gcp/dataproc_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes > >> >>>> *airflow/* > >> >>>>>>> *operators/gcp/bigtable_operator.py* > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > >> >>>>>>> *operators/ssh_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> *Option C*. Group operators/sensors/hooks in > >> >>>> airflow/operators(sensors, > >> >>>>>>> hooks)*/<PROVIDER>.* Non-cloud provider ones are moved to > >> >>>>>>> airflow/operators(sensors/hooks), Keep the prefix. > >> >>>>>>> > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py > >> >>>>>>> becomes airflow/operators/**aws/aws_sns_publish_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py* > >> >>>>>>> becomes *airflow/operators/gcp/gcp_dataproc_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes > >> >>>> *airflow/* > >> >>>>>>> *operators/gcp/gcp_bigtable_operator.py* > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > >> >>>>>>> *operators/ssh_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> *Option D.* Group operators/sensors/hooks in > >> >>>> *airflow/<PROVIDER>*/operators(sensors, > >> >>>>>>> hooks). Non-cloud provider ones are moved to > >> >>>>>>> airflow/operators(sensors/hooks). Drop the prefix. > >> >>>>>>> > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py > >> >>>>>>> becomes airflow/aws/operators/**sns_publish_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py* > >> >>>>>>> becomes *airflow/gcp/operators/dataproc_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes > >> >>>> *airflow/* > >> >>>>>>> *gcp/operators/bigtable_operator.py* > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > >> >>>>>>> *operators/ssh_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> *Option E.* Group operators/sensors/hooks in > >> >>>> *airflow/<PROVIDER>*/operators(sensors, > >> >>>>>>> hooks). Non-cloud provider ones are moved to > >> >>>>>>> airflow/operators(sensors/hooks).* Keep the prefix*. > >> >>>>>>> > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/operators/sns_publish_operator.py > >> >>>>>>> becomes airflow/aws/operators/aws_**sns_publish_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> - *airflow/contrib/operators/dataproc_operator.py* > >> >>>>>>> becomes *airflow/gcp/operators/gcp_dataproc_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/operators/gcp_bigtable_operator.py *becomes > >> >>>> *airflow/* > >> >>>>>>> *gcp/operators/gcp_bigtable_operator.py* > >> >>>>>>> - > >> >>>>>>> *airflow/contrib/operators/ssh_operator.py *becomes *airflow/* > >> >>>>>>> *operators/ssh_operator.py* > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> Shall I proceed with updating the page ? > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> J. > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> On Mon, Jul 29, 2019 at 11:18 AM Kaxil Naik < > kaxiln...@gmail.com> > >> >>>> wrote: > >> >>>>>>> > >> >>>>>>>> Thanks Jarek, adding my vote. > >> >>>>>>>> > >> >>>>>>>> On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor < > >> >> a...@apache.org> > >> >>>> wrote: > >> >>>>>>>> > >> >>>>>>>>> Thanks Jarek, much clearer. I have voted there too. > >> >>>>>>>>> > >> >>>>>>>>> -ash > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>>> On 29 Jul 2019, at 08:13, Driesprong, Fokko > >> >> <fo...@driesprong.frl > >> >>>> > >> >>>>>>>>> wrote: > >> >>>>>>>>>> > >> >>>>>>>>>> Thanks Jarek, the Wiki works much better for me. I've added > my > >> >>> vote > >> >>>>>>>> there > >> >>>>>>>>>> as well. > >> >>>>>>>>>> > >> >>>>>>>>>> You've convinced me on the second case. I've changed my vote > on > >> >>>> Case 2 > >> >>>>>>>>> from > >> >>>>>>>>>> A to B :-) > >> >>>>>>>>>> > >> >>>>>>>>>> Maybe we should leave the vote open a bit longer since it > >> >>> involves > >> >>>>>>>> quite > >> >>>>>>>>>> some reading, and I would like to get some proper feedback > and > >> >>>>>>>> opinions > >> >>>>>>>>> on > >> >>>>>>>>>> this. Plus I only see two votes right now. If we don't think > >> >> this > >> >>>>>>>>> through, > >> >>>>>>>>>> it might be that we're having this vote again in the near > >> >> future > >> >>>> :-) > >> >>>>>>>>>> > >> >>>>>>>>>> Cheers, Fokko > >> >>>>>>>>>> > >> >>>>>>>>>> Op zo 28 jul. 2019 om 10:49 schreef Jarek Potiuk < > >> >>>>>>>>> jarek.pot...@polidea.com>: > >> >>>>>>>>>> > >> >>>>>>>>>>> I updated the AIP-21 proposal in the way that should answer > >> >> all > >> >>>> the > >> >>>>>>>>>>> concerns in this thread. > >> >>>>>>>>>>> > >> >>>>>>>>>>> NOTE! There is an action for all of those who are > interested: > >> >>>> Please > >> >>>>>>>>>>> fill-in your voting in the voting table till Tuesday 30th of > >> >>> July > >> >>>> 6pm > >> >>>>>>>>> CEST > >> >>>>>>>>>>> (5pm BST, 9 am PST). > >> >>>>>>>>>>> > >> >>>>>>>>>>> I created a summary of the proposal > >> >>>>>>>>>>> < > >> >>>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>> > >> >>>> > >> >>> > >> >> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Summaryoftheproposal > >> >>>>>>>>>>>> > >> >>>>>>>>>>> in table form - which contains also examples for each case. > >> >>>>>>>>>>> > >> >>>>>>>>>>> I added a voting table > >> >>>>>>>>>>> < > >> >>>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>> > >> >>>> > >> >>> > >> >> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths#AIP-21:Changesinimportpaths-Voting > >> >>>>>>>>>>>> > >> >>>>>>>>>>> where you should add your votes: > >> >>>>>>>>>>> > >> >>>>>>>>>>> I propose this voting mechanism: > >> >>>>>>>>>>> > >> >>>>>>>>>>> Voting will take place till *Tuesday* 30 Jul 2019 6pm CEST > (5 > >> >>> pm > >> >>>>>>>> BST, > >> >>>>>>>>> 9am > >> >>>>>>>>>>> PST) . > >> >>>>>>>>>>> > >> >>>>>>>>>>> After the choice, the final consistent set of choices will > be > >> >>>>>>>> announced > >> >>>>>>>>>>> (taking into account majority of binding vote, also > including > >> >>>>>>>> potential > >> >>>>>>>>>>> vetos and consistency between the choices. Non-binding votes > >> >>> will > >> >>>> be > >> >>>>>>>>> taken > >> >>>>>>>>>>> into account in case there is a draw. The final set of > choices > >> >>>> will > >> >>>>>>>> be > >> >>>>>>>>>>> announced at devlist thread > >> >>>>>>>>>>> < > >> >>>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>> > >> >>>> > >> >>> > >> >> > >> > https://lists.apache.org/thread.html/4cedc94bee53ad908eee8333a56b58be8b5641881e73f69b97e436a9@%3Cdev.airflow.apache.org%3E > >> >>>>>>>>>>>> > >> >>>>>>>>>>> after > >> >>>>>>>>>>> the voting completes. > >> >>>>>>>>>>> > >> >>>>>>>>>>> Unless there is a veto raised to the final proposal, the > final > >> >>>>>>>> proposal > >> >>>>>>>>> is > >> >>>>>>>>>>> accepted by Lazy Consensus > >> >>>>>>>>>>> <https://community.apache.org/committers/lazyConsensus.html > > > >> >>> on > >> >>>>>>>>> *Friday* > >> >>>>>>>>>>> 02 > >> >>>>>>>>>>> Aug 2019 at 6pm CEST (5 pm BST, 9am PST). > >> >>>>>>>>>>> > >> >>>>>>>>>>> Please let me know if what I proposed can be further > improved > >> >> to > >> >>>>>>>> avoid > >> >>>>>>>>>>> chaos. > >> >>>>>>>>>>> > >> >>>>>>>>>>> J. > >> >>>>>>>>>>> > >> >>>>>>>>>>> On Sat, Jul 27, 2019 at 11:41 AM Jarek Potiuk < > >> >>>>>>>> jarek.pot...@polidea.com > >> >>>>>>>>>> > >> >>>>>>>>>>> wrote: > >> >>>>>>>>>>> > >> >>>>>>>>>>>> Ok. I see that indeed voting could have been organised a > bit > >> >>>> better > >> >>>>>>>> - > >> >>>>>>>>> dev > >> >>>>>>>>>>>> list does not seem to cope well with such a complex case > :). > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> As Kamil noticed - we already have a formal AIP-21 > >> >>>>>>>>>>>> < > >> >>>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>> > >> >>>> > >> >>> > >> >> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> in the Wiki - which I should have mentioned before. I have > >> >> not > >> >>>> much > >> >>>>>>>>> time > >> >>>>>>>>>>>> today - but tomorrow I will try to summarise the options - > >> >>> based > >> >>>> on > >> >>>>>>>>> some > >> >>>>>>>>>>>> real code examples (to make it easier to digest). I think > the > >> >>>> best > >> >>>>>>>>>>> approach > >> >>>>>>>>>>>> will be to make a voting matrix in the AIP-21 Wiki page > where > >> >>>> people > >> >>>>>>>>> will > >> >>>>>>>>>>>> be able to state their preferences for each case (by > adding a > >> >>>> row to > >> >>>>>>>>> the > >> >>>>>>>>>>>> table). I think that might provide much better clarity and > a > >> >>>> single > >> >>>>>>>>> place > >> >>>>>>>>>>>> which will show the current aggregated state of voting. > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> However - as Ash also stated - some of the options are > >> >>>>>>>> inter-dependent > >> >>>>>>>>> so > >> >>>>>>>>>>>> at the end we will have to adjust the results in case they > >> >> are > >> >>>> not > >> >>>>>>>>>>>> consistent - and make final voting on aggregated set of > >> >> cases > >> >>> - > >> >>>>>>>> but I > >> >>>>>>>>>>>> think in this case following "lasy consensus" - i,e. if > noone > >> >>>>>>>> objects > >> >>>>>>>>> to > >> >>>>>>>>>>>> final set of cases, we will proceed with it.. > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> Do you think that will work better ? > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> J. > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> Principal Software Engineer > >> >>>>>>>>>>>> Phone: +48660796129 > >> >>>>>>>>>>>> > >> >>>>>>>>>>>> sob., 27 lip 2019, 09:26 użytkownik Kamil Breguła < > >> >>>>>>>>>>>> kamil.breg...@polidea.com> napisał: > >> >>>>>>>>>>>> > >> >>>>>>>>>>>>> Document is also available as AIP on wiki: > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>> > >> >>>> > >> >>> > >> >> > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>> On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko > >> >>>>>>>> <fo...@driesprong.frl > >> >>>>>>>>>> > >> >>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>>>> I agree with all, this is a bit chaotic. For the sake of > >> >>>> clarity, > >> >>>>>>>> I > >> >>>>>>>>>>>>> would > >> >>>>>>>>>>>>>> suggest moving the Google Doc to the Wiki as an AIP. > >> >> Create a > >> >>>>>>>>>>> structural > >> >>>>>>>>>>>>>> code improvement AIP and then and add a matrix of > >> >> users/cases > >> >>>>>>>> where > >> >>>>>>>>>>>>>> everybody can add his vote per case. This gives also the > >> >>>>>>>> opportunity > >> >>>>>>>>>>> for > >> >>>>>>>>>>>>>> changing your vote on a specific case. WDYT? > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Cheers, Fokko > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> Op vr 26 jul. 2019 om 22:28 schreef Chen Tong < > >> >>>> cix...@gmail.com>: > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> I agree with Ash. It is clear to vote on each case > rather > >> >>> than > >> >>>>>>>> all > >> >>>>>>>>>>>>>>> together. > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor < > >> >>>>>>>> a...@apache.org> > >> >>>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> A number of proposals overlap or add on to each other, > >> >> so I > >> >>>>>>>> think > >> >>>>>>>>>>>>> (?) a > >> >>>>>>>>>>>>>>>> single vote makes sense. > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> To be clear, my vote is -1 until it's clear exactly > what > >> >> we > >> >>>> are > >> >>>>>>>>>>>>> voting > >> >>>>>>>>>>>>>> on. > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> On 26 July 2019 20:39:56 BST, Kaxil Naik < > >> >>>> kaxiln...@gmail.com> > >> >>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>>>>> I agree with Ash and instead of voting on all 7 Cases > >> >>>> together, > >> >>>>>>>>>>> how > >> >>>>>>>>>>>>>>>>> about > >> >>>>>>>>>>>>>>>>> separate vote emails for all the cases? Each email can > >> >>> have > >> >>>> the > >> >>>>>>>>>>>>>>>>> description > >> >>>>>>>>>>>>>>>>> copied over from the doc. > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> It would probably be a bit easier to track a decision > in > >> >>> the > >> >>>>>>>>>>> future > >> >>>>>>>>>>>>> as > >> >>>>>>>>>>>>>>>>> well. > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> What do you guys think? @Jarek Potiuk < > >> >>>>>>>> jarek.pot...@polidea.com> > >> >>>>>>>>>>>>>>>>> @Driesprong, > >> >>>>>>>>>>>>>>>>> Fokko <fo...@driesprong.frl> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> On Sat, Jul 27, 2019 at 1:03 AM Kaxil Naik < > >> >>>>>>>> kaxiln...@gmail.com> > >> >>>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> *Case #1* airflow.contrib.{resources} : *Solution A * > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> *Case #2* git *_{operator/sensor}{/s}.py : *Solution > A > >> >> * > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> *Case #3* {aws/azure/gcp}_* : *Solution C* > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> *Case #4* Separate namespace for resources : > >> >>>>>>>>>>>>>>>>>> *Solution A* > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> *Case #5* *Operator : *Solution A* > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> *Case #7* : *Solution A* > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 11:09 PM Daniel Standish > >> >>>>>>>>>>>>>>>>> <dpstand...@gmail.com> > >> >>>>>>>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> I have tried to add some clarification to Jarek's > >> >>> summary, > >> >>>>>>>> but > >> >>>>>>>>>>> I > >> >>>>>>>>>>>>> am > >> >>>>>>>>>>>>>>>>> a > >> >>>>>>>>>>>>>>>>>>> little fuzzy on exact proposal for case 3 so > hopefully > >> >>>> Jarek > >> >>>>>>>>>>> can > >> >>>>>>>>>>>>>>>>> further > >> >>>>>>>>>>>>>>>>>>> clarify. > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> According to my reading, in this motion cases 4,5,6 > >> >> are > >> >>>> all > >> >>>>>>>>>>>>> either > >> >>>>>>>>>>>>>>>>>>> proposal > >> >>>>>>>>>>>>>>>>>>> *rejections* or otherwise have no effect, so > attention > >> >>>> can be > >> >>>>>>>>>>>>>>>>> focused on > >> >>>>>>>>>>>>>>>>>>> cases 1,2,3 and 7. > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> *Motion is to adopt the following:* > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> Case 1 - Solution A - Get rid of contrib > >> >>>>>>>>>>>>>>>>>>> All objects will be moved out of contrib folder and > >> >> into > >> >>>>>>>>>>>>> respective > >> >>>>>>>>>>>>>>>>>>> non-contrib folder. > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> Case 2 - Solution A - Remove object type suffix from > >> >>>> modules > >> >>>>>>>>>>>>>>>>>>> Example: > >> >>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator becomes > >> >>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator > >> >>>>>>>>>>>>>>>>>>> Example: > >> >>>>>>>>>>>>>>>>>>> Airflow.hooks.foo_hook.FooOperator becomes > >> >>>>>>>>>>>>> airflow.hooks.foo.FooHook > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> Case 3 - conventions for cloud provider etc prefixes > >> >>>>>>>>>>>>>>>>>>> *I am not sure exactly what the proposal is.* > >> >>>>>>>>>>>>>>>>>>> Example case is > >> >>>>>>>>>>>>> airflow/contrib/operators/gcp_bigtable_operator.py > >> >>>>>>>>>>>>>>>>>>> Since motion adopts case 1 solution A, we can assume > >> >> it > >> >>> is > >> >>>>>>>> now > >> >>>>>>>>>>>>> named > >> >>>>>>>>>>>>>>>>> like > >> >>>>>>>>>>>>>>>>>>> so: airflow/operators/gcp_bigtable_operator.py > >> >>>>>>>>>>>>>>>>>>> So what is proposed for this case? > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> Case 4 - Solution B - *no change* > >> >>>>>>>>>>>>>>>>>>> This proposal was about namespaces but the motion is > >> >> to > >> >>>>>>>> reject > >> >>>>>>>>>>>>> the > >> >>>>>>>>>>>>>>>>>>> proposal. > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> Case 5 - Solution B - *no change* > >> >>>>>>>>>>>>>>>>>>> The proposal was to remove class type suffix e.g. > >> >> remove > >> >>>> the > >> >>>>>>>>>>>>>>>>> Operator in > >> >>>>>>>>>>>>>>>>>>> FooOperator, but the motion is to reject this > proposal > >> >>> -- > >> >>>>>>>> i.e. > >> >>>>>>>>>>>>>>>>> motion is > >> >>>>>>>>>>>>>>>>>>> no > >> >>>>>>>>>>>>>>>>>>> change on this case, i.e. we resolve to keep > >> >> "Operator" > >> >>>> (and > >> >>>>>>>>>>>>>>>>> "Sensor") > >> >>>>>>>>>>>>>>>>>>> suffixes on class names > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> Case 6 - *No change* > >> >>>>>>>>>>>>>>>>>>> This case concerns object naming inconsistencies, > but > >> >>>> motion > >> >>>>>>>>>>> does > >> >>>>>>>>>>>>>>>>> not > >> >>>>>>>>>>>>>>>>>>> enact > >> >>>>>>>>>>>>>>>>>>> any specific proposal. > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> Case 7 - Solution A - use warnings.warn template for > >> >>>>>>>>>>> deprecation > >> >>>>>>>>>>>>>>>>> warnings > >> >>>>>>>>>>>>>>>>>>> (instead of zope) > >> >>>>>>>>>>>>>>>>>>> Example: > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> # in old_package/old_mod.py > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> import warnings > >> >>>>>>>>>>>>>>>>>>> from new_package.new_mod import * > >> >>>>>>>>>>>>>>>>>>> warnings.warn("old_package.old_mod has moved to > >> >>>>>>>>>>>>> new_package.new_mod. > >> >>>>>>>>>>>>>>>>>>> Import > >> >>>>>>>>>>>>>>>>>>> of " > >> >>>>>>>>>>>>>>>>>>> "old_package.old_mod will become unsupported in > >> >> version > >> >>>> 2", > >> >>>>>>>>>>>>>>>>>>> DeprecationWarning, 2) > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> Reference implementation here: > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>> > >> >>>> > >> >>> > >> >> > >> > https://github.com/mik-laj/airflow-deprecation-sample/tree/master/solution1 > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> FWIW +1 (non-binding) on 1,2,7 -- unsure on 3. > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> I am very happy that this motion now gets rid of > >> >>> contrib. > >> >>>>>>>>>>> Thank > >> >>>>>>>>>>>>> you > >> >>>>>>>>>>>>>>>>>>> Sergio > >> >>>>>>>>>>>>>>>>>>> for turning the tide with your effective > argumentation > >> >>> ;) > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 3:16 AM Ash Berlin-Taylor < > >> >>>>>>>>>>>>> a...@apache.org> > >> >>>>>>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> Could someone summarise what the proposed name > >> >> changes > >> >>>> are, > >> >>>>>>>>>>>>> here, > >> >>>>>>>>>>>>>>>>> in > >> >>>>>>>>>>>>>>>>>>> this > >> >>>>>>>>>>>>>>>>>>>> thread? Pointing at a google doc and only giving > >> >>> partial > >> >>>>>>>>>>>>> examples > >> >>>>>>>>>>>>>>>>> is 1) > >> >>>>>>>>>>>>>>>>>>> a > >> >>>>>>>>>>>>>>>>>>>> bit difficult to quickly work out what we are > voting > >> >>> on, > >> >>>> and > >> >>>>>>>>>>> 2) > >> >>>>>>>>>>>>>>>>> using a > >> >>>>>>>>>>>>>>>>>>>> google doc isn't great for a longer term record. > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> Thanks, > >> >>>>>>>>>>>>>>>>>>>> Ash > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> On 26 Jul 2019, at 09:14, Jarek Potiuk > >> >>>>>>>>>>>>>>>>> <jarek.pot...@polidea.com> > >> >>>>>>>>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> I would like to recast the vote. Let's start from > 0 > >> >> :) > >> >>>> . I > >> >>>>>>>>>>>>> would > >> >>>>>>>>>>>>>>>>> like > >> >>>>>>>>>>>>>>>>>>> to > >> >>>>>>>>>>>>>>>>>>>>> keep the July 30th 6pm CEST date, and at least > three > >> >>> +1 > >> >>>>>>>>>>>>>>>>> (binding) > >> >>>>>>>>>>>>>>>>>>> votes > >> >>>>>>>>>>>>>>>>>>>> are > >> >>>>>>>>>>>>>>>>>>>>> needed to pass. I marked in red the modifications > to > >> >>> the > >> >>>>>>>>>>>>>>>>> previous > >> >>>>>>>>>>>>>>>>>>>> proposal. > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> Here is the proposal (details here > >> >>>>>>>>>>>>>>>>>>>>> < > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>> > >> >>>> > >> >>> > >> >> > >> > https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit#heading=h.j4jc3i70qszo > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> ) > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> - *Case 1: A: Get rid of contrib,* > >> >>>>>>>>>>>>>>>>>>>>> - *Case 2: A: > >> >>> Airflow.operators.foo_operator.FooOperator > >> >>>>>>>>>>>>> could > >> >>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator* > >> >>>>>>>>>>>>>>>>>>>>> - *Case 3: C: > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>> > airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator > >> >>>>>>>>>>>>>>>>>>> could > >> >>>>>>>>>>>>>>>>>>>>> become > >> >>> airflow.gcp.operators.bigtable.BigTableOperator* > >> >>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction* > >> >>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") > >> >> suffixes > >> >>> on > >> >>>>>>>>>>>>> class > >> >>>>>>>>>>>>>>>>>>> names* > >> >>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on > >> >>> case-by-case > >> >>>>>>>>>>>>> (and > >> >>>>>>>>>>>>>>>>> my team > >> >>>>>>>>>>>>>>>>>>>> can > >> >>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)* > >> >>>>>>>>>>>>>>>>>>>>> - *Case 7: Native python solution (with better IDE > >> >>>>>>>>>>>>>>>>> integration)* > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> This is my binding (+1) vote. > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> On Fri, Jul 26, 2019 at 10:08 AM Jarek Potiuk < > >> >>>>>>>>>>>>>>>>>>> jarek.pot...@polidea.com> > >> >>>>>>>>>>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> Hey Felix, > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> For 7* -> I am in favour of trying the native one > >> >> as > >> >>>> well > >> >>>>>>>>>>> (I > >> >>>>>>>>>>>>>>>>> use IDE > >> >>>>>>>>>>>>>>>>>>>> quite > >> >>>>>>>>>>>>>>>>>>>>>> often ;) ) > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> J. > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko > >> >>>>>>>>>>>>>>>>>>> <fo...@driesprong.frl > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> wrote: > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> Thanks Kamil for putting the document together. > I > >> >>>> wasn't > >> >>>>>>>>>>>>> able > >> >>>>>>>>>>>>>>>>> to > >> >>>>>>>>>>>>>>>>>>>> respond > >> >>>>>>>>>>>>>>>>>>>>>>> earlier since I was giving Airflow workshops > >> >>>> throughout > >> >>>>>>>>>>>>> Europe > >> >>>>>>>>>>>>>>>>> :-) > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 1: *Solution A → Remove everything from > >> >>> contrib > >> >>>>>>>>>>> into > >> >>>>>>>>>>>>> a > >> >>>>>>>>>>>>>>>>> single > >> >>>>>>>>>>>>>>>>>>>>>>> package. > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> To make it easier to the end-user, my preference > >> >>>> would be > >> >>>>>>>>>>>>> to > >> >>>>>>>>>>>>>>>>> get > >> >>>>>>>>>>>>>>>>>>> rid of > >> >>>>>>>>>>>>>>>>>>>>>>> the > >> >>>>>>>>>>>>>>>>>>>>>>> contrib package altogether. What's in contrib or > >> >> not > >> >>>> is a > >> >>>>>>>>>>>>>>>>> tricky > >> >>>>>>>>>>>>>>>>>>>> question, > >> >>>>>>>>>>>>>>>>>>>>>>> I think this is already reflected in the > document > >> >>>> since > >> >>>>>>>>>>>>>>>>> "maturity" > >> >>>>>>>>>>>>>>>>>>> is > >> >>>>>>>>>>>>>>>>>>>> in > >> >>>>>>>>>>>>>>>>>>>>>>> quotes. What would be the moment to transfer the > >> >>>> operator > >> >>>>>>>>>>>>> to > >> >>>>>>>>>>>>>>>>> the > >> >>>>>>>>>>>>>>>>>>> core > >> >>>>>>>>>>>>>>>>>>>> or > >> >>>>>>>>>>>>>>>>>>>>>>> vice versa? I'm afraid that renaming contrib to > >> >>>>>>>>>>> incubating > >> >>>>>>>>>>>>>>>>> will > >> >>>>>>>>>>>>>>>>>>> confuse > >> >>>>>>>>>>>>>>>>>>>>>>> the > >> >>>>>>>>>>>>>>>>>>>>>>> user even more. For people who aren't as known > >> >> with > >> >>>>>>>>>>>>> Airflow it > >> >>>>>>>>>>>>>>>>> isn't > >> >>>>>>>>>>>>>>>>>>>> that > >> >>>>>>>>>>>>>>>>>>>>>>> transparent which operator lives where, > especially > >> >>> if > >> >>>> you > >> >>>>>>>>>>>>>>>>> don't use > >> >>>>>>>>>>>>>>>>>>> an > >> >>>>>>>>>>>>>>>>>>>> IDE > >> >>>>>>>>>>>>>>>>>>>>>>> such as Ash ;) Besides that I don't think it is > >> >>> worth > >> >>>>>>>>>>>>> changing > >> >>>>>>>>>>>>>>>>> the > >> >>>>>>>>>>>>>>>>>>>> public > >> >>>>>>>>>>>>>>>>>>>>>>> API just for a different name. > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> Ok. I am convinced. I will re-cast the vote with > >> >>> this. > >> >>>> It > >> >>>>>>>>>>>>> will > >> >>>>>>>>>>>>>>>>> make > >> >>>>>>>>>>>>>>>>>>>> easier > >> >>>>>>>>>>>>>>>>>>>>>> as we will not have to define other rules as > those > >> >>>> that we > >> >>>>>>>>>>>>>>>>> already > >> >>>>>>>>>>>>>>>>>>> have. > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 2:* Slight preference to Solution B (let's > >> >>> say a > >> >>>>>>>>>>> +0) > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> I like to search on Github with t, and then > search > >> >>> for > >> >>>>>>>>>>>>>>>>> BashOperator. > >> >>>>>>>>>>>>>>>>>>>> After > >> >>>>>>>>>>>>>>>>>>>>>>> the change, you should search for Operator/Bash. > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> Jarek, I'm confused with your vote on this, from > >> >>> your > >> >>>>>>>>>>> mail > >> >>>>>>>>>>>>> you > >> >>>>>>>>>>>>>>>>>>> state: - > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 2: B: > >> >>> Airflow.operators.foo_operator.FooOperator > >> >>>>>>>>>>>>> could > >> >>>>>>>>>>>>>>>>> become > >> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo.FooOperator*. But the B > >> >>>> solution is > >> >>>>>>>>>>>>>>>>> keeping it > >> >>>>>>>>>>>>>>>>>>>> the > >> >>>>>>>>>>>>>>>>>>>>>>> same, so that would mean - *Case 2: B: > >> >>>>>>>>>>>>>>>>>>>>>>> Airflow.operators.foo_operator.FooOperator > *would > >> >>> stay > >> >>>>>>>>>>>>>>>>>>>>>>> airflow.operators.foo_operator*.FooOperator* > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> My mistake. It was of course A. I think there > is a > >> >>>> little > >> >>>>>>>>>>>>>>>>> confusion > >> >>>>>>>>>>>>>>>>>>>> here. > >> >>>>>>>>>>>>>>>>>>>>>> This case is not for changing BashOperator into > >> >> Bash > >> >>>> but > >> >>>>>>>>>>> for > >> >>>>>>>>>>>>>>>>> changing > >> >>>>>>>>>>>>>>>>>>>> the > >> >>>>>>>>>>>>>>>>>>>>>> *module* name not the *class* name. So > >> >>>>>>>>>>>>>>>>> bash_operator.BashOperator -> > >> >>>>>>>>>>>>>>>>>>>>>> bash.BashOperator :) . you will still search for > >> >>>>>>>>>>>>> BashOperator > >> >>>>>>>>>>>>>>>>> in this > >> >>>>>>>>>>>>>>>>>>>> case. > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 3:* I'm leaning towards B > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> Mostly because it isn't as trivial to decide > when > >> >> it > >> >>>> gets > >> >>>>>>>>>>>>> its > >> >>>>>>>>>>>>>>>>> own > >> >>>>>>>>>>>>>>>>>>>> package > >> >>>>>>>>>>>>>>>>>>>>>>> or not. Besides the cloud providers, there are > >> >>>> companies > >> >>>>>>>>>>>>> like > >> >>>>>>>>>>>>>>>>>>>> Databricks > >> >>>>>>>>>>>>>>>>>>>>>>> and Qubole which might also end up with their > own > >> >>>> package > >> >>>>>>>>>>>>>>>>> because > >> >>>>>>>>>>>>>>>>>>> their > >> >>>>>>>>>>>>>>>>>>>>>>> integration is getting richer over time. Moving > >> >> this > >> >>>> is a > >> >>>>>>>>>>>>> bit > >> >>>>>>>>>>>>>>>>>>> painful > >> >>>>>>>>>>>>>>>>>>>>>>> because we need to deprecate the old path and > move > >> >>>>>>>>>>>>> everything > >> >>>>>>>>>>>>>>>>> which > >> >>>>>>>>>>>>>>>>>>>>>>> introduces noise into version control etc. > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> I'd say, we should go for C. As I proposed. It's > >> >> not > >> >>> as > >> >>>>>>>>>>>>>>>>> difficult as > >> >>>>>>>>>>>>>>>>>>> it > >> >>>>>>>>>>>>>>>>>>>>>> seems to group operators in packages and it has > >> >>>> numerous > >> >>>>>>>>>>>>>>>>> advantages. > >> >>>>>>>>>>>>>>>>>>> The > >> >>>>>>>>>>>>>>>>>>>>>> nice thing about deprecations - we can do them in > >> >> the > >> >>>>>>>>>>>>>>>>> __init__.py of > >> >>>>>>>>>>>>>>>>>>> the > >> >>>>>>>>>>>>>>>>>>>>>> packages which causes the noise fairly small (Git > >> >>> will > >> >>>>>>>>>>>>> actually > >> >>>>>>>>>>>>>>>>>>> realise > >> >>>>>>>>>>>>>>>>>>>>>> that the file has been moved and you can follow > >> >> that. > >> >>>>>>>>>>> Maybe > >> >>>>>>>>>>>>> not > >> >>>>>>>>>>>>>>>>> as > >> >>>>>>>>>>>>>>>>>>> super > >> >>>>>>>>>>>>>>>>>>>>>> easy to merge changes later to 1.10.4 but it's > >> >> not a > >> >>>>>>>>>>> rocket > >> >>>>>>>>>>>>>>>>> science > >> >>>>>>>>>>>>>>>>>>>> either. > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> *Case 7:* Python native solution > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> Yes. > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> --- > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> Apart from all, great work! > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> Cheers, Fokko > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> Op di 23 jul. 2019 om 21:27 schreef Felix > >> >> Uellendall > >> >>>>>>>>>>>>>>>>>>>>>>> <felue...@pm.me.invalid > >> >>>>>>>>>>>>>>>>>>>>>>>> : > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>> I have no strong "No" against any proposed > change > >> >>> of > >> >>>>>>>>>>> these > >> >>>>>>>>>>>>>>>>> cases. > >> >>>>>>>>>>>>>>>>>>> So I > >> >>>>>>>>>>>>>>>>>>>>>>> go > >> >>>>>>>>>>>>>>>>>>>>>>>> with +1 (non-binding). > >> >>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>> P.S. Thanks Jarek for bringing this up again > and > >> >>> your > >> >>>>>>>>>>>>> intense > >> >>>>>>>>>>>>>>>>> work > >> >>>>>>>>>>>>>>>>>>>>>>> towards > >> >>>>>>>>>>>>>>>>>>>>>>>> airflow currently :) and thanks to Kamil for > even > >> >>>>>>>>>>> creating > >> >>>>>>>>>>>>>>>>> this > >> >>>>>>>>>>>>>>>>>>>>>>> document. I > >> >>>>>>>>>>>>>>>>>>>>>>>> like how the code is getting more and more > >> >>> consistent > >> >>>>>>>>>>> and > >> >>>>>>>>>>>>>>>>> clean :) > >> >>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>> Kind regards, > >> >>>>>>>>>>>>>>>>>>>>>>>> Felix > >> >>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>> Sent from ProtonMail mobile > >> >>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>> -------- Original Message -------- > >> >>>>>>>>>>>>>>>>>>>>>>>> On Jul 23, 2019, 18:34, Jarek Potiuk wrote: > >> >>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>>> Hello everyone, > >> >>>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>>> This email is calling a vote on the changes in > >> >>>> import > >> >>>>>>>>>>>>> paths. > >> >>>>>>>>>>>>>>>>> It's > >> >>>>>>>>>>>>>>>>>>>> been > >> >>>>>>>>>>>>>>>>>>>>>>>>> discussed in > >> >>>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>> > >> >>>> > >> >>> > >> >> > >> > https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E > >> >>>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>>> The vote will last for at least 1 week (July > >> >> 30th > >> >>>> 6pm > >> >>>>>>>>>>>>> CEST), > >> >>>>>>>>>>>>>>>>> and > >> >>>>>>>>>>>>>>>>>>> at > >> >>>>>>>>>>>>>>>>>>>>>>> least > >> >>>>>>>>>>>>>>>>>>>>>>>>> three +1 (binding) votes have been cast. > >> >>>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>>> The proposal to vote is based on the document > >> >> from > >> >>>>>>>>>>> Kamil > >> >>>>>>>>>>>>>>>>>>>>>>>>> < > >> >>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>> > >> >>>> > >> >>> > >> >> > >> > https://docs.google.com/document/d/1F8zve5S78DXcjpPttW89HnqT0M0iKjT6fo9jX57Ef6A/edit# > >> >>>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>>> The proposed solution is: > >> >>>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 1: B: Contrib vs not: we move all that > >> >> are > >> >>>>>>>>>>> "well" > >> >>>>>>>>>>>>>>>>> tested > >> >>>>>>>>>>>>>>>>>>> and > >> >>>>>>>>>>>>>>>>>>>>>>>>> rename contrib to "incubating" or similar.* > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 2: B: > >> >>>>>>>>>>> Airflow.operators.foo_operator.FooOperator > >> >>>>>>>>>>>>>>>>> could > >> >>>>>>>>>>>>>>>>>>>>>>>>> become airflow.operators.foo.FooOperator* > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 3: C: > >> >>>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>> > airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator > >> >>>>>>>>>>>>>>>>>>>> could > >> >>>>>>>>>>>>>>>>>>>>>>>>> become > >> >>>> airflow.gcp.operators.bigtable.BigTableOperator* > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 4: B: no namespace introduction* > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 5: B: Keep "Operator" (and "Sensor") > >> >>>> suffixes > >> >>>>>>>>>>> on > >> >>>>>>>>>>>>>>>>> class > >> >>>>>>>>>>>>>>>>>>> names* > >> >>>>>>>>>>>>>>>>>>>>>>>>> - *Case 6: We will treat isolated cases on > >> >>>> case-by-case > >> >>>>>>>>>>>>> (and > >> >>>>>>>>>>>>>>>>> my > >> >>>>>>>>>>>>>>>>>>> team > >> >>>>>>>>>>>>>>>>>>>>>>> can > >> >>>>>>>>>>>>>>>>>>>>>>>>> do the job of GCP-related operators)* > >> >>>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>>> This is my (binding) +1 vote. > >> >>>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>>> Best regards, > >> >>>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>>> J. > >> >>>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>>> -- > >> >>>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk > >> >>>>>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | > Principal > >> >>>>>>>>>>> Software > >> >>>>>>>>>>>>>>>>> Engineer > >> >>>>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>>>>> M: [+48 660 796 129](tel:+48660796129) > >> >>>>>>>>>>>>>>>>>>>>>>> <[+48660796129](tel:+48660796129)> > >> >>>>>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >> >>>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> -- > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> Jarek Potiuk > >> >>>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal > >> >>>> Software > >> >>>>>>>>>>>>>>>>> Engineer > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> > >> >>>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> -- > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> Jarek Potiuk > >> >>>>>>>>>>>>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal > >> >>> Software > >> >>>>>>>>>>>>>> Engineer > >> >>>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>>> M: +48 660 796 129 <+48660796129> > >> >>>>>>>>>>>>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>>> -- > >> >>>>>>>>>>>>>>>>>> *Kaxil Naik* > >> >>>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer* > >> >>>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* > >> >>> Apache > >> >>>>>>>>>>> Spark > >> >>>>>>>>>>>>> & > >> >>>>>>>>>>>>>>>>> Neo4j > >> >>>>>>>>>>>>>>>>>> Developer > >> >>>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil > >> >>>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>>>> -- > >> >>>>>>>>>>>>>>>>> *Kaxil Naik* > >> >>>>>>>>>>>>>>>>> *Big Data Consultant | DevOps Data Engineer* > >> >>>>>>>>>>>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* > >> >>> Apache > >> >>>>>>>> Spark > >> >>>>>>>>>>> & > >> >>>>>>>>>>>>>>>>> Neo4j > >> >>>>>>>>>>>>>>>>> Developer > >> >>>>>>>>>>>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil > >> >>>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> >>>>>>>>>>>> > >> >>>>>>>>>>> > >> >>>>>>>>>>> -- > >> >>>>>>>>>>> > >> >>>>>>>>>>> Jarek Potiuk > >> >>>>>>>>>>> Polidea <https://www.polidea.com/> | Principal Software > >> >>> Engineer > >> >>>>>>>>>>> > >> >>>>>>>>>>> M: +48 660 796 129 <+48660796129> > >> >>>>>>>>>>> [image: Polidea] <https://www.polidea.com/> > >> >>>>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>>> > >> >>>>>>>> > >> >>>>>>>> -- > >> >>>>>>>> *Kaxil Naik* > >> >>>>>>>> *Big Data Consultant | DevOps Data Engineer* > >> >>>>>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache > >> >> Spark & > >> >>>> Neo4j > >> >>>>>>>> Developer > >> >>>>>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil > >> >>>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> -- > >> >>>>>>> > >> >>>>>>> Jarek Potiuk > >> >>>>>>> Polidea <https://www.polidea.com/> | Principal Software > Engineer > >> >>>>>>> > >> >>>>>>> M: +48 660 796 129 <+48660796129> > >> >>>>>>> [image: Polidea] <https://www.polidea.com/> > >> >>>>>>> > >> >>>>>>> > >> >>>>>> > >> >>>>>> -- > >> >>>>>> > >> >>>>>> Jarek Potiuk > >> >>>>>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >> >>>>>> > >> >>>>>> M: +48 660 796 129 <+48660796129> > >> >>>>>> [image: Polidea] <https://www.polidea.com/> > >> >>>>> > >> >>>> > >> >>> > >> >> > >> > >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
