Re: [VOTE] Changes in import paths

Hey Fokko, Just to alleviate your concerns a bit - we are going to make it backwards compatible for a start. And we will just start with GCP operators together with Kamil and Tomek, to show how it works (and that it's not that difficult in fact - maybe even automate it it if needed - as we usually

Re: [VOTE] Changes in import paths

Hi Jarek, No worries. I must admit that I've lost track a bit when the case 3+4 got combined. Maybe for the future better to let the current vote run and maybe reinitiate a new vote if there are new options. Currently, it is quite easy to think of a few obvious groups, but it is so hard to keep t

Re: [VOTE] Changes in import paths

Hey Fokko, Really sorry, to miss this . I see now that you have not voted because you thought it will be transferred automatically -it was thought as not and additional option but combined replacement for (3+4) cases. This was after I realised that there was a veto for namespaces from Ash and I

Re: [VOTE] Changes in import paths

Hi Jarek, Just wondering how the vote is unanimous. On case 3 I've voted for B. I still believe introducing namespaces is hard to maintain, as I explained earlier: Mostly because it isn't as trivial to decide when it gets its own package or not. Besides the cloud providers, there are companies lik

Re: [VOTE] Changes in import paths

Seems that after this months of discussion we got an unanimous voting result. I summarised it in the AIP-21 and changed its status to "Accepted". Thanks for all your feedback! We will start soon moving GCP opera

Re: [VOTE] Changes in import paths

Yep. Agree with Ash on it. There are a number of 'action' operators specific for cloud providers and these should be our target. The transfer ones require another AIP (A lot of that already discussed in AIP-8 https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100827303). J. Principa

Re: [VOTE] Changes in import paths

This is a good idea for now. I'm also not overly concerned about these few non-cloud examples - FTPtoS3Operator can stay where it is and doesn't have to move under 'aws.' to my mind. Longer term I'd like to go back to making the "transfer/copy/transform" operators "composable" so that we can ha

Re: [VOTE] Changes in import paths

Maybe we can put all those AtoB operators under one name like “transfer”, then it would be easier to look for such operator? Best, Tomek On Tue, 30 Jul 2019 at 21:39, Chen Tong wrote: > Daniel mentioned a good point. Such composed operator may also involves > both cloud and non-cloud provider s

Re: [VOTE] Changes in import paths

Daniel mentioned a good point. Such composed operator may also involves both cloud and non-cloud provider saying FTPtoS3Operator. Should it in AWS folder? Also, saying in the future, another cloud provider is growing large enough. Will we rename all plugins related to this provider? What's the cri

Re: [VOTE] Changes in import paths

One wrinkle with case 3+4 option D is inter-provider operators. Mainly it's storage I think e.g. XToS3Operator or XToGCSOperator where the X is a service some different provider. Maybe the rule should be to locate the operator according to the first provider referenced. So e.g. s3_to_gcs_transfe

Re: [VOTE] Changes in import paths

Yes. All changes will be backwards compatible. In the case of using the old path, a message containing a proposal for change will be reported to the user. I prepared an example of how to change the name of a class in a case with the use of a native solution. Source code: https://github.com/mik-la

Re: [VOTE] Changes in import paths

Just cos I'm not sure it's _explicitly_ stated, but all of the moves will have a deprecation of the old name right? 3+4 case D gets my vote too. -a > On 30 Jul 2019, at 09:58, Jarek Potiuk wrote: > > I went ahead and updated the page (just to speed it up) as I think it > really makes sense to

Re: [VOTE] Changes in import paths

I went ahead and updated the page (just to speed it up) as I think it really makes sense to join those two cases (and I do not see any drawbacks - I think the options we have cover all possible approaches) and we can always go back if we need to. https://cwiki.apache.org/confluence/display/AIRFLOW

Re: [VOTE] Changes in import paths

I think almost everyone voted and we have almost perfect consensus. We all agree amongst other on moving all operators out of contrib (Great!). The only doubts are for *Case 3* (Cloud provider prefix) and *Case 4* (Using Namespaces). I think there was actually an overlap between those two. Also As

Re: [VOTE] Changes in import paths

Thanks Jarek, adding my vote. On Mon, Jul 29, 2019 at 2:40 PM Ash Berlin-Taylor wrote: > Thanks Jarek, much clearer. I have voted there too. > > -ash > > > > On 29 Jul 2019, at 08:13, Driesprong, Fokko > wrote: > > > > Thanks Jarek, the Wiki works much better for me. I've added my vote there >

Re: [VOTE] Changes in import paths

Thanks Jarek, much clearer. I have voted there too. -ash > On 29 Jul 2019, at 08:13, Driesprong, Fokko wrote: > > Thanks Jarek, the Wiki works much better for me. I've added my vote there > as well. > > You've convinced me on the second case. I've changed my vote on Case 2 from > A to B :-) >

Re: [VOTE] Changes in import paths

Thanks Jarek, the Wiki works much better for me. I've added my vote there as well. You've convinced me on the second case. I've changed my vote on Case 2 from A to B :-) Maybe we should leave the vote open a bit longer since it involves quite some reading, and I would like to get some proper feed

Re: [VOTE] Changes in import paths

I updated the AIP-21 proposal in the way that should answer all the concerns in this thread. NOTE! There is an action for all of those who are interested: Please fill-in your voting in the voting table till Tuesday 30th of July 6pm CEST (5pm BST, 9 am PST). I created a summary of the proposal

Re: [VOTE] Changes in import paths

Ok. I see that indeed voting could have been organised a bit better - dev list does not seem to cope well with such a complex case :). As Kamil noticed - we already have a formal AIP-21 in the Wiki - which I sh

Re: [VOTE] Changes in import paths

Document is also available as AIP on wiki: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-21%3A+Changes+in+import+paths On Sat, Jul 27, 2019, 9:07 AM Driesprong, Fokko wrote: > I agree with all, this is a bit chaotic. For the sake of clarity, I would > suggest moving the Google Doc to t

Re: [VOTE] Changes in import paths

I agree with all, this is a bit chaotic. For the sake of clarity, I would suggest moving the Google Doc to the Wiki as an AIP. Create a structural code improvement AIP and then and add a matrix of users/cases where everybody can add his vote per case. This gives also the opportunity for changing yo

Re: [VOTE] Changes in import paths

I agree with Ash. It is clear to vote on each case rather than all together. On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor wrote: > A number of proposals overlap or add on to each other, so I think (?) a > single vote makes sense. > > To be clear, my vote is -1 until it's clear exactly what

Re: [VOTE] Changes in import paths

A number of proposals overlap or add on to each other, so I think (?) a single vote makes sense. To be clear, my vote is -1 until it's clear exactly what we are voting on. On 26 July 2019 20:39:56 BST, Kaxil Naik wrote: >I agree with Ash and instead of voting on all 7 Cases together, how >about

Re: [VOTE] Changes in import paths

I agree with Ash and instead of voting on all 7 Cases together, how about separate vote emails for all the cases? Each email can have the description copied over from the doc. It would probably be a bit easier to track a decision in the future as well. What do you guys think? @Jarek Potiuk @Dri

Re: [VOTE] Changes in import paths

*Case #1* airflow.contrib.{resources} : *Solution A * *Case #2* git *_{operator/sensor}{/s}.py : *Solution A * *Case #3* {aws/azure/gcp}_* : *Solution C* *Case #4* Separate namespace for resources : *Solution A* *Case #5* *Operator : *Solution A* *Case #7* : *Solution A* On Fri, Jul 26, 2019

Re: [VOTE] Changes in import paths

I have tried to add some clarification to Jarek's summary, but I am a little fuzzy on exact proposal for case 3 so hopefully Jarek can further clarify. According to my reading, in this motion cases 4,5,6 are all either proposal *rejections* or otherwise have no effect, so attention can be focused

Re: [VOTE] Changes in import paths

Could someone summarise what the proposed name changes are, here, in this thread? Pointing at a google doc and only giving partial examples is 1) a bit difficult to quickly work out what we are voting on, and 2) using a google doc isn't great for a longer term record. Thanks, Ash > On 26 Jul 2

Re: [VOTE] Changes in import paths

I would like to recast the vote. Let's start from 0 :) . I would like to keep the July 30th 6pm CEST date, and at least three +1 (binding) votes are needed to pass. I marked in red the modifications to the previous proposal. Here is the proposal (details here

Re: [VOTE] Changes in import paths

Hey Felix, For 7* -> I am in favour of trying the native one as well (I use IDE quite often ;) ) J. On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko wrote: > Thanks Kamil for putting the document together. I wasn't able to respond > earlier since I was giving Airflow workshops throughout Euro

Re: [VOTE] Changes in import paths

Thanks Kamil for putting the document together. I wasn't able to respond earlier since I was giving Airflow workshops throughout Europe :-) *Case 1: *Solution A → Remove everything from contrib into a single package. To make it easier to the end-user, my preference would be to get rid of the cont

Re: [VOTE] Changes in import paths

I have no strong "No" against any proposed change of these cases. So I go with +1 (non-binding). P.S. Thanks Jarek for bringing this up again and your intense work towards airflow currently :) and thanks to Kamil for even creating this document. I like how the code is getting more and more cons

[VOTE] Changes in import paths

Hello everyone, This email is calling a vote on the changes in import paths. It's been discussed in https://lists.apache.org/thread.html/4e648d9421c792d4537f5ac66f1a16dce468f816fc5221a9f9db9433@%3Cdev.airflow.apache.org%3E The vote will last for at least 1 week (July 30th 6pm CEST), and at least