Re: Airflow DAG Serialisation

Hi Kaxi, Just sent out the AIP: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in+DB+using+JSON+for+Airflow+Webserver+and+%28optional%29+Scheduler Thanks! Zhou On Fri, Jul 26, 2019 at 1:33 PM Zhou Fang wrote: > Hi Kaxil, > > We are also working on persisting DAGs

Re: Airflow DAG Serialisation

Hi Kaxil, We are also working on persisting DAGs into DB using JSON for Airflow webserver in Google Composer. We target at minimizing the change to the current Airflow code. Happy to get synced on this! Here is our progress: (1) Serializing DAGs using Pickle to be used in webserver It has been

Re: [VOTE] Changes in import paths

I agree with Ash. It is clear to vote on each case rather than all together. On Fri, Jul 26, 2019 at 3:54 PM Ash Berlin-Taylor wrote: > A number of proposals overlap or add on to each other, so I think (?) a > single vote makes sense. > > To be clear, my vote is -1 until it's clear exactly what

Re: Need a JIRA ticked for adding comments?

Commenting does require an account, which you can create. This is due to a lot of spam about a year ago. Verstuurd vanaf mijn iPad > Op 26 jul. 2019 om 20:45 heeft Qingping Hou het volgende > geschreven: > > I believe docs and comment change doesn't require a ticket. > >> On Fri, Jul 26,

Re: [VOTE] Changes in import paths

A number of proposals overlap or add on to each other, so I think (?) a single vote makes sense. To be clear, my vote is -1 until it's clear exactly what we are voting on. On 26 July 2019 20:39:56 BST, Kaxil Naik wrote: >I agree with Ash and instead of voting on all 7 Cases together, how

Re: [VOTE] Changes in import paths

I agree with Ash and instead of voting on all 7 Cases together, how about separate vote emails for all the cases? Each email can have the description copied over from the doc. It would probably be a bit easier to track a decision in the future as well. What do you guys think? @Jarek Potiuk

Re: [VOTE] Changes in import paths

*Case #1* airflow.contrib.{resources} : *Solution A * *Case #2* git *_{operator/sensor}{/s}.py : *Solution A * *Case #3* {aws/azure/gcp}_* : *Solution C* *Case #4* Separate namespace for resources : *Solution A* *Case #5* *Operator : *Solution A* *Case #7* : *Solution A* On Fri, Jul 26, 2019

Re: Need a JIRA ticked for adding comments?

I believe docs and comment change doesn't require a ticket. On Fri, Jul 26, 2019 at 10:31 AM Lin Jiqiang wrote: > > Hello, > > I would like to add some comments for parameters > to make it easier to know what type the parameters are. > > I know that I should create a JIRA ticket for changing

Re: [VOTE] Changes in import paths

I have tried to add some clarification to Jarek's summary, but I am a little fuzzy on exact proposal for case 3 so hopefully Jarek can further clarify. According to my reading, in this motion cases 4,5,6 are all either proposal *rejections* or otherwise have no effect, so attention can be focused

Need a JIRA ticked for adding comments?

Hello, I would like to add some comments for parameters to make it easier to know what type the parameters are. I know that I should create a JIRA ticket for changing code, but need I create one for adding comments? Thanks, Lin

Re: Airflow DAG Serialisation

Great to see this happening! On Fri, Jul 26, 2019 at 8:54 AM Jarek Potiuk wrote: > Great! That's definitely one of the most painful aspects of Airflow. Happy > to help/comment/take part in the discussions and later in the > implementation. > > On Fri, Jul 26, 2019 at 4:48 PM Deng Xiaodong

Re: [DISCUSS] Apache Airflow website requirements

Seems a suitable and reasonable approach, esp given current circumstances. I would imagine the desire would be to then update anything in other languages each time English updated? Or on a daily basis? Ongoing would absolutely be important - in a predictable manner - rather than just running

Re: contribute a sensor

Just follow https://github.com/apache/airflow/blob/master/CONTRIBUTING.md :). I think it should have everything you need. BTW. We have on-going discussion about removing contrib folder altogether, so I think you should be better-off adding directly to "sensors" package. J. On Fri, Jul 26, 2019

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

Update: I added appropriate comment in the GitLab CI issue about PRs and we are getting attention of Jason Lenny - director of Product Management @ GitLab. Let's hope they prioritise it quickly enough. Speaking of potential complexity/Maintenance - in order to alleviate any maintenance worries, I

Re: Airflow DAG Serialisation

Great! That's definitely one of the most painful aspects of Airflow. Happy to help/comment/take part in the discussions and later in the implementation. On Fri, Jul 26, 2019 at 4:48 PM Deng Xiaodong wrote: > Yes, it’s something really worth exploring. > > Recently I just finished a POC on DAG

Re: Airflow DAG Serialisation

Yes, it’s something really worth exploring. Recently I just finished a POC on DAG serialisation as well, while the main objective was to allow “rewriting” some components, like worker (for specific operators/sensors), in languages other than Python. We’re also choosing JSON, so that most

Re: Airflow DAG Serialisation

This is awesome, and will bring tons of value to all Airflow users, thank you for driving this! On Fri, Jul 26, 2019 at 5:37 PM Kaxil Naik wrote: > Hi all, > > We, at Astronomer, are going to spend time working on DAG Serialisation. > There are 2 AIPs that are somewhat related to what we plan

Airflow DAG Serialisation

Hi all, We, at Astronomer, are going to spend time working on DAG Serialisation. There are 2 AIPs that are somewhat related to what we plan to work on: - AIP-18 Persist all information from DAG file in DB

contribute a sensor

Hello, I'd like to contribute a sensor to contrib.sensors Is there a process around this? Thanks, Neil

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

GKE and its own CI will allow us to solve other problems - building and publishing documentation from the master branch. Currently, building is done using the RTD service. Unfortunately, our project is too large and often the documentation is not built properly.

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

> Do we _specifically_ have to use GCS for images? We use > docker.io/apache/airflow right now? > Does that have to change? > We do not use GCS specifically. We use GCR (Google Container Registry) which has the same APIs as docker.io to store/retrieve images (it

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

On Fri, Jul 26, 2019 at 11:35 AM Kamil Breguła wrote: > Response inline. > > On Fri, Jul 26, 2019 at 10:58 AM Driesprong, Fokko > wrote: > > > > Nice document Jarek. > > > > We should look at the pro's and con's regarding moving away from Travis. > > The process for Airflow, and also many other

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

Do we _specifically_ have to use GCS for images? We use docker.io/apache/airflow right now? Does that have to change? What process handles the mirroring between Github and Gitlab? What ways might it go wrong? Is there any way we can avoid needing to mirror the

Re: [VOTE] Changes in import paths

Could someone summarise what the proposed name changes are, here, in this thread? Pointing at a google doc and only giving partial examples is 1) a bit difficult to quickly work out what we are voting on, and 2) using a google doc isn't great for a longer term record. Thanks, Ash > On 26 Jul

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

Response inline. On Fri, Jul 26, 2019 at 10:58 AM Driesprong, Fokko wrote: > > Nice document Jarek. > > We should look at the pro's and con's regarding moving away from Travis. > The process for Airflow, and also many other OSS projects, is to first > develop on your local fork. If everything

Re: [Discuss] AIP-23 Proposal "Migration out of Travis CI"

Nice document Jarek. We should look at the pro's and con's regarding moving away from Travis. The process for Airflow, and also many other OSS projects, is to first develop on your local fork. If everything looks good, open a PR to the main repo. This reduces the noise we have on the project

Re: [2.0 spring cleaning] Changes in import paths

Valid points Sergio. I changed my mind and agree that getting rid of contrib is a better idea. On Thu, Jul 25, 2019 at 7:39 PM Sergio Kef wrote: > My 2 cents regarding contrib. > > I don't see many benefits of keeping a contrib module. As a user it is > super confusing that for example we have

Re: [VOTE] Changes in import paths

Hey Felix, For 7* -> I am in favour of trying the native one as well (I use IDE quite often ;) ) J. On Wed, Jul 24, 2019 at 9:34 AM Driesprong, Fokko wrote: > Thanks Kamil for putting the document together. I wasn't able to respond > earlier since I was giving Airflow workshops throughout