Creating an external dag that is time aware

2019-04-01 Thread Paul Tremblay
I have a simple pipeline that consists of of two tasks. The second task fetches an FTP file and stores on Google Cloud. The first task is an external dag that waits for the file to be ready. I am wondering if I can pass the execution date to the external dag. Right now, my code uses the dag.start_

Re: FAB - New REST API in the works

2019-04-01 Thread danielvazgaspar
On 2019/04/01 17:14:28, Maxime Beauchemin wrote: > Hey! > > I wanted to point out that there's awesome work taking place in FAB around > a new REST API provided by the framework, and ways to extend it. > > Daniel Gaspar (cced) is working on this currently, and looking for input on > design /

Re: FAB - New REST API in the works

2019-04-01 Thread Ash Berlin-Taylor
I'll comment on the PR tomorrow, but from a quick scan i didn't see anything about OpenAPI/Swagger which would be a great addition. -a On 1 April 2019 18:14:28 BST, Maxime Beauchemin wrote: >Hey! > >I wanted to point out that there's awesome work taking place in FAB >around >a new REST API pro

FAB - New REST API in the works

2019-04-01 Thread Maxime Beauchemin
Hey! I wanted to point out that there's awesome work taking place in FAB around a new REST API provided by the framework, and ways to extend it. Daniel Gaspar (cced) is working on this currently, and looking for input on design / implementation. Check it out and chime in on the PR https://github

Re: 1.10.3beta2 "snapshot" for available testing

2019-04-01 Thread Jiajie Zhong
Awesome and Thanks, Ash Best wish. -- Jiajie From: Ash Berlin-Taylor Sent: Monday, April 1, 2019 23:43 To: dev@airflow.apache.org Subject: 1.10.3beta2 "snapshot" for available testing Hi Everyone, Following on form 1.10.3b1 I've pulled in another 84 (!) commits,

Re: 1.10.3beta2 "snapshot" for available testing

2019-04-01 Thread Jiajie Zhong
Awesome and Thanks for what you did. Best wish. -- Jiajie From: Ash Berlin-Taylor Sent: Monday, April 1, 2019 23:43 To: dev@airflow.apache.org Subject: 1.10.3beta2 "snapshot" for available testing Hi Everyone, Following on form 1.10.3b1 I've pulled in another 84

1.10.3beta2 "snapshot" for available testing

2019-04-01 Thread Ash Berlin-Taylor
Hi Everyone, Following on form 1.10.3b1 I've pulled in another 84 (!) commits, mostly around doc fixes or code-reorg. I will start the RC process this week. In order to distinguish it from an actual (apache) release it is: 1. Marked as beta (python package managers do not install beta versions

Re: API Reference - current confusion and improvement plan

2019-04-01 Thread Ash Berlin-Taylor
That got fixed - thanks for the reminder :) > On 29 Mar 2019, at 17:07, Jiajie Zhong wrote: > > Awesome Ash ??, expecting beta2. > And don't forget some minor bug still in beta1 > > https://issues.apache.org/jira/browse/AIRFLOW-3623?focusedCommentId=16803866&page=com.atlassian.jira.plugin.syst

Re: Task middleware or task hooks as a way to increase observability

2019-04-01 Thread Stachurski, Stephan
If this is a common pattern, then I think it's worth considering the level of effort to bake this into airflow, both for developer convenience and safety. If it's baked into airflow, a developer working on dags in the cluster will be saved from accidentally overriding important callbacks that feed

Re: Task middleware or task hooks as a way to increase observability

2019-04-01 Thread Stachurski, Stephan
I'm not opposed to the idea of building on the existing statsd metrics, but they are really not anywhere close to the richness and configurability that I think I would want for my use case. If you look at the existing metrics, they are all about looking at the health of an airflow cluster in aggre

Re: Task middleware or task hooks as a way to increase observability

2019-04-01 Thread Greg Neiheisel
Hey Stephan, not sure if you've seen it or not, but Airflow has some built in support for exporting statsd metrics. We run everything in Kubernetes and are also heavy users of Prometheus. We've had a pretty solid experience using the statsd-exporter (https://github.com/prometheus/statsd_exporter),

Re: Hangout: Learn how to hack on Airflow (recording available)

2019-04-01 Thread Ash Berlin-Taylor
The recording has been uploaded here: https://www.youtube.com/watch?v=UBtXaPEg9yE The text may not be clear at various points, sorry about that; I'll try and get around to writing up a blog post or two of things. -ash > On 30 Mar 2019, at 05:07, Kevin Yang wrote: > > Thanks Ash, love to see

Re: [VOTE] Accept AIP-3: Drop support for Python 2

2019-04-01 Thread Bas Harenslak
Great news! Happy to remove the Python 2 code ;-) Btw that list (https://jira.apache.org/jira/browse/AIRFLOW-4196) was a brain-dump. If there’s anything missing, feel free to create more issues. Bas On 1 Apr 2019, at 11:03, Driesprong, Fokko mailto:fo...@driesprong.frl>> wrote: Hi all, I'm h

Re: [DISCUSS] Utility DAGs

2019-04-01 Thread Ash Berlin-Taylor
> For example, say we want to support retention in some Airflow tables such > as task_instance, dag_run and log, it seems reasonable to me to create a > DAG to periodically clean up the tables I guess you mean something like https://github.com/teamclairvoyant/airflow-maintenance-dags/tree/master/

Re: [VOTE] Accept AIP-3: Drop support for Python 2

2019-04-01 Thread Driesprong, Fokko
Hi all, I'm happy to inform you all that the vote has been passed! From Airflow 2.0 we will drop support for Python 2 (and this is not an April Fools' joke :-). With a total 18 +1 votes, and a single +0 vote (at least a positive zero) from Ash. With the input of Bas Harenslak, I've composed a no