Re: Graduation resolution passed - Airflow is a TLP

2018-12-20 Thread Driesprong, Fokko
Awesome! Congrats! Cheers, Fokko Op do 20 dec. 2018 om 22:40 schreef Sid Anand > YaaY! > > -s > > On Thu, Dec 20, 2018 at 1:13 PM Jakob Homan wrote: > > > Hey all- > >The Board minutes haven't been published yet (probably due to > > Holiday-related slowness), but I can see through the

Refactor models.py

2018-12-06 Thread Driesprong, Fokko
Hi All, I think it is time to refactor the infamous models.py. This file is far too big, and it only keeps growing. My suggestion is to create a new package, called models, which will contain all the orm classes (the ones with __tablename__ in the class). And for example the BaseOperator to the

Re: [DISCUSS] Apache Airflow graduation from the incubator

2018-12-02 Thread Driesprong, Fokko
> mainly > > >>>>>> concerns filling in some items, maybe here and there creation some > > >>>>>> documentation but I don't think much. @Kaxil, @Tao: are you > willing > > to pick > > >>>>>> this up? @Sid ca

Re: [VOTE] Graduate the Apache Airflow as a TLP

2018-11-30 Thread Driesprong, Fokko
+1 binding Op vr 30 nov. 2018 om 23:05 schreef Bolke de Bruin > +1, binding > > Yahoo! :-) > > Verstuurd vanaf mijn iPad > > > Op 30 nov. 2018 om 22:48 heeft Tao Feng het > volgende geschreven: > > > > +1 (binding) > > > > Thanks Jakob and everyone! > > > >> On Fri, Nov 30, 2018 at 1:33 PM

Re: [DISCUSS] Apache Airflow graduation from the incubator

2018-11-27 Thread Driesprong, Fokko
+1 from my side. Would be awesome to graduate Airflow If time allows, I'll also dive into CD30. Cheers, Fokko Op di 27 nov. 2018 om 16:21 schreef Ash Berlin-Taylor : > Oarsome Bolke, thanks for starting this. > > It looks like we are closer than I thought! > > We can use those security lists

Re: Remove airflow from pypi

2018-11-27 Thread Driesprong, Fokko
02 AM, George Leslie-Waksman > > wrote: > > > > > > It's probably a good idea to put something at "airflow", even if it > > > just fails to install and tells people to install apache-airflow > > > instead. > > > > > > If not, there's

Re: Remove airflow from pypi

2018-11-23 Thread Driesprong, Fokko
s, I went ahead and deleted it. > > > > On Fri, Nov 23, 2018 at 10:18 AM Bolke de Bruin > wrote: > > > > > Agree! This is even a security issue. > > > > > > Sent from my iPhone > > > > > > > On 23 Nov 2018, at 15:29, Driesprong, Fok

Remove airflow from pypi

2018-11-23 Thread Driesprong, Fokko
Hi all, I think we should remove airflow (not apache-airflow) from Pypi. I still get questions from people who accidentally install Airflow 1.8.0. I see this is maintained by mistercrunch, artwr, aeon. Anyone any objections? Cheers, Fokko

Re: Task got executed even when it's only immediate parent was in upstream failed state

2018-11-18 Thread Driesprong, Fokko
this > in the changelog. > > > > > > > Regards, > > Abhishek > > On 18 November 2018 at 8:54:43 AM, Driesprong, Fokko (fo...@driesprong.frl) > wrote: > > Hi Abhishek, > > Do you think it would be difficult to verify if this behavior still exists > in 1.10? Ma

Re: Fusing operators together

2018-11-18 Thread Driesprong, Fokko
Hi Shubham, I think the EmrStepOperator and EmrStepSensor are a clear exception. Most operators wait until the operation has finished successfully. For example, the DruidOperator will block until the indexing job has successfully finished:

Re: Task got executed even when it's only immediate parent was in upstream failed state

2018-11-18 Thread Driesprong, Fokko
Hi Abhishek, Do you think it would be difficult to verify if this behavior still exists in 1.10? Maybe write a dummy dag, and run it using Docker. Cheers, Fokko Op vr 16 nov. 2018 om 21:45 schreef Abhishek Sinha : > Airflow Version 1.8.2, Celery and Postgres > > Task (and its downstream nodes)

Re: [VOTE] Airflow 1.10.1 RC2

2018-11-18 Thread Driesprong, Fokko
A +1 from my side as well. Thanks for picking this up Ash. Just checked the new release using Docke r, everything seems to work. Cheers, Fokko Op za 17 nov. 2018 om 16:43 schreef Deng Xiaodong : > Even

Re: Can I change the subject line for alert emails

2018-11-13 Thread Driesprong, Fokko
Thanks. Alek, can you rebase the PR, and resolve conflicts? Op ma 12 nov. 2018 om 19:42 schreef Alek Storm : > I have a PR I need to rebase that would enable this (we had the exact same > requirement): https://github.com/apache/incubator-airflow/pull/2338 > > Alek > > On Mon, Nov 12, 2018 at

Re: REST API roadmap/plan?

2018-11-05 Thread Driesprong, Fokko
Thanks Matthew, We're working on getting the experimental API. This API should be used for both the GUI and can be used for external systems to interface with Airflow (triggering dags for example). Verdan was working on this, but for now, this process is a bit stuck again. It turns out that data

Re: Apache Airflow / Cloud Composer workshops Amsterdam

2018-10-15 Thread Driesprong, Fokko
t; Sounds like a great event! Will any of the talks/workshops be > streamed/livecast/recorded for those of us who can't make it to Amsterdam? > > - Ben > > On Fri, Oct 12, 2018 at 12:40 PM Driesprong, Fokko > wrote: > > > Hi all, > > > > From October

Re: Something May Be Wrong with the Travis CI Tests

2018-10-14 Thread Driesprong, Fokko
w what you think. > (I will start from DockerOperator first though). > > > XD > > On Sat, Oct 13, 2018 at 8:02 PM Driesprong, Fokko > wrote: > > > Hi XD, > > > > Very good point. I was looking into this recently, but since time is a > > limit

Re: Something May Be Wrong with the Travis CI Tests

2018-10-13 Thread Driesprong, Fokko
Hi XD, Very good point. I was looking into this recently, but since time is a limited matter, I did not really dig into it. It has to do with the test discovery. The python_operator does not match the given pattern test*.py:

Apache Airflow / Cloud Composer workshops Amsterdam

2018-10-12 Thread Driesprong, Fokko
Hi all, >From October 15-19, 2018, GoDataFest takes place in Amsterdam, The Netherlands. This week is dedicated to data technology and features free talks, training sessions and workshops. Leading tech companies, like AWS (Monday, October 15), Dataiku (Tuesday, October 16), Databricks

Re: "setup.py test" is being naughty

2018-10-12 Thread Driesprong, Fokko
We're working hard to get rid of the tight Travis integration and moving to a Docker based setup. I think it should be very easy to get a Docker up and running which is packed with the required dependencies. Unfortunately we're not there yet. Also the tox layer feels a bit redundant to me, since

Re: Pinning dependencies for Apache Airflow

2018-10-04 Thread Driesprong, Fokko
Hi Jarek, Thanks for bringing this up. I missed the discussion on Slack since I'm on holiday, but I saw the thread and it was way too interesting, and therefore this email :) This is actually something that we need to address asap. Like you mention, we saw it earlier that specific transient

Re: It's very hard to become a committer on the project

2018-09-23 Thread Driesprong, Fokko
Many thanks for the effort Stefan, I just went through them and updated the status accordingly in Jira. Thanks! Cheers, Fokko Op za 22 sep. 2018 om 23:25 schreef Stefan Seelmann : > On 9/20/18 10:02 PM, Driesprong, Fokko wrote: > > us still have a full time job on the side :) Tomo

Re: Fundamental change - Separate DAG name and id.

2018-09-20 Thread Driesprong, Fokko
I like the dag_id for both the name and as an unique identifier. If you change the dag in such a way, that it deserves a new name, you probably want to create a new dag anyway. If you want to give some additional context, you can use the description field:

Re: It's very hard to become a committer on the project

2018-09-20 Thread Driesprong, Fokko
Some history. Recently we've moved the to Gitbox and moved away from the Apache repo itself to simplify the setup. Before that, the code on Github was merely a mirror of the Apache git repo. Before Gitbox there was a Python script that would like the issues to the

Re: Database referral integrity

2018-09-18 Thread Driesprong, Fokko
I'm in favor of having referential integrity. It will add some load in having to enforce the referential integrity, but it will also make sure that the database stays clean. Also in Airflow we use transactions which will make sure that the integrity checks are not validated on every statement, but

Re: Guidelines on Contrib vs Non-contrib

2018-09-18 Thread Driesprong, Fokko
I fully agree with using plain Python modules :) I don't think a lot of hooks/operators graduate to core since it will break the import. A few of them, for example Databricks and the Google hooks are mature enough. For me the main point is having test coverage and a stable API. Cheers, Fokko Op

Re: Call for fixes for Airflow 1.10.1

2018-09-12 Thread Driesprong, Fokko
nsure that he/she knows what the state is. > > > > B > > > > Op zo 9 sep. 2018 18:02 schreef Driesprong, Fokko >: > > Can we add this one as well? > > > > https://github.com/apache/incubator-airflow/pull/3862 < > https://github.com/apache/inc

Re: Call for fixes for Airflow 1.10.1

2018-09-09 Thread Driesprong, Fokko
Can we add this one as well? https://github.com/apache/incubator-airflow/pull/3862 https://issues.apache.org/jira/browse/AIRFLOW-1917 I'm happy to cherry pick them onto the 1.10.1 by myself as well. Any idea when we will start this branch? Cheers, Fokko Op do 6 sep. 2018 om 08:08 schreef Deng

Re: re run build

2018-09-05 Thread Driesprong, Fokko
Hi, This PR has to rebase onto master first to get rid of the conflicts. Cheers, Fokko Op wo 5 sep. 2018 om 09:20 schreef airflowuser : > Can someone assist with rerun build for: > https://github.com/apache/incubator-airflow/pull/2488 > > tried to send it to committers mailing list but for

Re: Call for fixes for Airflow 1.10.1

2018-09-03 Thread Driesprong, Fokko
Ash, thanks for picking this up! Maybe add this one: https://github.com/apache/incubator-airflow/pull/3804 https://issues.apache.org/jira/browse/AIRFLOW-2866 Cheers, Fokko Op ma 3 sep. 2018 om 14:47 schreef Deng Xiaodong : > Hi Ash, > > JIRA ticket 2922

Re: WebHdfsSensor doesn't support HDFS HA

2018-08-29 Thread Driesprong, Fokko
Hi Manu, Thanks for raising this question. There is a PR for moving to hdfs3. There is code in the existing codebase, which support HA

Re: [RESULT][VOTE] Release Airflow 1.10.0

2018-08-27 Thread Driesprong, Fokko
wrote: > > Done for Bolke, Fokko and kaxil. > > Best, > Arthur > > On Sun, Aug 26, 2018 at 3:08 AM Driesprong, Fokko > > wrote: > > > Gentle ping! Would be awesome to get 1.10 on Pypi :-) > > > > Op wo 22 aug.

Re: [RESULT][VOTE] Release Airflow 1.10.0

2018-08-26 Thread Driesprong, Fokko
gt; B. > > Sent from my iPhone > > > > > Kaxil Naik > > Data Reply > 2nd Floor, Nova South > 160 Victoria Street, Westminster > London SW1E 5LB - UK > phone: +44 (0)20 7730 6000 > k.n...@reply.com > www.reply.com > On 22 Aug 2018, at 16:13, Driespr

Amsterdam Apache Airflow meetup

2018-08-25 Thread Driesprong, Fokko
Hi all, Due a cancellation there is a speaker slot available at the Amsterdam Meetup the 12th of September: https://www.meetup.com/Amsterdam-Airflow-meetup/events/253673642/ If you're interested in speaking let me know. It can be quite broad, from how you're using Airflow within your

Re: [RESULT][VOTE] Release Airflow 1.10.0

2018-08-22 Thread Driesprong, Fokko
Certainly: https://github.com/apache/incubator-airflow/releases/tag/1.10.0 Cheers, Fokko Op wo 22 aug. 2018 om 15:18 schreef Ash Berlin-Taylor : > Could you push the git tag too please Fokko/Bolke? > > -ash > > > On 22 Aug 2018, at 08:16, Driesprong, Fokko > wrot

Re: PR Review Dashboard?

2018-08-22 Thread Driesprong, Fokko
s week > and I’ll report back. > > On Tue, Aug 7, 2018 at 12:15 AM Driesprong, Fokko > wrote: > > > Hi Holden, > > > > Thanks for reaching out. Recently we've moved to Apache Gitbox ( > > https://gitbox.apache.org/), so we use the Github UI directly instead o

Re: Jira cleanup and triage

2018-08-22 Thread Driesprong, Fokko
Hi Gerardo, Thanks for bringing this up. This is actually a good point. Recently we've moved the Apache Airflow repo to the Gitbox repo ( https://gitbox.apache.org/). Before with the Apache repo, the Github repo was just a mirror of the Apache one. Now we do everything on Github itself. We still

Re: [RESULT][VOTE] Release Airflow 1.10.0

2018-08-22 Thread Driesprong, Fokko
Thanks Max, My PyPI ID is Fokko Cheers, Fokko 2018-08-21 22:49 GMT+02:00 Maxime Beauchemin : > I can, what's your PyPI ID? > > Max > > On Mon, Aug 20, 2018 at 2:11 PM Driesprong, Fokko > wrote: > > > Thanks Bolke! > > > > I've just pushed the

Re: Regarding airflow 1.10

2018-08-21 Thread Driesprong, Fokko
Hi Hemanth, Thanks for the question. There is the KubernetesExecutor which just uses a container with Airflow to schedule a task on the kubernetes cluster. This container will then execute a single task, and then the Executor checks the exitcode of the container. There is also the

Re: updating dags best practice

2018-08-21 Thread Driesprong, Fokko
Hi Aleksander, What kind of executor are you using? This is really important when choosing a strategy. For most of the Airflow deployments that I've worked with simply uses git to deploy the master branch. This is with a LocalExecutor. Cheers, Fokko Op di 21 aug. 2018 om 15:25 schreef Dev

Re: [RESULT][VOTE] Release Airflow 1.10.0

2018-08-20 Thread Driesprong, Fokko
Thanks Bolke! I've just pushed the artifacts to Apache Dist: https://dist.apache.org/repos/dist/release/incubator/airflow/1.10.0-incubating/ I don't have any access to pypi, this means that I'm not able to upload the artifacts over there. Anyone in the position to grand me access or upload it to

Use Docker for running Airflow tests

2018-08-19 Thread Driesprong, Fokko
Hi all, Gerardo is doing some awesome work on Dockerizing the CI pipeline. The PR is still open here: https://github.com/apache/incubator-airflow/pull/3393 To make this work, and not store any Docker images in private repositories, I've created a new repository which stores the Docker image

Re: Apache Airflow welcome new committer/PMC member : Feng Tao (a.k.a. feng-tao)

2018-08-12 Thread Driesprong, Fokko
t; wrote: > > > > > > > > > > > Congratulations, Feng! > > > > > > > > > > > > *Taylor Edmiston* > > > > > > Blog <https://blog.tedmiston.com/> | CV > > > > > > <https://stack

Re: [VOTE] Airflow 1.10.0rc4

2018-08-09 Thread Driesprong, Fokko
Good point Bolke, Sid, seems that there are still a few issues with Tenacity as well , therefore I would like to change my vote: +1 (binding) Cheers, Fokko 2018-08-09 14:08 GMT+02:00 Ash Berlin-Taylor : > +0.5 (binding) from me. > > Tested

Re: [VOTE] Airflow 1.10.0rc4

2018-08-08 Thread Driesprong, Fokko
-1 (binding) Sorry Bolke for not checking this earlier. In rc3 we've replaced some of the reserved keywords. But still I'm unable to run a simple dag in the 1.10 rc4 release under Python 3.7: MacBook-Pro-van-Fokko:sdh-api-pobt fokkodriesprong$ docker run -e SLUGIFY_USES_TEXT_UNIDECODE=yes -t -i

Re: PR Review Dashboard?

2018-08-07 Thread Driesprong, Fokko
Hi Holden, Thanks for reaching out. Recently we've moved to Apache Gitbox ( https://gitbox.apache.org/), so we use the Github UI directly instead of having to merge using a CLI ( https://github.com/apache/incubator-airflow/blob/master/dev/airflow-pr). Not sure if we're already up to the game of

Re: Apache Airflow welcome new committer/PMC member : Feng Tao (a.k.a. feng-tao)

2018-08-03 Thread Driesprong, Fokko
Welcome Feng! Awesome to have you on board! 2018-08-03 10:41 GMT+02:00 Naik Kaxil : > Hi Airflow'ers, > > > > Please join the Apache Airflow PMC in welcoming its newest member and > > co-committer, Feng Tao (a.k.a. feng-tao). > > > > Welcome Feng, great to have you

Re: [VOTE] Airflow 1.10.0rc3

2018-08-03 Thread Driesprong, Fokko
+1 Binding Installed it using: SLUGIFY_USES_TEXT_UNIDECODE=yes pip install https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc3/apache-airflow-1.10.0rc3+incubating-bin.tar.gz Cheers, Fokko 2018-08-03 9:47 GMT+02:00 Bolke de Bruin : > Hey all, > > I have cut Airflow 1.10.0 RC3.

Re: We've migrated to Github to repo!

2018-08-01 Thread Driesprong, Fokko
low debate > here...] > > Max > > On Tue, Jul 31, 2018 at 12:37 PM Driesprong, Fokko > wrote: > > > Hi Max, > > > > You're right. I just started plowing though my mailbox and merged a > commit > > without squash and merge, but it changes history as you

Re: We've migrated to Github to repo!

2018-07-31 Thread Driesprong, Fokko
Hi Max, You're right. I just started plowing though my mailbox and merged a commit without squash and merge, but it changes history as you mention. Nice thing of Github is if you change it, it remembers your preference which is Squash and Merge :-) Love the Gitbox so far, great work! Cheers,

Re: Kerberos and Airflow

2018-07-26 Thread Driesprong, Fokko
Hi Ry, You should ask Bolke de Bruin. He's really experienced with Kerberos and he did also the implementation for Airflow. Beside that he worked also on implementing Kerberos in Ambari. Just want to let you know. Cheers, Fokko Op do 26 jul. 2018 om 23:03 schreef Ry Walker > Hi everyone - > >

Re: Airflow's JS code (and dependencies) manageable via npm and webpack

2018-07-23 Thread Driesprong, Fokko
​Nice work Verdan. The frontend really needed some love, thank you for picking this up. Maybe we should also think deprecating the old www. Keeping both of the UI's is something that takes a lot of time. Maybe after the release of 1.10 we can think of moving to Airflow 2.0, and removing the old

Re: [VOTE] Airflow 1.10.0rc2

2018-07-16 Thread Driesprong, Fokko
My vote is ​+1 (binding)​ Cheers, Fokko 2018-07-16 9:52 GMT+02:00 Driesprong, Fokko : > Awesome Bolke! > > I don't have a production Airflow at hand right now, but I've ran some > simple tests against Python 2.7, 3.5, 3.6 and it all looked fine. > > +1 From my side > > C

Re: [VOTE] Airflow 1.10.0rc2

2018-07-16 Thread Driesprong, Fokko
Awesome Bolke! I don't have a production Airflow at hand right now, but I've ran some simple tests against Python 2.7, 3.5, 3.6 and it all looked fine. +1 From my side Cheers, Fokko 2018-07-15 20:05 GMT+02:00 Bolke de Bruin : > Hey all, > > I have cut Airflow 1.10.0 RC2. This email is calling

Re: [Proposal] Explicit re-schedule of sensors

2018-07-15 Thread Driesprong, Fokko
Thanks Stefan for picking this up. The sensors are in desperate need for some redesign for the aforementioned reasons. Please note this ticket: https://issues.apache.org/jira/browse/AIRFLOW-2001 It addresses the same issue. Regarding the open question. I would be reluctant for introducing new

Re: [VOTE] Airflow 1.10.0rc1

2018-07-15 Thread Driesprong, Fokko
does not work yet, but it isn't supported anyway. I would like to have https://github.com/apache/incubator-airflow/pull/3604 in RC2. It isn't a critical bug, but it looks messy. Cheers, Fokko 2018-07-14 0:06 GMT+02:00 Driesprong, Fokko : > Thanks Bolke for all the effort. > > I t

Re: [VOTE] Airflow 1.10.0rc1

2018-07-13 Thread Driesprong, Fokko
> Verstuurd vanaf mijn iPad > > > Op 13 jul. 2018 om 22:23 heeft Driesprong, Fokko > het volgende geschreven: > > > > Ok, I've did some testing. > > > > 1.10 works fine with the LocalExecutor. With the SequentialExecutor it > does > > not pick up any task, e

Re: [VOTE] Airflow 1.10.0rc1

2018-07-13 Thread Driesprong, Fokko
re not picked up. If you put a dag in the normal dag > folder it works fine. > >> > >> Please create a jira for this @fokko. A pr would be appreciated. > >> > >> B. > >> > >> Sent from my iPhone > >> > >>> On 13 Jul 2

Re: [VOTE] Airflow 1.10.0rc1

2018-07-13 Thread Driesprong, Fokko
With the SequentialExecutor the webserver also acts as the scheduler (without parallelism) 2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson : > Need to run the scheduler also right? > > -- > Carl Johan Gustavsson > > On 13 July 2018 at 15:37:22, Driesprong, Fokko (fo...@dries

Re: [VOTE] Airflow 1.10.0rc1

2018-07-13 Thread Driesprong, Fokko
I've installed Airflow, but using a clean install and the sequential executor, the tasks are not being picked up. This is easy to replicate: docker run -t -i --rm -p 8080:8080 python:3.5 bash pip install

Re: Apache Airflow 1.10.0b3

2018-07-13 Thread Driesprong, Fokko
Ah, I picked the wrong tar.gz, Thanks B Cheers, Fokko 2018-07-13 15:07 GMT+02:00 Bolke de Bruin : > Please use the Rc Fokko. This has been fixed in the RC. > > B. > > Verstuurd vanaf mijn iPad > > > Op 13 jul. 2018 om 15:01 heeft Driesprong, Fokko > het volgende

Re: Apache Airflow 1.10.0b3

2018-07-13 Thread Driesprong, Fokko
Hi all, I just tried to install a local Airflow instance within Docker, using Python 3.5, but I got an error on initdb. Anyone else experiencing this? It has to do with the schema migrations. Cheers, Fokko 2018-07-09 22:56 GMT+02:00 Taylor Edmiston : > We discussed the splitting out of hooks &

Re: DagRunOperator - target dag tasks are not triggered - possible timezone conflict

2018-07-06 Thread Driesprong, Fokko
Hi Niranda, What version of Airflow are you running? There are a lot of improvements coming up in Airflow 1.10 regarding timezones. https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L77-L79 Cheers, Fokko 2018-07-05 14:57 GMT+02:00 Niranda

Re: Airflow 1.10.0

2018-06-15 Thread Driesprong, Fokko
> > > > Sorry that I've been MIA for a while. Just wanted to check in on 1.10, > and > > its current status. > > > > Cheers, > > Chris > > > > On Tue, May 1, 2018 at 11:18 PM Driesprong, Fokko > > wrote: > > > >> Hi Bryon, > &

Re: Master is broken

2018-06-12 Thread Driesprong, Fokko
Hi Gerardo, I totally agree that when master turns red, we should stop merging and fix the build or revert the commit that broke the build. I think one of the underlying problems is having flaky tests, I tried to fix a few of those, but they are quite persistent. Sometimes it is hard to

Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator

2018-05-30 Thread Driesprong, Fokko
Edmiston* > Blog <https://blog.tedmiston.com/> | CV > <https://stackoverflow.com/cv/taylor> | LinkedIn > <https://www.linkedin.com/in/tedmiston/> | AngelList > <https://angel.co/taylor> | Stack Overflow > <https://stackoverflow.com/users/149

Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator

2018-05-30 Thread Driesprong, Fokko
Hi Craig, This is something that needs to be fixed. I agree with you this is very dirty. In your installation you're not installing the kubernetes stuff, so the KubernetesPodOperator is ignored. We need to figure out how to have example dags that are not compatible with the vanilla installation,

Re: HttpSensor raising exception with status=403

2018-05-28 Thread Driesprong, Fokko
Hi Pedro, You could just create a CustomHttpHook and place it on your pythonpath, then you should also create a CustomHttpSensor. Hope this helps. Cheers, Fokko 2018-05-26 2:48 GMT+02:00 Pedro Machado : > Hi, > > I am using HttpSensor to look for a file. The webserver is

Re: How to wait for external process

2018-05-28 Thread Driesprong, Fokko
Hi Stefan, Afaik there isn't a more efficient way of doing this. DAGs that are relying on a lot of sensors are experiencing the same issues. The only way right now, I can think of, is doing updating the state directly in the database. But then you need to know what you are doing. I can image that

Re: 1.10.0 Release

2018-05-26 Thread Driesprong, Fokko
Hi Kaxil, Good point. Right now Bolke and I are a bit busy with the PyData talk that we're giving tomorrow in Amsterdam. I've did a `git checkout v1-10-test && git reset --hard apache/master` a couple of times to make it up to date with master. We have to be careful to set the version number

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

2018-05-22 Thread Driesprong, Fokko
>> 3CCAO2yDybRq2VUM1JYo_6VT_H8Ca7Lu8af6H-2CZKQzYT6xYGU-g%40mail.gmail.com > %3E > >>> > >>> > >>>> On 6 Mar 2018, at 09:57, Ash Berlin-Taylor > <ash_airflowlist@firemirror. > >>> com> wrote: > >>>> > >>&g

Re: Airflow with Celery

2018-05-16 Thread Driesprong, Fokko
I had similar issues with Airflow running the Celery executor. The celery_result_backend should be a persistent database like Postgres or MySql. What broker are you using? I would recommend using Redis or RabbitMQ, depending on what you like the most. Cheers, Fokko 2018-05-15 21:12 GMT+02:00

Re: Python3 and sensors module

2018-05-16 Thread Driesprong, Fokko
Hi Cindy, The other sensors should work under Python3. We try to support Python3 as much as possible, but sometimes libraries are used that are not compatible. Could you describe what you are running into? Cheers, Fokko 2018-05-16 5:36 GMT+02:00 Cindy Rottinghuis :

Re: Airflow Docker Container

2018-05-14 Thread Driesprong, Fokko
Hi Daniel, My dear colleague from GoDataDriven, Bas Harenslak, started on building an official Docker container on the Dockerhub. I've put him in the CC. In the end I strongly believe the image should end up in the official Docker repository: https://github.com/docker-library/official-images

Re: 答复: Airflow REST API proof of concept.

2018-05-11 Thread Driesprong, Fokko
Hi Luke, This is the REST api for the new UI: https://github.com/apache/incubator-airflow/blob/master/airflow/www_rbac/api/experimental/endpoints.py RBAC = Role Based Access Control, the fine grained security model based on the fabmanager. Recently we've added some endpoints to it. In the end

Re: Airflow 1.10.0

2018-05-02 Thread Driesprong, Fokko
Hi Bryon, We'll be releasing the RC's soon. If there aren't much issues with the RC's, it will be released quickly. But we need the community to test these. Cheers, Fokko 2018-05-01 20:57 GMT+02:00 Wicklund, Bryon : > Hey I was wondering if you had a date in mind or

Re: Problem with SparkSubmit

2018-04-28 Thread Driesprong, Fokko
Hi Anton, Which version of Airflow are you running? Cheers, Fokko 2018-04-27 10:24 GMT+02:00 Anton Mushin : > Hi all, > I have problem with spark operator. I get exception > > user@host:/# airflow test myDAG myTask 2018-04-26 > [2018-04-26 15:32:11,279] {driver.py:120}

Re: Use KubernetesExecutor to launch tasks into a Dask cluster in Kubernetes

2018-04-28 Thread Driesprong, Fokko
Also one of the main benefits of the Kubernetes Executor is having a Docker image that contains all the dependencies that you need for your job. Personally I would switch to Kubernetes when it leaves the experimental stage. Cheers, Fokko 2018-04-28 16:27 GMT+02:00 Kyle Hamlin

Re: k8s example DAGs

2018-04-23 Thread Driesprong, Fokko
Hi Ruslan, This is a good point. I also get No module named kubernetes exceptions when running the initdb. This should be fixed. Could you create a Jira ticket for this? Cheers, Fokko 2018-04-23 4:28 GMT+02:00 Ruslan Dautkhanov : > Is it possible to make kubernetes

Run duration

2018-04-19 Thread Driesprong, Fokko
Hi all, I have a question regarding the run duration. Is anyone still using this? You can use this to kill the scheduler after a predefined set of seconds. I know this was implemented in the early days to force a restart of the Airflow scheduler once a while. Recently I was at a customer, which

Re: Trouble with remote s3 logging

2018-04-17 Thread Driesprong, Fokko
Hi Kyle, Thanks for reaching out. This looks like a bug. Could you open a Jira issue describing this issue? This seems to be something that should be rather easy fo fix. Since it tries to append a tuple instead of a string. I would expect that more people would encounter this bug. Cheers, Fokko

Re: Give it up for Fokko!

2018-04-14 Thread Driesprong, Fokko
Thanks you all for the kind words. I really love the energetic community around Airflow and all the cool stuff that we're working on. I find it truly amazing how we build such an awesome product with great people from all around the world! Cheers, Fokko 2018-04-14 1:56 GMT+02:00 Sid Anand

Re: Cancel a Running dag

2018-04-13 Thread Driesprong, Fokko
Like Bolke said, it has been fixed in master. One of the perquisites is support by the operator. For example, the Spark operator has implemented how to kill the Spark job on YARN, Local and Kubernetes. If you are running something else, you might want to check if this is implemented. Implemented

Re: Readthedocs

2018-04-10 Thread Driesprong, Fokko
Thanks Kaxil! Cheers, Fokko 2018-04-10 14:19 GMT+02:00 Naik Kaxil <k.n...@reply.com>: > Added you. > > https://readthedocs.org/projects/airflow/ > > Regards, > Kaxil > > > On 10/04/2018, 13:04, "fo...@driesprongen.nl on behalf of Driesprong, > Fokk

Readthedocs

2018-04-10 Thread Driesprong, Fokko
Hi all, Would it be possible to give me access to the Readthedocs environment. I'm working on improving the docs a bit, but I can't see the output of the docs generation. Cheers, Fokko

Re: Issue : Merges are not syncing back to master

2018-04-09 Thread Driesprong, Fokko
Thanks for picking this up Sid! Cheers, Fokko 2018-04-06 7:22 GMT+02:00 Sid Anand : > Heads up. Apache Airflow github master > is behind Apache Airflow > master >

Re: Running dag run doesn't schedule task

2018-03-19 Thread Driesprong, Fokko
Hi David, First I would update to Apache Airflow 1.9.0, there have been a lot of fixes between 1.8.2 and 1.9.0. Just to see if the bug is still in there. Cheers, Fokko 2018-03-18 19:41 GMT+01:00 David Capwell : > Thanks for the reply > > Our script doesn't set it so should

Re: Very looong py files

2018-03-11 Thread Driesprong, Fokko
Maybe wait until this one has been merged 浪 https://github.com/apache/incubator-airflow/pull/3116/files Cheers, Fokko Op zo 11 mrt. 2018 om 21:34 schreef Driesprong, Fokko <fo...@driesprong.frl> > Hi Bruno, > > I agree that there are files that are too big as you menti

Re: Very looong py files

2018-03-11 Thread Driesprong, Fokko
Hi Bruno, I agree that there are files that are too big as you mentioned. Sometimes structures are being refactored like I did with the sensors. The problem is that if you refactor one of these files, most PR requests will have merge conflicts. But this should not be seen as an impediment to

Re: Ash Berlin-Taylor joins Apache Airflow as committer and PPMC member

2018-02-24 Thread Driesprong, Fokko
Welcome Ash! 2018-02-23 18:45 GMT+01:00 Andy Hadjigeorgiou : > Congrats Ash! > > > On Feb 23, 2018, at 12:19 PM, Taylor Edmiston > wrote: > > > > Congrats, Ash! Very exciting. > > > > *Taylor Edmiston* > > TEdmiston.com |

Re: Scheduling error that seems to be related to @once since Airflow v1.9 upgrade

2018-02-12 Thread Driesprong, Fokko
Hi Aaron, It looks like you hit a bug: https://issues.apache.org/jira/browse/AIRFLOW-1977 Currently I don't have a solution for this at hand, I have to dig into the code. If the DAG is scheduled one, you might want to try setting it to None instead of @once:

Re: Airflow Documentation - Readthedocs

2018-02-11 Thread Driesprong, Fokko
e GitHub home (Readme) are all broken. > > Should I change the all the links to readthedocs? Or can someone who > has access to PyPI create the docs? > > What do you suggest? > > Regards, > Kaxil > > On 11/02/2018, 06:03, "Dri

Re: Airflow Documentation - Readthedocs

2018-02-10 Thread Driesprong, Fokko
gt; > > >>>>> Best, > > >>>>> Arthur > > >>>>> > > >>>>> On Sun, Feb 4, 2018 at 8:14 AM, Bolke de Bruin < > > >> bdbr...@gmail.com> > > >>>> wro

Re: Airflow Documentation - Readthedocs

2018-02-04 Thread Driesprong, Fokko
This is a good question, I see that artwr and Maxime are owners of the project, maybe they can fix the build? Or add some more Airflow committers as readthedocs project owners. I get quite some questions from colleagues because they are reading old docs :) Cheers, Fokko 2018-02-04 14:32

Fixed Travis permission problem

2018-01-31 Thread Driesprong, Fokko
Hi all, I've hotfixed the CI permission problem: Exception: Traceback (most recent call last): File "/home/travis/build/apache/incubator-airflow/.tox/py35-backend_mysql/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main status = self.run(options, args) File

Re: Airflow 1.8 CeleryExecutor: multiple servers in broker_url

2018-01-17 Thread Driesprong, Fokko
Hi Andrii, When looking in the Airflow code, I'd say this should workd: https://github.com/apache/incubator-airflow/blob/v1-8-stable/airflow/executors/celery_executor.py#L41 Which version of celery are you using? Please note, when you are updating to 1.9.1, the config of Celery changes to make

Re: Airflow 1.10

2018-01-14 Thread Driesprong, Fokko
I think 1.10 is a good idea. I'm working on this refactoring of the sensor structure: https://github.com/apache/incubator-airflow/pull/2875 Would be awesome to get this in. At my current project we use sensors in a few places, but still there is some work to be done. For example, don't allocate

Re: Fix on_kill command for operators

2018-01-08 Thread Driesprong, Fokko
Yes, for Spark this should work. Depending on the operator and the implementation: https://github.com/apache/incubator-airflow/blob/3e6babe8ed8f8f281b67aa3f4e03bf3cfc1bcbaa/airflow/contrib/hooks/spark_submit_hook.py#L412-L428 However this is a big change in behaviour. I'm curious about the

Re: Airflow 1.9.0 is released

2018-01-03 Thread Driesprong, Fokko
Awesome work! Cheers, Fokko Op wo 3 jan. 2018 om 18:50 schreef Chris Riccomini > Hey all, > > I have updated the docs as well: > > https://airflow.incubator.apache.org/ > > Cheers, > Chris > > On Wed, Jan 3, 2018 at 9:30 AM, Sid Anand wrote: > > >

Re: [VOTE] Airflow 1.9.0rc8

2017-12-15 Thread Driesprong, Fokko
+1 binding Op vr 15 dec. 2017 om 23:39 schreef Bolke de Bruin > +1, binding > > Checked sigs, version, source is there (did not check build), bin is there. > > Bolke > > Verstuurd vanaf mijn iPad > > > Op 15 dec. 2017 om 23:31 heeft Joy Gao het volgende >

Re: [VOTE] Airflow 1.9.0rc6

2017-12-12 Thread Driesprong, Fokko
+1 from my side Cheers, Fokko Op di 12 dec. 2017 om 17:28 schreef Ash Berlin-Taylor < ash_airflowl...@firemirror.com> > +0.5 from me. > > Our big test will come on Thursday morning, but looking good so far for > the small daily dags we've got are running okay, logs are showing up, and > making

  1   2   >