Should next_ds be set to execution_date for manually triggered runs?

next_ds is useful when you need cron-style scheduling, e.g. a task that runs for date "X" uses that date for its logic, e.g. send an email to users saying the run that was supposed to run for date "X" has completed. The problem is it doesn't behave as expected when it comes to manually triggered ru

Re: Will Airflow 2.0.0 support Python 2.7?

I think we should probably drop Python2.7 support for 2.0 since it will be quite a large undertaking and I expect it to take a long time to complete since we should batch as many as the non-backwards incompatible changes we want to make as possible. Even if we drop support a little bit earlier than

Re: [DISCUSS] Apache Airflow graduation from the incubator

+1 thank you for everyone who helped drive this (especially Bolke!). On Tue, Nov 27, 2018 at 1:02 AM Tao Feng wrote: > And happy to help out any items for the TLP push. > > On Mon, Nov 26, 2018 at 9:53 PM Tao Feng wrote: > > > +1 as well. (do we need to start another vote thread per Jakob's > >

Re: Remove airflow from pypi

This could potentially break builds for some users but I feel the pros mentioned outweigh this, I went ahead and deleted it. On Fri, Nov 23, 2018 at 10:18 AM Bolke de Bruin wrote: > Agree! This is even a security issue. > > Sent from my iPhone > > > On 23 Nov 2018, at 15:29, Driesprong, Fokko >

Re: Pinning dependencies for Apache Airflow

Relevant discussion about this: https://github.com/apache/incubator-airflow/pull/1809#issuecomment-257502174 On Thu, Oct 4, 2018 at 11:25 AM Jarek Potiuk wrote: > TL;DR; A change is coming in the way how dependencies/requirements are > specified for Apache Airflow - they will be fixed rather tha

Re: Why not mark inactive DAGs in the main scheduler loop?

Blog <https://blog.tedmiston.com/> | CV > > <https://stackoverflow.com/cv/taylor> | LinkedIn > > <https://www.linkedin.com/in/tedmiston/> | AngelList > > <https://angel.co/taylor> | Stack Overflow > > <https://stackoverflow.com/users/149428/taylor

Why not mark inactive DAGs in the main scheduler loop?

I see some PRs creating endpoints to delete DAGs and other things related to manually deleting DAGs from the DB, but is there a good reason why we can't just move the deactivating DAG logic into the main scheduler loop? The scheduler already has some code like this, but it only runs when the Sched

Re: Broken DAG message won't go away in webserver

The scheduler should clear import errors for DAGfiles that no longer exist or DAGs that have fixed the import errors. Take a look at https://github.com/apache/incubator-airflow/blob/master/airflow/jobs.py#L761 and where it is called. If it's not working there might be a bug. On Fri, Aug 10, 2018

Re: Kerberos and Airflow

something up. We can then shoot holes in it (i think > you have a point on the crypto) and maybe do some hacking on it. This could > be part of the hackathon in sept in SF, I’m sure some other people would > have an interest in it as well. > > B. > > Verstuurd vanaf mijn

Re: The need for LocalTaskJob

Alex (cc'd) brought this up to me about this a while ago too, and I agreed with him. It is definitely something we should do, I remember there were some things that were a bit tricky about removing the intermediate process and would be a bit of work to fix (something about the tasks needing to hear

Re: Kerberos and Airflow

(which could even in the DAG or > part of the DAG definition) we could specify which fields would be used > for hash calculation. We could even specify > static hashes. This would give flexibility as to what freedom the users > have in the auto-generated DAGS. > > Something like t

Re: Apache Airflow welcome new committer/PMC member : Feng Tao (a.k.a. feng-tao)

Welcome Feng, awesome work :)! On Fri, Aug 3, 2018 at 11:20 AM Taylor Edmiston wrote: > Congratulations, Feng! > > *Taylor Edmiston* > Blog | CV > | LinkedIn > | AngelList >

Re: Kerberos and Airflow

per > generated > DAG / task and verify those with an established version (1st run?). This > limits the > attack vector. > > A DagSerializer would be great, but I think it solves a different issue > and the above > is somewhat simpler to implement? > > Bolke > > >

Re: Kerberos and Airflow

dags in > Airflow. We could enforce authentication on the DAG. It was kind of ruled > out in favor of git time machines although these never happened afaik ;-). > > BTW: I have updated my implementation with protobuf. Metadata is now > available at executor and task. > > > > On 29 J

Re: Kerberos and Airflow

n-mode>). > > Why not work on this together? We need it as well. Airflow as it is now we > consider the biggest security threat and it is really hard to secure it. > The above would definitely be a serious improvement. Another step would be > to stop Tasks from accessing the

Re: Kerberos and Airflow

upplied credentials and > populate > > a hash with “conn_ids”. Hooks normally call BaseHook.get_connection > anyway, so > > it shouldnt be too hard and should in principle not require changes to > the hooks > > themselves if they are well behaved. > > > > B. &g

Re: Kerberos and Airflow

e hard to make secure (due to the possibility a process will list > all files in /tmp and get credentials through that). Maybe storing the > keytab with a password and having the password in the KEYRING might work. > Something to find out. > > B. > > Verstuurd vanaf mijn iPad &g

Re: Kerberos and Airflow

I'm curious if you had any ideas in terms of ideas to enable multi-tenancy with respect to Kerberos in Airflow. On Fri, Jul 27, 2018 at 2:38 PM Bolke de Bruin wrote: > Cool. The doc will need some refinement as it isn't entirely accurate. In > addition we need to separate between Airflow as a cl

Re: Give it up for Fokko!

Agreed, thanks for being super active reviewing PRs and JIRAs! On Sat, Apr 14, 2018 at 12:48 AM Bolke de Bruin wrote: > Hear hear Fokko. So I assume you will be joining us in SF around October > ;-)?? > > > On 14 Apr 2018, at 09:16, Driesprong, Fokko > wrote: > > > > Thanks you all for the kind

Re: Airflow Scalability with Local Executor

The LocalExecutor is great for running small numbers of DAGs/tasks, but it is more of a starter executor meant to made Airflow work out of the box. I would recommend switching to a different executor like the CeleryExecutor. You are certainly right that there is room for reducing the memory footpr

Re: 4/17 Airflow Meetup Slides

Thanks Sid! On Wed, Dec 27, 2017 at 3:13 PM Sid Anand wrote: > Thx for posting the slides : I've added them to the Announcements page : > > https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-Nov1,2017 > > On Thu, Dec 7, 2017 at 12:09 PM, D

Re: Switching minimum support version from python 3.4 -> 3.5

Sounds good to me On Tue, Dec 19, 2017 at 10:18 AM Chris Riccomini wrote: > :thumbsup: > > On Tue, Dec 19, 2017 at 5:19 AM, Bolke de Bruin wrote: > > > Hi All, > > > > We have some issues on Travis with issues around distributed and task, > > these are related to Python 3.4. As Python 3.4 is no

4/17 Airflow Meetup Slides

Here are the presentations from the 4/17 Airflow meetup at Airbnb. Unfortunately we weren't able to get the recording software working for the talk. Thank you all for coming!

Re: Experimental API

FWIW I am hoping we can change this insecure-by-default for 2.0, and there is already some stuff in the Airflow config that lets you do this out of the box if you tweak a couple of config values (e.g. check out secure_mode that we can hopefully build upon). On Mon, Oct 30, 2017 at 3:22 PM Bolke de

Airflow Meetup @ Airbnb - Mon Dec 4th

Hey guys, we are doing an Airflow meetup at Airbnb on December 4th! All are welcome, and food will be provided. Please RSVP and see the details/agenda at https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/events/244525050/ Sincerely, Dan

Re: Meetup Interest?

New Relic > > > > On Fri, Oct 13, 2017 at 1:03 PM, Christopher Bockman < > > > > ch...@fathomhealth.co> wrote: > > > > > > > > +1 as a vote. > > > > > > > > We're very actively working on Kube+Airflow, so would be particu

Meetup Interest?

Is there interest in doing an Airflow meet-up? Airbnb can host one in San Francisco. Some talk ideas can include the progress on Kubernetes integration and Scaling & Operations with Airflow. If you want to see other topics covered, feel free to suggest them!

Re: Apache Airflow welcome new committer/PMC member : Fokko Driespong (a.k.a. fokko)

Welcome! On Wed, Oct 4, 2017 at 2:31 PM Maxime Beauchemin wrote: > Welcome on board Fokko! > > Max > > On Wed, Oct 4, 2017 at 2:18 PM, Chris Riccomini > wrote: > > > Welcome!! > > > > On Wed, Oct 4, 2017 at 12:51 PM, Sid Anand wrote: > > > > > Folks, > > > Please join the Apache Airflow PMC in

Re: Some random fun

Haha, this is great. On Mon, Sep 25, 2017 at 11:37 AM Shah Altaf wrote: > **CupcakeSensor activated** > > > > On Mon, Sep 25, 2017 at 7:31 PM Laura Lorenz > wrote: > > > Just thought everyone here would appreciate the nerdy party our data team > > threw ourselves for completing a milestone on a

Re: Airflow 1.8.2 released

+1, this was a lot more work than anticipated. On Tue, Sep 5, 2017 at 10:10 AM Chris Riccomini wrote: > Thanks so much for slogging through this! > > On Mon, Sep 4, 2017 at 10:26 AM, Sumit Maheshwari > wrote: > > > Awesome!! > > > > Thanks a lot Max for being the RM for this release. > > > > >

Re: Airflow + Kubernetes update meeting

Works for me as well! On Tue, Sep 5, 2017 at 10:43 AM Daniel Imberman wrote: > @Marc we will make sure to record the meeting/supply notes. This should be > a pretty straightforward update/overview meeint. > @ChrisB this meeting will be a virtual meeting, though Bloomberg is > definitely interest

Re: Airflow + Kubernetes Talk video

Thanks for organizing, and leading this effort in general! On Fri, Jul 28, 2017 at 2:40 PM, Daniel Imberman wrote: > Hi guys! > > Thank you again to everyone who attended the talk yesterday. I've posted > the video of the conversation to youtube, and will soon add the video and > slides to the a

Re: Role Based Access Control for Airflow UI

that are not CRUD). Implementation wise it > > > probably > > > > > >>>>>> simplifies what we need to do. Maybe - using Max’s early POC > > as > > > an > > > > > >>>>> example > > > > > >

Re: Airflow + Kubernetes discussion

I'm in. On Thu, Jul 20, 2017 at 4:30 PM, Daniel Imberman wrote: > Glad to hear that people are interested! I've created a google calendar > event and messaged everyone in this thread, if anyone else would like to > join please let me know! > > On Thu, Jul 20, 2017 at 1:04 PM Bolke de Bruin wrot

Re: Good practices around restarting scheduler and server of airflow

Airbnb switched to running the scheduler without restarts about a month ago, and it is working well. The one issue you might hit is if you are using a version of Celery with a bug (where workers will pull off tasks even if they don't have space to run them), tasks can potentially be lost once they

Re: What argument does -A / --ignore_all_dependencies expect when triggering via airflow run?

Probably caused by the issue being fixed here: https://github.com/apache/i ncubator-airflow/pull/2327 On Wed, Jul 12, 2017 at 4:25 AM, Tobias Feldhaus < tobias.feldh...@localsearch.ch> wrote: > I am trying to force airflow to run a task so that the depends_on_past > setting of the DAG is honoured

Re: airflow backfill seems to ignore -I

Airflow dependencies were simplified a bit, -i no longer ignores failed state tasks, check out the -A flag which ignores pretty much all dependencies (including the failed state tasks), though depending on the version you are using there is a bug that is being fixed here: https://github.com/apache/

Re: Airflow Logging Improvements

cmanaha/ >> python-elasticsearch-logger <https://github.com/cmanaha/ >> python-elasticsearch-logger> for example. >> >> I then can be convinced to add something like “attempt”, but probably >> there are more friendly ways to solve it at that time. In addition >>

Re: Airflow Logging Improvements

Responding to some of Bolke's concerns in the github PR for this change: > Mmm still not convinced. Especially on elastic search it is just easier to use the start_date to shard on. sharding on start_date isn't great because there is still some risk of collisions and it means that we are coupling

Re: Role Based Access Control for Airflow UI

Looks good to me in general, thanks for putting this together! I think the ability to integrate with external RBAC systems like LDAP is important (i.e. the Airflow DB should not be decoupled with the RBAC database wherever possible). I wouldn't be too worried about the permissions about refreshin

Re: Discussion on Airflow 1.8.1 RC2

8.0 to 1.8.1 regression then I would also like to >> know, cause I made that change and feel responsible for it. >> >> Cheers >> Bolke >> >> >> On 3 May 2017, at 22:13, Dan Davydov wrote: >> >> cc Alex and Rui who were working on fixes, I'm

Re: Discussion on Airflow 1.8.1 RC2

ly). > > Cheers > Bolke > > > On 3 May 2017, at 20:28, Dan Davydov wrote: > > We saw several regressions moving from 1.8.0 to 1.8.1 the first time we > tried, and while I think we merged all our fixes to master (not sure if > they all made it into 1.8.1 however), we

Re: dag file processing times

pendencies in a zip you should be fine with the > above. > > Or if we start using manifests that list the files that are needed in a > > dag... > > > > > > Sent from my iPhone > > > > > On 24 Apr 2017, at 22:46, Dan Davydov > > wrote: &

Re: dag file processing times

One idea to solve this is to use a daemon that uses inotify to watch for changes in files and then reprocesses just those files. The hard part is without any kind of dependency/build system for DAGs it can be hard to tell which DAGs depend on which files. On Mon, Apr 24, 2017 at 1:21 PM, Gerard To

Welcome @saguziel as a committer and PMC member!

Alex (@saguziel - AirBnB) has been making contributions and reviews for quite a long time now and I'm very happy to say he has just become an official committer and PMC member. He has ~13 commits, most of which are to the core of Airflow, and has been active reviewing open source PRs, contributing

Re: 1.8.1 release

rowse/AIRFLOW-1004 > >> https://issues.apache.org/jira/browse/AIRFLOW-1003 > >> https://issues.apache.org/jira/browse/AIRFLOW-1001 > >> https://issues.apache.org/jira/browse/AIRFLOW-1015 > > > > > > It would be great to have at least some of them fixed i

Re: 1.8.1 release

Here is my list for targeted 1.8.1 fixes: https://issues.apache.org/jira/browse/AIRFLOW-982 https://issues.apache.org/jira/browse/AIRFLOW-983 https://issues.apache.org/jira/browse/AIRFLOW-1019 (and in general the slow startup time from this new logic of orphaned/reset task) https://issues.apache.or

Re: [RESULT][VOTE]Release Airflow 1.8.0 based on Airflow 1.8.0rc5

That's reasonable (treating it a bug instead of a change in behavior). Full speed ahead! On Thu, Mar 16, 2017 at 9:01 AM, Bolke de Bruin wrote: > Hello, > > Apache Airflow (incubating) 1.8.0 (RC5) has been accepted. > > 9 “+1” votes received: > > - Maxime Beauchemin (binding) > - Chris Riccomini

Re: Make Scheduler More Centralized

I'm not convinced that this would add *that* much more load, we could probably change this functionality now if we wanted to. Just my two cents. On Thu, Mar 16, 2017 at 4:06 PM, Rui Wang wrote: > Thanks all your comments! > > Then looks like we should focus on scalability of scheduler now rather

Re: Airflow Committers: Landscape checks doing more harm than good?

+1 as well though I have found it useful on larger PRs to help me catch some issues so it probably makes sense to make to add the travis linting at the same time we remove it in landscape. Not sure how much usability we lose by losing the landscape UI but I like that all of the errors would be in o

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

oor so we can > focus on these bug fixes for 1.8.1. > > Bolke > > Sent from my iPhone > > > On 15 Mar 2017, at 18:25, Dan Davydov > wrote: > > > > Another issue we are seeing is > > https://issues.apache.org/jira/browse/AIRFLOW-992 - tasks that have both &g

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

igger rule but again I feel this is hacky and really we should have the old ALL_SUCCESS (default) and a new ALL_SUCCESS_OR_SKIPPED trigger rule if desired. On Wed, Mar 15, 2017 at 6:25 PM, Dan Davydov wrote: > Another issue we are seeing is https://issues.apache.org/ > jira/browse/AIRFLOW-

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

; > >>>> DAG > > >>>>>> Run > > >>>>>>>> goes back to Running, DOES NOT requeue or execute the task at > all. > > >>>> The > > >>>>>> DAG > > >>>>>>>> Run the returns from Running to Success though it nev

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc5

I'll test this on staging as soon as I get a chance (the testing is non-blocking on the rc5). Bolke very much in particular :). On Mon, Mar 13, 2017 at 10:46 AM, Jeremiah Lowin wrote: > +1 (binding) extremely impressed by the work and diligence all contributors > have put in to getting these blo

Re: High load in CPU of MySQL when running airflow

term for us, I will be just running the SQL command directly > from MySQL CLI and then setup an airflow job to do that periodically. > > Thanks. > -Jason > > On Tue, Mar 7, 2017 at 5:47 PM, Dan Davydov invalid> > wrote: > > > FWIW we use the following D

Re: High load in CPU of MySQL when running airflow

FWIW we use the following DAG at Airbnb to reap the task instances table (this is a stopgap): # DAG to delete old TIs so that UI operations on the webserver are fast. This DAG is a # stopgap, ideally we would make the UI not query all task instances and add indexes to # the task_instance table whe

Re: Proposal to simplify start/end dates

e. non cron > syntax? > > (Just thinking out loud) > > B. > > Sent from my iPhone > > > On 7 Mar 2017, at 22:27, Dan Davydov > wrote: > > > > Sure thing. > > > > Current Behavior: > > - User creates DAG with default_args start date to

Re: Proposal to simplify start/end dates

lls now from the calculation? Can you provide an example? > > Bolke > > > On 7 Mar 2017, at 20:38, Dan Davydov > wrote: > > > > A very common source of confusion for our users is when they specify > > start_date in default_args but not in their DAG arguments and th

Proposal to simplify start/end dates

A very common source of confusion for our users is when they specify start_date in default_args but not in their DAG arguments and then try to change this start_date to move the execution of their DAG forward (e.g. from 2015 to 2016). This doesn't work because the logic that is used to calculate th

Re: Airflow running different with different user id ?

Within a couple of weeks. On Fri, Mar 3, 2017 at 12:34 PM, Michael Gong wrote: > When approximately will it be released? > > Sent from my PP•KING™ smartphone > > On Mar 3, 2017 1:42 PM, Dan Davydov > wrote: > Yes it is starting on 1.8.0 which will be released soon,

Re: Airflow running different with different user id ?

Yes it is starting on 1.8.0 which will be released soon, you can look in the documentation/grep for "run_as". On Mar 3, 2017 8:50 AM, "Michael Gong" wrote: > Hi, > > > Suppose I have 1 airflow instance running 2 different DAGs, is it possible > to specify the 2 DAGs running under 2 different ids

Re: Getting to RC5: Update

anymore in > the dag (or so it should :)). > > Bolke > > > On 1 Mar 2017, at 19:55, Dan Davydov > wrote: > > > > We are seeing another major issue with backfills where task instances are > > being deleted and marked as "removed", I am still inv

Re: Getting to RC5: Update

We are seeing another major issue with backfills where task instances are being deleted and marked as "removed", I am still investigating. Let's keep discussion about these in https://issues.apache.org/jira/browse/AIRFLOW-921 and the subtask comments to have it one place. I will look at the other p

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

d the subdag, do not zoom in, but click on the subdag in the > main dag. Use mark success there. It will then allow you to mark all tasks > successful that are part of the subdag. > > Do we still consider this a blocker? Imho, no as a workaround seems to > exist. > > - Bol

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

lke de Bruin wrote: > > > > I'll have a look. I verified and the code is there to take of this. > > > > B. > > > > Sent from my iPhone > > > >> On 27 Feb 2017, at 22:34, Dan Davydov > wrote: > >> > >> Repro steps: >

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

; > Can you elaborate on 2, cause I thought I specifically took care of that. > > Cheers > Bolke > > Sent from my iPhone > > > On 27 Feb 2017, at 20:27, Dan Davydov > wrote: > > > > I created https://issues.apache.org/jira/browse/AIRFLOW-921 to track the &g

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

ng > the > >> impact it is a blocker or not. We don’t often use backfills and > definitely > >> not at your size, so that is why it didn’t pop up with us. I’m assuming > >> blocker for now, btw. > >>> - Speculation on the High DB Load. I’m not sure what your benchma

Re: Cutting down on testing time

This looks like a great effort to me at least in the short term (in the long term I think most of the integration tests should be run together if the infra allows this). Another thing we could start looking into is parallelizing tests (though this may require beefier machines from Travis). On Sat,

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

; > > wrote: > > > > > > > > Should clarify this occurs when a dagrun does not have a start date, > > not > > > a > > > > dag (which makes it even less likely to happen). I don't think this > is > > a > > > > block

Re: scheduler running on multiple nodes

nd of > failover or to distribute execution of jobs? > > Thanks! > On Fri, Feb 24, 2017 at 3:49 AM Dan Davydov invalid> > wrote: > > > Fwiw Airbnb was running multiple schedulers for a short while on 1.7.1 > and > > we didn't seem to have issues. > > > >

Re: scheduler running on multiple nodes

Fwiw Airbnb was running multiple schedulers for a short while on 1.7.1 and we didn't seem to have issues. On Feb 24, 2017 12:25 AM, "Bolke de Bruin" wrote: > While I agree with the assessment of Sid that a lot has changed and we do > not officially test on multiple schedulers, many changes were

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

7.1.* behaves one way, then 1.8.0 behaves > >> another, then 1.8.1 goes back to the old way again. I guess I'm just not > >> that worried about it. > >> > >> Anyway.. take it or leave it. :) > >> > >> Cheers, > >> Chris > >> > >>

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Here is the DAG: http://imgur.com/a/zXXsS On Thu, Feb 23, 2017 at 12:18 PM, Arthur Wiedmer wrote: > Dan, > > Inline images get stripped by the mailing server. You will have to upload > to imgur or something. > > Best > Arthur > > On Feb 23, 2017 12:13 PM, "Dan

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

mption that such a task should exist (ie. a task was added to a Dag in > a later stage). > > > > On 23 Feb 2017, at 20:15, Dan Davydov > wrote: > > > > Some more issues found by our users in addition to the one Alex reported > > and the UI issue when a dagrun doe

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

have some concern that this change > https://github.com/apache/incubator-airflow/pull/1939 > [AIRFLOW-679] may be having issues because we are seeing lots of double > triggers > of tasks and tasks being killed as a result. > > > > > > On Wed, Feb 22, 2017 4:35 PM, Dan Davy

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

you to go through one > release > > >> cycle? Oh, and do you have a documented step by step process for > > releasing? > > >> I'd like to add the Pypi part to this doc and add committers that are > > >> interested to have rights on the project on P

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

Should clarify this occurs when a dagrun does not have a start date, not a dag (which makes it even less likely to happen). I don't think this is a blocker for releasing. On Wed, Feb 22, 2017 at 1:15 PM, Dan Davydov wrote: > I rolled this out in our prod and the webservers failed to loa

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

17, at 08:20, Bolke de Bruin wrote: > > > > > > Hello, > > > > > > Apache Airflow (incubating) 1.8.0 (based on RC4) has been accepted. > > > > > > 9 “+1” votes received: > > > > > > - Maxime Beauchemin (binding) > > &g

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc4

+1 (binding). Mark success works great now, thanks to Bolke for fixing. On Fri, Feb 17, 2017 at 12:22 AM, Bolke de Bruin wrote: > Dear All, > > I have made the FOURTH RELEASE CANDIDATE of Airflow 1.8.0 available at: > https://dist.apache.org/repos/dist/dev/incubator/airflow/ < > https://dist.apa

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

gt; https://github.com/apache/incubator-airflow/pull/2075> > > > > Is (part of) the fix. I can include it retroactively if needed, but I > don’t consider it blocking. > > > > Bolke > > > > > >> On 13 Feb 2017, at 20:56, Dan Davydov <mailto:dan.

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

m> > > > > > > >>>> wrote: > > > > > > >>>> > > > > > > >>>>> I am running LocalExecutor and not doing crazy things but > use > > > DAG > > > > > > >>>>> generation

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

Our staging looks good, all the DAGs there pass. +1 (binding) On Fri, Feb 10, 2017 at 10:21 AM, Chris Riccomini wrote: > Running in all environments. Will vote after the weekend to make sure > things are working properly, but so far so good. > > On Fri, Feb 10, 2017 at 6:05 AM, Bolke de Bruin w

Re: Airflow 1.8.0 Release Candidate 1

Bolke, attached is the patch for the cgroups fix. Let me know which branches you would like me to merge it to. If anyone has complaints about the patch let me know (but it does not touch the core of airflow, only the new cgroups task runner). On Mon, Feb 6, 2017 at 4:24 PM, siddharth anand wrote:

Re: Flow-based Airflow?

Woops looks like I replied to the wrong thread! Thanks Bolke. On Mon, Feb 6, 2017 at 1:42 PM, Bolke de Bruin wrote: > Dataflow or 1.8? > > Sent from my iPhone > > > On 6 Feb 2017, at 22:35, Dan Davydov > wrote: > > > > We have been running in our staging and

Re: Airflow 1.8.0 Release Candidate 1

On the Airbnb side we should be good once https://github.com/apache/ incubator-airflow/pull/2057/ is merged. On Mon, Feb 6, 2017 at 9:23 AM, Chris Riccomini wrote: > Upgraded to RC1 in all environments this morning. So far so good. > > On Fri, Feb 3, 2017 at 6:04 PM, Jeremiah Lowin wrote: > > >

Re: Flow-based Airflow?

We have been running in our staging and have found a couple of issues. I will report back with them soon. On Thu, Feb 2, 2017 at 2:23 PM, Jeremiah Lowin wrote: > Very good point -- however I'm hesitant to overcomplicate the base class. > At the moment users only have to override "serialize()" an

Re: Airflow 1.8.0 Release Candidate 1

+1, this requires a lot more work than appears on the surface. On Thu, Feb 2, 2017 at 12:43 PM, Arthur Wiedmer wrote: > Bolke, > > Thank you again for leading this effort. This has been quite the journey. > > Best, > Arthur > > On Thu, Feb 2, 2017 at 11:50 AM, Bolke de Bruin wrote: > > > Hi All

Re: Airflow 1.8.0 BETA 5

> "/usr/local/lib/python2.7/site-packages/airflow-1.8.0b5+ > apache.incubating-py2.7.egg/airflow/models.py", > line 264, in process_file > m = imp.load_source(mod_name, filepath) > File "/home/airflow/dags/marketing_segmentation.py", line 17, in &

Re: Airflow 1.8.0 BETA 5

The latest commit fixed a regression since 1.7 that files with parsing errors no longer showed up on the UI. On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel wrote: > Just installed beta 5 on our dev environment it lighted up as a christmas > tree. I got a a screen full of import errors. I see th

Re: Experiences with 1.8.0

I'd be happy to lend a hand fixing these issues and hopefully some others are too. Do you mind creating jiras for these since you have the full context? I have created a JIRA for (1) and have assigned it to myself: https://issues.apache.org/jira/browse/AIRFLOW-780 On Fri, Jan 20, 2017 at 1:01 AM,

Re: Airflow 1.8.0 BETA 1

So it is, my bad. Bad skills with ctrl-f :). On Tue, Jan 17, 2017 at 3:31 PM, Bolke de Bruin wrote: > Arthur's change is already in! > > B. > > Sent from my iPhone > > > On 17 Jan 2017, at 22:20, Dan Davydov > wrote: > > > > Would be good to c

Re: Airflow 1.8.0 BETA 1

Would be good to cherrypick Arthur's fix into here if possible: https://github.com/apache/incubator-airflow/pull/1973/files (commit 43bf89d) The impersonation stuff should be wrapping up shortly pending Bolke's comments. Also agreed with Max on the thanks. Thanks Alex too for the change log! On

Re: Airflow 1.8.0 alpha 4

The task dependency engine code is well commented, but I can provide a high level overview specifically for developers if there is interest (note that this would be the first documentation of it's kind in that it would be developer-only documentation). The disadvantage is that it would create dupli

Re: Last minute open meetup speaking slot

Thank you Arthur for stepping up, and my sincere apologies. I just don't want to get anyone sick. I hope to present the talk at the next meetup. On Jan 10, 2017 11:22 PM, "George Leslie-Waksman" wrote: > Listing updated. Thanks for stepping in on short notice. > > On Tue, Jan 10, 2017 at 8:15 PM

Re: Subsequent Airflow Meetup: 2017/01/11

orge, > > > > Confirmed - would like give a talk. Thanks, > > > > Kevin Mandich > > > > On Tue, Jan 3, 2017 at 5:40 AM, Dan Davydov > .invalid> > > wrote: > > > > > Confirmed. > > > > > > On Sun, Jan 1, 2017 at 9:16 PM

Re: Airflow 1.8.0 Alpha 1

; > > Can I ask that you create a 1.8 alpha 1 branch in the git repo? This > will > > > make it easier for us to track what changes are getting cherry picked > > into > > > the branch, and will also make it easier for users to pip install, if > > they > >

Re: Airflow 1.8.0 Alpha 1

verify and let me know if something is > off. I’m still waiting for access to the incubating dist repository. > > Bolke > > > > On 3 Jan 2017, at 14:38, Dan Davydov > wrote: > > > > I have also started on this effort, recently Alex Guziel and I have been > &g

Re: Subsequent Airflow Meetup: 2017/01/11

you come up for air, it looks like both Dan(Airbnb) and Kevin(Agari) > > have talk ideas. > > > > -s > > > > On Wed, Nov 16, 2016 at 11:50 PM, Dan Davydov < > > dan.davy...@airbnb.com.invalid> wrote: > > > > > Based on chatting with a couple of pe

Re: Airflow 1.8.0 Alpha 1

I have also started on this effort, recently Alex Guziel and I have been pushing Airbnb's custom cherries onto master to get Airbnb back onto master in order for us to do a release. I think it might make sense to wait for these two commits to get merged in since they would be quite nice to have fo

Re: Integration test env

This is extremely generous of you! I do agree with the approach of trying to get funding from Apache and having shared resources (e.g. so that we don't depend on any one company or individual for the uptime of the integration environment, plus so we would have public cloud integration potentially).

  1   2   >