Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

Thanks! I am holding off the IPMC until we resolved the “mark success” issue. I feel (and others with me) too many are reliant on this feature. I am reworking the functionality, but it will take me at least another day unfortunately. - Bolke > On 13 Feb 2017, at 22:20, Maxime Beauchemin > wro

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

Infinitely grateful for your work! It's a nice demonstration on how one person can move mountains. Thanks Bolke! Max On Mon, Feb 13, 2017 at 1:06 PM, Chris Riccomini wrote: > Hey Bolke, > > Fantastic work. Thanks so much for your effort--it's a huge service to the > community. > > Thanks! > >

Re: [RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

Hey Bolke, Fantastic work. Thanks so much for your effort--it's a huge service to the community. Thanks! Cheers, Chris On Mon, Feb 13, 2017 at 11:57 AM, Bolke de Bruin wrote: > Hello Airflowers :-), > > Apache Airflow (incubating) 1.8.0 (RC3) has been accepted. > > 9 “+1” votes received: > >

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

You raise a good point on the “extra_task” thing. I havent tested that. I am discussing matters on gitter with Sid as well (although he is in a meeting now). I am holding off raising the vote on the IPMC (*sigh* ;) ). Bolke > On 13 Feb 2017, at 21:40, Dan Davydov wrote: > > I feel like there

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

I feel like there might be enough reliance on these features to merge these in, e.g. mark-successing a non-existent task to prevent it from running. I'm curious what others think. Also isn't mark success still needed for when you add a new task with depends_on_past to an existing dag or is that fix

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

A little bit more background on the issue. Mark success sits in views.py as “def success”. The code should mark a task “successful”, with optional upstream and downstream tasks as well. Even for tasks in the future (up until datetime.now() ) and past. It was often used to kick off the first of d

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

https://github.com/apache/incubator-airflow/pull/2075 Is (part of) the fix. I can include it retroactively if needed, but I don’t consider it blocking. Bolke > On 13 Feb 2017, at 20:56, Dan Davydov wrote: > > Can you give more details

[RESULT] [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

Hello Airflowers :-), Apache Airflow (incubating) 1.8.0 (RC3) has been accepted. 9 “+1” votes received: - Maxime Beauchemin (binding) - Chris Riccomini (binding) - Arthur Wiedmer (binding) - Dan Davydov (binding) - Jeremiah Lowin (binding) - Siddharth Anand (binding) - Alex van Boxel (binding) -

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

Hi Siddharth,I found the same issue on "Mark Success". I have opened two tickets [https://issues.apache.org/jira/browse/AIRFLOW-869] and [https://issues.apache.org/jira/browse/AIRFLOW-870], and I have some thoughts on the fixes in PRs (see comments in the ticket for the PR link). But I guess th

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

Can you give more details/a repro case Sid? FWIW mark success and clear both work for me. On Mon, Feb 13, 2017 at 11:51 AM, siddharth anand wrote: > Folks! > I need to change my vote.. -1 (Binding). > > > Mark Success/Clear is broken in the UI. It's a regression. > > -s > > On Mon, Feb 13, 2017

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

Folks! I need to change my vote.. -1 (Binding). Mark Success/Clear is broken in the UI. It's a regression. -s On Mon, Feb 13, 2017 at 10:53 AM, Alex Van Boxel wrote: > +1 (binding) > > On Mon, Feb 13, 2017 at 7:45 PM siddharth anand wrote: > > > +1 (binding) > > > > On Mon, Feb 13, 2017 at 8

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

+1 (binding) On Mon, Feb 13, 2017 at 7:45 PM siddharth anand wrote: > +1 (binding) > > On Mon, Feb 13, 2017 at 8:57 AM, Chris Riccomini > wrote: > > > +1 (binding) > > > > On Sun, Feb 12, 2017 at 8:54 AM, Jeremiah Lowin > wrote: > > > > > Interesting -- I also run on Kubernetes with a git-sync

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

+1 (binding) On Mon, Feb 13, 2017 at 8:57 AM, Chris Riccomini wrote: > +1 (binding) > > On Sun, Feb 12, 2017 at 8:54 AM, Jeremiah Lowin wrote: > > > Interesting -- I also run on Kubernetes with a git-sync sidecar, but the > > containers wait for the synced repo to apprar before starting since i

Re: Celery or Dask?

As far as I know I'm the only person using Dask with Airflow at the moment. I've been using Dask for a variety of other (non-Airflow) tasks and have found it to be a great tool. However, it's important to note that Celery is a much more mature project with finer control over how tasks are executed.

Re: [VOTE] Release Airflow 1.8.0 based on Airflow 1.8.0rc3

+1 (binding) On Sun, Feb 12, 2017 at 8:54 AM, Jeremiah Lowin wrote: > Interesting -- I also run on Kubernetes with a git-sync sidecar, but the > containers wait for the synced repo to apprar before starting since it > contains some dependencies -- I assume that's why I didn't experience the > sa

Re: Celery or Dask?

Dask just landed in master. So no Celery is the most used option to scale-out. Always interested in what you are running into, but please be prepared to provide a lot of info on your setup. - Boke > On 13 Feb 2017, at 17:01, EKC (Erik Cederstrand) wrote: > > Hello all, > > > I'm investigati

Celery or Dask?

Hello all, I'm investigating why some of our DAGs are not being scheduled properly ( ran into https://issues.apache.org/jira/browse/AIRFLOW-342, among other things). Coupled with comments on this list, I'm getting the impression that Celery is a second-class citizen and core developers are mai