Re: Airflow 1.8.0 BETA 5

Rc 1 isn't out yet, differences will be small. Sent from my iPhone > On 31 Jan 2017, at 22:00, Chris Riccomini wrote: > > I'm still on b5. Should I upgrade to rc1, or is it still in progress? > >> On Tue, Jan 31, 2017 at 12:26 PM, Alex Van Boxel

Re: Airflow 1.8.0 BETA 5

I'm still on b5. Should I upgrade to rc1, or is it still in progress? On Tue, Jan 31, 2017 at 12:26 PM, Alex Van Boxel wrote: > I identified my root cause: it's was a problem at my site with a faulty > rebase. So all green. > > On Tue, Jan 31, 2017 at 8:52 PM Alex Van Boxel

Re: Airflow 1.8.0 BETA 5

I identified my root cause: it's was a problem at my site with a faulty rebase. So all green. On Tue, Jan 31, 2017 at 8:52 PM Alex Van Boxel wrote: > So bumped to RC1 and this seems fine. I don't get it. > > So it's a go. Sorry for the noise, but better safe then sorry. I also

Re: Airflow 1.8.0 BETA 5

So bumped to RC1 and this seems fine. I don't get it. So it's a go. Sorry for the noise, but better safe then sorry. I also found that the scheduler logs setting are not equal from the normal logging so that's also a go. You get green light from me. On Tue, Jan 31, 2017 at 8:32 PM Bolke de

Re: Airflow 1.8.0 BETA 5

And the scheduler doesn't log anything in beta 4? Sent from my iPhone > On 31 Jan 2017, at 19:26, Alex Van Boxel wrote: > > I see it in the scheduler and in the UI. Currently trying to do a new > upgrade. > >> On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin

Re: Airflow 1.8.0 BETA 5

I see it in the scheduler and in the UI. Currently trying to do a new upgrade. On Tue, Jan 31, 2017 at 8:00 PM Bolke de Bruin wrote: > Please note I will be holding off on the RC, before we understand this > issue better. > > Bolke > > > On 31 Jan 2017, at 18:06, Bolke de

Re: Airflow 1.8.0 BETA 5

Please note I will be holding off on the RC, before we understand this issue better. Bolke > On 31 Jan 2017, at 18:06, Bolke de Bruin wrote: > > Hey Alex, > > Could it actually be that Airflow is doing the right thing? Earlier it was > swallowing the errors. Where do you

Re: Airflow 1.8.0 BETA 5

Hey Alex, Could it actually be that Airflow is doing the right thing? Earlier it was swallowing the errors. Where do you see the errors? In the scheduler logs, UI, processor logs? - Bolke > On 31 Jan 2017, at 16:25, Alex Van Boxel wrote: > > I'll try to identify the core

Re: Airflow 1.8.0 BETA 5

Data point: b5 is working fine for me so far. On Tue, Jan 31, 2017 at 8:25 AM, Alex Van Boxel wrote: > I'll try to identify the core problem > > On Tue, Jan 31, 2017, 16:43 Bolke de Bruin wrote: > > > Hey Alex > > > > Can you provide some info on the

Re: Airflow 1.8.0 BETA 5

I'll try to identify the core problem On Tue, Jan 31, 2017, 16:43 Bolke de Bruin wrote: > Hey Alex > > Can you provide some info on the scheduler paths thing. I don't have/see > that issue. Do you mean cli paths or by cfg? Jira would be nice in any case. > > I don't think the

Re: Airflow 1.8.0 BETA 5

Hey Alex Can you provide some info on the scheduler paths thing. I don't have/see that issue. Do you mean cli paths or by cfg? Jira would be nice in any case. I don't think the dag processor respects cli parameters. Bolke Sent from my iPhone > On 31 Jan 2017, at 15:10, Alex Van Boxel

Re: Airflow 1.8.0 BETA 5

@Alex I'm not able to reproduce locally (assuming the two python files are in the same folder or is on your PYTHONPATH). I don't see that import error anyways. Just in case, what is your complete DAG definition? Is anyone else able to repro? On Mon, Jan 30, 2017 at 3:09 PM, Alex Van Boxel

Re: Airflow 1.8.0 BETA 5

Well this means none of my DAG's work anymore: you just can do this anymore: file bqschema.py with def marketing_segment(): return [ {"name": "user_id", "type": "integer", "mode": "nullable"}, {"name": "bucket_date", "type": "timestamp", "mode": "nullable"}, {"name":

Re: Airflow 1.8.0 BETA 5

We're upgraded to b5 in dev/prod as of an hour ago. Will keep you posted. On Mon, Jan 30, 2017 at 2:45 PM, Dan Davydov wrote: > The latest commit fixed a regression since 1.7 that files with parsing > errors no longer showed up on the UI. > > On Mon, Jan 30,

Re: Airflow 1.8.0 BETA 5

The latest commit fixed a regression since 1.7 that files with parsing errors no longer showed up on the UI. On Mon, Jan 30, 2017 at 2:42 PM, Alex Van Boxel wrote: > Just installed beta 5 on our dev environment it lighted up as a christmas > tree. I got a a screen full of

Re: Airflow 1.8.0 BETA 5

Just installed beta 5 on our dev environment it lighted up as a christmas tree. I got a a screen full of import errors. I see that the latest commit did something with import errors... is it coorect?! On Sun, Jan 29, 2017 at 4:37 PM Bolke de Bruin wrote: > Hey Boris > > The

Re: Airflow 1.8.0 BETA 5

Hey Boris The scheduler is a bit more aggressive and can use multiple processors, so higher CPU usage is actually a good thing. I case it is really out of hand look at the new scheduler options and heartbeat options (see PR for updating.md not in the beta yet). Bolke Sent from my iPhone >

Re: Airflow 1.8.0 BETA 5

I am not sure if it is my config or something, but looks like after the upgrade and start of scheduler, airflow would totally hose CPU. The reason is two new examples that start running right away - latest only and latest with trigger. Once I pause them, CPU goes back to idle. Is this because now