Re: [ANNOUNCE] Please welcome new Airflow committer Bas Harenslak

2019-04-23 Thread Chao-Han Tsai
Congrats Bas! On Tue, Apr 23, 2019 at 8:00 AM Deng Xiaodong wrote: > Congrats Bas! > > XD > > > On 23 Apr 2019, at 6:26 PM, Kaxil Naik wrote: > > > > Congrats Bas :) > > > > On Tue, Apr 23, 2019 at 10:25 AM Jarek Potiuk > > wrote: > > > >> Congrats Bas! > >> > >> J. > >> > >> On Tue, Apr 23, 2

Re: [External] Re: scheduler logging

2019-04-23 Thread Frank Maritato
ok, yes, looks like 1.10.3 resolves this. Thanks for the help! On Tue, Apr 23, 2019 at 2:32 PM Kaxil Naik wrote: > Yes, please update us if 1.10.3 resolves that for you or if you face > same problem. > > On Tue, Apr 23, 2019 at 10:30 PM Frank Maritato > wrote: > > > Ok, I'll update to 1.10.3 a

Re: [External] Re: scheduler logging

2019-04-23 Thread Kaxil Naik
Yes, please update us if 1.10.3 resolves that for you or if you face same problem. On Tue, Apr 23, 2019 at 10:30 PM Frank Maritato wrote: > Ok, I'll update to 1.10.3 and see if I see the same behavior. > > On Tue, Apr 23, 2019 at 2:28 PM Kaxil Naik wrote: > > > It works for me i.e in *1.10.3 *i

Re: [External] Re: scheduler logging

2019-04-23 Thread Frank Maritato
Ok, I'll update to 1.10.3 and see if I see the same behavior. On Tue, Apr 23, 2019 at 2:28 PM Kaxil Naik wrote: > It works for me i.e in *1.10.3 *if I changed the "logging_level = > WARN" in *airflow.cfg > *file. I no longer get *info* logs in logs/schedule or in the UI. > > > > On Tue, Apr 23,

Re: [External] Re: scheduler logging

2019-04-23 Thread Kaxil Naik
It works for me i.e in *1.10.3 *if I changed the "logging_level = WARN" in *airflow.cfg *file. I no longer get *info* logs in logs/schedule or in the UI. On Tue, Apr 23, 2019 at 10:08 PM Kaxil Naik wrote: > On having a closer look, only cli.py was using the LOGGING_LEVEL from > settings.py >

Re: [External] Re: scheduler logging

2019-04-23 Thread Frank Maritato
Yes, I've tried this. I created config/log_config.py with the 1.10.0 contents of that file and with airflow.cfg:core.logging_level = ERROR and I still get the INFO log messages when I run: airflow scheduler -D On Tue, Apr 23, 2019 at 2:09 PM Kaxil Naik wrote: > On having a closer look, only cl

How to achieve Airflow worker fault tolerance?

2019-04-23 Thread Kevin Lam
Hi all, We currently run Airflow as a Deployment in a kubernetes cluster. We run a single airflow worker with the CeleryExecutor and Redis for Messaging (similar to what is outlined in https://github.com/puckel/docker-airflow). We are trying to use airflow in a maximally fault-tolerant fashion. S

Re: [External] Re: scheduler logging

2019-04-23 Thread Kaxil Naik
On having a closer look, only cli.py was using the LOGGING_LEVEL from settings.py The logger is configured at https://github.com/apache/airflow/blob/master/airflow/config_templates/airflow_local_settings.py Check the following note in UPDATING.md: https://github.com/apache/airflow/blob/85899b3ae

Re: scheduler logging

2019-04-23 Thread Daniel Standish
Re: > Can you confirm if that log message is actually affecting behaviour, or > does it just look bad? > -ash I have not noticed any issues. My guess is it just looks bad. I am not sure which process it's coming from. On Tue, Apr 23, 2019 at 12:13 PM Ash Berlin-Taylor wrote: > Can you conf

Re: [External] Re: scheduler logging

2019-04-23 Thread Frank Maritato
Thank you!! On Tue, Apr 23, 2019 at 1:54 PM Kaxil Naik wrote: > @Frank - It is still the case. I will fix this in master and should be > fixed in 1.10.4 > > On Tue, Apr 23, 2019 at 9:38 PM Frank Maritato > wrote: > > > sorry, it's line 93, not 35. > > > > On Tue, Apr 23, 2019 at 1:34 PM Frank M

Re: [External] Re: scheduler logging

2019-04-23 Thread Kaxil Naik
@Frank - It is still the case. I will fix this in master and should be fixed in 1.10.4 On Tue, Apr 23, 2019 at 9:38 PM Frank Maritato wrote: > sorry, it's line 93, not 35. > > On Tue, Apr 23, 2019 at 1:34 PM Frank Maritato > wrote: > > > Ok, I figured out why this is happening. settings.py:35 i

Re: [External] Re: scheduler logging

2019-04-23 Thread Frank Maritato
sorry, it's line 93, not 35. On Tue, Apr 23, 2019 at 1:34 PM Frank Maritato wrote: > Ok, I figured out why this is happening. settings.py:35 is hard coding the > logging level to INFO. I hacked it locally to: > > LOGGING_LEVEL = conf.get('core', 'logging_level') > > and now all the INFO logs are

Re: [External] Re: scheduler logging

2019-04-23 Thread Frank Maritato
Ok, I figured out why this is happening. settings.py:35 is hard coding the logging level to INFO. I hacked it locally to: LOGGING_LEVEL = conf.get('core', 'logging_level') and now all the INFO logs are gone. Is this fixed in a later version of airflow? On Tue, Apr 23, 2019 at 1:09 PM Frank Mari

Re: [External] Re: scheduler logging

2019-04-23 Thread Frank Maritato
Those two issues don't seem to be present in 1.10.0. I only see one message for "db connection invalidated" and I don't see anything about harvesting. I have tried using a custom logger to see if that effects the output to airflow-scheduler.{log|out} but so far I haven't had any luck. The logs tha

Re: [PLEASE PARTICIPATE] Brainstorming ideas for Season of Docs

2019-04-23 Thread Jarek Potiuk
Great! On Tue, Apr 23, 2019 at 7:02 PM Aizhamal Nurmamat kyzy wrote: > We submitted the application. Fingers crossed! > > Accepted mentoring organizations will be announced on April 30, 2019 at > 12:00 UTC. I will update you all in this thread with the results. > > Thanks everybody, > Aizhamal >

Re: scheduler logging

2019-04-23 Thread Ash Berlin-Taylor
Can you confirm if that log message is actually affecting behaviour, or does it just look bad? -ash On 23 April 2019 19:38:57 BST, Daniel Standish wrote: >I noticed two issues in 1.10.2, one of which has been resolved, but I >am >not sure if they were present in 1.10.0, or if they are the messa

Re: scheduler logging

2019-04-23 Thread Daniel Standish
I noticed two issues in 1.10.2, one of which has been resolved, but I am not sure if they were present in 1.10.0, or if they are the messages that are bothering you. 1. "Harvesting DAG parsing results" was printed every 2 seconds or something. This was resolved in commit [AIRFLOW-3911] Change Harv

Re: [ANNOUNCE] Please welcome new Airflow committer Bas Harenslak

2019-04-23 Thread Aizhamal Nurmamat kyzy
Congratulations Bas! Thank you for all your contributions:) On Tue, Apr 23, 2019 at 02:48 Szymon Przedwojski < szymon.przedwoj...@polidea.com> wrote: > Congratulations Bas! > > Szymon Przedwojski > Polidea | Software Engineer > > M: +48 500 330 790 > E: szymon.przedwoj...@polidea.com > > > On 23

Re: [PLEASE PARTICIPATE] Brainstorming ideas for Season of Docs

2019-04-23 Thread Aizhamal Nurmamat kyzy
We submitted the application. Fingers crossed! Accepted mentoring organizations will be announced on April 30, 2019 at 12:00 UTC. I will update you all in this thread with the results. Thanks everybody, Aizhamal On Mon, Apr 22, 2019 at 4:36 PM Aizhamal Nurmamat kyzy wrote: > Hi all, > > +Ash B

Re: scheduler logging

2019-04-23 Thread Bolke de Bruin
It's probably better to create a custom logging.conf and use that instead. B. Op di 23 apr. 2019 18:13 schreef Frank Maritato : > No one else has this issue? Or no one has a solution? > > > On Wed, Apr 17, 2019 at 5:49 PM Frank Maritato > wrote: > > > Hi All, > > > > We are running airflow 1.10

Re: scheduler logging

2019-04-23 Thread Frank Maritato
No one else has this issue? Or no one has a solution? On Wed, Apr 17, 2019 at 5:49 PM Frank Maritato wrote: > Hi All, > > We are running airflow 1.10.0 and I'm wondering how I can turn off or turn > down the logging for the scheduler? I tried setting logging_level=WARN in > airflow.cfg and rest

Re: [ANNOUNCE] Please welcome new Airflow committer Bas Harenslak

2019-04-23 Thread Deng Xiaodong
Congrats Bas! XD > On 23 Apr 2019, at 6:26 PM, Kaxil Naik wrote: > > Congrats Bas :) > > On Tue, Apr 23, 2019 at 10:25 AM Jarek Potiuk > wrote: > >> Congrats Bas! >> >> J. >> >> On Tue, Apr 23, 2019 at 11:03 AM Driesprong, Fokko >> wrote: >> >>> All, >>> >>> I'm pleased to announce that

Re: Proposal: Automatically mark stale PRs in github

2019-04-23 Thread Jiajie Zhong
I think we should change stale-bot strategy to auto close PR, If 30 days is too short for contributions, is 60 or 90 days make sence? In addition, I notice that we have some PR pass CI but none review it or let a suggest on it. So could we add a bot auto remind committer if PR pass CI but no on

Re: [ANNOUNCE] Please welcome new Airflow committer Bas Harenslak

2019-04-23 Thread Kaxil Naik
Congrats Bas :) On Tue, Apr 23, 2019 at 10:25 AM Jarek Potiuk wrote: > Congrats Bas! > > J. > > On Tue, Apr 23, 2019 at 11:03 AM Driesprong, Fokko > wrote: > > > All, > > > > I'm pleased to announce that the Airflow PMC has voted in Bas Harenslak > to > > be a committer. Bas's been actively inv

Re: [ANNOUNCE] Please welcome new Airflow committer Bas Harenslak

2019-04-23 Thread Szymon Przedwojski
Congratulations Bas! Szymon Przedwojski Polidea | Software Engineer M: +48 500 330 790 E: szymon.przedwoj...@polidea.com > On 23 Apr 2019, at 11:30, Jiajie Zhong wrote: > > Congratulations, Bas ! > > Best wish. > -- Jiajie > > From: Jarek Potiuk > Sent: Tues

Re: [ANNOUNCE] Please welcome new Airflow committer Bas Harenslak

2019-04-23 Thread Jiajie Zhong
Congratulations, Bas ! Best wish. -- Jiajie From: Jarek Potiuk Sent: Tuesday, April 23, 2019 17:25 To: dev@airflow.apache.org Subject: Re: [ANNOUNCE] Please welcome new Airflow committer Bas Harenslak Congrats Bas! J. On Tue, Apr 23, 2019 at 11:03 AM Driesprong

Re: [ANNOUNCE] Please welcome new Airflow committer Bas Harenslak

2019-04-23 Thread Jarek Potiuk
Congrats Bas! J. On Tue, Apr 23, 2019 at 11:03 AM Driesprong, Fokko wrote: > All, > > I'm pleased to announce that the Airflow PMC has voted in Bas Harenslak to > be a committer. Bas's been actively involved in the project for some time > now, contributing patches and participating in the maili

[ANNOUNCE] Please welcome new Airflow committer Bas Harenslak

2019-04-23 Thread Driesprong, Fokko
All, I'm pleased to announce that the Airflow PMC has voted in Bas Harenslak to be a committer. Bas's been actively involved in the project for some time now, contributing patches and participating in the mailing list, the Slack channels, and the AIP process. Please join in me congratulating Bas f

Re: Proposal: Automatically mark stale PRs in github

2019-04-23 Thread airflowuser
Since there are many many open PRs in the repo it can be hard for committers to keep track (I think that you are keeping tack by the mailing list which sometimes can easily be missed). It may be easier to tack using the filter of recently updated (see image) I hoped that some day this will be

Re: Proposal: Automatically mark stale PRs in github

2019-04-23 Thread Ash Berlin-Taylor
As a user/reporter on other opensource projects I would personally see auto-close after 30 days to be far too aggressive to the point of being unfriendly to contributions. Unless we get markedly better at merging PRs I wouldn't want to see us mark as stale so quickly. -ash > On 22 Apr 2019, a