Re: Stalled and slow Travis CI builds

Hello Everyone, We just merged to master a possible fix provided by Travis CI engineers ( https://issues.apache.org/jira/browse/AIRFLOW-4511) . Can you please all rebase your changes onto latest master and trigger many builds so that we can see if it fixes the problem. J. On Fri, May 10, 2019

Re: Cron schedule with DST-aware timezone

The idea is obviously that with a Timedelta you want to say "add 24h". With DST this shifts the actual point in local time (in local time 17.00 can become 16.00/18.00). Cron schedules specify points in local time so 17.00h always stays 17.00h. There is nothing to make configurable here. Otherw

Re: Cron schedule with DST-aware timezone

The distinction could come from being non-UTC VS UTC. You can schedule a DAG at any time UTC and it would follow the 24-hour intervals but if it is non-UTC it would follow DST changes when relevant (by timezone). You could technically make this behavior configurable and follow the former path as

Re: Status of a Dagrun code

Here it is: https://github.com/PolideaInternal/airflow/blob/master/airflow/models/dagrun.py#L242 On Mon, May 13, 2019 at 10:50 PM Reuben Sawyer wrote: > Where is the code that decides how to set the state of the overall dagrun > to success? On 1.10.3 (localexecutor) I am seeing my dags intermitt

Re: Cron schedule with DST-aware timezone

Just to add to that: there are already tests that tests this behaviour: https://github.com/PolideaInternal/airflow/blob/master/tests/models/test_dag.py#L749 and indeed - the schedule follows DST changes rather than discards the DST time. I think it is generally a good idea to follow DST (for clari

Re: Cron schedule with DST-aware timezone

It would be great if people can provide failing unit tests as PR with clear expectations stated out as code. It makes it easier for people to get consensus on expectations and for anyone to jump in and implement a fix. Max On Mon, May 13, 2019 at 12:48 PM David Klosowski wrote: > Damian is corr

Status of a Dagrun code

Where is the code that decides how to set the state of the overall dagrun to success? On 1.10.3 (localexecutor) I am seeing my dags intermittently being set to Success even if some of its tasks are in Removed state or if all tasks are in None state.

Re: Cron schedule with DST-aware timezone

Damian is correct. We've observed that exact behavior and noticed the timedelta logic is dubiously broken for DST but works for CRON. On Mon, May 13, 2019 at 12:38 PM Shaw, Damian P. < damian.sha...@credit-suisse.com> wrote: > Hi, > > I’m not part of the Airflow team but I came to the same concl

RE: Cron schedule with DST-aware timezone

It is assumed that you want to have cron always at the exact *local time*. So in your case always 17.00h. I see that indeed the documentation is not correct here. It actually observes DST. Cheers Bolke On 13 May 2019 at 21:38:04, Shaw, Damian P. (damian.sha...@credit-suisse.com) wrote: Hi, I’m

RE: Cron schedule with DST-aware timezone

Hi, I’m not part of the Airflow team but I came to the same conclusion, that the behavior is the opposite of what the documentation specifies. When using the cron notation then DST is properly honored, when using timedelta it is not. I played around with the DAG.following_schedule method to sat

Cron schedule with DST-aware timezone

Hi team, I have a question about the expected behavior of Airflow scheduler when the schedule_interval is a cron expression and the start_date is in a timezone with DST. Based on the Airflow documentation https://airflow.apache.org/timezone.html#cron-schedules, the DST change will be ignored if s

Proposal: Create epics on the airflow ASF jira

Hi everyone, I've been working on cleaning up the backlog of tickets for airflow, and I'm running into an issue where it's pretty hard to filter tickets by topic. I've created this ticket to allow us to create epics for airflow tickets, but also

Tasks that run just once

I'm trying to design a set of DAGs to do a one create and backfill of a set of tables in BigQuery and then perform periodic loads into those tables. I can't quite get it to work the way I want to and I'm wondering if other people have solved similar problems. The parameters are as follows: I have

Re: Clarification request from Season of doc Mentor (SDOC2019)

Thank you for your response. I have gone through the links and i have started writing my proposal . Just wanted to know, 1) What is expected of a proposal . 2) On what factors will the proposal be judged. Thanks, Yajna. On Sat, May 11, 2019 at 4:16 AM Aizhamal Nurmamat kyzy wrote: > Hi Yanja, >

Re: [DISCUSS] Change all example_dags's start_date to specific datetime

I found out two example dag with .now in our code base https://github.com/apache/airflow/blob/1f192121b0c769cf07af052327f503b3a47c77a0/airflow/example_dags/example_trigger_target_dag.py#L47 https://github.com/apache/airflow/blob/1f192121b0c769cf07af052327f503b3a47c77a0/airflow/example_dags/example

Re: [DISCUSS] Change all example_dags's start_date to specific datetime

I know catchup_by=true by default, but I think when user run example dag, their just want to test how Airflow run, and thier would stop it manually. But if user use start_date=airflow.utils.dates.days_ago(2) as their production dag would cause unexpected behavior. IMO, if we use fixed date in e