Max,
The schedule interval is 1 day.
Sent from my iPhone
> On 30-Oct-2018, at 9:29 PM, Maxime Beauchemin
> wrote:
>
> Also what's your schedule interval? I'm just trying to confirm that this
> isn't a "run every minute, or anytime someone blinks" kind of DAG.
>
> Max
>
> On Tue, Oct 30,
Hi,
Recently I've upgraded an Airflow installation from 1.8.0 to 1.10.0 (yes, I
know, it took some time to get there) and, while the upgrade was mostly
painless, there was one behavior change that I wasn't quite able to
understand why it happened and what the best approach is to get back to the
They are :-)
I might have to adjust some tests, let's see. Will do that tomorrow if required.
B.
> On 30 Oct 2018, at 21:53, Ash Berlin-Taylor wrote:
>
> Fair :) Timezones are _hard_
>
> Giving it a look now.
>
> -ash
>
>> On 30 Oct 2018, at 20:50, Bolke de Bruin wrote:
>>
>> The reason
Fair :) Timezones are _hard_
Giving it a look now.
-ash
> On 30 Oct 2018, at 20:50, Bolke de Bruin wrote:
>
> The reason for not passing a TZ aware object is, is that many libraries make
> mistakes (pytz, arrow etc) when doing transitions hence to use of pendulum
> which seem most complete.
The reason for not passing a TZ aware object is, is that many libraries make
mistakes (pytz, arrow etc) when doing transitions hence to use of pendulum
which seem most complete. I don’t know what croniter is relying on and I don’t
want to find out ;-).
B.
> On 30 Oct 2018, at 21:13, Ash
Patch available at:
https://github.com/apache/incubator-airflow/pull/4117
please test.
B.
> On 30 Oct 2018, at 21:14, Bolke de Bruin wrote:
>
> We should just pass it the UTC date (we should never use local time except at
> the user interface). I’m testing a patch right now.
>
> B.
>
>>
I think if we give croniter a tz-aware DT in the local tz it will deal with DST
(i.e. will give 2:55 CEST followed by 2:00 CET) and then we convert it to UTC
for return - but right now we are giving it a TZ-unaware local time.
I think.
Ash
On 30 October 2018 19:40:27 GMT, Bolke de Bruin
We should just pass it the UTC date (we should never use local time except at
the user interface). I’m testing a patch right now.
B.
> On 30 Oct 2018, at 21:13, Ash Berlin-Taylor wrote:
>
> I think if we give croniter a tz-aware DT in the local tz it will deal with
> DST (i.e. will give 2:55
I think we should use the UTC date for cron instead of the naive local date
time. I will check of croniter implements this so we can rely on that.
B.
> On 28 Oct 2018, at 02:09, Bolke de Bruin wrote:
>
> I wonder how to treat this:
>
> This is what I think happens (need to verify more, but I
I think DAGs will start running by themselves again as soon as the interval has
passed that it has already seen. So depending your schedule this can take a
while.
B.
> On 30 Oct 2018, at 19:53, a...@apache.org wrote:
>
> 1.10.1 isn't out yet, so 1.10.0 :)
>
> I think this only affects days
I wonder how to treat this:
This is what I think happens (need to verify more, but I am pretty sure) the
specified DAG should run every 5 minutes. At DST change (3AM -> 2AM) we
basically hit a schedule that we have already seen. 2AM -> 3AM has already
happened. Obviously the intention is to
Hi Airflow Devs:
Is this timezone issue in Airflow version 1.10.0 or only in 1.10.1?
Thanks.
Regards,
David
On Tue, Oct 30, 2018 at 11:11 AM Bolke de Bruin wrote:
> we specifically remove timezone info to determine the next schedule. Ie.
> cron sets exact date times so tz info should not
we specifically remove timezone info to determine the next schedule. Ie. cron
sets exact date times so tz info should not make sense. I’m going to have a
look now.
> On 30 Oct 2018, at 19:01, Ash Berlin-Taylor wrote:
>
> I've done a bit more digging - the issue is of our tz-aware handling
I've done a bit more digging - the issue is of our tz-aware handling inside
following_schedule (and previous schedule) - causing it to loop.
This section of the croniter docs seems relevant
https://github.com/kiorky/croniter#about-dst
Be sure to init your croniter instance with a TZ aware
Oh that’s a great environment to start digging. Thanks. I’ll have a look.
B.
Verstuurd vanaf mijn iPad
> Op 30 okt. 2018 om 18:25 heeft Ash Berlin-Taylor het
> volgende geschreven:
>
> This line in airflow.jobs (line 874 in my checkout) is causing the loop:
>
>last_run =
This line in airflow.jobs (line 874 in my checkout) is causing the loop:
last_run = dag.get_last_dagrun(session=session)
if last_run and next_run_date:
while next_run_date <= last_run.execution_date:
next_run_date =
Hi, kaczors on gitter has produced a minmal reproduction case:
https://github.com/kaczors/airflow_1_10_tz_bug
Rough repro steps: In a VM, with time syncing disabled, and configured with
system timezone of Europe/Zurich (or any other CEST one) run
- `date 10280250.00`
- initdb, start
Also what's your schedule interval? I'm just trying to confirm that this
isn't a "run every minute, or anytime someone blinks" kind of DAG.
Max
On Tue, Oct 30, 2018 at 5:48 AM Brian Greene <
br...@heisenbergwoodworking.com> wrote:
> How do you trigger it externally?
>
> We have several custom
How do you trigger it externally?
We have several custom operators that trigger other jobs and we had to be
really careful or we’d get duplicate keys for the dag run and it would fail to
kick off.
One scheduler, but we saw it repeatedly and have it noted as a thing to watch
out for.
Brian
19 matches
Mail list logo
