Hi Ryan,
I really like the idea of exposing some more scheduler details. More
transparency in scheduling also in the UI would help the user in (1) seeing and
understanding what is going on and (2) reduces the need to crawl for logs and
raise support tickets if status is “strange”. I often also
Over the last couple weeks I've come across a rather tricky problem a few
times. One DAG run gets "stuck" in the queued state, while subsequent DAG
runs will be stuck running (screenshot below). One of these issues was
caused by `max_active_runs` being met when a task instance from a
previously
> I guess I remain concerned the zoneinfo API isn't sufficient for many
places where the pendulum is
I think you try to compare a bit different things. ZoneInfo provides tzinfo
implementation (
https://docs.python.org/3/library/datetime.html#tzinfo-objects) and it is
not related to the datetime
> We do not use pickling by default as we roll our own serialization
format. We probably just need the key (zoneinfo.key).
Yeah, for sure we don't need to rely on pickling, if I don't miss anything
than in Airflow 3 pickling serialisation should be removed. I just use as
reference info which
I believe some discussion on this has already occurred:
https://github.com/apache/airflow/issues/19450 (in fact I commented some
concerns on it
https://github.com/apache/airflow/issues/19450#issuecomment-1536494966)
I guess I remain concerned the zoneinfo API isn't sufficient for many places
FYI:
I've just added:
https://github.com/apache/airflow/pull/34667
which documents how to use newer timezone information with Pendulum.
Also work seems to be progressing (albeit slowly) on Pendulum 3:
https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677
Bolke
On Thu,
for serialization I am not too worried about ZoneInfo. We do not use
pickling by default as we roll our own serialization format. We probably
just need the key (zoneinfo.key).
I'm not sure what happened about this:
https://github.com/sdispater/pendulum/issues/590
Bolke
On Thu, 28 Sept 2023 at
I agree with all problems that you mention about datetime tz-aware data.
I lived for almost 30 years in a country which had in different periods of
time up to 10 time zones, and on a regular basis changed it (merge/unmerge)
, disable DST, temporarily enable DST. In addition I also worked in a
Tested out my changes: Defining priorityClasses for airflow services and
workers (#31615) and Ability to inject extra containers into pgbouncer
(#33686).
Changes seem to work as expected.
+1 (non binding)
Thanks,
Amogh Desai
On Thu, Sep 28, 2023 at 5:10 AM Jed Cunningham
wrote:
> Hello Apache
Thanks for starting the discussion Andrey.
Some background on the choice for Pendulum at the time. In the early days
of Airflow it wasn't timezone aware. Originating from Airbnb which had a
reasonable mature data organization the view was everything needs to be in
UTC. According to Maxime the
This discussion is more about the known problem of pendulum and how we
could deal with it and maybe how we (as Community) might help autor.
The library is mostly supported by a single author Sébastien Eustace (
https://github.com/sdispater) and it seems like we bump into the situation
which is
11 matches
Mail list logo
