Just to clarify I'd like us to consider the possibility that no new pendulum would be released or released at the end of 2024, like a pessimistic scenario: - What should we do in this case? - Work out a backup plan.
---- Best Wishes *Andrey Anshin* On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <[email protected]> wrote: > Also I think TP - had a document in the past (years ago) describing a > draft of a more complete alternative we can take to approach datetime vs. > pendulum dichotomy. I cannot easily find the document and discussion - but > I do remember it was proposing some interesting changes in the approach of > Airflow to have an abstraction layer on top of it (as far as I remember). > Maybe we can resurrect that idea if TP might find the proposal ? > > On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <[email protected]> wrote: > > > Hi, > > > > I agree that the current speed of development of Pendulum leaves > something > > to be desired. However, I think we should not underestimate the effort of > > replacing it. It is not just a matter of %s/pendulum/datetime/g so to > say. > > If we are *truly* thinking about moving to native datetime / zoneinfo etc > > we need *extensive* tests, basically copying what pendulum does to check > > its behavior. The reason is that native implementations in the past made > > serious mistakes and I do not put a lot of trust in them. > > > > An abstraction or vendoring in could be alternatives, but they bring > their > > own significant challenges. > > > > I have re-raised the issue here: > > > > https://github.com/sdispater/pendulum/issues/753 > > > > The upside is that it seems the amount of issues with the beta is > limited, > > so hopefully the maintainers can spend a couple of cycles to address > them. > > > > Bolke > > > > On Fri, 17 Nov 2023 at 10:28, Andrey Anshin <[email protected]> > > wrote: > > > > > There is no changes in stable pendulum so let's try to continue this > > > discussion and start think about "Plan B" > > > > > > Just a reminder: > > > - pendulum 2.1.2 released 3 years ago (at the time Airflow 1.10.x) > > > - pendulum 2 doesn't work well in Python 3.12, this is a showstopper > for > > > the support Python 3.12 > > > - pendulum 2 have memory leaks ( > > > https://github.com/sdispater/pendulum/issues/720), and Airflow use > > > approach > > > to achieve this leaks, especially in K8S executor > > > - pendulum 2 doesn't use system tzdata by default, but we have a > > > workaround, thanks Bolke for the documentation > > > - pendulum it is a core dependency of Airflow, and I guess we can't > > > remove/replace it without breaking changes. > > > > > > So my proposal if things won't change in the near future then we need > to > > > start removing the pendulum from the core and replace it with the > native > > > datetime / zoneinfo / tzinfo. But maybe we have another way to resolve > > this > > > without breaking changes? Because for me it would be a little weird if > > > removal pendulum would be a main driver for release Airflow 3 > > > > > > > > > > > > > > > ---- > > > Best Wishes > > > *Andrey Anshin* > > > > > > > > > > > > On Thu, 28 Sept 2023 at 13:01, Andrey Anshin <[email protected] > > > > > wrote: > > > > > > > This discussion is more about the known problem of pendulum and how > we > > > > could deal with it and maybe how we (as Community) might help autor. > > > > > > > > The library is mostly supported by a single author Sébastien Eustace > ( > > > > https://github.com/sdispater) and it seems like we bump into the > > > > situation which is described in xkcd #2347 ( > > > > https://imgs.xkcd.com/comics/dependency.png). To be honest it is not > > > > something new when library mainly supported by one author so there is > > > > always a risk that the library will no longer be supported / > abandoned > > > > And if takes in account that pendulum provides core functionality in > > > > Airflow it could have dramatical impact in the future. > > > > > > > > Pendulum is a really nice library which helps a lot of developers to > > work > > > > with dates/datetimes. However there is one major problem, the last > > > release > > > > of this library happened more than 3 years ago ( > > > > https://pypi.org/project/pendulum/#history) in the time when Airflow > > > > 1.10.11 was released > > > > > > > > Fortunately, the project is not abandoned and on a regular basis > > commits > > > > add into the master branch. However these commits are not included > into > > > any > > > > final release and that's why some things related to datetime don't > work > > > as > > > > expected in Airflow. There are list of known (for me) issues which > are > > > > affect Airflow > > > > > > > > *Memory Leak on parse*: > > > > - https://github.com/sdispater/pendulum/issues/720, this one fixed > 2 > > > > years ago but not available yet ( > > > > https://github.com/sdispater/pendulum/pull/563). Since we use parse > > > dates > > > > in airflow codebase: datetime parameters and datetime in logs this > one > > > > could be a reason for memory leakage in Airflow: > > > > - https://github.com/apache/airflow/discussions/24694 > > > > - https://github.com/apache/airflow/discussions/28597 > > > > > > > > *Incorrect time zones*, known issues and should be already fixed in > > > > master branch > > > > - https://github.com/sdispater/pendulum/issues/700, Mexico do not > use > > > DST > > > > anymore > > > > - https://github.com/sdispater/pendulum/issues/706, Egypt reinstate > > DST > > > > > > > > We add clarification in https://github.com/apache/airflow/pull/30467 > , > > > > however it seems like there is no other way rather than patching > > Pendulum > > > > right now. > > > > > > > > All these issues should be solved as soon as pendulum 3 is released. > > The > > > > current announced estimation is end of september/ beginning of > October: > > > > > > https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677 > > > > > > > > So in theory we would have a fixed version of pendulum soon, and it > > might > > > > break something in Airflow but from my point of view it is better > than > > > > current status. > > > > > > > > However there might be a situation where the release of the pendulum > > > would > > > > be postponed, so maybe better to have a backup plan. What could we do > > in > > > > this case? > > > > > > > > Maybe we should start to use zoneinfo.ZoneInfo instead of pendulum > > > > datetime? https://github.com/apache/airflow/issues/19450 > > > > Pros: > > > > - stdlib (python 3.9+) > > > > - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo > > > > > > > > Cons: > > > > - Current serialization model can't deal with backport packages. E.g. > > > > timezone which are serialized in backport_zoneinfo can't be > > deserialized > > > in > > > > zoneinfo > > > > > > > > Maybe we should replace parse datetime with another solution. Does > > anyone > > > > know a good replacement? > > > > > > > > Maybe someone from Airflow Community could propose their help with > > > > maintenance of library: > > > > - https://github.com/sdispater/pendulum/issues/590 > > > > > > > > Maybe we should get rid of the pendulum at all, as a last resort > > > solution. > > > > I can't imagine how we could do that, because a lot of stuff depends > on > > > the > > > > pendulum and removing it would be a breaking change. > > > > > > > > ---- > > > > Best Wishes > > > > *Andrey Anshin* > > > > > > > > > > > > > > > > > -- > > > > -- > > Bolke de Bruin > > [email protected] > > >
