Also I think TP - had a document in the past (years ago) describing a draft of a more complete alternative we can take to approach datetime vs. pendulum dichotomy. I cannot easily find the document and discussion - but I do remember it was proposing some interesting changes in the approach of Airflow to have an abstraction layer on top of it (as far as I remember). Maybe we can resurrect that idea if TP might find the proposal ?
On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <bdbr...@gmail.com> wrote: > Hi, > > I agree that the current speed of development of Pendulum leaves something > to be desired. However, I think we should not underestimate the effort of > replacing it. It is not just a matter of %s/pendulum/datetime/g so to say. > If we are *truly* thinking about moving to native datetime / zoneinfo etc > we need *extensive* tests, basically copying what pendulum does to check > its behavior. The reason is that native implementations in the past made > serious mistakes and I do not put a lot of trust in them. > > An abstraction or vendoring in could be alternatives, but they bring their > own significant challenges. > > I have re-raised the issue here: > > https://github.com/sdispater/pendulum/issues/753 > > The upside is that it seems the amount of issues with the beta is limited, > so hopefully the maintainers can spend a couple of cycles to address them. > > Bolke > > On Fri, 17 Nov 2023 at 10:28, Andrey Anshin <andrey.ans...@taragol.is> > wrote: > > > There is no changes in stable pendulum so let's try to continue this > > discussion and start think about "Plan B" > > > > Just a reminder: > > - pendulum 2.1.2 released 3 years ago (at the time Airflow 1.10.x) > > - pendulum 2 doesn't work well in Python 3.12, this is a showstopper for > > the support Python 3.12 > > - pendulum 2 have memory leaks ( > > https://github.com/sdispater/pendulum/issues/720), and Airflow use > > approach > > to achieve this leaks, especially in K8S executor > > - pendulum 2 doesn't use system tzdata by default, but we have a > > workaround, thanks Bolke for the documentation > > - pendulum it is a core dependency of Airflow, and I guess we can't > > remove/replace it without breaking changes. > > > > So my proposal if things won't change in the near future then we need to > > start removing the pendulum from the core and replace it with the native > > datetime / zoneinfo / tzinfo. But maybe we have another way to resolve > this > > without breaking changes? Because for me it would be a little weird if > > removal pendulum would be a main driver for release Airflow 3 > > > > > > > > > > ---- > > Best Wishes > > *Andrey Anshin* > > > > > > > > On Thu, 28 Sept 2023 at 13:01, Andrey Anshin <andrey.ans...@taragol.is> > > wrote: > > > > > This discussion is more about the known problem of pendulum and how we > > > could deal with it and maybe how we (as Community) might help autor. > > > > > > The library is mostly supported by a single author Sébastien Eustace ( > > > https://github.com/sdispater) and it seems like we bump into the > > > situation which is described in xkcd #2347 ( > > > https://imgs.xkcd.com/comics/dependency.png). To be honest it is not > > > something new when library mainly supported by one author so there is > > > always a risk that the library will no longer be supported / abandoned > > > And if takes in account that pendulum provides core functionality in > > > Airflow it could have dramatical impact in the future. > > > > > > Pendulum is a really nice library which helps a lot of developers to > work > > > with dates/datetimes. However there is one major problem, the last > > release > > > of this library happened more than 3 years ago ( > > > https://pypi.org/project/pendulum/#history) in the time when Airflow > > > 1.10.11 was released > > > > > > Fortunately, the project is not abandoned and on a regular basis > commits > > > add into the master branch. However these commits are not included into > > any > > > final release and that's why some things related to datetime don't work > > as > > > expected in Airflow. There are list of known (for me) issues which are > > > affect Airflow > > > > > > *Memory Leak on parse*: > > > - https://github.com/sdispater/pendulum/issues/720, this one fixed 2 > > > years ago but not available yet ( > > > https://github.com/sdispater/pendulum/pull/563). Since we use parse > > dates > > > in airflow codebase: datetime parameters and datetime in logs this one > > > could be a reason for memory leakage in Airflow: > > > - https://github.com/apache/airflow/discussions/24694 > > > - https://github.com/apache/airflow/discussions/28597 > > > > > > *Incorrect time zones*, known issues and should be already fixed in > > > master branch > > > - https://github.com/sdispater/pendulum/issues/700, Mexico do not use > > DST > > > anymore > > > - https://github.com/sdispater/pendulum/issues/706, Egypt reinstate > DST > > > > > > We add clarification in https://github.com/apache/airflow/pull/30467, > > > however it seems like there is no other way rather than patching > Pendulum > > > right now. > > > > > > All these issues should be solved as soon as pendulum 3 is released. > The > > > current announced estimation is end of september/ beginning of October: > > > > https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677 > > > > > > So in theory we would have a fixed version of pendulum soon, and it > might > > > break something in Airflow but from my point of view it is better than > > > current status. > > > > > > However there might be a situation where the release of the pendulum > > would > > > be postponed, so maybe better to have a backup plan. What could we do > in > > > this case? > > > > > > Maybe we should start to use zoneinfo.ZoneInfo instead of pendulum > > > datetime? https://github.com/apache/airflow/issues/19450 > > > Pros: > > > - stdlib (python 3.9+) > > > - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo > > > > > > Cons: > > > - Current serialization model can't deal with backport packages. E.g. > > > timezone which are serialized in backport_zoneinfo can't be > deserialized > > in > > > zoneinfo > > > > > > Maybe we should replace parse datetime with another solution. Does > anyone > > > know a good replacement? > > > > > > Maybe someone from Airflow Community could propose their help with > > > maintenance of library: > > > - https://github.com/sdispater/pendulum/issues/590 > > > > > > Maybe we should get rid of the pendulum at all, as a last resort > > solution. > > > I can't imagine how we could do that, because a lot of stuff depends on > > the > > > pendulum and removing it would be a breaking change. > > > > > > ---- > > > Best Wishes > > > *Andrey Anshin* > > > > > > > > > > > -- > > -- > Bolke de Bruin > bdbr...@gmail.com >
