I think we miss important insight - straight from the source. I believe it's time to be candid and simply ask questions for the future of Pendulum directly where we should - ie. we should just ask maintainers.
I've just started a very candid and open discussion there - https://github.com/sdispater/pendulum/discussions/771 . I linked back to this discussion and explained where we are coming from. I even offered an option, that maybe - if they accept some help and would be open to make a few of us maintainers of Pendulum, that I will ask if some of the maintainers here in Airflow would like to step up (I would be willing to - for one) and help to at least move Pendulum through 3.12 and maybe keep it running for as long as it will be needed to maintain it. I think if they would be open to some of us helping them - this might be actually the "cheapest" option for us to be honest at least in a mid-term if we could gain influence and even take part and speed up releases and merging of issues that are blocking us (or would be blocking us in the future). I tried to be very friendly but candid and direct on what kind of problems it creates us and also expressed our thanks for supporting us for so long (I do believe they deserve it). I think the sole (pretty much) maintainer of Pendulum might really have a hard time maintaining and keeping it up for years - without being paid for and thanked. I hope - maybe all that they need are a few words of encouragement and thanks and realising that others depend on their work - but also seeing that they are understood, empathised with and that there are viable alternatives they might follow might be helpful for them to be also candid and respond. I hope my intentions will be understood - that I am not complaining, but even trying to help. Let\s see where it gets us. J. On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin <andrey.ans...@taragol.is> wrote: > Just to clarify I'd like us to consider the possibility that no new > pendulum would be released or released at the end of 2024, like a > pessimistic scenario: > - What should we do in this case? > - Work out a backup plan. > > > ---- > Best Wishes > *Andrey Anshin* > > > > On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <ja...@potiuk.com> wrote: > > > Also I think TP - had a document in the past (years ago) describing a > > draft of a more complete alternative we can take to approach datetime vs. > > pendulum dichotomy. I cannot easily find the document and discussion - > but > > I do remember it was proposing some interesting changes in the approach > of > > Airflow to have an abstraction layer on top of it (as far as I remember). > > Maybe we can resurrect that idea if TP might find the proposal ? > > > > On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <bdbr...@gmail.com> > wrote: > > > > > Hi, > > > > > > I agree that the current speed of development of Pendulum leaves > > something > > > to be desired. However, I think we should not underestimate the effort > of > > > replacing it. It is not just a matter of %s/pendulum/datetime/g so to > > say. > > > If we are *truly* thinking about moving to native datetime / zoneinfo > etc > > > we need *extensive* tests, basically copying what pendulum does to > check > > > its behavior. The reason is that native implementations in the past > made > > > serious mistakes and I do not put a lot of trust in them. > > > > > > An abstraction or vendoring in could be alternatives, but they bring > > their > > > own significant challenges. > > > > > > I have re-raised the issue here: > > > > > > https://github.com/sdispater/pendulum/issues/753 > > > > > > The upside is that it seems the amount of issues with the beta is > > limited, > > > so hopefully the maintainers can spend a couple of cycles to address > > them. > > > > > > Bolke > > > > > > On Fri, 17 Nov 2023 at 10:28, Andrey Anshin <andrey.ans...@taragol.is> > > > wrote: > > > > > > > There is no changes in stable pendulum so let's try to continue this > > > > discussion and start think about "Plan B" > > > > > > > > Just a reminder: > > > > - pendulum 2.1.2 released 3 years ago (at the time Airflow 1.10.x) > > > > - pendulum 2 doesn't work well in Python 3.12, this is a showstopper > > for > > > > the support Python 3.12 > > > > - pendulum 2 have memory leaks ( > > > > https://github.com/sdispater/pendulum/issues/720), and Airflow use > > > > approach > > > > to achieve this leaks, especially in K8S executor > > > > - pendulum 2 doesn't use system tzdata by default, but we have a > > > > workaround, thanks Bolke for the documentation > > > > - pendulum it is a core dependency of Airflow, and I guess we can't > > > > remove/replace it without breaking changes. > > > > > > > > So my proposal if things won't change in the near future then we need > > to > > > > start removing the pendulum from the core and replace it with the > > native > > > > datetime / zoneinfo / tzinfo. But maybe we have another way to > resolve > > > this > > > > without breaking changes? Because for me it would be a little weird > if > > > > removal pendulum would be a main driver for release Airflow 3 > > > > > > > > > > > > > > > > > > > > ---- > > > > Best Wishes > > > > *Andrey Anshin* > > > > > > > > > > > > > > > > On Thu, 28 Sept 2023 at 13:01, Andrey Anshin < > andrey.ans...@taragol.is > > > > > > > wrote: > > > > > > > > > This discussion is more about the known problem of pendulum and how > > we > > > > > could deal with it and maybe how we (as Community) might help > autor. > > > > > > > > > > The library is mostly supported by a single author Sébastien > Eustace > > ( > > > > > https://github.com/sdispater) and it seems like we bump into the > > > > > situation which is described in xkcd #2347 ( > > > > > https://imgs.xkcd.com/comics/dependency.png). To be honest it is > not > > > > > something new when library mainly supported by one author so there > is > > > > > always a risk that the library will no longer be supported / > > abandoned > > > > > And if takes in account that pendulum provides core functionality > in > > > > > Airflow it could have dramatical impact in the future. > > > > > > > > > > Pendulum is a really nice library which helps a lot of developers > to > > > work > > > > > with dates/datetimes. However there is one major problem, the last > > > > release > > > > > of this library happened more than 3 years ago ( > > > > > https://pypi.org/project/pendulum/#history) in the time when > Airflow > > > > > 1.10.11 was released > > > > > > > > > > Fortunately, the project is not abandoned and on a regular basis > > > commits > > > > > add into the master branch. However these commits are not included > > into > > > > any > > > > > final release and that's why some things related to datetime don't > > work > > > > as > > > > > expected in Airflow. There are list of known (for me) issues which > > are > > > > > affect Airflow > > > > > > > > > > *Memory Leak on parse*: > > > > > - https://github.com/sdispater/pendulum/issues/720, this one fixed > > 2 > > > > > years ago but not available yet ( > > > > > https://github.com/sdispater/pendulum/pull/563). Since we use > parse > > > > dates > > > > > in airflow codebase: datetime parameters and datetime in logs this > > one > > > > > could be a reason for memory leakage in Airflow: > > > > > - https://github.com/apache/airflow/discussions/24694 > > > > > - https://github.com/apache/airflow/discussions/28597 > > > > > > > > > > *Incorrect time zones*, known issues and should be already fixed in > > > > > master branch > > > > > - https://github.com/sdispater/pendulum/issues/700, Mexico do not > > use > > > > DST > > > > > anymore > > > > > - https://github.com/sdispater/pendulum/issues/706, Egypt > reinstate > > > DST > > > > > > > > > > We add clarification in > https://github.com/apache/airflow/pull/30467 > > , > > > > > however it seems like there is no other way rather than patching > > > Pendulum > > > > > right now. > > > > > > > > > > All these issues should be solved as soon as pendulum 3 is > released. > > > The > > > > > current announced estimation is end of september/ beginning of > > October: > > > > > > > > > https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677 > > > > > > > > > > So in theory we would have a fixed version of pendulum soon, and it > > > might > > > > > break something in Airflow but from my point of view it is better > > than > > > > > current status. > > > > > > > > > > However there might be a situation where the release of the > pendulum > > > > would > > > > > be postponed, so maybe better to have a backup plan. What could we > do > > > in > > > > > this case? > > > > > > > > > > Maybe we should start to use zoneinfo.ZoneInfo instead of pendulum > > > > > datetime? https://github.com/apache/airflow/issues/19450 > > > > > Pros: > > > > > - stdlib (python 3.9+) > > > > > - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo > > > > > > > > > > Cons: > > > > > - Current serialization model can't deal with backport packages. > E.g. > > > > > timezone which are serialized in backport_zoneinfo can't be > > > deserialized > > > > in > > > > > zoneinfo > > > > > > > > > > Maybe we should replace parse datetime with another solution. Does > > > anyone > > > > > know a good replacement? > > > > > > > > > > Maybe someone from Airflow Community could propose their help with > > > > > maintenance of library: > > > > > - https://github.com/sdispater/pendulum/issues/590 > > > > > > > > > > Maybe we should get rid of the pendulum at all, as a last resort > > > > solution. > > > > > I can't imagine how we could do that, because a lot of stuff > depends > > on > > > > the > > > > > pendulum and removing it would be a breaking change. > > > > > > > > > > ---- > > > > > Best Wishes > > > > > *Andrey Anshin* > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > -- > > > Bolke de Bruin > > > bdbr...@gmail.com > > > > > >
