Thanks Jarek On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> I think we miss important insight - straight from the source. > > I believe it's time to be candid and simply ask questions for the future of > Pendulum directly where we should - ie. we should just ask maintainers. > > I've just started a very candid and open discussion there - > https://github.com/sdispater/pendulum/discussions/771 . > > I linked back to this discussion and explained where we are coming from. I > even offered an option, that maybe - if they accept some help and would be > open to make a few of us maintainers of Pendulum, that I will ask if some > of the maintainers here in Airflow would like to step up (I would be > willing to - for one) and help to at least move Pendulum through 3.12 and > maybe keep it running for as long as it will be needed to maintain it. > > I think if they would be open to some of us helping them - this might be > actually the "cheapest" option for us to be honest at least in a mid-term > if we could gain influence and even take part and speed up releases and > merging of issues that are blocking us (or would be blocking us in the > future). > > I tried to be very friendly but candid and direct on what kind of problems > it creates us and also expressed our thanks for supporting us for so long > (I do believe they deserve it). I think the sole (pretty much) maintainer > of Pendulum might really have a hard time maintaining and keeping it up for > years - without being paid for and thanked. I hope - maybe all that they > need are a few words of encouragement and thanks and realising that others > depend on their work - but also seeing that they are understood, empathised > with and that there are viable alternatives they might follow might be > helpful for them to be also candid and respond. I hope my intentions will > be understood - that I am not complaining, but even trying to help. > > Let\s see where it gets us. > > J. > > On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin <andrey.ans...@taragol.is> > wrote: > > > Just to clarify I'd like us to consider the possibility that no new > > pendulum would be released or released at the end of 2024, like a > > pessimistic scenario: > > - What should we do in this case? > > - Work out a backup plan. > > > > > > ---- > > Best Wishes > > *Andrey Anshin* > > > > > > > > On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > Also I think TP - had a document in the past (years ago) describing a > > > draft of a more complete alternative we can take to approach datetime > vs. > > > pendulum dichotomy. I cannot easily find the document and discussion - > > but > > > I do remember it was proposing some interesting changes in the approach > > of > > > Airflow to have an abstraction layer on top of it (as far as I > remember). > > > Maybe we can resurrect that idea if TP might find the proposal ? > > > > > > On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <bdbr...@gmail.com> > > wrote: > > > > > > > Hi, > > > > > > > > I agree that the current speed of development of Pendulum leaves > > > something > > > > to be desired. However, I think we should not underestimate the > effort > > of > > > > replacing it. It is not just a matter of %s/pendulum/datetime/g so to > > > say. > > > > If we are *truly* thinking about moving to native datetime / zoneinfo > > etc > > > > we need *extensive* tests, basically copying what pendulum does to > > check > > > > its behavior. The reason is that native implementations in the past > > made > > > > serious mistakes and I do not put a lot of trust in them. > > > > > > > > An abstraction or vendoring in could be alternatives, but they bring > > > their > > > > own significant challenges. > > > > > > > > I have re-raised the issue here: > > > > > > > > https://github.com/sdispater/pendulum/issues/753 > > > > > > > > The upside is that it seems the amount of issues with the beta is > > > limited, > > > > so hopefully the maintainers can spend a couple of cycles to address > > > them. > > > > > > > > Bolke > > > > > > > > On Fri, 17 Nov 2023 at 10:28, Andrey Anshin < > andrey.ans...@taragol.is> > > > > wrote: > > > > > > > > > There is no changes in stable pendulum so let's try to continue > this > > > > > discussion and start think about "Plan B" > > > > > > > > > > Just a reminder: > > > > > - pendulum 2.1.2 released 3 years ago (at the time Airflow 1.10.x) > > > > > - pendulum 2 doesn't work well in Python 3.12, this is a > showstopper > > > for > > > > > the support Python 3.12 > > > > > - pendulum 2 have memory leaks ( > > > > > https://github.com/sdispater/pendulum/issues/720), and Airflow use > > > > > approach > > > > > to achieve this leaks, especially in K8S executor > > > > > - pendulum 2 doesn't use system tzdata by default, but we have a > > > > > workaround, thanks Bolke for the documentation > > > > > - pendulum it is a core dependency of Airflow, and I guess we can't > > > > > remove/replace it without breaking changes. > > > > > > > > > > So my proposal if things won't change in the near future then we > need > > > to > > > > > start removing the pendulum from the core and replace it with the > > > native > > > > > datetime / zoneinfo / tzinfo. But maybe we have another way to > > resolve > > > > this > > > > > without breaking changes? Because for me it would be a little weird > > if > > > > > removal pendulum would be a main driver for release Airflow 3 > > > > > > > > > > > > > > > > > > > > > > > > > ---- > > > > > Best Wishes > > > > > *Andrey Anshin* > > > > > > > > > > > > > > > > > > > > On Thu, 28 Sept 2023 at 13:01, Andrey Anshin < > > andrey.ans...@taragol.is > > > > > > > > > wrote: > > > > > > > > > > > This discussion is more about the known problem of pendulum and > how > > > we > > > > > > could deal with it and maybe how we (as Community) might help > > autor. > > > > > > > > > > > > The library is mostly supported by a single author Sébastien > > Eustace > > > ( > > > > > > https://github.com/sdispater) and it seems like we bump into the > > > > > > situation which is described in xkcd #2347 ( > > > > > > https://imgs.xkcd.com/comics/dependency.png). To be honest it is > > not > > > > > > something new when library mainly supported by one author so > there > > is > > > > > > always a risk that the library will no longer be supported / > > > abandoned > > > > > > And if takes in account that pendulum provides core functionality > > in > > > > > > Airflow it could have dramatical impact in the future. > > > > > > > > > > > > Pendulum is a really nice library which helps a lot of developers > > to > > > > work > > > > > > with dates/datetimes. However there is one major problem, the > last > > > > > release > > > > > > of this library happened more than 3 years ago ( > > > > > > https://pypi.org/project/pendulum/#history) in the time when > > Airflow > > > > > > 1.10.11 was released > > > > > > > > > > > > Fortunately, the project is not abandoned and on a regular basis > > > > commits > > > > > > add into the master branch. However these commits are not > included > > > into > > > > > any > > > > > > final release and that's why some things related to datetime > don't > > > work > > > > > as > > > > > > expected in Airflow. There are list of known (for me) issues > which > > > are > > > > > > affect Airflow > > > > > > > > > > > > *Memory Leak on parse*: > > > > > > - https://github.com/sdispater/pendulum/issues/720, this one > fixed > > > 2 > > > > > > years ago but not available yet ( > > > > > > https://github.com/sdispater/pendulum/pull/563). Since we use > > parse > > > > > dates > > > > > > in airflow codebase: datetime parameters and datetime in logs > this > > > one > > > > > > could be a reason for memory leakage in Airflow: > > > > > > - https://github.com/apache/airflow/discussions/24694 > > > > > > - https://github.com/apache/airflow/discussions/28597 > > > > > > > > > > > > *Incorrect time zones*, known issues and should be already fixed > in > > > > > > master branch > > > > > > - https://github.com/sdispater/pendulum/issues/700, Mexico do > not > > > use > > > > > DST > > > > > > anymore > > > > > > - https://github.com/sdispater/pendulum/issues/706, Egypt > > reinstate > > > > DST > > > > > > > > > > > > We add clarification in > > https://github.com/apache/airflow/pull/30467 > > > , > > > > > > however it seems like there is no other way rather than patching > > > > Pendulum > > > > > > right now. > > > > > > > > > > > > All these issues should be solved as soon as pendulum 3 is > > released. > > > > The > > > > > > current announced estimation is end of september/ beginning of > > > October: > > > > > > > > > > > > https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677 > > > > > > > > > > > > So in theory we would have a fixed version of pendulum soon, and > it > > > > might > > > > > > break something in Airflow but from my point of view it is better > > > than > > > > > > current status. > > > > > > > > > > > > However there might be a situation where the release of the > > pendulum > > > > > would > > > > > > be postponed, so maybe better to have a backup plan. What could > we > > do > > > > in > > > > > > this case? > > > > > > > > > > > > Maybe we should start to use zoneinfo.ZoneInfo instead of > pendulum > > > > > > datetime? https://github.com/apache/airflow/issues/19450 > > > > > > Pros: > > > > > > - stdlib (python 3.9+) > > > > > > - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo > > > > > > > > > > > > Cons: > > > > > > - Current serialization model can't deal with backport packages. > > E.g. > > > > > > timezone which are serialized in backport_zoneinfo can't be > > > > deserialized > > > > > in > > > > > > zoneinfo > > > > > > > > > > > > Maybe we should replace parse datetime with another solution. > Does > > > > anyone > > > > > > know a good replacement? > > > > > > > > > > > > Maybe someone from Airflow Community could propose their help > with > > > > > > maintenance of library: > > > > > > - https://github.com/sdispater/pendulum/issues/590 > > > > > > > > > > > > Maybe we should get rid of the pendulum at all, as a last resort > > > > > solution. > > > > > > I can't imagine how we could do that, because a lot of stuff > > depends > > > on > > > > > the > > > > > > pendulum and removing it would be a breaking change. > > > > > > > > > > > > ---- > > > > > > Best Wishes > > > > > > *Andrey Anshin* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > -- > > > > Bolke de Bruin > > > > bdbr...@gmail.com > > > > > > > > > >
