An option would be to fork Pendulum? It is MIT Licensed, I don't know whether that poses problems to get in ASF?
If forking (?) which is somewhat non-ideal, would we want that 'in' airflow? If not 'in' airflow, I wonder if ASF incubator would accept a forked project? [ anecdotally Linux Foundation has incubated Starrocks which is a fork of Apache Doris, so there's precedence for this to be OK in the wider OSS ecosystem ]. On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk <[email protected]> wrote: > I have not heard back from maintainers, just a comment from someone else > who suggested donating a pendulum to the ASF (which is kinda interesting > idea). I followed up, let's see. I think if we do not hear back and there > will be another week or two where "Pendulum 3 supporting 3.12 support "is > coming" but no-one knows when, effectively means that we should look for an > alternative like NOW. > > J. > > > > On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish > <[email protected]> wrote: > > > Thanks Jarek > > > > On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <[email protected]> wrote: > > > > > I think we miss important insight - straight from the source. > > > > > > I believe it's time to be candid and simply ask questions for the > future > > of > > > Pendulum directly where we should - ie. we should just ask > maintainers. > > > > > > I've just started a very candid and open discussion there - > > > https://github.com/sdispater/pendulum/discussions/771 . > > > > > > I linked back to this discussion and explained where we are coming > from. > > I > > > even offered an option, that maybe - if they accept some help and would > > be > > > open to make a few of us maintainers of Pendulum, that I will ask if > some > > > of the maintainers here in Airflow would like to step up (I would be > > > willing to - for one) and help to at least move Pendulum through 3.12 > and > > > maybe keep it running for as long as it will be needed to maintain it. > > > > > > I think if they would be open to some of us helping them - this might > be > > > actually the "cheapest" option for us to be honest at least in a > mid-term > > > if we could gain influence and even take part and speed up releases and > > > merging of issues that are blocking us (or would be blocking us in the > > > future). > > > > > > I tried to be very friendly but candid and direct on what kind of > > problems > > > it creates us and also expressed our thanks for supporting us for so > long > > > (I do believe they deserve it). I think the sole (pretty much) > > maintainer > > > of Pendulum might really have a hard time maintaining and keeping it up > > for > > > years - without being paid for and thanked. I hope - maybe all that > they > > > need are a few words of encouragement and thanks and realising that > > others > > > depend on their work - but also seeing that they are understood, > > empathised > > > with and that there are viable alternatives they might follow might be > > > helpful for them to be also candid and respond. I hope my intentions > will > > > be understood - that I am not complaining, but even trying to help. > > > > > > Let\s see where it gets us. > > > > > > J. > > > > > > On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin < > [email protected]> > > > wrote: > > > > > > > Just to clarify I'd like us to consider the possibility that no new > > > > pendulum would be released or released at the end of 2024, like a > > > > pessimistic scenario: > > > > - What should we do in this case? > > > > - Work out a backup plan. > > > > > > > > > > > > ---- > > > > Best Wishes > > > > *Andrey Anshin* > > > > > > > > > > > > > > > > On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <[email protected]> wrote: > > > > > > > > > Also I think TP - had a document in the past (years ago) > > describing a > > > > > draft of a more complete alternative we can take to approach > datetime > > > vs. > > > > > pendulum dichotomy. I cannot easily find the document and > discussion > > - > > > > but > > > > > I do remember it was proposing some interesting changes in the > > approach > > > > of > > > > > Airflow to have an abstraction layer on top of it (as far as I > > > remember). > > > > > Maybe we can resurrect that idea if TP might find the proposal ? > > > > > > > > > > On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin <[email protected]> > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I agree that the current speed of development of Pendulum leaves > > > > > something > > > > > > to be desired. However, I think we should not underestimate the > > > effort > > > > of > > > > > > replacing it. It is not just a matter of %s/pendulum/datetime/g > so > > to > > > > > say. > > > > > > If we are *truly* thinking about moving to native datetime / > > zoneinfo > > > > etc > > > > > > we need *extensive* tests, basically copying what pendulum does > to > > > > check > > > > > > its behavior. The reason is that native implementations in the > past > > > > made > > > > > > serious mistakes and I do not put a lot of trust in them. > > > > > > > > > > > > An abstraction or vendoring in could be alternatives, but they > > bring > > > > > their > > > > > > own significant challenges. > > > > > > > > > > > > I have re-raised the issue here: > > > > > > > > > > > > https://github.com/sdispater/pendulum/issues/753 > > > > > > > > > > > > The upside is that it seems the amount of issues with the beta is > > > > > limited, > > > > > > so hopefully the maintainers can spend a couple of cycles to > > address > > > > > them. > > > > > > > > > > > > Bolke > > > > > > > > > > > > On Fri, 17 Nov 2023 at 10:28, Andrey Anshin < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > There is no changes in stable pendulum so let's try to continue > > > this > > > > > > > discussion and start think about "Plan B" > > > > > > > > > > > > > > Just a reminder: > > > > > > > - pendulum 2.1.2 released 3 years ago (at the time Airflow > > 1.10.x) > > > > > > > - pendulum 2 doesn't work well in Python 3.12, this is a > > > showstopper > > > > > for > > > > > > > the support Python 3.12 > > > > > > > - pendulum 2 have memory leaks ( > > > > > > > https://github.com/sdispater/pendulum/issues/720), and Airflow > > use > > > > > > > approach > > > > > > > to achieve this leaks, especially in K8S executor > > > > > > > - pendulum 2 doesn't use system tzdata by default, but we have > a > > > > > > > workaround, thanks Bolke for the documentation > > > > > > > - pendulum it is a core dependency of Airflow, and I guess we > > can't > > > > > > > remove/replace it without breaking changes. > > > > > > > > > > > > > > So my proposal if things won't change in the near future then > we > > > need > > > > > to > > > > > > > start removing the pendulum from the core and replace it with > the > > > > > native > > > > > > > datetime / zoneinfo / tzinfo. But maybe we have another way to > > > > resolve > > > > > > this > > > > > > > without breaking changes? Because for me it would be a little > > weird > > > > if > > > > > > > removal pendulum would be a main driver for release Airflow 3 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ---- > > > > > > > Best Wishes > > > > > > > *Andrey Anshin* > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, 28 Sept 2023 at 13:01, Andrey Anshin < > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > This discussion is more about the known problem of pendulum > and > > > how > > > > > we > > > > > > > > could deal with it and maybe how we (as Community) might help > > > > autor. > > > > > > > > > > > > > > > > The library is mostly supported by a single author Sébastien > > > > Eustace > > > > > ( > > > > > > > > https://github.com/sdispater) and it seems like we bump into > > the > > > > > > > > situation which is described in xkcd #2347 ( > > > > > > > > https://imgs.xkcd.com/comics/dependency.png). To be honest > it > > is > > > > not > > > > > > > > something new when library mainly supported by one author so > > > there > > > > is > > > > > > > > always a risk that the library will no longer be supported / > > > > > abandoned > > > > > > > > And if takes in account that pendulum provides core > > functionality > > > > in > > > > > > > > Airflow it could have dramatical impact in the future. > > > > > > > > > > > > > > > > Pendulum is a really nice library which helps a lot of > > developers > > > > to > > > > > > work > > > > > > > > with dates/datetimes. However there is one major problem, the > > > last > > > > > > > release > > > > > > > > of this library happened more than 3 years ago ( > > > > > > > > https://pypi.org/project/pendulum/#history) in the time when > > > > Airflow > > > > > > > > 1.10.11 was released > > > > > > > > > > > > > > > > Fortunately, the project is not abandoned and on a regular > > basis > > > > > > commits > > > > > > > > add into the master branch. However these commits are not > > > included > > > > > into > > > > > > > any > > > > > > > > final release and that's why some things related to datetime > > > don't > > > > > work > > > > > > > as > > > > > > > > expected in Airflow. There are list of known (for me) issues > > > which > > > > > are > > > > > > > > affect Airflow > > > > > > > > > > > > > > > > *Memory Leak on parse*: > > > > > > > > - https://github.com/sdispater/pendulum/issues/720, this one > > > fixed > > > > > 2 > > > > > > > > years ago but not available yet ( > > > > > > > > https://github.com/sdispater/pendulum/pull/563). Since we > use > > > > parse > > > > > > > dates > > > > > > > > in airflow codebase: datetime parameters and datetime in logs > > > this > > > > > one > > > > > > > > could be a reason for memory leakage in Airflow: > > > > > > > > - https://github.com/apache/airflow/discussions/24694 > > > > > > > > - https://github.com/apache/airflow/discussions/28597 > > > > > > > > > > > > > > > > *Incorrect time zones*, known issues and should be already > > fixed > > > in > > > > > > > > master branch > > > > > > > > - https://github.com/sdispater/pendulum/issues/700, Mexico > do > > > not > > > > > use > > > > > > > DST > > > > > > > > anymore > > > > > > > > - https://github.com/sdispater/pendulum/issues/706, Egypt > > > > reinstate > > > > > > DST > > > > > > > > > > > > > > > > We add clarification in > > > > https://github.com/apache/airflow/pull/30467 > > > > > , > > > > > > > > however it seems like there is no other way rather than > > patching > > > > > > Pendulum > > > > > > > > right now. > > > > > > > > > > > > > > > > All these issues should be solved as soon as pendulum 3 is > > > > released. > > > > > > The > > > > > > > > current announced estimation is end of september/ beginning > of > > > > > October: > > > > > > > > > > > > > > > > > > > > https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677 > > > > > > > > > > > > > > > > So in theory we would have a fixed version of pendulum soon, > > and > > > it > > > > > > might > > > > > > > > break something in Airflow but from my point of view it is > > better > > > > > than > > > > > > > > current status. > > > > > > > > > > > > > > > > However there might be a situation where the release of the > > > > pendulum > > > > > > > would > > > > > > > > be postponed, so maybe better to have a backup plan. What > could > > > we > > > > do > > > > > > in > > > > > > > > this case? > > > > > > > > > > > > > > > > Maybe we should start to use zoneinfo.ZoneInfo instead of > > > pendulum > > > > > > > > datetime? https://github.com/apache/airflow/issues/19450 > > > > > > > > Pros: > > > > > > > > - stdlib (python 3.9+) > > > > > > > > - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo > > > > > > > > > > > > > > > > Cons: > > > > > > > > - Current serialization model can't deal with backport > > packages. > > > > E.g. > > > > > > > > timezone which are serialized in backport_zoneinfo can't be > > > > > > deserialized > > > > > > > in > > > > > > > > zoneinfo > > > > > > > > > > > > > > > > Maybe we should replace parse datetime with another solution. > > > Does > > > > > > anyone > > > > > > > > know a good replacement? > > > > > > > > > > > > > > > > Maybe someone from Airflow Community could propose their help > > > with > > > > > > > > maintenance of library: > > > > > > > > - https://github.com/sdispater/pendulum/issues/590 > > > > > > > > > > > > > > > > Maybe we should get rid of the pendulum at all, as a last > > resort > > > > > > > solution. > > > > > > > > I can't imagine how we could do that, because a lot of stuff > > > > depends > > > > > on > > > > > > > the > > > > > > > > pendulum and removing it would be a breaking change. > > > > > > > > > > > > > > > > ---- > > > > > > > > Best Wishes > > > > > > > > *Andrey Anshin* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > -- > > > > > > Bolke de Bruin > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > >
