Agree. I think this is the best way forward.- we have to bite the bullet of potential backwards compatibility issues and separating out a compat layer where we could optionally use pendulum instead would be the most "easy" approach for our users. Most of them would not even notice if we do a good job, but those who somehow depend on pendulum objects would get a possibility to switch back.
On Fri, Dec 1, 2023 at 1:09 PM Andrey Anshin <[email protected]> wrote: > IMHO, if we would like to replace pendulum to alternatives, the better > alternative would be combination of datetime.datetime + zoneinfo + > datetime.timezone > > We still do hacks around pendulum functional, which makes hacks around > datetime.datetime, so in my perspective better remove redundant layers and > use native objects directly. > > What confuses me the most is how we get around type changes in Tasks > Context. All dates return as pendulum.DateTime for a very long time > (literally for ages) and it is probably could be classified as breaking > changes, because it might break quite a bit users pipelines. It still could > be resolved as optional pendulum dependency + config option for datetime > types in context, which might help to get familiar types for a while. > > > On Wed, 29 Nov 2023 at 05:58, Jarek Potiuk <[email protected]> wrote: > > > "infamous fork of " should be "fork of infamous Akka" - just to be clear > :) > > > > On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk <[email protected]> wrote: > > > > > It's a very rare occurrence that ASF accepts forks. Usually it has to > be > > > willingly donated by those who own the IP rights, - mainly because of > > > trademark issues. Licence is one thing, but carrying things like the > name > > > of the project is not possible without clearing IP and donating the > > > project. > > > > > > It happened last time last year - with Pekko project - infamous fork or > > > Akka after Lightbend changed licensing for Akka - > > > > https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka > > > making it impossible to be used in ASF projects with the new licence. > But > > > it was a really "strong" case and the "we do not accept forks usually" > > was > > > quite a contentious issue (I followed the discussion - it was > fascinating > > > actually). It tool a LOT of time and effort for those who led the > effort > > > eventually - and also mostly because things had to be crystal-clear and > > not > > > "litigable" because the other side had some business reasons to make > the > > > licensing changes and they did not understood why it is impossible to > use > > > Akka as-is even with all the exclusions they offered to ASF - and > > > effectively it was a "hostile" move. > > > > > > I don't think it's "worth" it in this case to be honest - especially if > > > the maintainers are effectively "silent" and seem to ignore problems of > > one > > > of the really serious users they have. If they were willing to > cooperate, > > > we could do a lot, but when we are left alone, I think it will be far > > more > > > convenient for us to look for alternatives - to be honest. > > > > > > J. > > > > > > > > > On Wed, Nov 29, 2023 at 2:46 AM Austin Bennett <[email protected]> > > wrote: > > > > > >> An option would be to fork Pendulum? It is MIT Licensed, I don't know > > >> whether that poses problems to get in ASF? > > >> > > >> If forking (?) which is somewhat non-ideal, would we want that 'in' > > >> airflow? > > >> > > >> If not 'in' airflow, I wonder if ASF incubator would accept a forked > > >> project? [ anecdotally Linux Foundation has incubated Starrocks which > > is > > >> a > > >> fork of Apache Doris, so there's precedence for this to be OK in the > > wider > > >> OSS ecosystem ]. > > >> > > >> > > >> On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk <[email protected]> > wrote: > > >> > > >> > I have not heard back from maintainers, just a comment from someone > > else > > >> > who suggested donating a pendulum to the ASF (which is kinda > > interesting > > >> > idea). I followed up, let's see. I think if we do not hear back and > > >> there > > >> > will be another week or two where "Pendulum 3 supporting 3.12 > support > > >> "is > > >> > coming" but no-one knows when, effectively means that we should look > > >> for an > > >> > alternative like NOW. > > >> > > > >> > J. > > >> > > > >> > > > >> > > > >> > On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish > > >> > <[email protected]> wrote: > > >> > > > >> > > Thanks Jarek > > >> > > > > >> > > On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <[email protected]> > > wrote: > > >> > > > > >> > > > I think we miss important insight - straight from the source. > > >> > > > > > >> > > > I believe it's time to be candid and simply ask questions for > the > > >> > future > > >> > > of > > >> > > > Pendulum directly where we should - ie. we should just ask > > >> > maintainers. > > >> > > > > > >> > > > I've just started a very candid and open discussion there - > > >> > > > https://github.com/sdispater/pendulum/discussions/771 . > > >> > > > > > >> > > > I linked back to this discussion and explained where we are > coming > > >> > from. > > >> > > I > > >> > > > even offered an option, that maybe - if they accept some help > and > > >> would > > >> > > be > > >> > > > open to make a few of us maintainers of Pendulum, that I will > ask > > if > > >> > some > > >> > > > of the maintainers here in Airflow would like to step up (I > would > > be > > >> > > > willing to - for one) and help to at least move Pendulum through > > >> 3.12 > > >> > and > > >> > > > maybe keep it running for as long as it will be needed to > maintain > > >> it. > > >> > > > > > >> > > > I think if they would be open to some of us helping them - this > > >> might > > >> > be > > >> > > > actually the "cheapest" option for us to be honest at least in a > > >> > mid-term > > >> > > > if we could gain influence and even take part and speed up > > releases > > >> and > > >> > > > merging of issues that are blocking us (or would be blocking us > in > > >> the > > >> > > > future). > > >> > > > > > >> > > > I tried to be very friendly but candid and direct on what kind > of > > >> > > problems > > >> > > > it creates us and also expressed our thanks for supporting us > for > > so > > >> > long > > >> > > > (I do believe they deserve it). I think the sole (pretty much) > > >> > > maintainer > > >> > > > of Pendulum might really have a hard time maintaining and > keeping > > >> it up > > >> > > for > > >> > > > years - without being paid for and thanked. I hope - maybe all > > that > > >> > they > > >> > > > need are a few words of encouragement and thanks and realising > > that > > >> > > others > > >> > > > depend on their work - but also seeing that they are understood, > > >> > > empathised > > >> > > > with and that there are viable alternatives they might follow > > might > > >> be > > >> > > > helpful for them to be also candid and respond. I hope my > > intentions > > >> > will > > >> > > > be understood - that I am not complaining, but even trying to > > help. > > >> > > > > > >> > > > Let\s see where it gets us. > > >> > > > > > >> > > > J. > > >> > > > > > >> > > > On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin < > > >> > [email protected]> > > >> > > > wrote: > > >> > > > > > >> > > > > Just to clarify I'd like us to consider the possibility that > no > > >> new > > >> > > > > pendulum would be released or released at the end of 2024, > like > > a > > >> > > > > pessimistic scenario: > > >> > > > > - What should we do in this case? > > >> > > > > - Work out a backup plan. > > >> > > > > > > >> > > > > > > >> > > > > ---- > > >> > > > > Best Wishes > > >> > > > > *Andrey Anshin* > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <[email protected]> > > >> wrote: > > >> > > > > > > >> > > > > > Also I think TP - had a document in the past (years ago) > > >> > > describing a > > >> > > > > > draft of a more complete alternative we can take to approach > > >> > datetime > > >> > > > vs. > > >> > > > > > pendulum dichotomy. I cannot easily find the document and > > >> > discussion > > >> > > - > > >> > > > > but > > >> > > > > > I do remember it was proposing some interesting changes in > the > > >> > > approach > > >> > > > > of > > >> > > > > > Airflow to have an abstraction layer on top of it (as far > as I > > >> > > > remember). > > >> > > > > > Maybe we can resurrect that idea if TP might find the > > proposal ? > > >> > > > > > > > >> > > > > > On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin < > > >> [email protected]> > > >> > > > > wrote: > > >> > > > > > > > >> > > > > > > Hi, > > >> > > > > > > > > >> > > > > > > I agree that the current speed of development of Pendulum > > >> leaves > > >> > > > > > something > > >> > > > > > > to be desired. However, I think we should not > underestimate > > >> the > > >> > > > effort > > >> > > > > of > > >> > > > > > > replacing it. It is not just a matter of > > >> %s/pendulum/datetime/g > > >> > so > > >> > > to > > >> > > > > > say. > > >> > > > > > > If we are *truly* thinking about moving to native > datetime / > > >> > > zoneinfo > > >> > > > > etc > > >> > > > > > > we need *extensive* tests, basically copying what pendulum > > >> does > > >> > to > > >> > > > > check > > >> > > > > > > its behavior. The reason is that native implementations in > > the > > >> > past > > >> > > > > made > > >> > > > > > > serious mistakes and I do not put a lot of trust in them. > > >> > > > > > > > > >> > > > > > > An abstraction or vendoring in could be alternatives, but > > they > > >> > > bring > > >> > > > > > their > > >> > > > > > > own significant challenges. > > >> > > > > > > > > >> > > > > > > I have re-raised the issue here: > > >> > > > > > > > > >> > > > > > > https://github.com/sdispater/pendulum/issues/753 > > >> > > > > > > > > >> > > > > > > The upside is that it seems the amount of issues with the > > >> beta is > > >> > > > > > limited, > > >> > > > > > > so hopefully the maintainers can spend a couple of cycles > to > > >> > > address > > >> > > > > > them. > > >> > > > > > > > > >> > > > > > > Bolke > > >> > > > > > > > > >> > > > > > > On Fri, 17 Nov 2023 at 10:28, Andrey Anshin < > > >> > > > [email protected]> > > >> > > > > > > wrote: > > >> > > > > > > > > >> > > > > > > > There is no changes in stable pendulum so let's try to > > >> continue > > >> > > > this > > >> > > > > > > > discussion and start think about "Plan B" > > >> > > > > > > > > > >> > > > > > > > Just a reminder: > > >> > > > > > > > - pendulum 2.1.2 released 3 years ago (at the time > Airflow > > >> > > 1.10.x) > > >> > > > > > > > - pendulum 2 doesn't work well in Python 3.12, this is a > > >> > > > showstopper > > >> > > > > > for > > >> > > > > > > > the support Python 3.12 > > >> > > > > > > > - pendulum 2 have memory leaks ( > > >> > > > > > > > https://github.com/sdispater/pendulum/issues/720), and > > >> Airflow > > >> > > use > > >> > > > > > > > approach > > >> > > > > > > > to achieve this leaks, especially in K8S executor > > >> > > > > > > > - pendulum 2 doesn't use system tzdata by default, but > we > > >> have > > >> > a > > >> > > > > > > > workaround, thanks Bolke for the documentation > > >> > > > > > > > - pendulum it is a core dependency of Airflow, and I > guess > > >> we > > >> > > can't > > >> > > > > > > > remove/replace it without breaking changes. > > >> > > > > > > > > > >> > > > > > > > So my proposal if things won't change in the near future > > >> then > > >> > we > > >> > > > need > > >> > > > > > to > > >> > > > > > > > start removing the pendulum from the core and replace it > > >> with > > >> > the > > >> > > > > > native > > >> > > > > > > > datetime / zoneinfo / tzinfo. But maybe we have another > > way > > >> to > > >> > > > > resolve > > >> > > > > > > this > > >> > > > > > > > without breaking changes? Because for me it would be a > > >> little > > >> > > weird > > >> > > > > if > > >> > > > > > > > removal pendulum would be a main driver for release > > Airflow > > >> 3 > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > ---- > > >> > > > > > > > Best Wishes > > >> > > > > > > > *Andrey Anshin* > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > On Thu, 28 Sept 2023 at 13:01, Andrey Anshin < > > >> > > > > [email protected] > > >> > > > > > > > > >> > > > > > > > wrote: > > >> > > > > > > > > > >> > > > > > > > > This discussion is more about the known problem of > > >> pendulum > > >> > and > > >> > > > how > > >> > > > > > we > > >> > > > > > > > > could deal with it and maybe how we (as Community) > might > > >> help > > >> > > > > autor. > > >> > > > > > > > > > > >> > > > > > > > > The library is mostly supported by a single author > > >> Sébastien > > >> > > > > Eustace > > >> > > > > > ( > > >> > > > > > > > > https://github.com/sdispater) and it seems like we > bump > > >> into > > >> > > the > > >> > > > > > > > > situation which is described in xkcd #2347 ( > > >> > > > > > > > > https://imgs.xkcd.com/comics/dependency.png). To be > > >> honest > > >> > it > > >> > > is > > >> > > > > not > > >> > > > > > > > > something new when library mainly supported by one > > author > > >> so > > >> > > > there > > >> > > > > is > > >> > > > > > > > > always a risk that the library will no longer be > > >> supported / > > >> > > > > > abandoned > > >> > > > > > > > > And if takes in account that pendulum provides core > > >> > > functionality > > >> > > > > in > > >> > > > > > > > > Airflow it could have dramatical impact in the future. > > >> > > > > > > > > > > >> > > > > > > > > Pendulum is a really nice library which helps a lot of > > >> > > developers > > >> > > > > to > > >> > > > > > > work > > >> > > > > > > > > with dates/datetimes. However there is one major > > problem, > > >> the > > >> > > > last > > >> > > > > > > > release > > >> > > > > > > > > of this library happened more than 3 years ago ( > > >> > > > > > > > > https://pypi.org/project/pendulum/#history) in the > time > > >> when > > >> > > > > Airflow > > >> > > > > > > > > 1.10.11 was released > > >> > > > > > > > > > > >> > > > > > > > > Fortunately, the project is not abandoned and on a > > regular > > >> > > basis > > >> > > > > > > commits > > >> > > > > > > > > add into the master branch. However these commits are > > not > > >> > > > included > > >> > > > > > into > > >> > > > > > > > any > > >> > > > > > > > > final release and that's why some things related to > > >> datetime > > >> > > > don't > > >> > > > > > work > > >> > > > > > > > as > > >> > > > > > > > > expected in Airflow. There are list of known (for me) > > >> issues > > >> > > > which > > >> > > > > > are > > >> > > > > > > > > affect Airflow > > >> > > > > > > > > > > >> > > > > > > > > *Memory Leak on parse*: > > >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/720, > > this > > >> one > > >> > > > fixed > > >> > > > > > 2 > > >> > > > > > > > > years ago but not available yet ( > > >> > > > > > > > > https://github.com/sdispater/pendulum/pull/563). > Since > > we > > >> > use > > >> > > > > parse > > >> > > > > > > > dates > > >> > > > > > > > > in airflow codebase: datetime parameters and datetime > in > > >> logs > > >> > > > this > > >> > > > > > one > > >> > > > > > > > > could be a reason for memory leakage in Airflow: > > >> > > > > > > > > - https://github.com/apache/airflow/discussions/24694 > > >> > > > > > > > > - https://github.com/apache/airflow/discussions/28597 > > >> > > > > > > > > > > >> > > > > > > > > *Incorrect time zones*, known issues and should be > > already > > >> > > fixed > > >> > > > in > > >> > > > > > > > > master branch > > >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/700, > > >> Mexico > > >> > do > > >> > > > not > > >> > > > > > use > > >> > > > > > > > DST > > >> > > > > > > > > anymore > > >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/706, > > Egypt > > >> > > > > reinstate > > >> > > > > > > DST > > >> > > > > > > > > > > >> > > > > > > > > We add clarification in > > >> > > > > https://github.com/apache/airflow/pull/30467 > > >> > > > > > , > > >> > > > > > > > > however it seems like there is no other way rather > than > > >> > > patching > > >> > > > > > > Pendulum > > >> > > > > > > > > right now. > > >> > > > > > > > > > > >> > > > > > > > > All these issues should be solved as soon as pendulum > 3 > > is > > >> > > > > released. > > >> > > > > > > The > > >> > > > > > > > > current announced estimation is end of september/ > > >> beginning > > >> > of > > >> > > > > > October: > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > > https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677 > > >> > > > > > > > > > > >> > > > > > > > > So in theory we would have a fixed version of pendulum > > >> soon, > > >> > > and > > >> > > > it > > >> > > > > > > might > > >> > > > > > > > > break something in Airflow but from my point of view > it > > is > > >> > > better > > >> > > > > > than > > >> > > > > > > > > current status. > > >> > > > > > > > > > > >> > > > > > > > > However there might be a situation where the release > of > > >> the > > >> > > > > pendulum > > >> > > > > > > > would > > >> > > > > > > > > be postponed, so maybe better to have a backup plan. > > What > > >> > could > > >> > > > we > > >> > > > > do > > >> > > > > > > in > > >> > > > > > > > > this case? > > >> > > > > > > > > > > >> > > > > > > > > Maybe we should start to use zoneinfo.ZoneInfo instead > > of > > >> > > > pendulum > > >> > > > > > > > > datetime? > > https://github.com/apache/airflow/issues/19450 > > >> > > > > > > > > Pros: > > >> > > > > > > > > - stdlib (python 3.9+) > > >> > > > > > > > > - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo > > >> > > > > > > > > > > >> > > > > > > > > Cons: > > >> > > > > > > > > - Current serialization model can't deal with backport > > >> > > packages. > > >> > > > > E.g. > > >> > > > > > > > > timezone which are serialized in backport_zoneinfo > can't > > >> be > > >> > > > > > > deserialized > > >> > > > > > > > in > > >> > > > > > > > > zoneinfo > > >> > > > > > > > > > > >> > > > > > > > > Maybe we should replace parse datetime with another > > >> solution. > > >> > > > Does > > >> > > > > > > anyone > > >> > > > > > > > > know a good replacement? > > >> > > > > > > > > > > >> > > > > > > > > Maybe someone from Airflow Community could propose > their > > >> help > > >> > > > with > > >> > > > > > > > > maintenance of library: > > >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/590 > > >> > > > > > > > > > > >> > > > > > > > > Maybe we should get rid of the pendulum at all, as a > > last > > >> > > resort > > >> > > > > > > > solution. > > >> > > > > > > > > I can't imagine how we could do that, because a lot of > > >> stuff > > >> > > > > depends > > >> > > > > > on > > >> > > > > > > > the > > >> > > > > > > > > pendulum and removing it would be a breaking change. > > >> > > > > > > > > > > >> > > > > > > > > ---- > > >> > > > > > > > > Best Wishes > > >> > > > > > > > > *Andrey Anshin* > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > -- > > >> > > > > > > > > >> > > > > > > -- > > >> > > > > > > Bolke de Bruin > > >> > > > > > > [email protected] > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > >
