Yeah. Agree it's a challenge but I think it's also an opportunity to fix those - because whether we admit it, or not - we already have those problems - regardless of Pendulum's use.
If you look at the issues (especially ones created around DST time) - there are always one or two issues (even in latest versions of Airflow) where DST transition is causing people problems. And they are rarely solved/addressed. Example from last October: https://github.com/apache/airflow/issues/35272 Example from May 2022: https://github.com/apache/airflow/issues/23400 And Example from March 2020 ( which might or might not be fixed by the open PR): https://github.com/apache/airflow/issues/7999 J. On Fri, Dec 1, 2023 at 5:41 PM Bolke de Bruin <[email protected]> wrote: > The alternative is to vendor in or fork pendulum as part of the Airflow > project. Also a major undertaking but maybe less effort. > > We shouldn't just address this as a technical challenge. The question > becomes: does that combination fornally do DST transitions correctly, does > it do leap seconds correctly, does it add dates correctly, does it do > timezone traversal correctly and many many other things. > > So if we switch to the datetime + zone info + timezone we need to have a > test suite that verifies this. Otherwise we potentially expose our users to > different behavior IN their data. Not just a schedule by in their SQL or in > some scripting where logical date / execution date is used or where date > arithmetic is applied. > > So it is not changing one car for another. It is changing the means of > transportation and not knowing if you can get to all the places you want / > need to go as before. > > Honestly, I'm a bit scared (I do see the lack of responsiveness). I hope > someone can take that away. > > Bolke > > > Sent from my iPhone > > > On 1 Dec 2023, at 13:39, Jarek Potiuk <[email protected]> wrote: > > > > Agree. I think this is the best way forward.- we have to bite the > bullet of > > potential backwards compatibility issues and separating out a compat > layer > > where we could optionally use pendulum instead would be the most "easy" > > approach for our users. Most of them would not even notice if we do a > good > > job, but those who somehow depend on pendulum objects would get a > > possibility to switch back. > > > >> On Fri, Dec 1, 2023 at 1:09 PM Andrey Anshin <[email protected]> > >> wrote: > >> > >> IMHO, if we would like to replace pendulum to alternatives, the better > >> alternative would be combination of datetime.datetime + zoneinfo + > >> datetime.timezone > >> > >> We still do hacks around pendulum functional, which makes hacks around > >> datetime.datetime, so in my perspective better remove redundant layers > and > >> use native objects directly. > >> > >> What confuses me the most is how we get around type changes in Tasks > >> Context. All dates return as pendulum.DateTime for a very long time > >> (literally for ages) and it is probably could be classified as breaking > >> changes, because it might break quite a bit users pipelines. It still > could > >> be resolved as optional pendulum dependency + config option for datetime > >> types in context, which might help to get familiar types for a while. > >> > >> > >>> On Wed, 29 Nov 2023 at 05:58, Jarek Potiuk <[email protected]> wrote: > >>> > >>> "infamous fork of " should be "fork of infamous Akka" - just to be > clear > >> :) > >>> > >>>> On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk <[email protected]> > wrote: > >>> > >>>> It's a very rare occurrence that ASF accepts forks. Usually it has to > >> be > >>>> willingly donated by those who own the IP rights, - mainly because of > >>>> trademark issues. Licence is one thing, but carrying things like the > >> name > >>>> of the project is not possible without clearing IP and donating the > >>>> project. > >>>> > >>>> It happened last time last year - with Pekko project - infamous fork > or > >>>> Akka after Lightbend changed licensing for Akka - > >>>> > >> https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka > >>>> making it impossible to be used in ASF projects with the new licence. > >> But > >>>> it was a really "strong" case and the "we do not accept forks usually" > >>> was > >>>> quite a contentious issue (I followed the discussion - it was > >> fascinating > >>>> actually). It tool a LOT of time and effort for those who led the > >> effort > >>>> eventually - and also mostly because things had to be crystal-clear > and > >>> not > >>>> "litigable" because the other side had some business reasons to make > >> the > >>>> licensing changes and they did not understood why it is impossible to > >> use > >>>> Akka as-is even with all the exclusions they offered to ASF - and > >>>> effectively it was a "hostile" move. > >>>> > >>>> I don't think it's "worth" it in this case to be honest - especially > if > >>>> the maintainers are effectively "silent" and seem to ignore problems > of > >>> one > >>>> of the really serious users they have. If they were willing to > >> cooperate, > >>>> we could do a lot, but when we are left alone, I think it will be far > >>> more > >>>> convenient for us to look for alternatives - to be honest. > >>>> > >>>> J. > >>>> > >>>> > >>>> On Wed, Nov 29, 2023 at 2:46 AM Austin Bennett <[email protected]> > >>> wrote: > >>>> > >>>>> An option would be to fork Pendulum? It is MIT Licensed, I don't > know > >>>>> whether that poses problems to get in ASF? > >>>>> > >>>>> If forking (?) which is somewhat non-ideal, would we want that 'in' > >>>>> airflow? > >>>>> > >>>>> If not 'in' airflow, I wonder if ASF incubator would accept a forked > >>>>> project? [ anecdotally Linux Foundation has incubated Starrocks > which > >>> is > >>>>> a > >>>>> fork of Apache Doris, so there's precedence for this to be OK in the > >>> wider > >>>>> OSS ecosystem ]. > >>>>> > >>>>> > >>>>> On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk <[email protected]> > >> wrote: > >>>>> > >>>>>> I have not heard back from maintainers, just a comment from someone > >>> else > >>>>>> who suggested donating a pendulum to the ASF (which is kinda > >>> interesting > >>>>>> idea). I followed up, let's see. I think if we do not hear back and > >>>>> there > >>>>>> will be another week or two where "Pendulum 3 supporting 3.12 > >> support > >>>>> "is > >>>>>> coming" but no-one knows when, effectively means that we should look > >>>>> for an > >>>>>> alternative like NOW. > >>>>>> > >>>>>> J. > >>>>>> > >>>>>> > >>>>>> > >>>>>> On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish > >>>>>> <[email protected]> wrote: > >>>>>> > >>>>>>> Thanks Jarek > >>>>>>> > >>>>>>> On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <[email protected]> > >>> wrote: > >>>>>>> > >>>>>>>> I think we miss important insight - straight from the source. > >>>>>>>> > >>>>>>>> I believe it's time to be candid and simply ask questions for > >> the > >>>>>> future > >>>>>>> of > >>>>>>>> Pendulum directly where we should - ie. we should just ask > >>>>>> maintainers. > >>>>>>>> > >>>>>>>> I've just started a very candid and open discussion there - > >>>>>>>> https://github.com/sdispater/pendulum/discussions/771 . > >>>>>>>> > >>>>>>>> I linked back to this discussion and explained where we are > >> coming > >>>>>> from. > >>>>>>> I > >>>>>>>> even offered an option, that maybe - if they accept some help > >> and > >>>>> would > >>>>>>> be > >>>>>>>> open to make a few of us maintainers of Pendulum, that I will > >> ask > >>> if > >>>>>> some > >>>>>>>> of the maintainers here in Airflow would like to step up (I > >> would > >>> be > >>>>>>>> willing to - for one) and help to at least move Pendulum through > >>>>> 3.12 > >>>>>> and > >>>>>>>> maybe keep it running for as long as it will be needed to > >> maintain > >>>>> it. > >>>>>>>> > >>>>>>>> I think if they would be open to some of us helping them - this > >>>>> might > >>>>>> be > >>>>>>>> actually the "cheapest" option for us to be honest at least in a > >>>>>> mid-term > >>>>>>>> if we could gain influence and even take part and speed up > >>> releases > >>>>> and > >>>>>>>> merging of issues that are blocking us (or would be blocking us > >> in > >>>>> the > >>>>>>>> future). > >>>>>>>> > >>>>>>>> I tried to be very friendly but candid and direct on what kind > >> of > >>>>>>> problems > >>>>>>>> it creates us and also expressed our thanks for supporting us > >> for > >>> so > >>>>>> long > >>>>>>>> (I do believe they deserve it). I think the sole (pretty much) > >>>>>>> maintainer > >>>>>>>> of Pendulum might really have a hard time maintaining and > >> keeping > >>>>> it up > >>>>>>> for > >>>>>>>> years - without being paid for and thanked. I hope - maybe all > >>> that > >>>>>> they > >>>>>>>> need are a few words of encouragement and thanks and realising > >>> that > >>>>>>> others > >>>>>>>> depend on their work - but also seeing that they are understood, > >>>>>>> empathised > >>>>>>>> with and that there are viable alternatives they might follow > >>> might > >>>>> be > >>>>>>>> helpful for them to be also candid and respond. I hope my > >>> intentions > >>>>>> will > >>>>>>>> be understood - that I am not complaining, but even trying to > >>> help. > >>>>>>>> > >>>>>>>> Let\s see where it gets us. > >>>>>>>> > >>>>>>>> J. > >>>>>>>> > >>>>>>>> On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin < > >>>>>> [email protected]> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Just to clarify I'd like us to consider the possibility that > >> no > >>>>> new > >>>>>>>>> pendulum would be released or released at the end of 2024, > >> like > >>> a > >>>>>>>>> pessimistic scenario: > >>>>>>>>> - What should we do in this case? > >>>>>>>>> - Work out a backup plan. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ---- > >>>>>>>>> Best Wishes > >>>>>>>>> *Andrey Anshin* > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <[email protected]> > >>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Also I think TP - had a document in the past (years ago) > >>>>>>> describing a > >>>>>>>>>> draft of a more complete alternative we can take to approach > >>>>>> datetime > >>>>>>>> vs. > >>>>>>>>>> pendulum dichotomy. I cannot easily find the document and > >>>>>> discussion > >>>>>>> - > >>>>>>>>> but > >>>>>>>>>> I do remember it was proposing some interesting changes in > >> the > >>>>>>> approach > >>>>>>>>> of > >>>>>>>>>> Airflow to have an abstraction layer on top of it (as far > >> as I > >>>>>>>> remember). > >>>>>>>>>> Maybe we can resurrect that idea if TP might find the > >>> proposal ? > >>>>>>>>>> > >>>>>>>>>> On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin < > >>>>> [email protected]> > >>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi, > >>>>>>>>>>> > >>>>>>>>>>> I agree that the current speed of development of Pendulum > >>>>> leaves > >>>>>>>>>> something > >>>>>>>>>>> to be desired. However, I think we should not > >> underestimate > >>>>> the > >>>>>>>> effort > >>>>>>>>> of > >>>>>>>>>>> replacing it. It is not just a matter of > >>>>> %s/pendulum/datetime/g > >>>>>> so > >>>>>>> to > >>>>>>>>>> say. > >>>>>>>>>>> If we are *truly* thinking about moving to native > >> datetime / > >>>>>>> zoneinfo > >>>>>>>>> etc > >>>>>>>>>>> we need *extensive* tests, basically copying what pendulum > >>>>> does > >>>>>> to > >>>>>>>>> check > >>>>>>>>>>> its behavior. The reason is that native implementations in > >>> the > >>>>>> past > >>>>>>>>> made > >>>>>>>>>>> serious mistakes and I do not put a lot of trust in them. > >>>>>>>>>>> > >>>>>>>>>>> An abstraction or vendoring in could be alternatives, but > >>> they > >>>>>>> bring > >>>>>>>>>> their > >>>>>>>>>>> own significant challenges. > >>>>>>>>>>> > >>>>>>>>>>> I have re-raised the issue here: > >>>>>>>>>>> > >>>>>>>>>>> https://github.com/sdispater/pendulum/issues/753 > >>>>>>>>>>> > >>>>>>>>>>> The upside is that it seems the amount of issues with the > >>>>> beta is > >>>>>>>>>> limited, > >>>>>>>>>>> so hopefully the maintainers can spend a couple of cycles > >> to > >>>>>>> address > >>>>>>>>>> them. > >>>>>>>>>>> > >>>>>>>>>>> Bolke > >>>>>>>>>>> > >>>>>>>>>>> On Fri, 17 Nov 2023 at 10:28, Andrey Anshin < > >>>>>>>> [email protected]> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> There is no changes in stable pendulum so let's try to > >>>>> continue > >>>>>>>> this > >>>>>>>>>>>> discussion and start think about "Plan B" > >>>>>>>>>>>> > >>>>>>>>>>>> Just a reminder: > >>>>>>>>>>>> - pendulum 2.1.2 released 3 years ago (at the time > >> Airflow > >>>>>>> 1.10.x) > >>>>>>>>>>>> - pendulum 2 doesn't work well in Python 3.12, this is a > >>>>>>>> showstopper > >>>>>>>>>> for > >>>>>>>>>>>> the support Python 3.12 > >>>>>>>>>>>> - pendulum 2 have memory leaks ( > >>>>>>>>>>>> https://github.com/sdispater/pendulum/issues/720), and > >>>>> Airflow > >>>>>>> use > >>>>>>>>>>>> approach > >>>>>>>>>>>> to achieve this leaks, especially in K8S executor > >>>>>>>>>>>> - pendulum 2 doesn't use system tzdata by default, but > >> we > >>>>> have > >>>>>> a > >>>>>>>>>>>> workaround, thanks Bolke for the documentation > >>>>>>>>>>>> - pendulum it is a core dependency of Airflow, and I > >> guess > >>>>> we > >>>>>>> can't > >>>>>>>>>>>> remove/replace it without breaking changes. > >>>>>>>>>>>> > >>>>>>>>>>>> So my proposal if things won't change in the near future > >>>>> then > >>>>>> we > >>>>>>>> need > >>>>>>>>>> to > >>>>>>>>>>>> start removing the pendulum from the core and replace it > >>>>> with > >>>>>> the > >>>>>>>>>> native > >>>>>>>>>>>> datetime / zoneinfo / tzinfo. But maybe we have another > >>> way > >>>>> to > >>>>>>>>> resolve > >>>>>>>>>>> this > >>>>>>>>>>>> without breaking changes? Because for me it would be a > >>>>> little > >>>>>>> weird > >>>>>>>>> if > >>>>>>>>>>>> removal pendulum would be a main driver for release > >>> Airflow > >>>>> 3 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> ---- > >>>>>>>>>>>> Best Wishes > >>>>>>>>>>>> *Andrey Anshin* > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Thu, 28 Sept 2023 at 13:01, Andrey Anshin < > >>>>>>>>> [email protected] > >>>>>>>>>>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> This discussion is more about the known problem of > >>>>> pendulum > >>>>>> and > >>>>>>>> how > >>>>>>>>>> we > >>>>>>>>>>>>> could deal with it and maybe how we (as Community) > >> might > >>>>> help > >>>>>>>>> autor. > >>>>>>>>>>>>> > >>>>>>>>>>>>> The library is mostly supported by a single author > >>>>> Sébastien > >>>>>>>>> Eustace > >>>>>>>>>> ( > >>>>>>>>>>>>> https://github.com/sdispater) and it seems like we > >> bump > >>>>> into > >>>>>>> the > >>>>>>>>>>>>> situation which is described in xkcd #2347 ( > >>>>>>>>>>>>> https://imgs.xkcd.com/comics/dependency.png). To be > >>>>> honest > >>>>>> it > >>>>>>> is > >>>>>>>>> not > >>>>>>>>>>>>> something new when library mainly supported by one > >>> author > >>>>> so > >>>>>>>> there > >>>>>>>>> is > >>>>>>>>>>>>> always a risk that the library will no longer be > >>>>> supported / > >>>>>>>>>> abandoned > >>>>>>>>>>>>> And if takes in account that pendulum provides core > >>>>>>> functionality > >>>>>>>>> in > >>>>>>>>>>>>> Airflow it could have dramatical impact in the future. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Pendulum is a really nice library which helps a lot of > >>>>>>> developers > >>>>>>>>> to > >>>>>>>>>>> work > >>>>>>>>>>>>> with dates/datetimes. However there is one major > >>> problem, > >>>>> the > >>>>>>>> last > >>>>>>>>>>>> release > >>>>>>>>>>>>> of this library happened more than 3 years ago ( > >>>>>>>>>>>>> https://pypi.org/project/pendulum/#history) in the > >> time > >>>>> when > >>>>>>>>> Airflow > >>>>>>>>>>>>> 1.10.11 was released > >>>>>>>>>>>>> > >>>>>>>>>>>>> Fortunately, the project is not abandoned and on a > >>> regular > >>>>>>> basis > >>>>>>>>>>> commits > >>>>>>>>>>>>> add into the master branch. However these commits are > >>> not > >>>>>>>> included > >>>>>>>>>> into > >>>>>>>>>>>> any > >>>>>>>>>>>>> final release and that's why some things related to > >>>>> datetime > >>>>>>>> don't > >>>>>>>>>> work > >>>>>>>>>>>> as > >>>>>>>>>>>>> expected in Airflow. There are list of known (for me) > >>>>> issues > >>>>>>>> which > >>>>>>>>>> are > >>>>>>>>>>>>> affect Airflow > >>>>>>>>>>>>> > >>>>>>>>>>>>> *Memory Leak on parse*: > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/720, > >>> this > >>>>> one > >>>>>>>> fixed > >>>>>>>>>> 2 > >>>>>>>>>>>>> years ago but not available yet ( > >>>>>>>>>>>>> https://github.com/sdispater/pendulum/pull/563). > >> Since > >>> we > >>>>>> use > >>>>>>>>> parse > >>>>>>>>>>>> dates > >>>>>>>>>>>>> in airflow codebase: datetime parameters and datetime > >> in > >>>>> logs > >>>>>>>> this > >>>>>>>>>> one > >>>>>>>>>>>>> could be a reason for memory leakage in Airflow: > >>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/24694 > >>>>>>>>>>>>> - https://github.com/apache/airflow/discussions/28597 > >>>>>>>>>>>>> > >>>>>>>>>>>>> *Incorrect time zones*, known issues and should be > >>> already > >>>>>>> fixed > >>>>>>>> in > >>>>>>>>>>>>> master branch > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/700, > >>>>> Mexico > >>>>>> do > >>>>>>>> not > >>>>>>>>>> use > >>>>>>>>>>>> DST > >>>>>>>>>>>>> anymore > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/706, > >>> Egypt > >>>>>>>>> reinstate > >>>>>>>>>>> DST > >>>>>>>>>>>>> > >>>>>>>>>>>>> We add clarification in > >>>>>>>>> https://github.com/apache/airflow/pull/30467 > >>>>>>>>>> , > >>>>>>>>>>>>> however it seems like there is no other way rather > >> than > >>>>>>> patching > >>>>>>>>>>> Pendulum > >>>>>>>>>>>>> right now. > >>>>>>>>>>>>> > >>>>>>>>>>>>> All these issues should be solved as soon as pendulum > >> 3 > >>> is > >>>>>>>>> released. > >>>>>>>>>>> The > >>>>>>>>>>>>> current announced estimation is end of september/ > >>>>> beginning > >>>>>> of > >>>>>>>>>> October: > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>> > >>> > https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677 > >>>>>>>>>>>>> > >>>>>>>>>>>>> So in theory we would have a fixed version of pendulum > >>>>> soon, > >>>>>>> and > >>>>>>>> it > >>>>>>>>>>> might > >>>>>>>>>>>>> break something in Airflow but from my point of view > >> it > >>> is > >>>>>>> better > >>>>>>>>>> than > >>>>>>>>>>>>> current status. > >>>>>>>>>>>>> > >>>>>>>>>>>>> However there might be a situation where the release > >> of > >>>>> the > >>>>>>>>> pendulum > >>>>>>>>>>>> would > >>>>>>>>>>>>> be postponed, so maybe better to have a backup plan. > >>> What > >>>>>> could > >>>>>>>> we > >>>>>>>>> do > >>>>>>>>>>> in > >>>>>>>>>>>>> this case? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Maybe we should start to use zoneinfo.ZoneInfo instead > >>> of > >>>>>>>> pendulum > >>>>>>>>>>>>> datetime? > >>> https://github.com/apache/airflow/issues/19450 > >>>>>>>>>>>>> Pros: > >>>>>>>>>>>>> - stdlib (python 3.9+) > >>>>>>>>>>>>> - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo > >>>>>>>>>>>>> > >>>>>>>>>>>>> Cons: > >>>>>>>>>>>>> - Current serialization model can't deal with backport > >>>>>>> packages. > >>>>>>>>> E.g. > >>>>>>>>>>>>> timezone which are serialized in backport_zoneinfo > >> can't > >>>>> be > >>>>>>>>>>> deserialized > >>>>>>>>>>>> in > >>>>>>>>>>>>> zoneinfo > >>>>>>>>>>>>> > >>>>>>>>>>>>> Maybe we should replace parse datetime with another > >>>>> solution. > >>>>>>>> Does > >>>>>>>>>>> anyone > >>>>>>>>>>>>> know a good replacement? > >>>>>>>>>>>>> > >>>>>>>>>>>>> Maybe someone from Airflow Community could propose > >> their > >>>>> help > >>>>>>>> with > >>>>>>>>>>>>> maintenance of library: > >>>>>>>>>>>>> - https://github.com/sdispater/pendulum/issues/590 > >>>>>>>>>>>>> > >>>>>>>>>>>>> Maybe we should get rid of the pendulum at all, as a > >>> last > >>>>>>> resort > >>>>>>>>>>>> solution. > >>>>>>>>>>>>> I can't imagine how we could do that, because a lot of > >>>>> stuff > >>>>>>>>> depends > >>>>>>>>>> on > >>>>>>>>>>>> the > >>>>>>>>>>>>> pendulum and removing it would be a breaking change. > >>>>>>>>>>>>> > >>>>>>>>>>>>> ---- > >>>>>>>>>>>>> Best Wishes > >>>>>>>>>>>>> *Andrey Anshin* > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Bolke de Bruin > >>>>>>>>>>> [email protected] > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
