"infamous fork of " should be "fork of infamous Akka" - just to be clear :)
On Wed, Nov 29, 2023 at 2:57 AM Jarek Potiuk <[email protected]> wrote: > It's a very rare occurrence that ASF accepts forks. Usually it has to be > willingly donated by those who own the IP rights, - mainly because of > trademark issues. Licence is one thing, but carrying things like the name > of the project is not possible without clearing IP and donating the > project. > > It happened last time last year - with Pekko project - infamous fork or > Akka after Lightbend changed licensing for Akka - > https://www.lightbend.com/blog/why-we-are-changing-the-license-for-akka > making it impossible to be used in ASF projects with the new licence. But > it was a really "strong" case and the "we do not accept forks usually" was > quite a contentious issue (I followed the discussion - it was fascinating > actually). It tool a LOT of time and effort for those who led the effort > eventually - and also mostly because things had to be crystal-clear and not > "litigable" because the other side had some business reasons to make the > licensing changes and they did not understood why it is impossible to use > Akka as-is even with all the exclusions they offered to ASF - and > effectively it was a "hostile" move. > > I don't think it's "worth" it in this case to be honest - especially if > the maintainers are effectively "silent" and seem to ignore problems of one > of the really serious users they have. If they were willing to cooperate, > we could do a lot, but when we are left alone, I think it will be far more > convenient for us to look for alternatives - to be honest. > > J. > > > On Wed, Nov 29, 2023 at 2:46 AM Austin Bennett <[email protected]> wrote: > >> An option would be to fork Pendulum? It is MIT Licensed, I don't know >> whether that poses problems to get in ASF? >> >> If forking (?) which is somewhat non-ideal, would we want that 'in' >> airflow? >> >> If not 'in' airflow, I wonder if ASF incubator would accept a forked >> project? [ anecdotally Linux Foundation has incubated Starrocks which is >> a >> fork of Apache Doris, so there's precedence for this to be OK in the wider >> OSS ecosystem ]. >> >> >> On Tue, Nov 28, 2023 at 6:41 PM Jarek Potiuk <[email protected]> wrote: >> >> > I have not heard back from maintainers, just a comment from someone else >> > who suggested donating a pendulum to the ASF (which is kinda interesting >> > idea). I followed up, let's see. I think if we do not hear back and >> there >> > will be another week or two where "Pendulum 3 supporting 3.12 support >> "is >> > coming" but no-one knows when, effectively means that we should look >> for an >> > alternative like NOW. >> > >> > J. >> > >> > >> > >> > On Wed, Nov 22, 2023 at 3:27 AM Daniel Standish >> > <[email protected]> wrote: >> > >> > > Thanks Jarek >> > > >> > > On Tue, Nov 21, 2023, 4:34 PM Jarek Potiuk <[email protected]> wrote: >> > > >> > > > I think we miss important insight - straight from the source. >> > > > >> > > > I believe it's time to be candid and simply ask questions for the >> > future >> > > of >> > > > Pendulum directly where we should - ie. we should just ask >> > maintainers. >> > > > >> > > > I've just started a very candid and open discussion there - >> > > > https://github.com/sdispater/pendulum/discussions/771 . >> > > > >> > > > I linked back to this discussion and explained where we are coming >> > from. >> > > I >> > > > even offered an option, that maybe - if they accept some help and >> would >> > > be >> > > > open to make a few of us maintainers of Pendulum, that I will ask if >> > some >> > > > of the maintainers here in Airflow would like to step up (I would be >> > > > willing to - for one) and help to at least move Pendulum through >> 3.12 >> > and >> > > > maybe keep it running for as long as it will be needed to maintain >> it. >> > > > >> > > > I think if they would be open to some of us helping them - this >> might >> > be >> > > > actually the "cheapest" option for us to be honest at least in a >> > mid-term >> > > > if we could gain influence and even take part and speed up releases >> and >> > > > merging of issues that are blocking us (or would be blocking us in >> the >> > > > future). >> > > > >> > > > I tried to be very friendly but candid and direct on what kind of >> > > problems >> > > > it creates us and also expressed our thanks for supporting us for so >> > long >> > > > (I do believe they deserve it). I think the sole (pretty much) >> > > maintainer >> > > > of Pendulum might really have a hard time maintaining and keeping >> it up >> > > for >> > > > years - without being paid for and thanked. I hope - maybe all that >> > they >> > > > need are a few words of encouragement and thanks and realising that >> > > others >> > > > depend on their work - but also seeing that they are understood, >> > > empathised >> > > > with and that there are viable alternatives they might follow might >> be >> > > > helpful for them to be also candid and respond. I hope my intentions >> > will >> > > > be understood - that I am not complaining, but even trying to help. >> > > > >> > > > Let\s see where it gets us. >> > > > >> > > > J. >> > > > >> > > > On Fri, Nov 17, 2023 at 5:32 PM Andrey Anshin < >> > [email protected]> >> > > > wrote: >> > > > >> > > > > Just to clarify I'd like us to consider the possibility that no >> new >> > > > > pendulum would be released or released at the end of 2024, like a >> > > > > pessimistic scenario: >> > > > > - What should we do in this case? >> > > > > - Work out a backup plan. >> > > > > >> > > > > >> > > > > ---- >> > > > > Best Wishes >> > > > > *Andrey Anshin* >> > > > > >> > > > > >> > > > > >> > > > > On Fri, 17 Nov 2023 at 16:33, Jarek Potiuk <[email protected]> >> wrote: >> > > > > >> > > > > > Also I think TP - had a document in the past (years ago) >> > > describing a >> > > > > > draft of a more complete alternative we can take to approach >> > datetime >> > > > vs. >> > > > > > pendulum dichotomy. I cannot easily find the document and >> > discussion >> > > - >> > > > > but >> > > > > > I do remember it was proposing some interesting changes in the >> > > approach >> > > > > of >> > > > > > Airflow to have an abstraction layer on top of it (as far as I >> > > > remember). >> > > > > > Maybe we can resurrect that idea if TP might find the proposal ? >> > > > > > >> > > > > > On Fri, Nov 17, 2023 at 1:06 PM Bolke de Bruin < >> [email protected]> >> > > > > wrote: >> > > > > > >> > > > > > > Hi, >> > > > > > > >> > > > > > > I agree that the current speed of development of Pendulum >> leaves >> > > > > > something >> > > > > > > to be desired. However, I think we should not underestimate >> the >> > > > effort >> > > > > of >> > > > > > > replacing it. It is not just a matter of >> %s/pendulum/datetime/g >> > so >> > > to >> > > > > > say. >> > > > > > > If we are *truly* thinking about moving to native datetime / >> > > zoneinfo >> > > > > etc >> > > > > > > we need *extensive* tests, basically copying what pendulum >> does >> > to >> > > > > check >> > > > > > > its behavior. The reason is that native implementations in the >> > past >> > > > > made >> > > > > > > serious mistakes and I do not put a lot of trust in them. >> > > > > > > >> > > > > > > An abstraction or vendoring in could be alternatives, but they >> > > bring >> > > > > > their >> > > > > > > own significant challenges. >> > > > > > > >> > > > > > > I have re-raised the issue here: >> > > > > > > >> > > > > > > https://github.com/sdispater/pendulum/issues/753 >> > > > > > > >> > > > > > > The upside is that it seems the amount of issues with the >> beta is >> > > > > > limited, >> > > > > > > so hopefully the maintainers can spend a couple of cycles to >> > > address >> > > > > > them. >> > > > > > > >> > > > > > > Bolke >> > > > > > > >> > > > > > > On Fri, 17 Nov 2023 at 10:28, Andrey Anshin < >> > > > [email protected]> >> > > > > > > wrote: >> > > > > > > >> > > > > > > > There is no changes in stable pendulum so let's try to >> continue >> > > > this >> > > > > > > > discussion and start think about "Plan B" >> > > > > > > > >> > > > > > > > Just a reminder: >> > > > > > > > - pendulum 2.1.2 released 3 years ago (at the time Airflow >> > > 1.10.x) >> > > > > > > > - pendulum 2 doesn't work well in Python 3.12, this is a >> > > > showstopper >> > > > > > for >> > > > > > > > the support Python 3.12 >> > > > > > > > - pendulum 2 have memory leaks ( >> > > > > > > > https://github.com/sdispater/pendulum/issues/720), and >> Airflow >> > > use >> > > > > > > > approach >> > > > > > > > to achieve this leaks, especially in K8S executor >> > > > > > > > - pendulum 2 doesn't use system tzdata by default, but we >> have >> > a >> > > > > > > > workaround, thanks Bolke for the documentation >> > > > > > > > - pendulum it is a core dependency of Airflow, and I guess >> we >> > > can't >> > > > > > > > remove/replace it without breaking changes. >> > > > > > > > >> > > > > > > > So my proposal if things won't change in the near future >> then >> > we >> > > > need >> > > > > > to >> > > > > > > > start removing the pendulum from the core and replace it >> with >> > the >> > > > > > native >> > > > > > > > datetime / zoneinfo / tzinfo. But maybe we have another way >> to >> > > > > resolve >> > > > > > > this >> > > > > > > > without breaking changes? Because for me it would be a >> little >> > > weird >> > > > > if >> > > > > > > > removal pendulum would be a main driver for release Airflow >> 3 >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > ---- >> > > > > > > > Best Wishes >> > > > > > > > *Andrey Anshin* >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > On Thu, 28 Sept 2023 at 13:01, Andrey Anshin < >> > > > > [email protected] >> > > > > > > >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > This discussion is more about the known problem of >> pendulum >> > and >> > > > how >> > > > > > we >> > > > > > > > > could deal with it and maybe how we (as Community) might >> help >> > > > > autor. >> > > > > > > > > >> > > > > > > > > The library is mostly supported by a single author >> Sébastien >> > > > > Eustace >> > > > > > ( >> > > > > > > > > https://github.com/sdispater) and it seems like we bump >> into >> > > the >> > > > > > > > > situation which is described in xkcd #2347 ( >> > > > > > > > > https://imgs.xkcd.com/comics/dependency.png). To be >> honest >> > it >> > > is >> > > > > not >> > > > > > > > > something new when library mainly supported by one author >> so >> > > > there >> > > > > is >> > > > > > > > > always a risk that the library will no longer be >> supported / >> > > > > > abandoned >> > > > > > > > > And if takes in account that pendulum provides core >> > > functionality >> > > > > in >> > > > > > > > > Airflow it could have dramatical impact in the future. >> > > > > > > > > >> > > > > > > > > Pendulum is a really nice library which helps a lot of >> > > developers >> > > > > to >> > > > > > > work >> > > > > > > > > with dates/datetimes. However there is one major problem, >> the >> > > > last >> > > > > > > > release >> > > > > > > > > of this library happened more than 3 years ago ( >> > > > > > > > > https://pypi.org/project/pendulum/#history) in the time >> when >> > > > > Airflow >> > > > > > > > > 1.10.11 was released >> > > > > > > > > >> > > > > > > > > Fortunately, the project is not abandoned and on a regular >> > > basis >> > > > > > > commits >> > > > > > > > > add into the master branch. However these commits are not >> > > > included >> > > > > > into >> > > > > > > > any >> > > > > > > > > final release and that's why some things related to >> datetime >> > > > don't >> > > > > > work >> > > > > > > > as >> > > > > > > > > expected in Airflow. There are list of known (for me) >> issues >> > > > which >> > > > > > are >> > > > > > > > > affect Airflow >> > > > > > > > > >> > > > > > > > > *Memory Leak on parse*: >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/720, this >> one >> > > > fixed >> > > > > > 2 >> > > > > > > > > years ago but not available yet ( >> > > > > > > > > https://github.com/sdispater/pendulum/pull/563). Since we >> > use >> > > > > parse >> > > > > > > > dates >> > > > > > > > > in airflow codebase: datetime parameters and datetime in >> logs >> > > > this >> > > > > > one >> > > > > > > > > could be a reason for memory leakage in Airflow: >> > > > > > > > > - https://github.com/apache/airflow/discussions/24694 >> > > > > > > > > - https://github.com/apache/airflow/discussions/28597 >> > > > > > > > > >> > > > > > > > > *Incorrect time zones*, known issues and should be already >> > > fixed >> > > > in >> > > > > > > > > master branch >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/700, >> Mexico >> > do >> > > > not >> > > > > > use >> > > > > > > > DST >> > > > > > > > > anymore >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/706, Egypt >> > > > > reinstate >> > > > > > > DST >> > > > > > > > > >> > > > > > > > > We add clarification in >> > > > > https://github.com/apache/airflow/pull/30467 >> > > > > > , >> > > > > > > > > however it seems like there is no other way rather than >> > > patching >> > > > > > > Pendulum >> > > > > > > > > right now. >> > > > > > > > > >> > > > > > > > > All these issues should be solved as soon as pendulum 3 is >> > > > > released. >> > > > > > > The >> > > > > > > > > current announced estimation is end of september/ >> beginning >> > of >> > > > > > October: >> > > > > > > > > >> > > > > > > >> > > > > >> > > >> https://github.com/sdispater/pendulum/issues/600#issuecomment-1711299677 >> > > > > > > > > >> > > > > > > > > So in theory we would have a fixed version of pendulum >> soon, >> > > and >> > > > it >> > > > > > > might >> > > > > > > > > break something in Airflow but from my point of view it is >> > > better >> > > > > > than >> > > > > > > > > current status. >> > > > > > > > > >> > > > > > > > > However there might be a situation where the release of >> the >> > > > > pendulum >> > > > > > > > would >> > > > > > > > > be postponed, so maybe better to have a backup plan. What >> > could >> > > > we >> > > > > do >> > > > > > > in >> > > > > > > > > this case? >> > > > > > > > > >> > > > > > > > > Maybe we should start to use zoneinfo.ZoneInfo instead of >> > > > pendulum >> > > > > > > > > datetime? https://github.com/apache/airflow/issues/19450 >> > > > > > > > > Pros: >> > > > > > > > > - stdlib (python 3.9+) >> > > > > > > > > - In pendulum 3.0 Timezone based on zoneinfo.Zoneinfo >> > > > > > > > > >> > > > > > > > > Cons: >> > > > > > > > > - Current serialization model can't deal with backport >> > > packages. >> > > > > E.g. >> > > > > > > > > timezone which are serialized in backport_zoneinfo can't >> be >> > > > > > > deserialized >> > > > > > > > in >> > > > > > > > > zoneinfo >> > > > > > > > > >> > > > > > > > > Maybe we should replace parse datetime with another >> solution. >> > > > Does >> > > > > > > anyone >> > > > > > > > > know a good replacement? >> > > > > > > > > >> > > > > > > > > Maybe someone from Airflow Community could propose their >> help >> > > > with >> > > > > > > > > maintenance of library: >> > > > > > > > > - https://github.com/sdispater/pendulum/issues/590 >> > > > > > > > > >> > > > > > > > > Maybe we should get rid of the pendulum at all, as a last >> > > resort >> > > > > > > > solution. >> > > > > > > > > I can't imagine how we could do that, because a lot of >> stuff >> > > > > depends >> > > > > > on >> > > > > > > > the >> > > > > > > > > pendulum and removing it would be a breaking change. >> > > > > > > > > >> > > > > > > > > ---- >> > > > > > > > > Best Wishes >> > > > > > > > > *Andrey Anshin* >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > -- >> > > > > > > >> > > > > > > -- >> > > > > > > Bolke de Bruin >> > > > > > > [email protected] >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
