> Beam is also multi language, which adjusts concerns. How does GRPC handle that? Or protos? (I'm sure there are other examples we can pull from...)
I'm not sure those are good projects to look at. They're likely much more concerned with binary compatibility and there's probably not much change in the user-facing API. Arrow is another multi-language project but I don't think we can learn much from it's versioning policy [1], which is much more concerned with binary compatibility than it is with API compatibility (for now). Perhaps one lesson is that they track a separate format version and API version. We could do something similar and have a separate version number for the Beam model protos. I'm not sure if that's relevant for this discussion or not. Spark may be a reasonable comparison since it provides an API in multiple languages, but that doesn't seem to have any bearing on it's versioning policy [2]. It sounds similar to Flink in that every minor release gets backported bugfixes for 18 months, but releases are slower (~6 months) so that's not as much of a burden. Brian [1] https://arrow.apache.org/docs/format/Versioning.html#backward-compatibility [2] https://spark.apache.org/versioning-policy.html On Thu, Apr 8, 2021 at 1:18 PM Robert Bradshaw <rober...@google.com> wrote: > Python (again a language) has a slower release cycle, fairly strict > backwards compatibility stance (with the ability to opt-in before changes > become the default) and clear ownership for maintenance of each minor > version until end-of-life (so each could be considered an LTS release). > https://devguide.python.org/devcycle/ > > Cython is more similar to Beam: best-effort compatibility, no LTS, but as > a code-generater rather than a runtime library a developer is mostly free > to upgrade at their own cadence regardless of the surrounding > ecosystem (including downstream projects that take them on as a > dependency). > > IIRC, Flink supports the latest N (3?) releases, which are infrequent > enough to cover about 12-18 months. > > My take is that Beam should be supportive of LTS releases, but we're not > in a position to commit to it (to the same level we commit to the 6-week > cut-from-head release cycle). But certain users of Beam (which have a large > overlap with the Beam community) could make such commitments as it helps > them (directly or indirectly). Let's give it a try. > > > On Thu, Apr 8, 2021 at 1:00 PM Robert Burke <rob...@frantil.com> wrote: > >> I don't know about other Apache projects but the Go Programming Language >> uses a slower release cadence, two releases a year. Only the last two >> releases are maintained with backports, except for significant security >> issues, which are backported as far back as possible. >> >> Further they divide each release period into clear 3 month "commit" and " >> fix" periods, with the main tree only open to new features in the commit >> phase with the fix phase largely frozen outside of discovered issues. >> >> Beam isn't entirely the same thing as a Programming Language, and >> certainly hasn't had such a rigorous commitment to forward compatibility as >> Go has. There's been a very strong expectation that Go programs must >> continue to compile, and function as expected through new versions of Go, >> and it's standard libraries (and to a lesser extent, arbitrary Go >> packages). But these tenets allow the Go maintainers and contributors to >> avoid LTS as a concept to worry about. >> >> The way I see it, it's not necessarily the number of releases that's the >> problem exactly, but the overall stability of the system. More frequent >> releases "feels" less stable than fewer, even if most of them are >> compatible ona code level. This will vary on the way language ecosystems >> handle dependencies though. >> >> Beam is also multi language, which adjusts concerns. How does GRPC handle >> that? Or protos? (I'm sure there are other examples we can pull from...) >> >> >> On Thu, Apr 8, 2021, 9:31 AM Kenneth Knowles <k...@apache.org> wrote: >> >>> The other obvious question, which I often ask and very rarely research: >>> what do other similar projects do? Does anyone know? >>> >>> On Thu, Apr 8, 2021 at 9:30 AM Kenneth Knowles <k...@apache.org> wrote: >>> >>>> I want to send a short summary of my views. There is a >>>> tension/contradiction that I think is OK: >>>> >>>> - An LTS series becomes more valuable when more stakeholders care for >>>> it. For those who don't care about LTS, the main thing would be to >>>> highlight major issues that are worth backporting. So having it part of how >>>> we think about things is helpful, even for the people who are not using it >>>> or supporting users of it. And cherrypicks are very easy to open. So I >>>> support building consensus around this. >>>> >>>> - Anyone can volunteer and perform a point release for any past >>>> branch. IMO if a stakeholder proposed to do the work, we (the Beam >>>> community) should welcome it. Certainly we have no ability to make anyone >>>> else do the point releases or directly fund any work. So I support >>>> welcoming such a contribution but not making unfunded commitments. >>>> >>>> In practice: I think if CompanyX has a customer who wants LTS support, >>>> CompanyX commits to the customer that they will perform LTS support and >>>> CompanyX can be sure that the Beam community welcomes such a contribution, >>>> and their customer can easily confirm Beam community's public position on >>>> it. >>>> >>>> Kenn >>>> >>>> On Thu, Apr 8, 2021 at 1:05 AM Jan Lukavský <je...@seznam.cz> wrote: >>>> >>>>> Understand that, my point was just that we cannot care about API >>>>> changes >>>>> only. What an LTS has to maintain is binary compatibility of state or >>>>> have any kind of upgrade process without the running Pipeline being >>>>> terminated and completely replaced with another one. Bootstrapping the >>>>> state is what is - from my point of view - the most expensive part of >>>>> upgrade with state-incompatibility. If an update to LTS breaks the >>>>> state >>>>> compatibility, then upgrading to the new (bugfixed) LTS version seems >>>>> to >>>>> be (about) the same pain as upgrading to the newest version of Beam. >>>>> That is what somewhat breaks the "LTS" expectations users might have. >>>>> But yes, fixes which break state might not be that frequent, but are >>>>> possible, that's for sure. In that case, we essentially could not >>>>> release a fix, even if it was critical security fix. My question was >>>>> then if these concerns will be somehow part of the proposed LTS >>>>> support >>>>> process. >>>>> >>>>> Jan >>>>> >>>>> On 4/7/21 4:23 PM, Elliotte Rusty Harold wrote: >>>>> > On Tue, Apr 6, 2021 at 9:43 PM Jan Lukavský <je...@seznam.cz> wrote: >>>>> >> Hi, >>>>> >> do we know what is the reason users stay on an older version of >>>>> Beam? My guess would be that it is not related to API changes, but more >>>>> likely to state incompatibility. Maybe if we could figure out a way which >>>>> would enable a smooth migration of state (and timers) between Beam >>>>> versions, that might help? The migration would probably have to be >>>>> runner-dependent, but Beam might offer some tooling to make this easier. >>>>> One example would be coder evolution, where we currently do not have the >>>>> option of "reading old way, writing new way" with some >>>>> "coder-version-registry". I suppose there might have been a discussion >>>>> about this in the past, does anyone know of any conclusion? >>>>> >> >>>>> > Different customers have different requirements. The projects and >>>>> > teams I'm talking to simply don't want to invest engineering time in >>>>> > migrating as often as they have to now. They want to be able to ship >>>>> a >>>>> > working product and not worry about it for a year or more. They >>>>> > recognize the possibility of unforeseen bugs and security issues that >>>>> > might require an update. However they'd like that update to be >>>>> minimal >>>>> > and not require them to update their own code. Nor do they want to >>>>> run >>>>> > the risk of performance regressions and newly introduced bugs and >>>>> > security issues. >>>>> > >>>>> >>>>
