As Kenn points clearly, everyone can do an Apache release of an earlier version, so this should cover most maintenance fixes for old versions. So any person (or company) can decide to work on supporting one version.
The real deal of having a LTS "backed by the community" is that ALL the community should care about backporting issues and that's exactly what made the previous LTS trial fail. Why should I (contributor/maintainer) care about backporting stuff if users (or my employer) can move rapidly to a more recent version and get even more additional benefits? It does not help the fact that backporting fixes has been absolutely painful in the past due to the rapid changes of Beam internals + build system + CI runs. But even if most people (or just one company) are interested on backporting issues and maintaining a LTS there is still more to clarify. The devil in the details: What are the goals or guarantees of the LTS version and how this impact the backporting of issues? Some already mentioned upgrade and state compatibility as candidates (that of course will require additional regression tests), but I can think about more mundane ones like can we upgrade dependency versions in the LTS for reasons other than security to make migration easier? What happens if changes in master introduce incompatible transitive (or not) APIs into the LTS version, should they be backported or no? There are without doubts more details to clarify. Other aspect of having a LTS version that has not been mentioned is that all the bugs that users report in advance because they are moving versions more regularly will be discovered and reported now one year later. This has a negative impact for the project quality too. We are not perfect and errors will happen, so the earlier we can find and fix them the better. It is up to us in the project that the quality is good enough so users are motivated to upgrade and don't prefer to stay in older versions just because of fear. On Tue, Apr 13, 2021 at 2:22 AM Robert Burke <rob...@frantil.com> wrote: > > I'll note that "binary compatibility" is can be substituted substitute for > "upgrade compatibility" or "state compatibility". > > On Mon, Apr 12, 2021, 5:04 PM Brian Hulette <bhule...@google.com> wrote: >> >> > Beam is also multi language, which adjusts concerns. How does GRPC handle >> > that? Or protos? (I'm sure there are other examples we can pull from...) >> >> I'm not sure those are good projects to look at. They're likely much more >> concerned with binary compatibility and there's probably not much change in >> the user-facing API. >> Arrow is another multi-language project but I don't think we can learn much >> from it's versioning policy [1], which is much more concerned with binary >> compatibility than it is with API compatibility (for now). Perhaps one >> lesson is that they track a separate format version and API version. We >> could do something similar and have a separate version number for the Beam >> model protos. I'm not sure if that's relevant for this discussion or not. >> >> Spark may be a reasonable comparison since it provides an API in multiple >> languages, but that doesn't seem to have any bearing on it's versioning >> policy [2]. It sounds similar to Flink in that every minor release gets >> backported bugfixes for 18 months, but releases are slower (~6 months) so >> that's not as much of a burden. >> >> Brian >> >> [1] >> https://arrow.apache.org/docs/format/Versioning.html#backward-compatibility >> [2] https://spark.apache.org/versioning-policy.html >> >> On Thu, Apr 8, 2021 at 1:18 PM Robert Bradshaw <rober...@google.com> wrote: >>> >>> Python (again a language) has a slower release cycle, fairly strict >>> backwards compatibility stance (with the ability to opt-in before changes >>> become the default) and clear ownership for maintenance of each minor >>> version until end-of-life (so each could be considered an LTS release). >>> https://devguide.python.org/devcycle/ >>> >>> Cython is more similar to Beam: best-effort compatibility, no LTS, but as a >>> code-generater rather than a runtime library a developer is mostly free to >>> upgrade at their own cadence regardless of the surrounding ecosystem >>> (including downstream projects that take them on as a dependency). >>> >>> IIRC, Flink supports the latest N (3?) releases, which are infrequent >>> enough to cover about 12-18 months. >>> >>> My take is that Beam should be supportive of LTS releases, but we're not in >>> a position to commit to it (to the same level we commit to the 6-week >>> cut-from-head release cycle). But certain users of Beam (which have a large >>> overlap with the Beam community) could make such commitments as it helps >>> them (directly or indirectly). Let's give it a try. >>> >>> >>> On Thu, Apr 8, 2021 at 1:00 PM Robert Burke <rob...@frantil.com> wrote: >>>> >>>> I don't know about other Apache projects but the Go Programming Language >>>> uses a slower release cadence, two releases a year. Only the last two >>>> releases are maintained with backports, except for significant security >>>> issues, which are backported as far back as possible. >>>> >>>> Further they divide each release period into clear 3 month "commit" and " >>>> fix" periods, with the main tree only open to new features in the commit >>>> phase with the fix phase largely frozen outside of discovered issues. >>>> >>>> Beam isn't entirely the same thing as a Programming Language, and >>>> certainly hasn't had such a rigorous commitment to forward compatibility >>>> as Go has. There's been a very strong expectation that Go programs must >>>> continue to compile, and function as expected through new versions of Go, >>>> and it's standard libraries (and to a lesser extent, arbitrary Go >>>> packages). But these tenets allow the Go maintainers and contributors to >>>> avoid LTS as a concept to worry about. >>>> >>>> The way I see it, it's not necessarily the number of releases that's the >>>> problem exactly, but the overall stability of the system. More frequent >>>> releases "feels" less stable than fewer, even if most of them are >>>> compatible ona code level. This will vary on the way language ecosystems >>>> handle dependencies though. >>>> >>>> Beam is also multi language, which adjusts concerns. How does GRPC handle >>>> that? Or protos? (I'm sure there are other examples we can pull from...) >>>> >>>> >>>> On Thu, Apr 8, 2021, 9:31 AM Kenneth Knowles <k...@apache.org> wrote: >>>>> >>>>> The other obvious question, which I often ask and very rarely research: >>>>> what do other similar projects do? Does anyone know? >>>>> >>>>> On Thu, Apr 8, 2021 at 9:30 AM Kenneth Knowles <k...@apache.org> wrote: >>>>>> >>>>>> I want to send a short summary of my views. There is a >>>>>> tension/contradiction that I think is OK: >>>>>> >>>>>> - An LTS series becomes more valuable when more stakeholders care for >>>>>> it. For those who don't care about LTS, the main thing would be to >>>>>> highlight major issues that are worth backporting. So having it part of >>>>>> how we think about things is helpful, even for the people who are not >>>>>> using it or supporting users of it. And cherrypicks are very easy to >>>>>> open. So I support building consensus around this. >>>>>> >>>>>> - Anyone can volunteer and perform a point release for any past branch. >>>>>> IMO if a stakeholder proposed to do the work, we (the Beam community) >>>>>> should welcome it. Certainly we have no ability to make anyone else do >>>>>> the point releases or directly fund any work. So I support welcoming >>>>>> such a contribution but not making unfunded commitments. >>>>>> >>>>>> In practice: I think if CompanyX has a customer who wants LTS support, >>>>>> CompanyX commits to the customer that they will perform LTS support and >>>>>> CompanyX can be sure that the Beam community welcomes such a >>>>>> contribution, and their customer can easily confirm Beam community's >>>>>> public position on it. >>>>>> >>>>>> Kenn >>>>>> >>>>>> On Thu, Apr 8, 2021 at 1:05 AM Jan Lukavský <je...@seznam.cz> wrote: >>>>>>> >>>>>>> Understand that, my point was just that we cannot care about API changes >>>>>>> only. What an LTS has to maintain is binary compatibility of state or >>>>>>> have any kind of upgrade process without the running Pipeline being >>>>>>> terminated and completely replaced with another one. Bootstrapping the >>>>>>> state is what is - from my point of view - the most expensive part of >>>>>>> upgrade with state-incompatibility. If an update to LTS breaks the state >>>>>>> compatibility, then upgrading to the new (bugfixed) LTS version seems to >>>>>>> be (about) the same pain as upgrading to the newest version of Beam. >>>>>>> That is what somewhat breaks the "LTS" expectations users might have. >>>>>>> But yes, fixes which break state might not be that frequent, but are >>>>>>> possible, that's for sure. In that case, we essentially could not >>>>>>> release a fix, even if it was critical security fix. My question was >>>>>>> then if these concerns will be somehow part of the proposed LTS support >>>>>>> process. >>>>>>> >>>>>>> Jan >>>>>>> >>>>>>> On 4/7/21 4:23 PM, Elliotte Rusty Harold wrote: >>>>>>> > On Tue, Apr 6, 2021 at 9:43 PM Jan Lukavský <je...@seznam.cz> wrote: >>>>>>> >> Hi, >>>>>>> >> do we know what is the reason users stay on an older version of >>>>>>> >> Beam? My guess would be that it is not related to API changes, but >>>>>>> >> more likely to state incompatibility. Maybe if we could figure out a >>>>>>> >> way which would enable a smooth migration of state (and timers) >>>>>>> >> between Beam versions, that might help? The migration would probably >>>>>>> >> have to be runner-dependent, but Beam might offer some tooling to >>>>>>> >> make this easier. One example would be coder evolution, where we >>>>>>> >> currently do not have the option of "reading old way, writing new >>>>>>> >> way" with some "coder-version-registry". I suppose there might have >>>>>>> >> been a discussion about this in the past, does anyone know of any >>>>>>> >> conclusion? >>>>>>> >> >>>>>>> > Different customers have different requirements. The projects and >>>>>>> > teams I'm talking to simply don't want to invest engineering time in >>>>>>> > migrating as often as they have to now. They want to be able to ship a >>>>>>> > working product and not worry about it for a year or more. They >>>>>>> > recognize the possibility of unforeseen bugs and security issues that >>>>>>> > might require an update. However they'd like that update to be minimal >>>>>>> > and not require them to update their own code. Nor do they want to run >>>>>>> > the risk of performance regressions and newly introduced bugs and >>>>>>> > security issues. >>>>>>> >
