Re: Reaching out to Greg KH for 6 year LTS kernel versions
On Fri, 12 Aug 2022 at 23:12, Florian Fainelli wrote: > > On 8/12/22 13:28, Robert Marko wrote: > > On Fri, 12 Aug 2022 at 21:45, Florian Fainelli wrote: > >> > >> On 8/12/22 11:09, Robert Marko wrote: > >>> On Fri, 12 Aug 2022 at 19:54, Florian Fainelli > >>> wrote: > > On 8/10/22 13:32, Robert Marko wrote: > > On Wed, 10 Aug 2022 at 22:30, Philip Prindeville > > wrote: > >> > >> Not to play the devil's advocate but... do we want old kernels hanging > >> out that long? > >> > >> Besides not encouraging people to update to new releases that mitigate > >> discovered CVE's, we'd also not pick up David Taht's excellent > >> improvements in Buffer Bloat. > > > > I have to agree with this. > > What would be the benefit for OpenWrt with having LTS kernels > > supported for 6 years? > > One aspect I could see is take for instance a device that is widely > popular amongst our user base as was TI's ar7 for instance a while back, > and for which we might have done a Linux 5.4, or 5.10 version at the > time but we do not wish to continue to maintain. > >>> > >>> I dont see how this is related to LTS kernel support. > >> > >> OK maybe I need to spell out my example more clearly. Let us say that we > >> have a release in 2019 of OpenWrt that supported TI AR7 based upon the > >> Linux 5.4 kernel. > >> > >> Fast forward to 2022, we decide to abandon TI AR7 in master and we stop > >> supporting it entirely for future releases. A very bad Linux security > >> problem show up, and because we care about our users, we keep updating > >> the LTS kernel that was used in the 2019.x release up until, say the > >> said kernel stops being supported. If that support time frame was 6 > >> years, then we would really be helping our users. > >> > >> Maybe the wider discussion to be had, is: > >> > >> - when and how do we decide to deprecate a given Linux port, I assume it > >> is when no more users show up or maybe more realistically when OpenWrt > >> developers just cannot keep up with updating those devices because they > >> no longer use those themselves > >> > >> - how long do we want to keep supporting past OpenWrt releases with > >> kernel updates, 2 years, 3/4 years, 6 years to match the kernel > >> lifecycle they are based upon? > > > > As an idea, I understand this, it would basically be an "LTS" OpenWrt > > release that > > would receive security-only updates. > > > > However, we had a long discussion on the IRC today and the resources are > > spread > > rather thin even currently with 2 or 3 releases being supported. > > > > If its gonna be a volunteer kind of no guarantees release, then maybe > > but I dont see > > how we can manage that as well. > > That is fair, if we are spread too thin, and clearly we are, then yes, I > agree we should focus on the latest releases and people who cannot > update for whatever reason, be it now unsupported hardware, or high > availability or whatever should find out a solution. It's open source > after all :) > > > >> > >> > >>> > > Being able to continue to deliver stable kernel updates in a stable > OpenWrt branch could be a good way for users to pick up their next xDSL > router since there are not so many out there that can actually run > OpenWrt compared to pure Wired/Wi-Fi for instance. > >>> > >>> I can agree with this. > >>> > > > Backporting stuff is already hard with only 2 LTS versions supported in > > OpenWrt. > > That argument I am sympathetic with, and the sheer amount of out of tree > patches we have in OpenWrt is not helping, in fact it definitively makes > it harder to regularly test, but still somehow we managed to do it. > > Since we will merge stable updates eventually, the point would be that > instead of testing those that are already released, we could try to test > the release candidates and report back anything we find? > >>> > >>> This is a good idea, not sure how we can do it within OpenWrt though with > >>> the amount of patches we have that make it a pain to bump kernels. > >> > >> Maybe we should give it a spin and see how painful that actually is and > >> then if we can sustain doing that over time it just becomes a thing a > >> group of volunteers can do? > >> > >> After all, we do go through that exercise fairly frequently. > > > > This looks similar to what we discussed to today on IRC, maybe having a more > > up-to-date development OpenWrt along longer lasting stable releases. > > > > As currently, OpenWrt is way out of date for kernel development but is > > moving > > too quickly for keeping the stable releases from regressing. > > Which is an interesting paradox. Agreed, however that is the conclusion we reached on the IRC after a long discussion. Its a compromise that makes both sides equally unhappy. Regards, Robert > -- > Florian ___
Re: Reaching out to Greg KH for 6 year LTS kernel versions
> On Aug 12, 2022, at 5:12 PM, Florian Fainelli wrote: > >> As an idea, I understand this, it would basically be an "LTS" OpenWrt >> release that >> would receive security-only updates. >> However, we had a long discussion on the IRC today and the resources are >> spread >> rather thin even currently with 2 or 3 releases being supported. >> If its gonna be a volunteer kind of no guarantees release, then maybe >> but I dont see >> how we can manage that as well. > > That is fair, if we are spread too thin, and clearly we are, then yes, I > agree we should focus on the latest releases and people who cannot update for > whatever reason, be it now unsupported hardware, or high availability or > whatever should find out a solution. It's open source after all :) I've been lurking in this discussion, but thought I would throw in this perspective: Who is asking for this? Could we ask them to quantify the benefit (to them) of a six-year LTS? Would they be willing to fund the effort required? (Some companies decide that Red Hat or Ubuntu are critical to their business, and hire people/assign developers to work to support those distributions...) Thanks for listening. Rich (Sorry for any duplicates - original message was in Rich Text...) ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Reaching out to Greg KH for 6 year LTS kernel versions
On 8/12/22 13:28, Robert Marko wrote: On Fri, 12 Aug 2022 at 21:45, Florian Fainelli wrote: On 8/12/22 11:09, Robert Marko wrote: On Fri, 12 Aug 2022 at 19:54, Florian Fainelli wrote: On 8/10/22 13:32, Robert Marko wrote: On Wed, 10 Aug 2022 at 22:30, Philip Prindeville wrote: Not to play the devil's advocate but... do we want old kernels hanging out that long? Besides not encouraging people to update to new releases that mitigate discovered CVE's, we'd also not pick up David Taht's excellent improvements in Buffer Bloat. I have to agree with this. What would be the benefit for OpenWrt with having LTS kernels supported for 6 years? One aspect I could see is take for instance a device that is widely popular amongst our user base as was TI's ar7 for instance a while back, and for which we might have done a Linux 5.4, or 5.10 version at the time but we do not wish to continue to maintain. I dont see how this is related to LTS kernel support. OK maybe I need to spell out my example more clearly. Let us say that we have a release in 2019 of OpenWrt that supported TI AR7 based upon the Linux 5.4 kernel. Fast forward to 2022, we decide to abandon TI AR7 in master and we stop supporting it entirely for future releases. A very bad Linux security problem show up, and because we care about our users, we keep updating the LTS kernel that was used in the 2019.x release up until, say the said kernel stops being supported. If that support time frame was 6 years, then we would really be helping our users. Maybe the wider discussion to be had, is: - when and how do we decide to deprecate a given Linux port, I assume it is when no more users show up or maybe more realistically when OpenWrt developers just cannot keep up with updating those devices because they no longer use those themselves - how long do we want to keep supporting past OpenWrt releases with kernel updates, 2 years, 3/4 years, 6 years to match the kernel lifecycle they are based upon? As an idea, I understand this, it would basically be an "LTS" OpenWrt release that would receive security-only updates. However, we had a long discussion on the IRC today and the resources are spread rather thin even currently with 2 or 3 releases being supported. If its gonna be a volunteer kind of no guarantees release, then maybe but I dont see how we can manage that as well. That is fair, if we are spread too thin, and clearly we are, then yes, I agree we should focus on the latest releases and people who cannot update for whatever reason, be it now unsupported hardware, or high availability or whatever should find out a solution. It's open source after all :) Being able to continue to deliver stable kernel updates in a stable OpenWrt branch could be a good way for users to pick up their next xDSL router since there are not so many out there that can actually run OpenWrt compared to pure Wired/Wi-Fi for instance. I can agree with this. Backporting stuff is already hard with only 2 LTS versions supported in OpenWrt. That argument I am sympathetic with, and the sheer amount of out of tree patches we have in OpenWrt is not helping, in fact it definitively makes it harder to regularly test, but still somehow we managed to do it. Since we will merge stable updates eventually, the point would be that instead of testing those that are already released, we could try to test the release candidates and report back anything we find? This is a good idea, not sure how we can do it within OpenWrt though with the amount of patches we have that make it a pain to bump kernels. Maybe we should give it a spin and see how painful that actually is and then if we can sustain doing that over time it just becomes a thing a group of volunteers can do? After all, we do go through that exercise fairly frequently. This looks similar to what we discussed to today on IRC, maybe having a more up-to-date development OpenWrt along longer lasting stable releases. As currently, OpenWrt is way out of date for kernel development but is moving too quickly for keeping the stable releases from regressing. Which is an interesting paradox. -- Florian ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Reaching out to Greg KH for 6 year LTS kernel versions
On 8/12/22 13:49, Philip Prindeville wrote: On Aug 12, 2022, at 11:54 AM, Florian Fainelli wrote: One aspect I could see is take for instance a device that is widely popular amongst our user base as was TI's ar7 for instance a while back, and for which we might have done a Linux 5.4, or 5.10 version at the time but we do not wish to continue to maintain. Per my previous email, that seems like something that the SoC manufacturer should be doing... Why is this our burden? Totally agree that SoC manufacturers share a fair amount of responsibility, there is no substitute for upstream and it associated benefits, but it is a commitment, and you are not stranger to this world. In terms of supporting a given piece of hardware though, I would decouple the timeline. There is the initial effort required to support your SoC and its many products which largely involves writing architecture specific code sometimes, DTS files, clock/SPI/USB/NAND/Ethernet MAC/PHY/Switch/Wi-Fi drivers for the most part. You would typically target a LTS kernel version for that effort such that you amortize that effort over a few years to come until the next LTS, and if possible in the meantime try to get your patches upstream such that the next LTS requires fewer patches to be carried over. Now, when new stable updates show up though, the amount of merge conflicts or SoC support code you need to adjust is minimal compared to the other parts of the kernel that got updated. So back to my example, we could have invested out of tree support for TI AR7 up until 5.4 and then decided to put it into an EOL/deep maintenance mode. Updating the stable 5.4 kernel would unlikely change the amount of AR7 specific patches, so the maintenance cost could be fairly low. Does that make sense? -- Florian ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Reaching out to Greg KH for 6 year LTS kernel versions
> On Aug 12, 2022, at 11:54 AM, Florian Fainelli wrote: > > One aspect I could see is take for instance a device that is widely popular > amongst our user base as was TI's ar7 for instance a while back, and for > which we might have done a Linux 5.4, or 5.10 version at the time but we do > not wish to continue to maintain. Per my previous email, that seems like something that the SoC manufacturer should be doing... Why is this our burden? ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Reaching out to Greg KH for 6 year LTS kernel versions
> On Aug 12, 2022, at 11:53 AM, Florian Fainelli wrote: > > OK, the 3-4 years time frame is not something that we currently have with > Linux kernels, it is either 2 years or 6 years, since 6 > 4, it sounds like > we still have some interest in having 6 LTS kernels, but maybe not have every > single LTS release be 6 years, right? Is this the REAL problem? That we need LTS releases? Or that SoC's like the MT76xx aren't able to keep up with new kernel releases? Would we need to have LTS support if chips like the AR7 and the MT7623 had ongoing kernel support from the manufacturers? Isn't this latter point the true root cause? -Philip ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Reaching out to Greg KH for 6 year LTS kernel versions
> On Aug 12, 2022, at 2:28 PM, Robert Marko wrote: > > On Fri, 12 Aug 2022 at 21:45, Florian Fainelli wrote: >> >> On 8/12/22 11:09, Robert Marko wrote: >>> On Fri, 12 Aug 2022 at 19:54, Florian Fainelli wrote: On 8/10/22 13:32, Robert Marko wrote: > On Wed, 10 Aug 2022 at 22:30, Philip Prindeville > wrote: >> >> Not to play the devil's advocate but... do we want old kernels hanging >> out that long? >> >> Besides not encouraging people to update to new releases that mitigate >> discovered CVE's, we'd also not pick up David Taht's excellent >> improvements in Buffer Bloat. > > I have to agree with this. > What would be the benefit for OpenWrt with having LTS kernels > supported for 6 years? One aspect I could see is take for instance a device that is widely popular amongst our user base as was TI's ar7 for instance a while back, and for which we might have done a Linux 5.4, or 5.10 version at the time but we do not wish to continue to maintain. >>> >>> I dont see how this is related to LTS kernel support. >> >> OK maybe I need to spell out my example more clearly. Let us say that we >> have a release in 2019 of OpenWrt that supported TI AR7 based upon the >> Linux 5.4 kernel. >> >> Fast forward to 2022, we decide to abandon TI AR7 in master and we stop >> supporting it entirely for future releases. A very bad Linux security >> problem show up, and because we care about our users, we keep updating >> the LTS kernel that was used in the 2019.x release up until, say the >> said kernel stops being supported. If that support time frame was 6 >> years, then we would really be helping our users. >> >> Maybe the wider discussion to be had, is: >> >> - when and how do we decide to deprecate a given Linux port, I assume it >> is when no more users show up or maybe more realistically when OpenWrt >> developers just cannot keep up with updating those devices because they >> no longer use those themselves >> >> - how long do we want to keep supporting past OpenWrt releases with >> kernel updates, 2 years, 3/4 years, 6 years to match the kernel >> lifecycle they are based upon? > > As an idea, I understand this, it would basically be an "LTS" OpenWrt > release that > would receive security-only updates. > > However, we had a long discussion on the IRC today and the resources are > spread > rather thin even currently with 2 or 3 releases being supported. > > If its gonna be a volunteer kind of no guarantees release, then maybe > but I dont see > how we can manage that as well. Chiming in as a software security developer... I think we're approaching this problem wrong. We need to make regular software updates easier, which we've not focused on in the past. Periodically "sysupgrade" gets damaged, and manages to brick devices. This should never happen. Ideally we should be moving towards unattended updates. Would this [kernel support] be as much of a burden if we worked harder at getting our patches upstreamed? And a 800lb gorilla is that companies like Netgear and D-Link don't arm-twist Mediatek to keep their SoC-specific patches upstreamed and their SoC's running more recent kernels. How much of this problem would go away if MediaTek supported 5.10 or 5.15 on all of their current SoC's and SoC's released in the last 5-7 years? Being able to continue to deliver stable kernel updates in a stable OpenWrt branch could be a good way for users to pick up their next xDSL router since there are not so many out there that can actually run OpenWrt compared to pure Wired/Wi-Fi for instance. >>> >>> I can agree with this. >>> > Backporting stuff is already hard with only 2 LTS versions supported in > OpenWrt. That argument I am sympathetic with, and the sheer amount of out of tree patches we have in OpenWrt is not helping, in fact it definitively makes it harder to regularly test, but still somehow we managed to do it. Since we will merge stable updates eventually, the point would be that instead of testing those that are already released, we could try to test the release candidates and report back anything we find? >>> >>> This is a good idea, not sure how we can do it within OpenWrt though with >>> the amount of patches we have that make it a pain to bump kernels. >> >> Maybe we should give it a spin and see how painful that actually is and >> then if we can sustain doing that over time it just becomes a thing a >> group of volunteers can do? >> >> After all, we do go through that exercise fairly frequently. > > This looks similar to what we discussed to today on IRC, maybe having a more > up-to-date development OpenWrt along longer lasting stable releases. I've not had time to keep up with IRC (I'm getting ready to move homes), so apologies if I'm rehashing some of this... > > As currently, OpenWrt is wa
Re: Reaching out to Greg KH for 6 year LTS kernel versions
On Fri, 12 Aug 2022 at 21:45, Florian Fainelli wrote: > > On 8/12/22 11:09, Robert Marko wrote: > > On Fri, 12 Aug 2022 at 19:54, Florian Fainelli wrote: > >> > >> On 8/10/22 13:32, Robert Marko wrote: > >>> On Wed, 10 Aug 2022 at 22:30, Philip Prindeville > >>> wrote: > > Not to play the devil's advocate but... do we want old kernels hanging > out that long? > > Besides not encouraging people to update to new releases that mitigate > discovered CVE's, we'd also not pick up David Taht's excellent > improvements in Buffer Bloat. > >>> > >>> I have to agree with this. > >>> What would be the benefit for OpenWrt with having LTS kernels > >>> supported for 6 years? > >> > >> One aspect I could see is take for instance a device that is widely > >> popular amongst our user base as was TI's ar7 for instance a while back, > >> and for which we might have done a Linux 5.4, or 5.10 version at the > >> time but we do not wish to continue to maintain. > > > > I dont see how this is related to LTS kernel support. > > OK maybe I need to spell out my example more clearly. Let us say that we > have a release in 2019 of OpenWrt that supported TI AR7 based upon the > Linux 5.4 kernel. > > Fast forward to 2022, we decide to abandon TI AR7 in master and we stop > supporting it entirely for future releases. A very bad Linux security > problem show up, and because we care about our users, we keep updating > the LTS kernel that was used in the 2019.x release up until, say the > said kernel stops being supported. If that support time frame was 6 > years, then we would really be helping our users. > > Maybe the wider discussion to be had, is: > > - when and how do we decide to deprecate a given Linux port, I assume it > is when no more users show up or maybe more realistically when OpenWrt > developers just cannot keep up with updating those devices because they > no longer use those themselves > > - how long do we want to keep supporting past OpenWrt releases with > kernel updates, 2 years, 3/4 years, 6 years to match the kernel > lifecycle they are based upon? As an idea, I understand this, it would basically be an "LTS" OpenWrt release that would receive security-only updates. However, we had a long discussion on the IRC today and the resources are spread rather thin even currently with 2 or 3 releases being supported. If its gonna be a volunteer kind of no guarantees release, then maybe but I dont see how we can manage that as well. > > > > > >> > >> Being able to continue to deliver stable kernel updates in a stable > >> OpenWrt branch could be a good way for users to pick up their next xDSL > >> router since there are not so many out there that can actually run > >> OpenWrt compared to pure Wired/Wi-Fi for instance. > > > > I can agree with this. > > > >> > >>> Backporting stuff is already hard with only 2 LTS versions supported in > >>> OpenWrt. > >> > >> That argument I am sympathetic with, and the sheer amount of out of tree > >> patches we have in OpenWrt is not helping, in fact it definitively makes > >> it harder to regularly test, but still somehow we managed to do it. > >> > >> Since we will merge stable updates eventually, the point would be that > >> instead of testing those that are already released, we could try to test > >> the release candidates and report back anything we find? > > > > This is a good idea, not sure how we can do it within OpenWrt though with > > the amount of patches we have that make it a pain to bump kernels. > > Maybe we should give it a spin and see how painful that actually is and > then if we can sustain doing that over time it just becomes a thing a > group of volunteers can do? > > After all, we do go through that exercise fairly frequently. This looks similar to what we discussed to today on IRC, maybe having a more up-to-date development OpenWrt along longer lasting stable releases. As currently, OpenWrt is way out of date for kernel development but is moving too quickly for keeping the stable releases from regressing. Regards, Robert > -- > Florian ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Reaching out to Greg KH for 6 year LTS kernel versions
On 8/12/22 11:09, Robert Marko wrote: On Fri, 12 Aug 2022 at 19:54, Florian Fainelli wrote: On 8/10/22 13:32, Robert Marko wrote: On Wed, 10 Aug 2022 at 22:30, Philip Prindeville wrote: Not to play the devil's advocate but... do we want old kernels hanging out that long? Besides not encouraging people to update to new releases that mitigate discovered CVE's, we'd also not pick up David Taht's excellent improvements in Buffer Bloat. I have to agree with this. What would be the benefit for OpenWrt with having LTS kernels supported for 6 years? One aspect I could see is take for instance a device that is widely popular amongst our user base as was TI's ar7 for instance a while back, and for which we might have done a Linux 5.4, or 5.10 version at the time but we do not wish to continue to maintain. I dont see how this is related to LTS kernel support. OK maybe I need to spell out my example more clearly. Let us say that we have a release in 2019 of OpenWrt that supported TI AR7 based upon the Linux 5.4 kernel. Fast forward to 2022, we decide to abandon TI AR7 in master and we stop supporting it entirely for future releases. A very bad Linux security problem show up, and because we care about our users, we keep updating the LTS kernel that was used in the 2019.x release up until, say the said kernel stops being supported. If that support time frame was 6 years, then we would really be helping our users. Maybe the wider discussion to be had, is: - when and how do we decide to deprecate a given Linux port, I assume it is when no more users show up or maybe more realistically when OpenWrt developers just cannot keep up with updating those devices because they no longer use those themselves - how long do we want to keep supporting past OpenWrt releases with kernel updates, 2 years, 3/4 years, 6 years to match the kernel lifecycle they are based upon? Being able to continue to deliver stable kernel updates in a stable OpenWrt branch could be a good way for users to pick up their next xDSL router since there are not so many out there that can actually run OpenWrt compared to pure Wired/Wi-Fi for instance. I can agree with this. Backporting stuff is already hard with only 2 LTS versions supported in OpenWrt. That argument I am sympathetic with, and the sheer amount of out of tree patches we have in OpenWrt is not helping, in fact it definitively makes it harder to regularly test, but still somehow we managed to do it. Since we will merge stable updates eventually, the point would be that instead of testing those that are already released, we could try to test the release candidates and report back anything we find? This is a good idea, not sure how we can do it within OpenWrt though with the amount of patches we have that make it a pain to bump kernels. Maybe we should give it a spin and see how painful that actually is and then if we can sustain doing that over time it just becomes a thing a group of volunteers can do? After all, we do go through that exercise fairly frequently. -- Florian ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Reaching out to Greg KH for 6 year LTS kernel versions
On Fri, 12 Aug 2022 at 19:54, Florian Fainelli wrote: > > On 8/10/22 13:32, Robert Marko wrote: > > On Wed, 10 Aug 2022 at 22:30, Philip Prindeville > > wrote: > >> > >> Not to play the devil's advocate but... do we want old kernels hanging out > >> that long? > >> > >> Besides not encouraging people to update to new releases that mitigate > >> discovered CVE's, we'd also not pick up David Taht's excellent > >> improvements in Buffer Bloat. > > > > I have to agree with this. > > What would be the benefit for OpenWrt with having LTS kernels > > supported for 6 years? > > One aspect I could see is take for instance a device that is widely > popular amongst our user base as was TI's ar7 for instance a while back, > and for which we might have done a Linux 5.4, or 5.10 version at the > time but we do not wish to continue to maintain. I dont see how this is related to LTS kernel support. > > Being able to continue to deliver stable kernel updates in a stable > OpenWrt branch could be a good way for users to pick up their next xDSL > router since there are not so many out there that can actually run > OpenWrt compared to pure Wired/Wi-Fi for instance. I can agree with this. > > > Backporting stuff is already hard with only 2 LTS versions supported in > > OpenWrt. > > That argument I am sympathetic with, and the sheer amount of out of tree > patches we have in OpenWrt is not helping, in fact it definitively makes > it harder to regularly test, but still somehow we managed to do it. > > Since we will merge stable updates eventually, the point would be that > instead of testing those that are already released, we could try to test > the release candidates and report back anything we find? This is a good idea, not sure how we can do it within OpenWrt though with the amount of patches we have that make it a pain to bump kernels. Regards, Robert > -- > Florian ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Reaching out to Greg KH for 6 year LTS kernel versions
On 8/10/22 13:32, Robert Marko wrote: On Wed, 10 Aug 2022 at 22:30, Philip Prindeville wrote: Not to play the devil's advocate but... do we want old kernels hanging out that long? Besides not encouraging people to update to new releases that mitigate discovered CVE's, we'd also not pick up David Taht's excellent improvements in Buffer Bloat. I have to agree with this. What would be the benefit for OpenWrt with having LTS kernels supported for 6 years? One aspect I could see is take for instance a device that is widely popular amongst our user base as was TI's ar7 for instance a while back, and for which we might have done a Linux 5.4, or 5.10 version at the time but we do not wish to continue to maintain. Being able to continue to deliver stable kernel updates in a stable OpenWrt branch could be a good way for users to pick up their next xDSL router since there are not so many out there that can actually run OpenWrt compared to pure Wired/Wi-Fi for instance. Backporting stuff is already hard with only 2 LTS versions supported in OpenWrt. That argument I am sympathetic with, and the sheer amount of out of tree patches we have in OpenWrt is not helping, in fact it definitively makes it harder to regularly test, but still somehow we managed to do it. Since we will merge stable updates eventually, the point would be that instead of testing those that are already released, we could try to test the release candidates and report back anything we find? -- Florian ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Reaching out to Greg KH for 6 year LTS kernel versions
On 8/10/22 15:27, Hauke Mehrtens wrote: On 8/9/22 01:15, Florian Fainelli wrote: Hi, Greg KH has communicated a few times before on his blog [1] that he is seeking the help of individuals and company to help him maintain the LTS kernels and allow them to be made 6 years instead of just the usual 2 years. 5.10 is a 6 year LTS, but 5.15 is not listed as such, although it certainly would make sense for it to be since we use 5.15 in OpenWrt. It would be good for the project to have a designated contact who can communicate the kernel version plan ahead of time, or once a LTS is picked up, we could sign up people to do regular testing of the stable release candidates? Thoughts? [1]: http://kroah.com/log/blog/2021/02/03/helping-out-with-lts-kernel-releases/ Hi Florian, I saw that you are often testing the stable rc version, this is really great. Someone could try to automate this in OpenWrt to automatically test the stable RC kernel versions, but it probably still needs some maintenance. Currently I do not see that we have the resources to do this. Yes I do try to test the release candidates when they show up, although that is done on the limited set of devices I have at work and at home usually, so it won't catch all of the possible regressions, and the tests are limited to kselftests and some occasional one offs that are device specific. I do not think OpenWrt needs 6 LTS years kernel support 3 to 4 years is fine for OpenWrt. If we do the 22.03 stable release in the next week we can declare 21.02 end of life by end of February 2023. We would have used kernel 5.4 only a bit over 3 years after its initial release is November 2019. OK, the 3-4 years time frame is not something that we currently have with Linux kernels, it is either 2 years or 6 years, since 6 > 4, it sounds like we still have some interest in having 6 LTS kernels, but maybe not have every single LTS release be 6 years, right? If you want to volunteer to help Greg KH more on this it would still be nice. I will continue to help with the stable candidates testing for the foreseeable future as it is largely automated and does not require much effort on my side, and will keep an open channel with Greg to tell him about what OpenWrt has selected. -- Florian ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Reaching out to Greg KH for 6 year LTS kernel versions
I am not on the openwrt-devel list from this acct... I tried to resubscribe but its taking too long... On Wed, Aug 10, 2022 at 1:29 PM Philip Prindeville wrote: > > Not to play the devil's advocate but... do we want old kernels hanging out > that long? People are still shipping 3.3 kernels. > Besides not encouraging people to update to new releases that mitigate > discovered CVE's, we'd also not pick up David Taht's excellent improvements > in Buffer Bloat. People keep giving me too much credit. There have been dozens of core contributors to this effort, 1000s of others, and we have billions of machines (not enough routers, tho!) now doing more of the right things. I haven't made a technical contribution in years. I( got a couple new things that I'd like to work on, but no funding). What was once a 5-7 lag in embedded from devel to distribution now seems closer to 10. I've gone political in search of finding ways to get bufferbloat fixes out there. Somehow convincing the world that in addition to burning billions on fiber buildouts ISPs should be supplying better routers has been a big goal, and I'm low on ideas on how to move forward on that. Certainly finding some way to get openwrt shippers like starlink to at least plan to upgrade to a modern openwrt rather than continue to ship LEDE is very desirable... 6 year old kernels at FCS noo... I thought ooka's new speedtest (which measures responsiveness on up/downloads now) would provide a tipping point. Certainly also preseem and libreqos are making an impact in the smaller ISP markets... In the last 6 months of coping with a string of regressions in openwrt's wifi[1], I've reflected on the waddington effect a lot. If you haven't heard of it, it applies a string of criteria to how and why you o prevenntative maintenence on airplanes: https://resources.savvyaviation.com/wp-content/uploads/articles_eaa/EAA_2011-03_the-waddington-effect.pdf As for 6 years on a LTS kernel, my instinct is NO! Progress MUST be made! We must keep working towards making kernel upgrades continuous and easy. But think deeply on the waddington effect. The cold and bitter reality though is if someone is willing to pay someone(s) to maintain a kernel for that long, and it keeps up with major security issues, it's no skin off our backs. I would prefer cash be injected into better review ( there are 1500 open issues and 300+ outstanding https://github.com/openwrt/openwrt/pulls right now), we found ways to attract, train, and retain good developers in "the FOSS way", and more companies and governments using openwrt found ways to support those, and "sufficiently rapid" change in the ever more fossilized FOSS ecosystem. [1] https://www.linkedin.com/feed/update/urn:li:activity:6963337312596369408/ > > > On Aug 8, 2022, at 5:15 PM, Florian Fainelli wrote: > > > > Hi, > > > > Greg KH has communicated a few times before on his blog [1] that he is > > seeking the help of individuals and company to help him maintain the LTS > > kernels and allow them to be made 6 years instead of just the usual 2 years. > > > > 5.10 is a 6 year LTS, but 5.15 is not listed as such, although it certainly > > would make sense for it to be since we use 5.15 in OpenWrt. > > > > It would be good for the project to have a designated contact who can > > communicate the kernel version plan ahead of time, or once a LTS is picked > > up, we could sign up people to do regular testing of the stable release > > candidates? > > > > Thoughts? > > > > [1]: > > http://kroah.com/log/blog/2021/02/03/helping-out-with-lts-kernel-releases/ > > -- > > Florian > > > > ___ > > openwrt-devel mailing list > > openwrt-devel@lists.openwrt.org > > https://lists.openwrt.org/mailman/listinfo/openwrt-devel > -- FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/ Dave Täht CEO, TekLibre, LLC ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Reaching out to Greg KH for 6 year LTS kernel versions
On 8/9/22 01:15, Florian Fainelli wrote: Hi, Greg KH has communicated a few times before on his blog [1] that he is seeking the help of individuals and company to help him maintain the LTS kernels and allow them to be made 6 years instead of just the usual 2 years. 5.10 is a 6 year LTS, but 5.15 is not listed as such, although it certainly would make sense for it to be since we use 5.15 in OpenWrt. It would be good for the project to have a designated contact who can communicate the kernel version plan ahead of time, or once a LTS is picked up, we could sign up people to do regular testing of the stable release candidates? Thoughts? [1]: http://kroah.com/log/blog/2021/02/03/helping-out-with-lts-kernel-releases/ Hi Florian, I saw that you are often testing the stable rc version, this is really great. Someone could try to automate this in OpenWrt to automatically test the stable RC kernel versions, but it probably still needs some maintenance. Currently I do not see that we have the resources to do this. I do not think OpenWrt needs 6 LTS years kernel support 3 to 4 years is fine for OpenWrt. If we do the 22.03 stable release in the next week we can declare 21.02 end of life by end of February 2023. We would have used kernel 5.4 only a bit over 3 years after its initial release is November 2019. If you want to volunteer to help Greg KH more on this it would still be nice. Hauke ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Reaching out to Greg KH for 6 year LTS kernel versions
On Wed, 10 Aug 2022 at 22:30, Philip Prindeville wrote: > > Not to play the devil's advocate but... do we want old kernels hanging out > that long? > > Besides not encouraging people to update to new releases that mitigate > discovered CVE's, we'd also not pick up David Taht's excellent improvements > in Buffer Bloat. I have to agree with this. What would be the benefit for OpenWrt with having LTS kernels supported for 6 years? Backporting stuff is already hard with only 2 LTS versions supported in OpenWrt. Regards, Robert > > > > On Aug 8, 2022, at 5:15 PM, Florian Fainelli wrote: > > > > Hi, > > > > Greg KH has communicated a few times before on his blog [1] that he is > > seeking the help of individuals and company to help him maintain the LTS > > kernels and allow them to be made 6 years instead of just the usual 2 years. > > > > 5.10 is a 6 year LTS, but 5.15 is not listed as such, although it certainly > > would make sense for it to be since we use 5.15 in OpenWrt. > > > > It would be good for the project to have a designated contact who can > > communicate the kernel version plan ahead of time, or once a LTS is picked > > up, we could sign up people to do regular testing of the stable release > > candidates? > > > > Thoughts? > > > > [1]: > > http://kroah.com/log/blog/2021/02/03/helping-out-with-lts-kernel-releases/ > > -- > > Florian > > > > ___ > > openwrt-devel mailing list > > openwrt-devel@lists.openwrt.org > > https://lists.openwrt.org/mailman/listinfo/openwrt-devel > > > ___ > openwrt-devel mailing list > openwrt-devel@lists.openwrt.org > https://lists.openwrt.org/mailman/listinfo/openwrt-devel ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: Reaching out to Greg KH for 6 year LTS kernel versions
Not to play the devil's advocate but... do we want old kernels hanging out that long? Besides not encouraging people to update to new releases that mitigate discovered CVE's, we'd also not pick up David Taht's excellent improvements in Buffer Bloat. > On Aug 8, 2022, at 5:15 PM, Florian Fainelli wrote: > > Hi, > > Greg KH has communicated a few times before on his blog [1] that he is > seeking the help of individuals and company to help him maintain the LTS > kernels and allow them to be made 6 years instead of just the usual 2 years. > > 5.10 is a 6 year LTS, but 5.15 is not listed as such, although it certainly > would make sense for it to be since we use 5.15 in OpenWrt. > > It would be good for the project to have a designated contact who can > communicate the kernel version plan ahead of time, or once a LTS is picked > up, we could sign up people to do regular testing of the stable release > candidates? > > Thoughts? > > [1]: > http://kroah.com/log/blog/2021/02/03/helping-out-with-lts-kernel-releases/ > -- > Florian > > ___ > openwrt-devel mailing list > openwrt-devel@lists.openwrt.org > https://lists.openwrt.org/mailman/listinfo/openwrt-devel ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
