Re: 5.10 LTS Kernel: 2 or 6 years?
Hi Hanjun, On 2/26/21 10:03 AM, Hanjun Guo wrote: > On 2021/2/19 22:45, Nikolai Kondrashov wrote: >> Would you be interested in working with the Linux Foundation KernelCI project >> on submitting your build and test results to the common database - KCIDB? > > Yes, we are willing to sent the test results to KCIDB. Wonderful! > For now, all the tests are inside the company, which blocks us to > directly sent the test results out of the test machine due to the > security policy, it takes us sometime to discuss how to send out > the test results, and we may need do the test in a public cloud. I understand. We faced the same dilemma at Red Hat with the CKI project, where I'm employed. In the end we just decided to publish the logs from the test lab (most sensitive part, I suppose) as they are, but then Red Hat is already a very open company. Here's an example of all logs we publish for a revision: https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/22/624489 And here's that revision in KCIDB: https://staging.kernelci.org:3000/d/revision/revision?orgId=1=kernelci04=093ae67c90a629599a0cb2300c5686ec7d8bbdd0 Take your time, there is a range of solutions to choose from, starting from isolating your test machines/boards, to running the test in the public cloud as you mention. Although the latter lacks the benefit of testing your own hardware, of course. Please also note that we require the tested kernel's code to be publicly available to accept results. >> We are working on aggregating results from various testing systems so we can >> provide a dashboard, and a single, aggregated e-mail report to subscribed >> maintainers and developers. >> >> We have a prototype dashboard at https://staging.kernelci.org:3000/ and are >> working hard on making the e-mail reports good enough to start reaching out to >> maintainers. >> >> We already have ARM, Google Syzbot, Gentoo GKernelCI, Red Hat CKI, and, of >> course, KernelCI native tests sending data to the database. Linaro Tuxsuite is >> starting sending today. We could use your data, and of course any development >> help you could spare :) > > How can we connect to the KCIDB? Can we send the test data out by email > first? The simplest way would be using our "kcidb-submit" command-line tool. Here's a minimal example of sending an empty report: echo '{"version":{"major":3,"minor":0}}' | kcidb-submit -p kernelci-production -t kernelci_new Another way could be using our Python 3 library, which is a little more involved. Here's an example sending the same empty report in Python: import kcidb client = kcidb.Client(project_id="kernelci-production", topic_name="kernelci_new") client.submit({"version":{"major":3,"minor":0}}) Finally, you can send (validated) data using the Google Cloud Pub/Sub service directly, using the library in one of the supported languages: https://cloud.google.com/pubsub/docs/tutorials The latter would be the least stable option (change-wise), but something e.g. Google Syzbot is already using. See more details in our Submission HOWTO: https://github.com/kernelci/kcidb/blob/main/SUBMISSION_HOWTO.md Although email submissions are theoretically possible (with due authentication, setup, etc.), they're not currently supported, and we would not put them at the top of our (rather large) priority list. Topmost item being getting the data we already have to developers. Would be excited to have you on board! Nick On 2/26/21 10:03 AM, Hanjun Guo wrote: > Hi Nick, > > Sorry for taking so long to reply you, we had discussions on how to > corporate with KCIDB, please see my comments inline. > > On 2021/2/19 22:45, Nikolai Kondrashov wrote: >> Hi Hanjun, >> >> On 2/19/21 10:54 AM, Hanjun Guo wrote: >> > In specific, we will start from the testing work, using HULK robot >> > (reports lots of bugs to mainline kernel) testing framework to test >> > compile, reboot, functional testing, and will extend to basic >> > performance regression testing in the future. >> >> I heard about Huawei ramping up kernel testing from someone at FOSDEM >> 2019. I wonder if it was you :) Nice to see your progress and the company >> stepping up to help with testing! > > I Cced Yongjun and Jinyue, they are the key persons :) > >> >> Would you be interested in working with the Linux Foundation KernelCI project >> on submitting your build and test results to the common database - KCIDB? > > Yes, we are willing to sent the test results to KCIDB. > > For now, all the tests are inside the company, which blocks us to > directly sent the test results out of the test machine due to the > security policy, it takes us sometime to discuss how to send out > the test results, and we may need do the test in a public cloud. > >> >> We are working on aggregating results from various testing systems so we can >> provide a dashboard, and a single, aggregated
Re: 5.10 LTS Kernel: 2 or 6 years?
Hi Nick, Sorry for taking so long to reply you, we had discussions on how to corporate with KCIDB, please see my comments inline. On 2021/2/19 22:45, Nikolai Kondrashov wrote: Hi Hanjun, On 2/19/21 10:54 AM, Hanjun Guo wrote: > In specific, we will start from the testing work, using HULK robot > (reports lots of bugs to mainline kernel) testing framework to test > compile, reboot, functional testing, and will extend to basic > performance regression testing in the future. I heard about Huawei ramping up kernel testing from someone at FOSDEM 2019. I wonder if it was you :) Nice to see your progress and the company stepping up to help with testing! I Cced Yongjun and Jinyue, they are the key persons :) Would you be interested in working with the Linux Foundation KernelCI project on submitting your build and test results to the common database - KCIDB? Yes, we are willing to sent the test results to KCIDB. For now, all the tests are inside the company, which blocks us to directly sent the test results out of the test machine due to the security policy, it takes us sometime to discuss how to send out the test results, and we may need do the test in a public cloud. We are working on aggregating results from various testing systems so we can provide a dashboard, and a single, aggregated e-mail report to subscribed maintainers and developers. We have a prototype dashboard at https://staging.kernelci.org:3000/ and are working hard on making the e-mail reports good enough to start reaching out to maintainers. We already have ARM, Google Syzbot, Gentoo GKernelCI, Red Hat CKI, and, of course, KernelCI native tests sending data to the database. Linaro Tuxsuite is starting sending today. We could use your data, and of course any development help you could spare :) How can we connect to the KCIDB? Can we send the test data out by email first? I wish I could show you my today's KCIDB presentation at DevConf.cz, but the recording is not out yet. Meanwhile you can take a look at our presentation at last year's Linux Plumbers: https://youtu.be/y9Glc90WUN0?t=10739 Or see our intro in an older blog post: https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/ Anyone wishing to contribute to KCIDB gets credentials and permissions to submit to our "playground" setup where they can send their data, see it in a dashboard, experiment without worrying about breaking anything, and decide if they like it or n > If you're interested, take a look at our Submission HOWTO: https://github.com/kernelci/kcidb/blob/v8/SUBMISSION_HOWTO.md and send an email to kerne...@groups.io (CC'd), or come over to the #kernelci channel on freenode.net! Thanks! Hanjun
Re: 5.10 LTS Kernel: 2 or 6 years?
On 2021/2/20 17:53, Greg Kroah-Hartman wrote: On Sat, Feb 20, 2021 at 03:02:54PM +0800, Hanjun Guo wrote: On 2021/2/19 17:08, Greg Kroah-Hartman wrote: On Fri, Feb 19, 2021 at 04:54:24PM +0800, Hanjun Guo wrote: Hi Greg, On 2021/1/26 15:29, Greg Kroah-Hartman wrote: [...] I want to see companies _using_ the kernel, and most importantly, _updating_ their devices with it, to know if it is worth to keep around for longer than 2 years. I also, hopefully, want to see how those companies will help me out in the testing and maintenance of that kernel version in order to make supporting it for 6 years actually possible. So, are you planning on using 5.10? Will you will be willing to help out in testing the -rc releases I make to let me know if there are any problems, and to help in pointing out and backporting any specific patches that your platforms need for that kernel release? We(Huawei) are willing to commit resources to help out in testing the stable -rc releases, and to help to backport patches for stable kernels. Wonderful! 5.10 stable kernel will be used for openEuler [1] kernel and also inside Huawei. From customer's feedback, it's very important to see the stable kernel we used to be maintained for 6 years in the community, and we will use 5.10 kernel for at least 6 years, so we are willing to help you and help ourselves :) In specific, we will start from the testing work, using HULK robot (reports lots of bugs to mainline kernel) testing framework to test compile, reboot, functional testing, and will extend to basic performance regression testing in the future. Great! Do you all need an email notification when the -rc releases come out for the stable trees, or can you trigger off of the -rc stable git tree? Different CI systems work in different ways :) We can trigger the test when you updated the -rc stable git tree, by monitoring new commits for the stable branches. So if you push all the commits at once for -rc stable branches, then our CI system can work well. I do push to the -rc branches, but those branches are rebased, and I do "intermediate" pushes as well. Meaning I push to have CI systems run on the existing patch queue at times that are not only the "main" -rc release periods. Watch the branches for a few weeks to get an idea of how they work if you are curious. Will run our CI system based on your -rc branch to see if anything doesn't work, then update our system as needed. Thanks Hanjun
Re: 5.10 LTS Kernel: 2 or 6 years?
On Mon, Feb 22, 2021 at 08:00:52AM -0600, Nishanth Aravamudan wrote: > Hi Greg, > > On 26.01.2021 [08:29:25 +0100], Greg Kroah-Hartman wrote: > > On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote: > > > Hi All, > > > > > > The 5.10 LTS kernel being officially LTS supported for 2 years > > > presents a problem: why would anyone select a 5.10 kernel with 2 > > > year LTS when 5.4 kernel has a 6 year LTS. > > > > > > If 5.10 is "actually" going to be supported for 6 years it would be > > > quite valuable to make such a declaration. > > > https://www.kernel.org/category/releases.html > > > > Why? What would that change? > > > > Ok, seriously, this happens every year, and every year we go through > > the same thing, it's not like this is somehow new, right? > > > > I want to see companies _using_ the kernel, and most importantly, > > _updating_ their devices with it, to know if it is worth to keep > > around for longer than 2 years. I also, hopefully, want to see how > > those companies will help me out in the testing and maintenance of > > that kernel version in order to make supporting it for 6 years > > actually possible. > > > > So, are you planning on using 5.10? Will you will be willing to help > > out in testing the -rc releases I make to let me know if there are any > > problems, and to help in pointing out and backporting any specific > > patches that your platforms need for that kernel release? > > > > When I get this kind of promises and support from companies, then I am > > glad to bump up the length of the kernel support from 2 to 6 years, > > and I mark it on the web site. Traditionally this happens in > > Febuary/March once I hear from enough companies. Can I count on your > > support in this endeavor? > > I am very sorry for the long delay on my end (I had privately e-mailed > Greg on January 28) -- DigitalOcean also intends to provide feedback and > testing on the 5.10 series. We intend to use it as the basis for our > next-next kernel and are very invested in ensuring the stability and > performance of the kernel. Great! If you need me to add your cc: to the -rc release announcements so that you have an easy email to respond to, just let me know and I can do so. Any ideas when the testing will start happening? There's a 5.10-rc release out there right now if you want to start today :) thanks, greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
On Fri, Feb 19, 2021 at 09:44:23AM -0800, Florian Fainelli wrote: > > > On 2/19/2021 7:53 AM, Greg Kroah-Hartman wrote: > > On Fri, Feb 19, 2021 at 07:05:41AM -0800, Florian Fainelli wrote: > >> > >> > >> On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote: > >>> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote: > On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote: > > On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote: > >> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote: > >>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: > As a company, we are most likely shooting ourselves in the foot by > not > having a point of coordination with the Linux Foundation and key > people > like you, Greg and other participants in the stable kernel. > >>> > >>> What does the LF have to do with this? > >>> > >>> We are here, on the mailing lists, working with everyone. Just test > >>> the > >>> -rc releases we make and let us know if they work or not for you, it's > >>> not a lot of "coordination" needed at all. > >>> > >>> Otherwise, if no one is saying that they are going to need these for 6 > >>> years and are willing to use it in their project (i.e. and test it), > >>> there's no need for us to maintain it for that long, right? > >> > >> Greg, please remember I expressed I really need them for slightly more > >> than > >> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time > >> permits if > >> this saves me from having to take over these kernels after you, like > >> in the > >> past, but I cannot engage on the regularity of my availability. > > > > Ok, great! > > > > That's one person/company saying they can help out (along with what CIP > > has been stating.) > > > > What about others? Broadcom started this conversation, odd that they > > don't seem to want to help out :) > Greg, I'm sorry but I'm not in a position to provide such a commitment. > >>> > >>> Ok, who at Broadcom do I need to talk to to get that type of commitment? > >> > >> I am not sure if I was too subtle before, we (Broadcom) cannot give you > >> an unified voice to speak with because we are divided in silos/business > >> units that make their independent decisions. > > > > That's fine, I'm totally used to that, large (and even small) companies > > always have different groups with different roadmaps and policies. > > > >> The group I work in (STB/CM, different from Scott's) is committed to > >> using the 5.10 kernel for 6 years and that is a decision that has been > >> taken. > > > > Great! Will you all be testing the -rc releases and letting me know how > > they work for your systems? > > Yes I will, can you add me to your CC list for the stable candidates? > That helps me not having to dig for those announcements specifically. Now added. Of course _after_ I just sent out a round of stable -rc releases, sorry about that :( greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
Hi Greg, On 26.01.2021 [08:29:25 +0100], Greg Kroah-Hartman wrote: > On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote: > > Hi All, > > > > The 5.10 LTS kernel being officially LTS supported for 2 years > > presents a problem: why would anyone select a 5.10 kernel with 2 > > year LTS when 5.4 kernel has a 6 year LTS. > > If 5.10 is "actually" going to be supported for 6 years it would be > > quite valuable to make such a declaration. > > https://www.kernel.org/category/releases.html > > Why? What would that change? > > Ok, seriously, this happens every year, and every year we go through > the same thing, it's not like this is somehow new, right? > > I want to see companies _using_ the kernel, and most importantly, > _updating_ their devices with it, to know if it is worth to keep > around for longer than 2 years. I also, hopefully, want to see how > those companies will help me out in the testing and maintenance of > that kernel version in order to make supporting it for 6 years > actually possible. > > So, are you planning on using 5.10? Will you will be willing to help > out in testing the -rc releases I make to let me know if there are any > problems, and to help in pointing out and backporting any specific > patches that your platforms need for that kernel release? > > When I get this kind of promises and support from companies, then I am > glad to bump up the length of the kernel support from 2 to 6 years, > and I mark it on the web site. Traditionally this happens in > Febuary/March once I hear from enough companies. Can I count on your > support in this endeavor? I am very sorry for the long delay on my end (I had privately e-mailed Greg on January 28) -- DigitalOcean also intends to provide feedback and testing on the 5.10 series. We intend to use it as the basis for our next-next kernel and are very invested in ensuring the stability and performance of the kernel. Thanks, Nish
Re: 5.10 LTS Kernel: 2 or 6 years?
On Saturday, February 20, 2021 6:05 PM, Greg Kroah-Hartman wrote: > On Sat, Feb 20, 2021 at 01:29:21PM +, Jari Ruusu wrote: > > I have been able to narrow the beginning of the problem to these kernels: > > 4.14.188 ... 4.14.202 > > Same "fix" that went info 4.14.y is also bugging 4.19.y kernels. > > Great, any chance you can narrow this down to the commit itself? I am not able to test WiFi on that laptop computer anymore, because that laptop now connects to world using wired connection. It was that WiFI->wired connection change that led me to realize the buggyness of in-tree iwlwifi in 4.19.y kernels. I did that narrowing to specific kernel versions by digging my archived backup files and their notes. No tests were run. Problems started triggering in those kernels that I mentioned in earlier email. That does not mean that the bugs were not already there in kernels older that those. Maybe some change just widened "window of opportunity" enough for me to see the issues. In-tree iwlwifi in 4.19.y is missing locking fixes. No amount of smooth-talking is going to change that. -- Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
Re: 5.10 LTS Kernel: 2 or 6 years?
On Sat, Feb 20, 2021 at 05:05:14PM +0100, Greg Kroah-Hartman wrote: > On Sat, Feb 20, 2021 at 01:29:21PM +, Jari Ruusu wrote: > > On Friday, February 19, 2021 5:23 PM, Jari Ruusu > > wrote: > > > My contribution here is trying to point you guys to right direction. > > > > I have been able to narrow the beginning of the problem to these kernels: > > 4.14.188 ... 4.14.202 > > > > Same "fix" that went info 4.14.y is also bugging 4.19.y kernels. > > Great, any chance you can narrow this down to the commit itself? What is strnage is that there was no iwlwifi driver change in this interval. Only iwlegacy (don't know if this one was used instead). Note, my laptop uses iwlwifi as well. I've met random issues with it a year ago when I started to use wifi, such as wifi randomly freezing during audio meetings, and automatically fixing itself after 1 minute. I also found that a down-up cycle on the interface would fix it. It happened less than once a day so it was not easy to diagnose, and given how crappy all wifi chips are, I naturally attributed this to the hardware. But since I upgraded to 5.4 months ago, I don't remember having met that issue anymore, so it was likely related to the driver. All I can say is that 4.19.68 was exhibiting this issue. I can't say for older ones because I didn't use wifi before. Just my two cents, Willy
Re: 5.10 LTS Kernel: 2 or 6 years?
On Sat, Feb 20, 2021 at 01:29:21PM +, Jari Ruusu wrote: > On Friday, February 19, 2021 5:23 PM, Jari Ruusu > wrote: > > My contribution here is trying to point you guys to right direction. > > I have been able to narrow the beginning of the problem to these kernels: > 4.14.188 ... 4.14.202 > > Same "fix" that went info 4.14.y is also bugging 4.19.y kernels. Great, any chance you can narrow this down to the commit itself? thanks, greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
On Friday, February 19, 2021 5:23 PM, Jari Ruusu wrote: > My contribution here is trying to point you guys to right direction. I have been able to narrow the beginning of the problem to these kernels: 4.14.188 ... 4.14.202 Same "fix" that went info 4.14.y is also bugging 4.19.y kernels. -- Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
Re: 5.10 LTS Kernel: 2 or 6 years?
On Sat, Feb 20, 2021 at 03:02:54PM +0800, Hanjun Guo wrote: > On 2021/2/19 17:08, Greg Kroah-Hartman wrote: > > On Fri, Feb 19, 2021 at 04:54:24PM +0800, Hanjun Guo wrote: > > > Hi Greg, > > > > > > On 2021/1/26 15:29, Greg Kroah-Hartman wrote: > > > [...] > > > > > > > > I want to see companies _using_ the kernel, and most importantly, > > > > _updating_ their devices with it, to know if it is worth to keep around > > > > for longer than 2 years. I also, hopefully, want to see how those > > > > companies will help me out in the testing and maintenance of that kernel > > > > version in order to make supporting it for 6 years actually possible. > > > > > > > > So, are you planning on using 5.10? Will you will be willing to help > > > > out in testing the -rc releases I make to let me know if there are any > > > > problems, and to help in pointing out and backporting any specific > > > > patches that your platforms need for that kernel release? > > > > > > We(Huawei) are willing to commit resources to help out in testing the > > > stable -rc releases, and to help to backport patches for stable kernels. > > > > Wonderful! > > > > > 5.10 stable kernel will be used for openEuler [1] kernel and also inside > > > Huawei. From customer's feedback, it's very important to see the stable > > > kernel we used to be maintained for 6 years in the community, and we > > > will use 5.10 kernel for at least 6 years, so we are willing to help > > > you and help ourselves :) > > > > > > In specific, we will start from the testing work, using HULK robot > > > (reports lots of bugs to mainline kernel) testing framework to test > > > compile, reboot, functional testing, and will extend to basic > > > performance regression testing in the future. > > > > Great! Do you all need an email notification when the -rc releases come > > out for the stable trees, or can you trigger off of the -rc stable git > > tree? Different CI systems work in different ways :) > > We can trigger the test when you updated the -rc stable git tree, > by monitoring new commits for the stable branches. So if you push > all the commits at once for -rc stable branches, then our CI system > can work well. I do push to the -rc branches, but those branches are rebased, and I do "intermediate" pushes as well. Meaning I push to have CI systems run on the existing patch queue at times that are not only the "main" -rc release periods. Watch the branches for a few weeks to get an idea of how they work if you are curious. > > And if you can reply to the -rc release emails with a "Tested-by:" tag, > > I will be glad to add that to the release commit when that happens to > > show that you all have tested the release. > > Thanks, will reply "Tested-by:" with -rc releases. We are working on > setting up the test farm and will report the test results in a week. Great, thanks! greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
On 2021/2/19 17:08, Greg Kroah-Hartman wrote: On Fri, Feb 19, 2021 at 04:54:24PM +0800, Hanjun Guo wrote: Hi Greg, On 2021/1/26 15:29, Greg Kroah-Hartman wrote: [...] I want to see companies _using_ the kernel, and most importantly, _updating_ their devices with it, to know if it is worth to keep around for longer than 2 years. I also, hopefully, want to see how those companies will help me out in the testing and maintenance of that kernel version in order to make supporting it for 6 years actually possible. So, are you planning on using 5.10? Will you will be willing to help out in testing the -rc releases I make to let me know if there are any problems, and to help in pointing out and backporting any specific patches that your platforms need for that kernel release? We(Huawei) are willing to commit resources to help out in testing the stable -rc releases, and to help to backport patches for stable kernels. Wonderful! 5.10 stable kernel will be used for openEuler [1] kernel and also inside Huawei. From customer's feedback, it's very important to see the stable kernel we used to be maintained for 6 years in the community, and we will use 5.10 kernel for at least 6 years, so we are willing to help you and help ourselves :) In specific, we will start from the testing work, using HULK robot (reports lots of bugs to mainline kernel) testing framework to test compile, reboot, functional testing, and will extend to basic performance regression testing in the future. Great! Do you all need an email notification when the -rc releases come out for the stable trees, or can you trigger off of the -rc stable git tree? Different CI systems work in different ways :) We can trigger the test when you updated the -rc stable git tree, by monitoring new commits for the stable branches. So if you push all the commits at once for -rc stable branches, then our CI system can work well. And if you can reply to the -rc release emails with a "Tested-by:" tag, I will be glad to add that to the release commit when that happens to show that you all have tested the release. Thanks, will reply "Tested-by:" with -rc releases. We are working on setting up the test farm and will report the test results in a week. And we will start from ARM64 and X86 architecture first, and then extend to other platforms. That's a good start, the useful ones :) For patch backporting, will send the bugfix patches (from mainline) we spotted, but I think this work may not doing in regular but will be triggered as needed. That's fine, it is not something that happens at a regular interval. Does this sound good to you? Yes it does, thank you so much. greg k-h Thanks Hanjun
Re: 5.10 LTS Kernel: 2 or 6 years?
On 2/19/2021 7:53 AM, Greg Kroah-Hartman wrote: > On Fri, Feb 19, 2021 at 07:05:41AM -0800, Florian Fainelli wrote: >> >> >> On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote: >>> On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote: On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote: > On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote: >> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote: >>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: As a company, we are most likely shooting ourselves in the foot by not having a point of coordination with the Linux Foundation and key people like you, Greg and other participants in the stable kernel. >>> >>> What does the LF have to do with this? >>> >>> We are here, on the mailing lists, working with everyone. Just test the >>> -rc releases we make and let us know if they work or not for you, it's >>> not a lot of "coordination" needed at all. >>> >>> Otherwise, if no one is saying that they are going to need these for 6 >>> years and are willing to use it in their project (i.e. and test it), >>> there's no need for us to maintain it for that long, right? >> >> Greg, please remember I expressed I really need them for slightly more >> than >> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits >> if >> this saves me from having to take over these kernels after you, like in >> the >> past, but I cannot engage on the regularity of my availability. > > Ok, great! > > That's one person/company saying they can help out (along with what CIP > has been stating.) > > What about others? Broadcom started this conversation, odd that they > don't seem to want to help out :) Greg, I'm sorry but I'm not in a position to provide such a commitment. >>> >>> Ok, who at Broadcom do I need to talk to to get that type of commitment? >> >> I am not sure if I was too subtle before, we (Broadcom) cannot give you >> an unified voice to speak with because we are divided in silos/business >> units that make their independent decisions. > > That's fine, I'm totally used to that, large (and even small) companies > always have different groups with different roadmaps and policies. > >> The group I work in (STB/CM, different from Scott's) is committed to >> using the 5.10 kernel for 6 years and that is a decision that has been >> taken. > > Great! Will you all be testing the -rc releases and letting me know how > they work for your systems? Yes I will, can you add me to your CC list for the stable candidates? That helps me not having to dig for those announcements specifically. > >> I could give you names of other decision makers in other business units >> I know who also deliver Linux for their respective business units >> however some of them may not make public appearances on mailing lists, >> let alone care about upstreaming their changes so I do not know whether >> a 6 years 5.10 kernel is even something they remotely entertain. > > That's fine, I'm not expecting emails from the list, we can take this > off-list if you like as it sounds like I need to talk to some different > managers there, right? :) I will put together a list and send you an email off list. -- Florian
Re: 5.10 LTS Kernel: 2 or 6 years?
On Fri, Feb 19, 2021 at 12:16:12PM +0100, Greg Kroah-Hartman wrote: > > Great! Can you run 'git bisect' on the 4.14.y stable tree to find the > offending change? To be fair, especially with WiFi bugs, you may need to run for hours or days before you are absolutely sure that a particular bisection point will result in the the locking/kernel crash bug to manifest itself. Worse, you have to be actively using the Wifi in order to see the problem, and in some cases, it only triggers when you switch between AP's, so you have to be actively using it in the work office, and taking it between conference rooms, only to see your machine crash taking your unsaved work, email drafts, etc. with it. That being said, users should at least report the bug. (That's what I did, when I ran into this a bunch of years ago, with an explanation of "I'm trying to do a bisect, but it may take a few weeks for me to figure out what the !@#!? is going on. In my case, I was trying to use upstream -rcX kernels to dogfood on my work laptop, but the principle is the same.) Ultiumately, I solved the problem, by switching laptops to one that didn't use an NVidia GPU (which sometimes forced me to stay 1-2 upstream versions behind, making life even more difficult when debugging these issues, until the out-of-tree video driver got updated to work with newer upstream), and which also had WiFi hardware which was less subject to these issues. It's unfortunate, but not all hardware is as well supported on Linux. And in my case, because $WORK was using Enterprise WiFi systems with AP's that don't get as much testing by developers, very few other people could repro the bugs. That's life, and sometimes the only solution is to switch hardware. And/or you just use a Chromebook in those sorts of situations, separating your work/enterprise and upstream development hardware, and be done with it. :-) Cheers, - Ted
Re: 5.10 LTS Kernel: 2 or 6 years?
On Fri, Feb 19, 2021 at 07:05:41AM -0800, Florian Fainelli wrote: > > > On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote: > > On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote: > >> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote: > >>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote: > On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote: > > On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: > >> As a company, we are most likely shooting ourselves in the foot by not > >> having a point of coordination with the Linux Foundation and key people > >> like you, Greg and other participants in the stable kernel. > > > > What does the LF have to do with this? > > > > We are here, on the mailing lists, working with everyone. Just test the > > -rc releases we make and let us know if they work or not for you, it's > > not a lot of "coordination" needed at all. > > > > Otherwise, if no one is saying that they are going to need these for 6 > > years and are willing to use it in their project (i.e. and test it), > > there's no need for us to maintain it for that long, right? > > Greg, please remember I expressed I really need them for slightly more > than > 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits > if > this saves me from having to take over these kernels after you, like in > the > past, but I cannot engage on the regularity of my availability. > >>> > >>> Ok, great! > >>> > >>> That's one person/company saying they can help out (along with what CIP > >>> has been stating.) > >>> > >>> What about others? Broadcom started this conversation, odd that they > >>> don't seem to want to help out :) > >> Greg, I'm sorry but I'm not in a position to provide such a commitment. > > > > Ok, who at Broadcom do I need to talk to to get that type of commitment? > > I am not sure if I was too subtle before, we (Broadcom) cannot give you > an unified voice to speak with because we are divided in silos/business > units that make their independent decisions. That's fine, I'm totally used to that, large (and even small) companies always have different groups with different roadmaps and policies. > The group I work in (STB/CM, different from Scott's) is committed to > using the 5.10 kernel for 6 years and that is a decision that has been > taken. Great! Will you all be testing the -rc releases and letting me know how they work for your systems? > I could give you names of other decision makers in other business units > I know who also deliver Linux for their respective business units > however some of them may not make public appearances on mailing lists, > let alone care about upstreaming their changes so I do not know whether > a 6 years 5.10 kernel is even something they remotely entertain. That's fine, I'm not expecting emails from the list, we can take this off-list if you like as it sounds like I need to talk to some different managers there, right? :) thanks, greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
On Friday, February 19, 2021 1:16 PM, Greg Kroah-Hartman wrote: > Great! Can you run 'git bisect' on the 4.14.y stable tree to find the > offending change? As I tried to explain earlier, that laptop's connection to world is no longer iwlwifi based but ethernet connection to fibre. I can't do much bisecting when iwlwifi is not being used at all. Sorry. My contribution here is trying to point you guys to right direction. -- Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
Re: 5.10 LTS Kernel: 2 or 6 years?
On 2/19/2021 12:25 AM, Greg Kroah-Hartman wrote: > On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote: >> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote: >>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote: On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote: > On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: >> As a company, we are most likely shooting ourselves in the foot by not >> having a point of coordination with the Linux Foundation and key people >> like you, Greg and other participants in the stable kernel. > > What does the LF have to do with this? > > We are here, on the mailing lists, working with everyone. Just test the > -rc releases we make and let us know if they work or not for you, it's > not a lot of "coordination" needed at all. > > Otherwise, if no one is saying that they are going to need these for 6 > years and are willing to use it in their project (i.e. and test it), > there's no need for us to maintain it for that long, right? Greg, please remember I expressed I really need them for slightly more than 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if this saves me from having to take over these kernels after you, like in the past, but I cannot engage on the regularity of my availability. >>> >>> Ok, great! >>> >>> That's one person/company saying they can help out (along with what CIP >>> has been stating.) >>> >>> What about others? Broadcom started this conversation, odd that they >>> don't seem to want to help out :) >> Greg, I'm sorry but I'm not in a position to provide such a commitment. > > Ok, who at Broadcom do I need to talk to to get that type of commitment? I am not sure if I was too subtle before, we (Broadcom) cannot give you an unified voice to speak with because we are divided in silos/business units that make their independent decisions. The group I work in (STB/CM, different from Scott's) is committed to using the 5.10 kernel for 6 years and that is a decision that has been taken. I could give you names of other decision makers in other business units I know who also deliver Linux for their respective business units however some of them may not make public appearances on mailing lists, let alone care about upstreaming their changes so I do not know whether a 6 years 5.10 kernel is even something they remotely entertain. Hope this helps. -- Florian
Re: 5.10 LTS Kernel: 2 or 6 years?
Hi Hanjun, On 2/19/21 10:54 AM, Hanjun Guo wrote: > In specific, we will start from the testing work, using HULK robot > (reports lots of bugs to mainline kernel) testing framework to test > compile, reboot, functional testing, and will extend to basic > performance regression testing in the future. I heard about Huawei ramping up kernel testing from someone at FOSDEM 2019. I wonder if it was you :) Nice to see your progress and the company stepping up to help with testing! Would you be interested in working with the Linux Foundation KernelCI project on submitting your build and test results to the common database - KCIDB? We are working on aggregating results from various testing systems so we can provide a dashboard, and a single, aggregated e-mail report to subscribed maintainers and developers. We have a prototype dashboard at https://staging.kernelci.org:3000/ and are working hard on making the e-mail reports good enough to start reaching out to maintainers. We already have ARM, Google Syzbot, Gentoo GKernelCI, Red Hat CKI, and, of course, KernelCI native tests sending data to the database. Linaro Tuxsuite is starting sending today. We could use your data, and of course any development help you could spare :) I wish I could show you my today's KCIDB presentation at DevConf.cz, but the recording is not out yet. Meanwhile you can take a look at our presentation at last year's Linux Plumbers: https://youtu.be/y9Glc90WUN0?t=10739 Or see our intro in an older blog post: https://foundation.kernelci.org/blog/2020/08/21/introducing-common-reporting/ Anyone wishing to contribute to KCIDB gets credentials and permissions to submit to our "playground" setup where they can send their data, see it in a dashboard, experiment without worrying about breaking anything, and decide if they like it or not. If you're interested, take a look at our Submission HOWTO: https://github.com/kernelci/kcidb/blob/v8/SUBMISSION_HOWTO.md and send an email to kerne...@groups.io (CC'd), or come over to the #kernelci channel on freenode.net! Nick On 2/19/21 10:54 AM, Hanjun Guo wrote: > Hi Greg, > > On 2021/1/26 15:29, Greg Kroah-Hartman wrote: > [...] >> >> I want to see companies _using_ the kernel, and most importantly, >> _updating_ their devices with it, to know if it is worth to keep around >> for longer than 2 years. I also, hopefully, want to see how those >> companies will help me out in the testing and maintenance of that kernel >> version in order to make supporting it for 6 years actually possible. >> >> So, are you planning on using 5.10? Will you will be willing to help >> out in testing the -rc releases I make to let me know if there are any >> problems, and to help in pointing out and backporting any specific >> patches that your platforms need for that kernel release? > > We(Huawei) are willing to commit resources to help out in testing the > stable -rc releases, and to help to backport patches for stable kernels. > > 5.10 stable kernel will be used for openEuler [1] kernel and also inside > Huawei. From customer's feedback, it's very important to see the stable > kernel we used to be maintained for 6 years in the community, and we > will use 5.10 kernel for at least 6 years, so we are willing to help > you and help ourselves :) > > In specific, we will start from the testing work, using HULK robot > (reports lots of bugs to mainline kernel) testing framework to test > compile, reboot, functional testing, and will extend to basic > performance regression testing in the future. > > And we will start from ARM64 and X86 architecture first, and then extend > to other platforms. > > For patch backporting, will send the bugfix patches (from mainline) > we spotted, but I think this work may not doing in regular but will > be triggered as needed. > > Does this sound good to you? > > Thanks > Hanjun > > [1]: https://openeuler.org/en/ > > ___ > linux-arm-kernel mailing list > linux-arm-ker...@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel >
Re: 5.10 LTS Kernel: 2 or 6 years?
On Fri, Feb 19, 2021 at 10:57:30AM +, Jari Ruusu wrote: > On Friday, February 19, 2021 12:37 PM, Greg Kroah-Hartman > wrote: > > Did you report that breakage to us and the developers of the driver? > > Sounds like a regression that people would love to hear about and get > > fixed. > > At that time I didn't know where the problem was, so I did not report it. > I only recently connected-the-dots that problem is in-tree iwlwifi, so I > am reporting it now. Great! Can you run 'git bisect' on the 4.14.y stable tree to find the offending change? thanks, greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
On Friday, February 19, 2021 12:37 PM, Greg Kroah-Hartman wrote: > Did you report that breakage to us and the developers of the driver? > Sounds like a regression that people would love to hear about and get > fixed. At that time I didn't know where the problem was, so I did not report it. I only recently connected-the-dots that problem is in-tree iwlwifi, so I am reporting it now. -- Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
Re: 5.10 LTS Kernel: 2 or 6 years?
On Fri, Feb 19, 2021 at 10:31:49AM +, Jari Ruusu wrote: > On Friday, February 19, 2021 10:22 AM, Greg Kroah-Hartman > wrote: > > That's not the goal of stable kernel releases/trees. If the driver > > version that is in 4.19.y does not work for you on release 4.19.0, odds > > of that "changing" in later stable releases is slim to none. > > But in-tree iwlwifi worked fine on 4.9.y and 4.14.y , and then got broken > in later 4.14.y versions. Did you report that breakage to us and the developers of the driver? Sounds like a regression that people would love to hear about and get fixed. > I tried later versions of 4.19.y in an attempt to > fix the breakage. Basically you are are saying that I should have NOT > attempted to upgrage 4.9.y -> 4.14.y -> 4.19.y Without anyone reporting a problem, how can anyone know to fix it? thanks, greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
On Friday, February 19, 2021 10:22 AM, Greg Kroah-Hartman wrote: > That's not the goal of stable kernel releases/trees. If the driver > version that is in 4.19.y does not work for you on release 4.19.0, odds > of that "changing" in later stable releases is slim to none. But in-tree iwlwifi worked fine on 4.9.y and 4.14.y , and then got broken in later 4.14.y versions. I tried later versions of 4.19.y in an attempt to fix the breakage. Basically you are are saying that I should have NOT attempted to upgrage 4.9.y -> 4.14.y -> 4.19.y -- Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
Re: 5.10 LTS Kernel: 2 or 6 years?
On Fri, Feb 19, 2021 at 04:54:24PM +0800, Hanjun Guo wrote: > Hi Greg, > > On 2021/1/26 15:29, Greg Kroah-Hartman wrote: > [...] > > > > I want to see companies _using_ the kernel, and most importantly, > > _updating_ their devices with it, to know if it is worth to keep around > > for longer than 2 years. I also, hopefully, want to see how those > > companies will help me out in the testing and maintenance of that kernel > > version in order to make supporting it for 6 years actually possible. > > > > So, are you planning on using 5.10? Will you will be willing to help > > out in testing the -rc releases I make to let me know if there are any > > problems, and to help in pointing out and backporting any specific > > patches that your platforms need for that kernel release? > > We(Huawei) are willing to commit resources to help out in testing the > stable -rc releases, and to help to backport patches for stable kernels. Wonderful! > 5.10 stable kernel will be used for openEuler [1] kernel and also inside > Huawei. From customer's feedback, it's very important to see the stable > kernel we used to be maintained for 6 years in the community, and we > will use 5.10 kernel for at least 6 years, so we are willing to help > you and help ourselves :) > > In specific, we will start from the testing work, using HULK robot > (reports lots of bugs to mainline kernel) testing framework to test > compile, reboot, functional testing, and will extend to basic > performance regression testing in the future. Great! Do you all need an email notification when the -rc releases come out for the stable trees, or can you trigger off of the -rc stable git tree? Different CI systems work in different ways :) And if you can reply to the -rc release emails with a "Tested-by:" tag, I will be glad to add that to the release commit when that happens to show that you all have tested the release. > And we will start from ARM64 and X86 architecture first, and then extend > to other platforms. That's a good start, the useful ones :) > For patch backporting, will send the bugfix patches (from mainline) > we spotted, but I think this work may not doing in regular but will > be triggered as needed. That's fine, it is not something that happens at a regular interval. > Does this sound good to you? Yes it does, thank you so much. greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
Hi Greg, On 2021/1/26 15:29, Greg Kroah-Hartman wrote: [...] I want to see companies _using_ the kernel, and most importantly, _updating_ their devices with it, to know if it is worth to keep around for longer than 2 years. I also, hopefully, want to see how those companies will help me out in the testing and maintenance of that kernel version in order to make supporting it for 6 years actually possible. So, are you planning on using 5.10? Will you will be willing to help out in testing the -rc releases I make to let me know if there are any problems, and to help in pointing out and backporting any specific patches that your platforms need for that kernel release? We(Huawei) are willing to commit resources to help out in testing the stable -rc releases, and to help to backport patches for stable kernels. 5.10 stable kernel will be used for openEuler [1] kernel and also inside Huawei. From customer's feedback, it's very important to see the stable kernel we used to be maintained for 6 years in the community, and we will use 5.10 kernel for at least 6 years, so we are willing to help you and help ourselves :) In specific, we will start from the testing work, using HULK robot (reports lots of bugs to mainline kernel) testing framework to test compile, reboot, functional testing, and will extend to basic performance regression testing in the future. And we will start from ARM64 and X86 architecture first, and then extend to other platforms. For patch backporting, will send the bugfix patches (from mainline) we spotted, but I think this work may not doing in regular but will be triggered as needed. Does this sound good to you? Thanks Hanjun [1]: https://openeuler.org/en/
Re: 5.10 LTS Kernel: 2 or 6 years?
On Fri, Feb 19, 2021 at 09:00:27AM +0100, Pavel Machek wrote: > Hi! > > > > > > For me > > > > > only way to get properly working WiFi on my laptop computer is to > > > > > compile that Intel out-of-tree version. Sad, but true. > > > > > > > > Why use 4.19.y on a laptop in the firstplace? That feels very wrong and > > > > is not the recommended thing to use the LTS kernels for. > > > > > > Well, that's actually what distributions are doing, for example Debian > > > 10.8 is on 4.19... > > > > There's 5.10 in buster-backports. That's probably the easiest way to get > > support for new HW. > > > > I can compile my own kernel, too. But if you go up the thread, it is > about iwlwifi becoming broken in 4.19, and Greg saying it is wrong > to put -stable on laptop. And -stable on laptop is norm, not the > exception. If distros want to "camp out" on an old kernel version, that's fine, as I describe in: http://www.kroah.com/log/blog/2018/08/24/what-stable-kernel-should-i-use/ many years ago. But for someone using a device where they want to use "new" hardware, that was made _after_ the kernel version was released, that's just someone who needs to pick a better distro :) thanks, greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote: > On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote: > > On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote: > >> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote: > >>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: > As a company, we are most likely shooting ourselves in the foot by not > having a point of coordination with the Linux Foundation and key people > like you, Greg and other participants in the stable kernel. > >>> > >>> What does the LF have to do with this? > >>> > >>> We are here, on the mailing lists, working with everyone. Just test the > >>> -rc releases we make and let us know if they work or not for you, it's > >>> not a lot of "coordination" needed at all. > >>> > >>> Otherwise, if no one is saying that they are going to need these for 6 > >>> years and are willing to use it in their project (i.e. and test it), > >>> there's no need for us to maintain it for that long, right? > >> > >> Greg, please remember I expressed I really need them for slightly more than > >> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if > >> this saves me from having to take over these kernels after you, like in the > >> past, but I cannot engage on the regularity of my availability. > > > > Ok, great! > > > > That's one person/company saying they can help out (along with what CIP > > has been stating.) > > > > What about others? Broadcom started this conversation, odd that they > > don't seem to want to help out :) > Greg, I'm sorry but I'm not in a position to provide such a commitment. Ok, who at Broadcom do I need to talk to to get that type of commitment? > My original question arose because the 5.10 kernel is declared as 2 years LTS > while older LTS kernels are now 6 years. > One problem this has created is requests to provide silicon support in an > older kernel version (for a new project) rather than starting from a newer > kernel version that more properly supports the (silicon and non-silicon) > features. Sounds like your development model is broken, again, who do I need to talk to in order to help you all fix this? thanks, greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
On Fri, Feb 19, 2021 at 07:10:35AM +, Jari Ruusu wrote: > On Thursday, February 18, 2021 7:44 PM, Greg Kroah-Hartman > wrote: > > > It was the other way around. Fine working in-tree driver got > > > broken by backported "fixes". I did mention bit-rot. > > > > It did? Please let us stable maintainers know about, we will always > > gladly revert problems patches. What commits caused the problem? > > I don't have a list of commits for you. It took me long time to > figure out that it was iwlwifi that was causing those problems. > > In-tree iwlwifi on 4.19.y kernels needs professional quality > locking audit and backporting of necessary fixes from upstream > Intel out-of-tree version. That's not the goal of stable kernel releases/trees. If the driver version that is in 4.19.y does not work for you on release 4.19.0, odds of that "changing" in later stable releases is slim to none. Especially without any specific bug reports or emails to the developers to tell them what is broken and not working for you. If however, you wish to stick with an out-of-tree driver, wonderful, that's your choice. But don't claim that somehow the stable kernel process is broken because of that, as it has nothing to do with this type of thing. Best of luck! greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
Hi! > > > > For me > > > > only way to get properly working WiFi on my laptop computer is to > > > > compile that Intel out-of-tree version. Sad, but true. > > > > > > Why use 4.19.y on a laptop in the firstplace? That feels very wrong and > > > is not the recommended thing to use the LTS kernels for. > > > > Well, that's actually what distributions are doing, for example Debian > > 10.8 is on 4.19... > > There's 5.10 in buster-backports. That's probably the easiest way to get > support for new HW. > I can compile my own kernel, too. But if you go up the thread, it is about iwlwifi becoming broken in 4.19, and Greg saying it is wrong to put -stable on laptop. And -stable on laptop is norm, not the exception. Pavel -- http://www.livejournal.com/~pavelmachek signature.asc Description: Digital signature
Re: 5.10 LTS Kernel: 2 or 6 years?
On Thursday, February 18, 2021 7:44 PM, Greg Kroah-Hartman wrote: > > It was the other way around. Fine working in-tree driver got > > broken by backported "fixes". I did mention bit-rot. > > It did? Please let us stable maintainers know about, we will always > gladly revert problems patches. What commits caused the problem? I don't have a list of commits for you. It took me long time to figure out that it was iwlwifi that was causing those problems. In-tree iwlwifi on 4.19.y kernels needs professional quality locking audit and backporting of necessary fixes from upstream Intel out-of-tree version. > So something in the 4.9.y and 4.14.y stable kernels caused a regression, > can you please do 'git bisect' to let us know what broke? My ability to do WiFi tests on that laptop computer are limited for now. Earlier that laptop's connectivity to world was through mobile WiFi router. That mobile WiFi router no longer has a SIM-card, so no connectivity through that anymore. That laptop's connectivity to world is now through wired ethernet to fiber. It was actually this switch to ethernet/fiber that made me realize the brokenness on in-tree iwlwifi on 4.19.y kernels. When in-tree WiFi was not used, those problems never triggered. Switched back to in-tree WiFi, and problems came back. Switched to out-of-tree Intel version of iwlwifi, problems went away again. Then I looked at the fixes in out-of-tree Intel version of iwlwifi that were missing in in-tree version, and I understood why that was so. Those stability issues on in-tree iwlwifi on 4.19.y kernels are difficult to trigger. Sometimes it may take days to trigger it once. Sometimes I was unlucky enough to trigger them more than once a day. I say that two weeks of operation without issues are needed to pronounce those issues gone. Currently, special arrangements are needed for me to test WiFi at all on that laptop computer, and those arrangements are something I am not willing to commit for multi-week testing run. Sorry. > And if 4.19.0 was always broken, why didn't you report that as well? I didn't test early 4.19.y kernels. -- Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
Re: 5.10 LTS Kernel: 2 or 6 years?
On Thursday 18 February 2021 21:55:34 Pavel Machek wrote: > Hi! > > > > For me > > > only way to get properly working WiFi on my laptop computer is to > > > compile that Intel out-of-tree version. Sad, but true. > > > > Why use 4.19.y on a laptop in the firstplace? That feels very wrong and > > is not the recommended thing to use the LTS kernels for. > > Well, that's actually what distributions are doing, for example Debian > 10.8 is on 4.19... There's 5.10 in buster-backports. That's probably the easiest way to get support for new HW. > I expect -stable is what most users are running on their notebooks. > > Best regards, > Pavel -- Ondrej Zary
Re: 5.10 LTS Kernel: 2 or 6 years?
Hi Willy, On 2021-02-18 1:00 p.m., Willy Tarreau wrote: > On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote: >> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote: >>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote: On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote: > On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: >> As a company, we are most likely shooting ourselves in the foot by not >> having a point of coordination with the Linux Foundation and key people >> like you, Greg and other participants in the stable kernel. > > What does the LF have to do with this? > > We are here, on the mailing lists, working with everyone. Just test the > -rc releases we make and let us know if they work or not for you, it's > not a lot of "coordination" needed at all. > > Otherwise, if no one is saying that they are going to need these for 6 > years and are willing to use it in their project (i.e. and test it), > there's no need for us to maintain it for that long, right? Greg, please remember I expressed I really need them for slightly more than 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if this saves me from having to take over these kernels after you, like in the past, but I cannot engage on the regularity of my availability. >>> >>> Ok, great! >>> >>> That's one person/company saying they can help out (along with what CIP >>> has been stating.) >>> >>> What about others? Broadcom started this conversation, odd that they >>> don't seem to want to help out :) >> Greg, I'm sorry but I'm not in a position to provide such a commitment. > > Are you at least in a position to defend that ? There are necessarily > some people in your company who understand the benefits of using open > source provdided for free by others and who understand that devoting > a few people's time to this task is extremely cheap compared to the > amount of work required by having to do it entirely yourself for a > lower quality. > >> My original question arose because the 5.10 kernel is declared as 2 years LTS >> while older LTS kernels are now 6 years. > (...) >> If all LTS kernels were declared as 3.5-4 years as Willy commented this would >> solve a few issues. 6 year LTS kernels would only have a maximum 1 year >> lifespan over the latest declared LTS kernel. Also, many products take a year >> or more to develop, there isn't any life left in an LTS kernel if it is only >> 2 years. > > We all have the same problem regarding this but how do you want Greg to > engage into such a task by himself if he's not certain he can count on > others to help ? The few of us having worked on extended kernels know > that there's a limit around 2.5 years beyond which backports become much > harder to perform and to test. Doing it every year would result in 6 LTS > kernels to maintain in addition to the last 1-2 stable ones. That becomes > a huge amount of work! I even think that having one LTS kernel every 2 > years but maintained one extra year (e.g. 5 vs 4 in my case) would reduce > the effort. > >> After 1-3 years of kernel age the relevant parties that want to invest and >> care about supporting specific kernel versions longer should become apparent >> and could commit to longer support. > > But that's exactly what's currently being done. Greg initially commits > to 2 years hoping to get some help to pursue this longer, and this causes > trouble to some of us not being certain upfront whether or not we're choosing > the right kernel. So only the solution I'm seeing is for Greg to know > early who jumps in so that those of us without the power or skill to > entirely maintain a kernel by themselves know early which version to > choose. Quite frankly if we ship an LTS kernel in a product, the least > we can do is to give back a little bit to make sure the situation remains > durable. > > As such even if you are not in a position to provide such a commitment, > I'd appreciate it if you would bring these arguments to those who are in > such a position, so that I don't end up as one of the too few ones having > to share a significant part of that task to make sure this valuable kernel > continues to exist. Thanks - will forward such info as necessary. > > Thanks, > Willy > smime.p7s Description: S/MIME Cryptographic Signature
Re: 5.10 LTS Kernel: 2 or 6 years?
On 2021-02-18 1:39 p.m., Sasha Levin wrote: > On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote: >> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote: >>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote: On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote: > On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: >> As a company, we are most likely shooting ourselves in the foot by not >> having a point of coordination with the Linux Foundation and key people >> like you, Greg and other participants in the stable kernel. > > What does the LF have to do with this? > > We are here, on the mailing lists, working with everyone. Just test the > -rc releases we make and let us know if they work or not for you, it's > not a lot of "coordination" needed at all. > > Otherwise, if no one is saying that they are going to need these for 6 > years and are willing to use it in their project (i.e. and test it), > there's no need for us to maintain it for that long, right? Greg, please remember I expressed I really need them for slightly more than 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if this saves me from having to take over these kernels after you, like in the past, but I cannot engage on the regularity of my availability. >>> >>> Ok, great! >>> >>> That's one person/company saying they can help out (along with what CIP >>> has been stating.) >>> >>> What about others? Broadcom started this conversation, odd that they >>> don't seem to want to help out :) >> Greg, I'm sorry but I'm not in a position to provide such a commitment. >> >> My original question arose because the 5.10 kernel is declared as 2 years >> LTS while older LTS kernels are now 6 years. >> One problem this has created is requests to provide silicon support in an >> older kernel version (for a new project) rather than starting from a newer >> kernel version that more properly supports the (silicon and non-silicon) >> features. > > So this sounds like you have boatloads of out-of-tree code and need a > stable kernel to avoid having to rebase that code. This is not why the > LTS trees are around. My question centered around the inconsistency of a 2 year LTS kernel and some of the problems of having older LTS kernels declared as 6 years. The discussion has gone sideways from there with many other assumptions. There is not always control over what kernel version a customer uses, how many hacks they have in it, when they upgrade it, what other vendor changes are in it, etc. But, will keep this email chain to forward to others. It may help influence some decisions. > > For new projects, the easiest route is to upstream your stuff and ship > the latest kernel. > >> If all LTS kernels were declared as 3.5-4 years as Willy commented this >> would solve a few issues. >> 6 year LTS kernels would only have a maximum 1 year lifespan over the latest >> declared LTS kernel. >> Also, many products take a year or more to develop, there isn't any life >> left in an LTS kernel if it is only 2 years. > > Products are supposed to upgrade their kernel. If you released something > with, for example, a 5.4 kernel, doesn't mean you're forever stuck on a > 5.4 kernel for that product. > >> After 1-3 years of kernel age the relevant parties that want to invest and >> care about supporting specific kernel versions longer should become apparent >> and could commit to longer support. Perhaps you move the burden of 6 years >> LTS elsewhere to longer term projects. But, I'm sure many are happy because >> you continue doing such a great job in a central location, especially those >> whose product lifespan is around 6 years. > > But this is exactly what's happening now: we support LTS kernels for two > years, and after that interested parties can figure it out on their own > if it's worth it for them to keep going. > smime.p7s Description: S/MIME Cryptographic Signature
Re: 5.10 LTS Kernel: 2 or 6 years?
On 2/18/2021 1:39 PM, Sasha Levin wrote: > On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote: >> On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote: >>> On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote: On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote: > On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: >> As a company, we are most likely shooting ourselves in the foot by >> not >> having a point of coordination with the Linux Foundation and key >> people >> like you, Greg and other participants in the stable kernel. > > What does the LF have to do with this? > > We are here, on the mailing lists, working with everyone. Just > test the > -rc releases we make and let us know if they work or not for you, it's > not a lot of "coordination" needed at all. > > Otherwise, if no one is saying that they are going to need these for 6 > years and are willing to use it in their project (i.e. and test it), > there's no need for us to maintain it for that long, right? Greg, please remember I expressed I really need them for slightly more than 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if this saves me from having to take over these kernels after you, like in the past, but I cannot engage on the regularity of my availability. >>> >>> Ok, great! >>> >>> That's one person/company saying they can help out (along with what CIP >>> has been stating.) >>> >>> What about others? Broadcom started this conversation, odd that they >>> don't seem to want to help out :) >> Greg, I'm sorry but I'm not in a position to provide such a commitment. >> >> My original question arose because the 5.10 kernel is declared as 2 >> years LTS while older LTS kernels are now 6 years. >> One problem this has created is requests to provide silicon support in >> an older kernel version (for a new project) rather than starting from >> a newer kernel version that more properly supports the (silicon and >> non-silicon) features. > > So this sounds like you have boatloads of out-of-tree code and need a > stable kernel to avoid having to rebase that code. This is not why the > LTS trees are around. > > For new projects, the easiest route is to upstream your stuff and ship > the latest kernel. Agreed, however no matter how good you are at upstreaming your changes, there is always a delta that you keep accumulating, it is just the reality of things be it because a specific piece of code was just harder to upstream you missed multiple kernel versions, or it was not suitable, or it changed, or anything. Not everything goes upstream even if that gets closer and closer to zero if you do a good job at it. > >> If all LTS kernels were declared as 3.5-4 years as Willy commented >> this would solve a few issues. >> 6 year LTS kernels would only have a maximum 1 year lifespan over the >> latest declared LTS kernel. >> Also, many products take a year or more to develop, there isn't any >> life left in an LTS kernel if it is only 2 years. > > Products are supposed to upgrade their kernel. If you released something > with, for example, a 5.4 kernel, doesn't mean you're forever stuck on a > 5.4 kernel for that product. The are lots of Android devices manufacturers that launch using of the 3 kernel versions supported by the given Android release they targeted and they do not update the kernel version ever after that. > >> After 1-3 years of kernel age the relevant parties that want to invest >> and care about supporting specific kernel versions longer should >> become apparent and could commit to longer support. Perhaps you move >> the burden of 6 years LTS elsewhere to longer term projects. But, I'm >> sure many are happy because you continue doing such a great job in a >> central location, especially those whose product lifespan is around 6 >> years. > > But this is exactly what's happening now: we support LTS kernels for two > years, and after that interested parties can figure it out on their own > if it's worth it for them to keep going. > -- Florian
Re: 5.10 LTS Kernel: 2 or 6 years?
On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote: On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote: On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote: On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote: On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: As a company, we are most likely shooting ourselves in the foot by not having a point of coordination with the Linux Foundation and key people like you, Greg and other participants in the stable kernel. What does the LF have to do with this? We are here, on the mailing lists, working with everyone. Just test the -rc releases we make and let us know if they work or not for you, it's not a lot of "coordination" needed at all. Otherwise, if no one is saying that they are going to need these for 6 years and are willing to use it in their project (i.e. and test it), there's no need for us to maintain it for that long, right? Greg, please remember I expressed I really need them for slightly more than 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if this saves me from having to take over these kernels after you, like in the past, but I cannot engage on the regularity of my availability. Ok, great! That's one person/company saying they can help out (along with what CIP has been stating.) What about others? Broadcom started this conversation, odd that they don't seem to want to help out :) Greg, I'm sorry but I'm not in a position to provide such a commitment. My original question arose because the 5.10 kernel is declared as 2 years LTS while older LTS kernels are now 6 years. One problem this has created is requests to provide silicon support in an older kernel version (for a new project) rather than starting from a newer kernel version that more properly supports the (silicon and non-silicon) features. So this sounds like you have boatloads of out-of-tree code and need a stable kernel to avoid having to rebase that code. This is not why the LTS trees are around. For new projects, the easiest route is to upstream your stuff and ship the latest kernel. If all LTS kernels were declared as 3.5-4 years as Willy commented this would solve a few issues. 6 year LTS kernels would only have a maximum 1 year lifespan over the latest declared LTS kernel. Also, many products take a year or more to develop, there isn't any life left in an LTS kernel if it is only 2 years. Products are supposed to upgrade their kernel. If you released something with, for example, a 5.4 kernel, doesn't mean you're forever stuck on a 5.4 kernel for that product. After 1-3 years of kernel age the relevant parties that want to invest and care about supporting specific kernel versions longer should become apparent and could commit to longer support. Perhaps you move the burden of 6 years LTS elsewhere to longer term projects. But, I'm sure many are happy because you continue doing such a great job in a central location, especially those whose product lifespan is around 6 years. But this is exactly what's happening now: we support LTS kernels for two years, and after that interested parties can figure it out on their own if it's worth it for them to keep going. -- Thanks, Sasha
Re: 5.10 LTS Kernel: 2 or 6 years?
On Thu, Feb 18, 2021 at 12:16:50PM -0800, Scott Branden wrote: > On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote: > > On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote: > >> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote: > >>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: > As a company, we are most likely shooting ourselves in the foot by not > having a point of coordination with the Linux Foundation and key people > like you, Greg and other participants in the stable kernel. > >>> > >>> What does the LF have to do with this? > >>> > >>> We are here, on the mailing lists, working with everyone. Just test the > >>> -rc releases we make and let us know if they work or not for you, it's > >>> not a lot of "coordination" needed at all. > >>> > >>> Otherwise, if no one is saying that they are going to need these for 6 > >>> years and are willing to use it in their project (i.e. and test it), > >>> there's no need for us to maintain it for that long, right? > >> > >> Greg, please remember I expressed I really need them for slightly more than > >> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if > >> this saves me from having to take over these kernels after you, like in the > >> past, but I cannot engage on the regularity of my availability. > > > > Ok, great! > > > > That's one person/company saying they can help out (along with what CIP > > has been stating.) > > > > What about others? Broadcom started this conversation, odd that they > > don't seem to want to help out :) > Greg, I'm sorry but I'm not in a position to provide such a commitment. Are you at least in a position to defend that ? There are necessarily some people in your company who understand the benefits of using open source provdided for free by others and who understand that devoting a few people's time to this task is extremely cheap compared to the amount of work required by having to do it entirely yourself for a lower quality. > My original question arose because the 5.10 kernel is declared as 2 years LTS > while older LTS kernels are now 6 years. (...) > If all LTS kernels were declared as 3.5-4 years as Willy commented this would > solve a few issues. 6 year LTS kernels would only have a maximum 1 year > lifespan over the latest declared LTS kernel. Also, many products take a year > or more to develop, there isn't any life left in an LTS kernel if it is only > 2 years. We all have the same problem regarding this but how do you want Greg to engage into such a task by himself if he's not certain he can count on others to help ? The few of us having worked on extended kernels know that there's a limit around 2.5 years beyond which backports become much harder to perform and to test. Doing it every year would result in 6 LTS kernels to maintain in addition to the last 1-2 stable ones. That becomes a huge amount of work! I even think that having one LTS kernel every 2 years but maintained one extra year (e.g. 5 vs 4 in my case) would reduce the effort. > After 1-3 years of kernel age the relevant parties that want to invest and > care about supporting specific kernel versions longer should become apparent > and could commit to longer support. But that's exactly what's currently being done. Greg initially commits to 2 years hoping to get some help to pursue this longer, and this causes trouble to some of us not being certain upfront whether or not we're choosing the right kernel. So only the solution I'm seeing is for Greg to know early who jumps in so that those of us without the power or skill to entirely maintain a kernel by themselves know early which version to choose. Quite frankly if we ship an LTS kernel in a product, the least we can do is to give back a little bit to make sure the situation remains durable. As such even if you are not in a position to provide such a commitment, I'd appreciate it if you would bring these arguments to those who are in such a position, so that I don't end up as one of the too few ones having to share a significant part of that task to make sure this valuable kernel continues to exist. Thanks, Willy
Re: 5.10 LTS Kernel: 2 or 6 years?
Hi! > > For me > > only way to get properly working WiFi on my laptop computer is to > > compile that Intel out-of-tree version. Sad, but true. > > Why use 4.19.y on a laptop in the firstplace? That feels very wrong and > is not the recommended thing to use the LTS kernels for. Well, that's actually what distributions are doing, for example Debian 10.8 is on 4.19... I expect -stable is what most users are running on their notebooks. Best regards, Pavel -- http://www.livejournal.com/~pavelmachek signature.asc Description: PGP signature
Re: 5.10 LTS Kernel: 2 or 6 years?
On 2021-02-18 10:36 a.m., Greg Kroah-Hartman wrote: > On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote: >> On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote: >>> On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: As a company, we are most likely shooting ourselves in the foot by not having a point of coordination with the Linux Foundation and key people like you, Greg and other participants in the stable kernel. >>> >>> What does the LF have to do with this? >>> >>> We are here, on the mailing lists, working with everyone. Just test the >>> -rc releases we make and let us know if they work or not for you, it's >>> not a lot of "coordination" needed at all. >>> >>> Otherwise, if no one is saying that they are going to need these for 6 >>> years and are willing to use it in their project (i.e. and test it), >>> there's no need for us to maintain it for that long, right? >> >> Greg, please remember I expressed I really need them for slightly more than >> 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if >> this saves me from having to take over these kernels after you, like in the >> past, but I cannot engage on the regularity of my availability. > > Ok, great! > > That's one person/company saying they can help out (along with what CIP > has been stating.) > > What about others? Broadcom started this conversation, odd that they > don't seem to want to help out :) Greg, I'm sorry but I'm not in a position to provide such a commitment. My original question arose because the 5.10 kernel is declared as 2 years LTS while older LTS kernels are now 6 years. One problem this has created is requests to provide silicon support in an older kernel version (for a new project) rather than starting from a newer kernel version that more properly supports the (silicon and non-silicon) features. If all LTS kernels were declared as 3.5-4 years as Willy commented this would solve a few issues. 6 year LTS kernels would only have a maximum 1 year lifespan over the latest declared LTS kernel. Also, many products take a year or more to develop, there isn't any life left in an LTS kernel if it is only 2 years. After 1-3 years of kernel age the relevant parties that want to invest and care about supporting specific kernel versions longer should become apparent and could commit to longer support. Perhaps you move the burden of 6 years LTS elsewhere to longer term projects. But, I'm sure many are happy because you continue doing such a great job in a central location, especially those whose product lifespan is around 6 years. > > thanks, > > greg k-h > smime.p7s Description: S/MIME Cryptographic Signature
Re: 5.10 LTS Kernel: 2 or 6 years?
On Thu, Feb 18, 2021 at 07:20:50PM +0100, Willy Tarreau wrote: > On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote: > > On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: > > > As a company, we are most likely shooting ourselves in the foot by not > > > having a point of coordination with the Linux Foundation and key people > > > like you, Greg and other participants in the stable kernel. > > > > What does the LF have to do with this? > > > > We are here, on the mailing lists, working with everyone. Just test the > > -rc releases we make and let us know if they work or not for you, it's > > not a lot of "coordination" needed at all. > > > > Otherwise, if no one is saying that they are going to need these for 6 > > years and are willing to use it in their project (i.e. and test it), > > there's no need for us to maintain it for that long, right? > > Greg, please remember I expressed I really need them for slightly more than > 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if > this saves me from having to take over these kernels after you, like in the > past, but I cannot engage on the regularity of my availability. Ok, great! That's one person/company saying they can help out (along with what CIP has been stating.) What about others? Broadcom started this conversation, odd that they don't seem to want to help out :) thanks, greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
On Thu, Feb 18, 2021 at 06:53:56PM +0100, Greg Kroah-Hartman wrote: > On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: > > As a company, we are most likely shooting ourselves in the foot by not > > having a point of coordination with the Linux Foundation and key people > > like you, Greg and other participants in the stable kernel. > > What does the LF have to do with this? > > We are here, on the mailing lists, working with everyone. Just test the > -rc releases we make and let us know if they work or not for you, it's > not a lot of "coordination" needed at all. > > Otherwise, if no one is saying that they are going to need these for 6 > years and are willing to use it in their project (i.e. and test it), > there's no need for us to maintain it for that long, right? Greg, please remember I expressed I really need them for slightly more than 3 years (say 3.5-4) :-) I'm fine with helping a bit more as time permits if this saves me from having to take over these kernels after you, like in the past, but I cannot engage on the regularity of my availability. Overall I think that a lot of people completely underestimate the amount of work it requires to maintain stable kernels, and how much it could be distributed. By having a bunch of users participate a little bit more (e.g. by sometimes backporting the patches that are essential to them, by testing what's relevant to their use case), it already offloads a lot of work. I don't think the extra work requires to be much organized if there are enough participants to share the efforts. Regards, Willy
Re: 5.10 LTS Kernel: 2 or 6 years?
Hi Florian! On Thu, Feb 18, 2021 at 09:42:00AM -0800, Florian Fainelli wrote: > > Other difficulty with the LTS version is the frequency it is updated. We > > would not > > pickup the changes that frequently to test. A quarterly, bi-annually, or > > when a critical fix > > is identified would be when we update and perform any meaningful testing > > when in maintainence. > > Well you really have the best of both worlds here, you are not forced to > update your downstream fork of the stable kernel every week, though > bonus points if you do. > > For instance I try to test all the stable release candidates for 4.9, > 5.4 and 5.10, and merge the stable tags every week when they show up. We > do release to our customers every 6 weeks though, so usually they will > jump several stable releases, and they are fine with that. > > Yes it takes time, but I would rather do that than have to continuously > respond to questions about is this CVE fixed in kernel X.Y.Z like it > used to be before we did that. It also helps catch problem faster before > customers do, the usual benefits of continuous integration/delivery. Totally agreed! In our use case at haproxy, it's slightly different but not that much. We're much less sensitive to kernel bugs except for the network parts and any long-term stability issue. However we're extremely sensitive to openssl bugs and haproxy bugs. Thus we use them as triggers to emit a new release and we systematically update the kernel to the latest one. Our local patches usually apply pretty well on top of that. We face maybe 1-3 rejects a year which take half an hour of extra manual work, and roughly one regression every 3 years, essentially caused by one of our local patches applying to the wrong place due to changes and not caught by QA tests before being put online. I think that in ~15 years (we started with 2.4), only a single customer was ever affected by a regression caused by the kernel, it is so low we could almost laugh about it. Quite frankly this is unrivaled, and it illustrates the huge benefit in almost blindly following LTS this way! More quality, less work, and faster response to critical bugs! For sure there's no "kernel hero" in our dev team, but who really wants that anyway ? Cheers, Willy
Re: 5.10 LTS Kernel: 2 or 6 years?
On 2/18/2021 9:53 AM, Greg Kroah-Hartman wrote: > On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: >> As a company, we are most likely shooting ourselves in the foot by not >> having a point of coordination with the Linux Foundation and key people >> like you, Greg and other participants in the stable kernel. > > What does the LF have to do with this? I did not know whether the commitment/plan to use a given LTS kernel had to be funneled through the Linux Foundation and then down to you, but I am happy that this can be done publicly via a mailing list instead. > > We are here, on the mailing lists, working with everyone. Just test the > -rc releases we make and let us know if they work or not for you, it's > not a lot of "coordination" needed at all. > > Otherwise, if no one is saying that they are going to need these for 6 > years and are willing to use it in their project (i.e. and test it), > there's no need for us to maintain it for that long, right? Right, that is straight forward, works for me! -- Florian
Re: 5.10 LTS Kernel: 2 or 6 years?
On Thu, Feb 18, 2021 at 09:21:13AM -0800, Florian Fainelli wrote: > As a company, we are most likely shooting ourselves in the foot by not > having a point of coordination with the Linux Foundation and key people > like you, Greg and other participants in the stable kernel. What does the LF have to do with this? We are here, on the mailing lists, working with everyone. Just test the -rc releases we make and let us know if they work or not for you, it's not a lot of "coordination" needed at all. Otherwise, if no one is saying that they are going to need these for 6 years and are willing to use it in their project (i.e. and test it), there's no need for us to maintain it for that long, right? thanks, greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
On Thu, Feb 18, 2021 at 05:19:54PM +, Jari Ruusu wrote: > On Thursday, February 18, 2021 4:33 PM, Willy Tarreau wrote: > > Usually you pick an LTS kernel for a specific hardware. If it > > works that's great. But you cannot expect hardware to suddenly start to > > work in the middle of a stable kernel. Sometimes it happens (PCI IDs) but > > that's basically all and that's not their purpose. > > It was the other way around. Fine working in-tree driver got > broken by backported "fixes". I did mention bit-rot. It did? Please let us stable maintainers know about, we will always gladly revert problems patches. What commits caused the problem? > In-tree iwlwifi worked half-ok on early 4.9.y stable. If > connection somehow de-autheticated (out of radio range or > whatever) it crashed the kernel spectacularly. Eventually that was > fixed and in-tree iwlwifi worked fine on 4.9.y and 4.14.y stable > kernels. On second half of year 2020 (don't remember exactly when) > iwlwifi started causing erratic behavior when some random process > terminated, as if some exit processing left some resources > un-freed or something weird like that. Upgraded to 4.19.y kernels > in hope to fix the issue. Nope, same problems continued there as > well. Replacing in-tree iwlwifi with out-of-tree upstream Intel > version solved the problem for me. So something in the 4.9.y and 4.14.y stable kernels caused a regression, can you please do 'git bisect' to let us know what broke? And if 4.19.0 was always broken, why didn't you report that as well? How about 5.11, have you tried that? If not, please do so and report it to the developers, otherwise how can it ever get fixed? thanks, greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
On 2/17/2021 11:48 AM, Scott Branden wrote: > Hi Greg, > > On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote: >> On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote: >>> On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote: Hi Greg, On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote: > On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote: >> Hi All, >> >> The 5.10 LTS kernel being officially LTS supported for 2 years presents >> a problem: >> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel >> has a 6 year LTS. > Because they want to use all of the latest stuff that 5.10 provides > them. Don't you want faster and more secure kernels for your devices? Yes, 5.10 is a more secure and less buggy kernel than 5.4. >>> >>> Great, use it, ship it to your customers and we are all happy. What do >>> you need me for any of this? :) >>> >> And AOSP has already declared the use >> of 5.10 kernel in their Android S and T releases. > Publically? Where? And is that really the name of the new Android > releases, I thought they switched to numbers now (hence the naming of > the current android-common kernel branches, marketing is fun...) https://source.android.com/devices/architecture/kernel/android-common Feature and launch kernels provides kernels supported per version. >>> >>> Oh nice, didn't know that. >>> >>> But note, Android kernels do not reflect the lifespan of LTS kernels. >>> If that were the case, I would still be supporting 3.18 as they are >>> doing that at the moment for their devices and customers, and will be >>> doing so for I think another full year. >>> >>> So while Android is nice to see here, remember that is what Google is >>> promising to support for their users. You can do the same thing for >>> your users, what do you need me here for this? You can do the same >>> thing that Google is doing for 3.18 right now, pick the stable fixes >>> from upstream, backport them, test them, and push them out to their >>> users. >>> >>> While Google is a great help to me in the LTS effort, providing huge >>> amounts of resources to enable my life easier with this (i.e. funding >>> Linaro's testing efforts), their promise to their customers/users does >>> not depend on me keeping LTS kernels alive, if I stopped tomorrow their >>> contracts are still in place and they know how to do this work >>> themselves (as is proof with 3.18). >>> >>> So you can provide the same kind of guarantee to support any kernel >>> version for any amount of time to any customer you like, it shouldn't >>> require me to do that work for you, right? >>> >> Is there some way we could make the LTS support more clear. >> A 2 year declaration is not LTS any more. > Not true at all, a "normal" stable kernel is dropped after the next > release happens, making their lifespan about 4 months long. 2 years is > much longer than 4 months, so it still is a "long term supported" kernel > in contrast, correct? Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it from 2 years. The timeframes are very different. >>> >>> At this point in time, anyone wanting a kernel longer than 2 years >>> should know how this all works. >>> >>> If not, please do some basic research, I have written whitepapers on >>> this and given numerous talks. The information is out there... >>> >> If 5.10 is "actually" going to be supported for 6 years it would be >> quite valuable to make such a declaration. >> https://www.kernel.org/category/releases.html > Why? What would that change? > > Ok, seriously, this happens every year, and every year we go through the > same thing, it's not like this is somehow new, right? No, but why do we need to keep playing the same game every year now. >>> >>> Because, 5.4 almost did not become "6 years" of support from me. That >>> was because in the beginning, no one said they were going to use it in >>> their devices and offer me help in testing and backporting. Only when I >>> knew for sure that we had people helping this out did I change the date >>> on kernel.org. >>> >>> So far the jury is still out for 5.10, are you willing to help with >>> this? If not, why are you willing to hope that others are going to do >>> your work for you? I am talking to some companies, but am not willing >>> to commit to anything in public just yet, because no one has committed >>> to me yet. >> >> Following up on this as I did not hear back from you. Are you and/or >> your company willing to help out with the testing of 5.10 to ensure that >> it is a LTS kernel? So far I have not had any companies agree to help >> out with this effort, which is sad to see as it seems that companies >> want 6 years of stable kernels, yet do not seem to be able to at the >> least, do a test-build/run of those kernels,
Re: 5.10 LTS Kernel: 2 or 6 years?
On 2/18/2021 8:51 AM, Sasha Levin wrote: > On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote: >> On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote: >>> Following up on this as I did not hear back from you. Are you and/or >>> your company willing to help out with the testing of 5.10 to ensure that >>> it is a LTS kernel? So far I have not had any companies agree to help >>> out with this effort, which is sad to see as it seems that companies >>> want 6 years of stable kernels, yet do not seem to be able to at the >>> least, do a test-build/run of those kernels, which is quite odd... >> I personally cannot commit to supporting this kernel for 6 years >> (and personally do not want to backport new features to a 6 year old >> kernel). >> And customers are finicky and ask for one thing and then change their >> mind later. > > Why would we commit to maintining an upstream LTS for 6 years then? If > no one ends up using it (and we don't want anyone using older LTS > kernels) we're still stuck maintaining it. > >> We'll have to see what decisions are made at a company level for this >> as there >> are added costs to run tests on LTS kernel branches. We already run >> extensive QA on > > This sounds very wrong: it's ok to get volunteers to commit to 6 years > while the company that is asking for it won't do the same? > > Shouldn't Broadcom commit to the work involved here first? There are different groups within Broadcom, and Scott and I belong to different groups, so I can only speak for mine. We can talk about why I do not use the same email domain as Scott if you would like, and no, I cannot help you fix the Broadcom Wi-Fi drivers :) My group is committed to using the 5.10 kernel for the next 6 years because of Android TV and we will do our best to test the 5.10 stable RC as they come. We are a small team of 7 people in the grand scheme of our larger business activities, and we get pulled into way too many things, so we may skip testing a few stable releases once in a while. I do not know how to make it more official than that. As a company, we are most likely shooting ourselves in the foot by not having a point of coordination with the Linux Foundation and key people like you, Greg and other participants in the stable kernel. The usual left hand and right hand not having yet discovered each other, hey hi there left hand! I can see about remedying that at least for the interest of the group I work in. -- Florian
Re: 5.10 LTS Kernel: 2 or 6 years?
On Thu, Feb 18, 2021 at 05:19:54PM +, Jari Ruusu wrote: > In-tree iwlwifi worked half-ok on early 4.9.y stable. If > connection somehow de-autheticated (out of radio range or > whatever) it crashed the kernel spectacularly. Eventually that was > fixed and in-tree iwlwifi worked fine on 4.9.y and 4.14.y stable > kernels. On second half of year 2020 (don't remember exactly when) > iwlwifi started causing erratic behavior when some random process > terminated, as if some exit processing left some resources > un-freed or something weird like that. So you bisected the stable kernel, and reported the problem so the problem commit could be dealt with? -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
Re: 5.10 LTS Kernel: 2 or 6 years?
On Thursday, February 18, 2021 4:33 PM, Willy Tarreau wrote: > Usually you pick an LTS kernel for a specific hardware. If it > works that's great. But you cannot expect hardware to suddenly start to > work in the middle of a stable kernel. Sometimes it happens (PCI IDs) but > that's basically all and that's not their purpose. It was the other way around. Fine working in-tree driver got broken by backported "fixes". I did mention bit-rot. In-tree iwlwifi worked half-ok on early 4.9.y stable. If connection somehow de-autheticated (out of radio range or whatever) it crashed the kernel spectacularly. Eventually that was fixed and in-tree iwlwifi worked fine on 4.9.y and 4.14.y stable kernels. On second half of year 2020 (don't remember exactly when) iwlwifi started causing erratic behavior when some random process terminated, as if some exit processing left some resources un-freed or something weird like that. Upgraded to 4.19.y kernels in hope to fix the issue. Nope, same problems continued there as well. Replacing in-tree iwlwifi with out-of-tree upstream Intel version solved the problem for me. -- Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
Re: 5.10 LTS Kernel: 2 or 6 years?
On 2/18/2021 3:31 AM, Willy Tarreau wrote: > On Thu, Feb 18, 2021 at 08:43:48AM +0100, Greg Kroah-Hartman wrote: >> On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote: >>> Other difficulty with the LTS version is the frequency it is updated. > > What a stange statement! So basically if fixes come in quickly so that > customers are not exposed too long to well-known issues, it's a difficulty ? > I guess by now every serious OS vendor provides at least weekly fixes, and > at an era where devices are all interconnected, it's really necessary > (unless of course you don't care about your customer's security). > >>> We would not >>> pickup the changes that frequently to test. A quarterly, bi-annually, or >>> when a critical fix >>> is identified would be when we update and perform any meaningful testing >>> when in maintainence. >> >> How are you "identifying" these "critical fixes"? We fix at least one >> known security issue a week, and probably multitudes of >> unknown-at-this-moment ones. How are you determining when you need to >> send a new base kernel update off to your customers? At such long >> intervals it feels like anyone using your kernel releases is woefully >> insecure. > > +1! It seems like this dangerous practice will never end :-( > > Let me explain a personal experience. When I took over 2.6.32 many years > ago, Greg asked me to adapt to the new maintenance process involving the > patch reviews. At first I feared that it would increase my amount of work. > And it did. But I also discovered how important these reviews were, because > I started to get lots of "don't take this one in this version" and more > importantly "if you merge this you'll need these ones as well". And very > quickly I discovered how bogus the branches I used to maintain before > had been, given the high feedback ratio! > > So based on this experience, I can assure anyone doing cherry-picks in > their garage from LTS kernels that they're doing crap and that they must > not distribute these kernels to anyone because THESE KERNELS ARE DANGEROUS. > It's even very easy to introduce vulnerabilities by doing this! Yes absolutely. > > The only set of fixes that can be trusted are the "official" stable > kernels, because they are the only ones that are approved by the patches > authors themselves. Well, let us say that the authors had a chance to review the backports being applied but given the volume maybe they did and silence means agreement, or maybe they did not get a chance to review those changes. Let us say that the trust level of the offical stable kernels is just the highest of all kernels that are out there? > Adding more stuff on top of stable kernels is fine > (and done at your own risk), but randomly dropping stuff from stable > kernels just because you don't think you need that is totally non-sense > and must not be done anymore! Yes, definitively not setting up for success. -- Florian
Re: 5.10 LTS Kernel: 2 or 6 years?
On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote: On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote: Following up on this as I did not hear back from you. Are you and/or your company willing to help out with the testing of 5.10 to ensure that it is a LTS kernel? So far I have not had any companies agree to help out with this effort, which is sad to see as it seems that companies want 6 years of stable kernels, yet do not seem to be able to at the least, do a test-build/run of those kernels, which is quite odd... I personally cannot commit to supporting this kernel for 6 years (and personally do not want to backport new features to a 6 year old kernel). And customers are finicky and ask for one thing and then change their mind later. Why would we commit to maintining an upstream LTS for 6 years then? If no one ends up using it (and we don't want anyone using older LTS kernels) we're still stuck maintaining it. We'll have to see what decisions are made at a company level for this as there are added costs to run tests on LTS kernel branches. We already run extensive QA on This sounds very wrong: it's ok to get volunteers to commit to 6 years while the company that is asking for it won't do the same? Shouldn't Broadcom commit to the work involved here first? whatever active development branches are in use and a subset on the mainline branch as well. QA resources are finite and committing those for 6 years is not something that makes sense if customers drop that kernel version. Testing of the LTS kernel changes really moves out of our hands and into the customer's testing after our major releases to them. Keep in mind that QA resources are generally more abundant than engineering resources that need to actually backport stuff to old kernels. -- Thanks, Sasha
Re: 5.10 LTS Kernel: 2 or 6 years?
On Thu, Feb 18, 2021 at 04:15:11PM +0200, Jari Ruusu wrote: > Willy Tarreau wrote: > > The only set of fixes that can be trusted are the "official" stable > > kernels, because they are the only ones that are approved by the patches > > authors themselves. Adding more stuff on top of stable kernels is fine > > (and done at your own risk), but randomly dropping stuff from stable > > kernels just because you don't think you need that is totally non-sense > > and must not be done anymore! > > This may be little bit off-topic... but stable kernel.org kernels > can also bit-rot badly because of "selective" backporting... as in > anything that does not apply cleanly gets dropped regardless of > how critical they are. Sure it will. And the huge difference is that it usually gets quickly spotted and fixed. For sensitive servers I tend to apply the principle of not necessarily updating to the latest stable kernel but one or two versions before it which nobody complained about. And I'm pretty fine with skipping a significant number of updates (we all do that anyway). > I will give you one example: Intel WiFi (iwlwifi) on 4.19.y > kernel.org stable kernels is currently missing many critical > locking fixes. As a result, that in-tree iwlwifi driver causes > erratic behavior to random unrelated processes, and has been doing > so for many months now. My not-so-politically correct opinion is > that in-tree iwlwifi is completely FUBAR unless someone steps up > to do professional quality backport of those locking fixes from > upstream out-of-tree Intel version [1] [2] of the driver. I see, and it happens with plenty of other drivers or subsystems. Is it in any way the stable branch's or stable maintainer's fault if someone doesn't correctly do the backporting job on their driver ? No. Is it expected that a driver works perfectly from its inclusion ? No. Is it expected that a driver can always be fixed without a significant rework that risks more breakage than fixes ? No. Some design limitations or errors can require so many changes that they're unfixable in place. I even had to *document* security issues in 2.4 because fixing them was riskier than keeping them. This happens in any piece of software. It's always been the case that some older kernels work less well than some newer ones due to limited features, partially wrong drivers etc, and getting better drivers is a valid reason for upgrading to a more recent one. However the older driver ought to continue to be maintained in a working state for those for whom it works fine. > For me > only way to get properly working WiFi on my laptop computer is to > compile that Intel out-of-tree version. Sad, but true. That's perfectly fine from my point of view. I've been doing the same for certain driver (e.g. e100 vs eepro100 15 years ago) and have been pleased to be able to stop using those out-of-tree versions. This is also in order to make this possible for those who need to do it that LTS kernels provide a lot of value: such out-of-tree drivers tend to take some time to resynchronize with latest updates, and once they're updated, you can use your machine for quite some time with them. Obviously if somemone is able to figure the required fixes for the locking bugs you mentioned above and to submit patches for stable branches, I'm sure Greg will appreciate! But maybe that's not fixable there and you need to upgrade. Usually you pick an LTS kernel for a specific hardware. If it works that's great. But you cannot expect hardware to suddenly start to work in the middle of a stable kernel. Sometimes it happens (PCI IDs) but that's basically all and that's not their purpose. Willy
Re: 5.10 LTS Kernel: 2 or 6 years?
Willy Tarreau wrote: > The only set of fixes that can be trusted are the "official" stable > kernels, because they are the only ones that are approved by the patches > authors themselves. Adding more stuff on top of stable kernels is fine > (and done at your own risk), but randomly dropping stuff from stable > kernels just because you don't think you need that is totally non-sense > and must not be done anymore! This may be little bit off-topic... but stable kernel.org kernels can also bit-rot badly because of "selective" backporting... as in anything that does not apply cleanly gets dropped regardless of how critical they are. I will give you one example: Intel WiFi (iwlwifi) on 4.19.y kernel.org stable kernels is currently missing many critical locking fixes. As a result, that in-tree iwlwifi driver causes erratic behavior to random unrelated processes, and has been doing so for many months now. My not-so-politically correct opinion is that in-tree iwlwifi is completely FUBAR unless someone steps up to do professional quality backport of those locking fixes from upstream out-of-tree Intel version [1] [2] of the driver. For me only way to get properly working WiFi on my laptop computer is to compile that Intel out-of-tree version. Sad, but true. [1] https://wireless.wiki.kernel.org/en/users/drivers/iwlwifi/core_release [2] https://git.kernel.org/pub/scm/linux/kernel/git/iwlwifi/backport-iwlwifi.git/ -- Jari Ruusu 4096R/8132F189 12D6 4C3A DCDA 0AA4 27BD ACDF F073 3C80 8132 F189
Re: 5.10 LTS Kernel: 2 or 6 years?
On Thu, Feb 18, 2021 at 04:15:11PM +0200, Jari Ruusu wrote: > Willy Tarreau wrote: > > The only set of fixes that can be trusted are the "official" stable > > kernels, because they are the only ones that are approved by the patches > > authors themselves. Adding more stuff on top of stable kernels is fine > > (and done at your own risk), but randomly dropping stuff from stable > > kernels just because you don't think you need that is totally non-sense > > and must not be done anymore! > > This may be little bit off-topic... but stable kernel.org kernels > can also bit-rot badly because of "selective" backporting... as in > anything that does not apply cleanly gets dropped regardless of > how critical they are. > > I will give you one example: Intel WiFi (iwlwifi) on 4.19.y > kernel.org stable kernels is currently missing many critical > locking fixes. Why has no one asked for the specific upstream commits to be backported if this is the case? > As a result, that in-tree iwlwifi driver causes > erratic behavior to random unrelated processes, and has been doing > so for many months now. My not-so-politically correct opinion is > that in-tree iwlwifi is completely FUBAR unless someone steps up > to do professional quality backport of those locking fixes from > upstream out-of-tree Intel version [1] [2] of the driver. Why does any out-of-tree driver come into play here? What is wrong with the in-kernel code? > For me > only way to get properly working WiFi on my laptop computer is to > compile that Intel out-of-tree version. Sad, but true. Why use 4.19.y on a laptop in the firstplace? That feels very wrong and is not the recommended thing to use the LTS kernels for. thanks, greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
Hi! > Why would you backport new features to an old kernel? That's not what > they are there for. For CIP project, this is one of advantages for "supported" boards, as we'll backport patches improving their support as long as those patches are mainline. > > If the CIP project has committed to 10 years you would think they would be > > in contact > > with you to add their support to the 5.10 LTS effort. > > They are doing testing right now, see the announcements where they test > each stable -rc release. But they have not talked to me about 5.10 > yet, Their model is that they will, somehow in a way that is yet to be > determined, take over maintaining these releases _after_ I drop them. > But they are only going to do so for a very specific hardware platform > or two, so anyone using anything other than their specific boards, is > going to be out of luck. I'd say this is a bit inaccurate :-). We care about different boards at armv7, armv8 and x86-64 architectures, which actually means we care about quite big part of the kernel already. Rough estimate is that 50% of patches in stable touch our configurations, I could collect better statistics. As you said, I'm not sure everything is fully determined at this point, but... we'll certainly try to support community effort for 4.4, 4.19 and 5.10 maintainance for as long as possible. It is well possible that we'll continue to maintain even configurations we can't test (we don't have s390 to test on) on best-effort basis (to help community and because applying a patch may be easier than determining if someone in CIP community depends on it). > Which makes sense, scoping support like this to those that actually care > about a specific hardware platform is much easier than the work that I > and Sasha do in support it for all platforms. So if you are interested > in 10 years, please work with CIP to get your platform into that list. And yes, this is very good suggestion. Best regards, Pavel PS: Again, I'm speaking for myself. -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany signature.asc Description: PGP signature
Re: 5.10 LTS Kernel: 2 or 6 years?
On Thu, Feb 18, 2021 at 08:43:48AM +0100, Greg Kroah-Hartman wrote: > On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote: > > Other difficulty with the LTS version is the frequency it is updated. What a stange statement! So basically if fixes come in quickly so that customers are not exposed too long to well-known issues, it's a difficulty ? I guess by now every serious OS vendor provides at least weekly fixes, and at an era where devices are all interconnected, it's really necessary (unless of course you don't care about your customer's security). > > We would not > > pickup the changes that frequently to test. A quarterly, bi-annually, or > > when a critical fix > > is identified would be when we update and perform any meaningful testing > > when in maintainence. > > How are you "identifying" these "critical fixes"? We fix at least one > known security issue a week, and probably multitudes of > unknown-at-this-moment ones. How are you determining when you need to > send a new base kernel update off to your customers? At such long > intervals it feels like anyone using your kernel releases is woefully > insecure. +1! It seems like this dangerous practice will never end :-( Let me explain a personal experience. When I took over 2.6.32 many years ago, Greg asked me to adapt to the new maintenance process involving the patch reviews. At first I feared that it would increase my amount of work. And it did. But I also discovered how important these reviews were, because I started to get lots of "don't take this one in this version" and more importantly "if you merge this you'll need these ones as well". And very quickly I discovered how bogus the branches I used to maintain before had been, given the high feedback ratio! So based on this experience, I can assure anyone doing cherry-picks in their garage from LTS kernels that they're doing crap and that they must not distribute these kernels to anyone because THESE KERNELS ARE DANGEROUS. It's even very easy to introduce vulnerabilities by doing this! The only set of fixes that can be trusted are the "official" stable kernels, because they are the only ones that are approved by the patches authors themselves. Adding more stuff on top of stable kernels is fine (and done at your own risk), but randomly dropping stuff from stable kernels just because you don't think you need that is totally non-sense and must not be done anymore! Willy
Re: 5.10 LTS Kernel: 2 or 6 years?
Hi! (Second attempt to reply, as first one is not in the archives?!) > > So far the jury is still out for 5.10, are you willing to help with > > this? If not, why are you willing to hope that others are going to do > > your work for you? I am talking to some companies, but am not willing > > to commit to anything in public just yet, because no one has committed > > to me yet. > > Following up on this as I did not hear back from you. Are you and/or > your company willing to help out with the testing of 5.10 to ensure that > it is a LTS kernel? So far I have not had any companies agree to help > out with this effort, which is sad to see as it seems that companies > want 6 years of stable kernels, yet do not seem to be able to at the > least, do a test-build/run of those kernels, which is quite odd... We expect to maintain 5.10.X kernel for 10 years, which means you can expect similar testing we do for 4.4 and 4.19 to happen for 5.10: https://www.cip-project.org/blog/2020/12/02/cip-to-embark-on-kernel-5-10-development-for-slts If Broadcom (or anyone else) needs long-term support for 5.10 kernel, they are welcome to explore/join the CIP project. That should happen even if 5.10-stable is only mainained for 2 years. More information can be found here: https://wiki.linuxfoundation.org/civilinfrastructureplatform/start https://wiki.linuxfoundation.org/civilinfrastructureplatform/ciptesting/cipreferencehardware Best regards, Pavel PS: Speaking for myself, but not saying anything not publicly known, I believe :-). I believe I can get more official statements if someone needs them. -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany signature.asc Description: Digital signature
Re: 5.10 LTS Kernel: 2 or 6 years?
On Wed, Feb 17, 2021 at 11:48:21AM -0800, Scott Branden wrote: > Hi Greg, > > On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote: > > On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote: > >> On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote: > >>> Hi Greg, > >>> > >>> > >>> On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote: > On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote: > > Hi All, > > > > The 5.10 LTS kernel being officially LTS supported for 2 years presents > > a problem: > > why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel > > has a 6 year LTS. > Because they want to use all of the latest stuff that 5.10 provides > them. Don't you want faster and more secure kernels for your devices? > >>> Yes, 5.10 is a more secure and less buggy kernel than 5.4. > >> > >> Great, use it, ship it to your customers and we are all happy. What do > >> you need me for any of this? :) > >> > > And AOSP has already declared the use > > of 5.10 kernel in their Android S and T releases. > Publically? Where? And is that really the name of the new Android > releases, I thought they switched to numbers now (hence the naming of > the current android-common kernel branches, marketing is fun...) > >>> https://source.android.com/devices/architecture/kernel/android-common > >>> Feature and launch kernels provides kernels supported per version. > >> > >> Oh nice, didn't know that. > >> > >> But note, Android kernels do not reflect the lifespan of LTS kernels. > >> If that were the case, I would still be supporting 3.18 as they are > >> doing that at the moment for their devices and customers, and will be > >> doing so for I think another full year. > >> > >> So while Android is nice to see here, remember that is what Google is > >> promising to support for their users. You can do the same thing for > >> your users, what do you need me here for this? You can do the same > >> thing that Google is doing for 3.18 right now, pick the stable fixes > >> from upstream, backport them, test them, and push them out to their > >> users. > >> > >> While Google is a great help to me in the LTS effort, providing huge > >> amounts of resources to enable my life easier with this (i.e. funding > >> Linaro's testing efforts), their promise to their customers/users does > >> not depend on me keeping LTS kernels alive, if I stopped tomorrow their > >> contracts are still in place and they know how to do this work > >> themselves (as is proof with 3.18). > >> > >> So you can provide the same kind of guarantee to support any kernel > >> version for any amount of time to any customer you like, it shouldn't > >> require me to do that work for you, right? > >> > > Is there some way we could make the LTS support more clear. > > A 2 year declaration is not LTS any more. > Not true at all, a "normal" stable kernel is dropped after the next > release happens, making their lifespan about 4 months long. 2 years is > much longer than 4 months, so it still is a "long term supported" kernel > in contrast, correct? > >>> Perhaps a new name needs to be made for "LTS" for 6 years to distinguish > >>> it from 2 years. > >>> The timeframes are very different. > >> > >> At this point in time, anyone wanting a kernel longer than 2 years > >> should know how this all works. > >> > >> If not, please do some basic research, I have written whitepapers on > >> this and given numerous talks. The information is out there... > >> > > If 5.10 is "actually" going to be supported for 6 years it would be > > quite valuable to make such a declaration. > > https://www.kernel.org/category/releases.html > Why? What would that change? > > Ok, seriously, this happens every year, and every year we go through the > same thing, it's not like this is somehow new, right? > >>> No, but why do we need to keep playing the same game every year now. > >> > >> Because, 5.4 almost did not become "6 years" of support from me. That > >> was because in the beginning, no one said they were going to use it in > >> their devices and offer me help in testing and backporting. Only when I > >> knew for sure that we had people helping this out did I change the date > >> on kernel.org. > >> > >> So far the jury is still out for 5.10, are you willing to help with > >> this? If not, why are you willing to hope that others are going to do > >> your work for you? I am talking to some companies, but am not willing > >> to commit to anything in public just yet, because no one has committed > >> to me yet. > > > > Following up on this as I did not hear back from you. Are you and/or > > your company willing to help out with the testing of 5.10 to ensure that > > it is a LTS kernel? So far I have not had any companies agree to help > > out with this effort, which is sad to see as it seems that companies > > want 6
Re: 5.10 LTS Kernel: 2 or 6 years?
Hi Greg, On 2021-02-17 1:40 a.m., Greg Kroah-Hartman wrote: > On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote: >> On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote: >>> Hi Greg, >>> >>> >>> On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote: On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote: > Hi All, > > The 5.10 LTS kernel being officially LTS supported for 2 years presents a > problem: > why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has > a 6 year LTS. Because they want to use all of the latest stuff that 5.10 provides them. Don't you want faster and more secure kernels for your devices? >>> Yes, 5.10 is a more secure and less buggy kernel than 5.4. >> >> Great, use it, ship it to your customers and we are all happy. What do >> you need me for any of this? :) >> > And AOSP has already declared the use > of 5.10 kernel in their Android S and T releases. Publically? Where? And is that really the name of the new Android releases, I thought they switched to numbers now (hence the naming of the current android-common kernel branches, marketing is fun...) >>> https://source.android.com/devices/architecture/kernel/android-common >>> Feature and launch kernels provides kernels supported per version. >> >> Oh nice, didn't know that. >> >> But note, Android kernels do not reflect the lifespan of LTS kernels. >> If that were the case, I would still be supporting 3.18 as they are >> doing that at the moment for their devices and customers, and will be >> doing so for I think another full year. >> >> So while Android is nice to see here, remember that is what Google is >> promising to support for their users. You can do the same thing for >> your users, what do you need me here for this? You can do the same >> thing that Google is doing for 3.18 right now, pick the stable fixes >> from upstream, backport them, test them, and push them out to their >> users. >> >> While Google is a great help to me in the LTS effort, providing huge >> amounts of resources to enable my life easier with this (i.e. funding >> Linaro's testing efforts), their promise to their customers/users does >> not depend on me keeping LTS kernels alive, if I stopped tomorrow their >> contracts are still in place and they know how to do this work >> themselves (as is proof with 3.18). >> >> So you can provide the same kind of guarantee to support any kernel >> version for any amount of time to any customer you like, it shouldn't >> require me to do that work for you, right? >> > Is there some way we could make the LTS support more clear. > A 2 year declaration is not LTS any more. Not true at all, a "normal" stable kernel is dropped after the next release happens, making their lifespan about 4 months long. 2 years is much longer than 4 months, so it still is a "long term supported" kernel in contrast, correct? >>> Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it >>> from 2 years. >>> The timeframes are very different. >> >> At this point in time, anyone wanting a kernel longer than 2 years >> should know how this all works. >> >> If not, please do some basic research, I have written whitepapers on >> this and given numerous talks. The information is out there... >> > If 5.10 is "actually" going to be supported for 6 years it would be quite > valuable to make such a declaration. > https://www.kernel.org/category/releases.html Why? What would that change? Ok, seriously, this happens every year, and every year we go through the same thing, it's not like this is somehow new, right? >>> No, but why do we need to keep playing the same game every year now. >> >> Because, 5.4 almost did not become "6 years" of support from me. That >> was because in the beginning, no one said they were going to use it in >> their devices and offer me help in testing and backporting. Only when I >> knew for sure that we had people helping this out did I change the date >> on kernel.org. >> >> So far the jury is still out for 5.10, are you willing to help with >> this? If not, why are you willing to hope that others are going to do >> your work for you? I am talking to some companies, but am not willing >> to commit to anything in public just yet, because no one has committed >> to me yet. > > Following up on this as I did not hear back from you. Are you and/or > your company willing to help out with the testing of 5.10 to ensure that > it is a LTS kernel? So far I have not had any companies agree to help > out with this effort, which is sad to see as it seems that companies > want 6 years of stable kernels, yet do not seem to be able to at the > least, do a test-build/run of those kernels, which is quite odd... I personally cannot commit to supporting this kernel for 6 years (and personally do not want to backport new features to a 6 year
Re: 5.10 LTS Kernel: 2 or 6 years?
On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote: > On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote: > > Hi Greg, > > > > > > On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote: > > > On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote: > > >> Hi All, > > >> > > >> The 5.10 LTS kernel being officially LTS supported for 2 years presents > > >> a problem: > > >> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel > > >> has a 6 year LTS. > > > Because they want to use all of the latest stuff that 5.10 provides > > > them. Don't you want faster and more secure kernels for your devices? > > Yes, 5.10 is a more secure and less buggy kernel than 5.4. > > Great, use it, ship it to your customers and we are all happy. What do > you need me for any of this? :) > > > >> And AOSP has already declared the use > > >> of 5.10 kernel in their Android S and T releases. > > > Publically? Where? And is that really the name of the new Android > > > releases, I thought they switched to numbers now (hence the naming of > > > the current android-common kernel branches, marketing is fun...) > > https://source.android.com/devices/architecture/kernel/android-common > > Feature and launch kernels provides kernels supported per version. > > Oh nice, didn't know that. > > But note, Android kernels do not reflect the lifespan of LTS kernels. > If that were the case, I would still be supporting 3.18 as they are > doing that at the moment for their devices and customers, and will be > doing so for I think another full year. > > So while Android is nice to see here, remember that is what Google is > promising to support for their users. You can do the same thing for > your users, what do you need me here for this? You can do the same > thing that Google is doing for 3.18 right now, pick the stable fixes > from upstream, backport them, test them, and push them out to their > users. > > While Google is a great help to me in the LTS effort, providing huge > amounts of resources to enable my life easier with this (i.e. funding > Linaro's testing efforts), their promise to their customers/users does > not depend on me keeping LTS kernels alive, if I stopped tomorrow their > contracts are still in place and they know how to do this work > themselves (as is proof with 3.18). > > So you can provide the same kind of guarantee to support any kernel > version for any amount of time to any customer you like, it shouldn't > require me to do that work for you, right? > > > >> Is there some way we could make the LTS support more clear. > > >> A 2 year declaration is not LTS any more. > > > Not true at all, a "normal" stable kernel is dropped after the next > > > release happens, making their lifespan about 4 months long. 2 years is > > > much longer than 4 months, so it still is a "long term supported" kernel > > > in contrast, correct? > > Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it > > from 2 years. > > The timeframes are very different. > > At this point in time, anyone wanting a kernel longer than 2 years > should know how this all works. > > If not, please do some basic research, I have written whitepapers on > this and given numerous talks. The information is out there... > > > >> If 5.10 is "actually" going to be supported for 6 years it would be > > >> quite valuable to make such a declaration. > > >> https://www.kernel.org/category/releases.html > > > Why? What would that change? > > > > > > Ok, seriously, this happens every year, and every year we go through the > > > same thing, it's not like this is somehow new, right? > > No, but why do we need to keep playing the same game every year now. > > Because, 5.4 almost did not become "6 years" of support from me. That > was because in the beginning, no one said they were going to use it in > their devices and offer me help in testing and backporting. Only when I > knew for sure that we had people helping this out did I change the date > on kernel.org. > > So far the jury is still out for 5.10, are you willing to help with > this? If not, why are you willing to hope that others are going to do > your work for you? I am talking to some companies, but am not willing > to commit to anything in public just yet, because no one has committed > to me yet. Following up on this as I did not hear back from you. Are you and/or your company willing to help out with the testing of 5.10 to ensure that it is a LTS kernel? So far I have not had any companies agree to help out with this effort, which is sad to see as it seems that companies want 6 years of stable kernels, yet do not seem to be able to at the least, do a test-build/run of those kernels, which is quite odd... If you want to point people at your company this link that explains it all in a single location instead of an email thread: http://www.kroah.com/log/blog/2021/02/03/helping-out-with-lts-kernel-releases/
Re: 5.10 LTS Kernel: 2 or 6 years?
Hi Greg, On Tue, Jan 26, 2021 at 07:51:18PM +0100, Greg Kroah-Hartman wrote: > > > Ok, seriously, this happens every year, and every year we go through the > > > same thing, it's not like this is somehow new, right? > > No, but why do we need to keep playing the same game every year now. > > Because, 5.4 almost did not become "6 years" of support from me. That > was because in the beginning, no one said they were going to use it in > their devices and offer me help in testing and backporting. Only when I > knew for sure that we had people helping this out did I change the date > on kernel.org. > > So far the jury is still out for 5.10, are you willing to help with > this? If not, why are you willing to hope that others are going to do > your work for you? I am talking to some companies, but am not willing > to commit to anything in public just yet, because no one has committed > to me yet. This is interesting, because I used to extend LTS kernels after you dropped them exactly for the reason I needed them. With 6 months of porting+testing, ~3 years of major version life in field, and roughly 6 extra months covering late starts or minor extensions to help match a deadline, I figured that 4 years was the minimum we needed for our products. As such, 6 years add some comfort, but as you probably remember I once got scared not knowing if 4.19 was going to be extended or not as we were having plenty in field and I did not plan to get back to that maintenance myself. These days, the 6 years duration allows us to skip one upgrade if we're too late, but we still prefer to upgrade every year. It's also true that before engaging in that direction before the official statement was on the site, each time I preferred to ask you how you felt about it, and I'm sure most major players decide after checking with you as well. For sure knowing better early would be much better but I understand your concerns about doing that job for free and for no reason. I initially got the impression that the extra resources you got were enough to help you and to keep me away from annoying you again with pushing my late branches, but it's certain that free time cannot extend forever. And You, Ben, Sasha and me definitely know how painful it is to backport past 2.5-3 years, especially with security stuff that everyone expects but often is the riskiest. While most of my time is spent in userland these days, I'm still willing to help as I can if that saves from from having to takeover an old kernel again on my week-ends. One of my impression, which might or might not be shared, is that the duration is more important than the frequency, which means that having one extended kernel every two years would bring much more value than maintaining all of them only 3 years, and would equally result in cutting the effort in half: with 6 years, you still have 5 years if you upgrade every two years. > What would you do if you were in my situation? I'd continue to ask for a shared effort for extended support, which everyone benefits from. Maybe one possibility would be to start gauging upfront around september whether or not the end-of-year's LTS should be an extended LTS or not, and if so, what's needed and who's willing to particpate. I suspect that numerous companies have available resources to help but are not even aware that they could help, and they're seeing something which works extremely well so there's nothing to try to improve. And if nobody provides resources, it means there's not enough interest to create yet another extra LTS kernel so better save the effort. Just my two cents, Willy
Re: 5.10 LTS Kernel: 2 or 6 years?
On Tue, Jan 26, 2021 at 10:30:16AM -0800, Scott Branden wrote: > Hi Greg, > > > On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote: > > On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote: > >> Hi All, > >> > >> The 5.10 LTS kernel being officially LTS supported for 2 years presents a > >> problem: > >> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has > >> a 6 year LTS. > > Because they want to use all of the latest stuff that 5.10 provides > > them. Don't you want faster and more secure kernels for your devices? > Yes, 5.10 is a more secure and less buggy kernel than 5.4. Great, use it, ship it to your customers and we are all happy. What do you need me for any of this? :) > >> And AOSP has already declared the use > >> of 5.10 kernel in their Android S and T releases. > > Publically? Where? And is that really the name of the new Android > > releases, I thought they switched to numbers now (hence the naming of > > the current android-common kernel branches, marketing is fun...) > https://source.android.com/devices/architecture/kernel/android-common > Feature and launch kernels provides kernels supported per version. Oh nice, didn't know that. But note, Android kernels do not reflect the lifespan of LTS kernels. If that were the case, I would still be supporting 3.18 as they are doing that at the moment for their devices and customers, and will be doing so for I think another full year. So while Android is nice to see here, remember that is what Google is promising to support for their users. You can do the same thing for your users, what do you need me here for this? You can do the same thing that Google is doing for 3.18 right now, pick the stable fixes from upstream, backport them, test them, and push them out to their users. While Google is a great help to me in the LTS effort, providing huge amounts of resources to enable my life easier with this (i.e. funding Linaro's testing efforts), their promise to their customers/users does not depend on me keeping LTS kernels alive, if I stopped tomorrow their contracts are still in place and they know how to do this work themselves (as is proof with 3.18). So you can provide the same kind of guarantee to support any kernel version for any amount of time to any customer you like, it shouldn't require me to do that work for you, right? > >> Is there some way we could make the LTS support more clear. > >> A 2 year declaration is not LTS any more. > > Not true at all, a "normal" stable kernel is dropped after the next > > release happens, making their lifespan about 4 months long. 2 years is > > much longer than 4 months, so it still is a "long term supported" kernel > > in contrast, correct? > Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it > from 2 years. > The timeframes are very different. At this point in time, anyone wanting a kernel longer than 2 years should know how this all works. If not, please do some basic research, I have written whitepapers on this and given numerous talks. The information is out there... > >> If 5.10 is "actually" going to be supported for 6 years it would be quite > >> valuable to make such a declaration. > >> https://www.kernel.org/category/releases.html > > Why? What would that change? > > > > Ok, seriously, this happens every year, and every year we go through the > > same thing, it's not like this is somehow new, right? > No, but why do we need to keep playing the same game every year now. Because, 5.4 almost did not become "6 years" of support from me. That was because in the beginning, no one said they were going to use it in their devices and offer me help in testing and backporting. Only when I knew for sure that we had people helping this out did I change the date on kernel.org. So far the jury is still out for 5.10, are you willing to help with this? If not, why are you willing to hope that others are going to do your work for you? I am talking to some companies, but am not willing to commit to anything in public just yet, because no one has committed to me yet. What would you do if you were in my situation? thanks, greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
Hi Greg, On 2021-01-25 11:29 p.m., Greg Kroah-Hartman wrote: > On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote: >> Hi All, >> >> The 5.10 LTS kernel being officially LTS supported for 2 years presents a >> problem: >> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a >> 6 year LTS. > Because they want to use all of the latest stuff that 5.10 provides > them. Don't you want faster and more secure kernels for your devices? Yes, 5.10 is a more secure and less buggy kernel than 5.4. You don't need to convince me which is the better kernel to start using. The kernel improves every version and we are at a point where regressions rarely happen. But, it is not our choice what version a customer chooses as they have many valid reasons to stabilize their code. They do have extensive QA cycles to go through and cannot simply pick up a newer kernel version to get a security fix or bug fix. As everyone, we also have testing and cannot test every kernel version extensively so must choose wisely what to test. We also track the mainline a test as well to ensure functionality - but stress testing is not done at the same level as a full QA cycle. Unfortunately some people think that 5.4 LTS is more stable than 5.10. And it also has a longer lifetime. So, people starting a new project today decide to use 5.4 (or even earlier) kernels as they deem these stable. For these reasons we would be forced to backport (upstreamed code) to an old kernel for new features and processors, etc not supported in 5.4. You can see how such a kernel would not be as stable as 5.10 if a single subtle fix is not backported. But, try convincing people to go with 5.10 with only a 2 year LTS lifespan vs. 5.4 with many more years. Hopefully you understand the kernel is not the product - the end solution is what is tested and QA'd. And, picking up small, safe LTS versions requires a much smaller QA cycle than a kernel upgrade with thousands of possible changes in the code every kernel version. > >> Yet, various unofficial reports indicate it will be supported for 6 years. > Rumors are nice, aren't they :) > >> And AOSP has already declared the use >> of 5.10 kernel in their Android S and T releases. > Publically? Where? And is that really the name of the new Android > releases, I thought they switched to numbers now (hence the naming of > the current android-common kernel branches, marketing is fun...) https://source.android.com/devices/architecture/kernel/android-common Feature and launch kernels provides kernels supported per version. > >> Is there some way we could make the LTS support more clear. >> A 2 year declaration is not LTS any more. > Not true at all, a "normal" stable kernel is dropped after the next > release happens, making their lifespan about 4 months long. 2 years is > much longer than 4 months, so it still is a "long term supported" kernel > in contrast, correct? Perhaps a new name needs to be made for "LTS" for 6 years to distinguish it from 2 years. The timeframes are very different. > >> If 5.10 is "actually" going to be supported for 6 years it would be quite >> valuable to make such a declaration. >> https://www.kernel.org/category/releases.html > Why? What would that change? > > Ok, seriously, this happens every year, and every year we go through the > same thing, it's not like this is somehow new, right? No, but why do we need to keep playing the same game every year now. > > I want to see companies _using_ the kernel, and most importantly, > _updating_ their devices with it, to know if it is worth to keep around > for longer than 2 years. I also, hopefully, want to see how those > companies will help me out in the testing and maintenance of that kernel > version in order to make supporting it for 6 years actually possible. > > So, are you planning on using 5.10? Will you will be willing to help > out in testing the -rc releases I make to let me know if there are any > problems, and to help in pointing out and backporting any specific > patches that your platforms need for that kernel release? > When I get this kind of promises and support from companies, then I am > glad to bump up the length of the kernel support from 2 to 6 years, and > I mark it on the web site. Traditionally this happens in Febuary/March > once I hear from enough companies. Can I count on your support in this > endeavor? It is a chicken and egg problem and 2 year LTS declaration is not helping customers commit to 5.10. We are using 5.10 right now and testing it. But, we don't make the end products. If customers decide to use 5.10 we will continue to test, support, and backport any specific patches to it. If they choose 5.4 that would be the one we would have to backport and try and support. The backport is not just for simple drivers. VFIO, IOMMU, and other unidentified architecture support would need to be backported. So 5.10 is the much better decision to pickup any LTS
Re: 5.10 LTS Kernel: 2 or 6 years?
On 1/25/2021 11:29 PM, Greg Kroah-Hartman wrote: > On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote: >> Hi All, >> >> The 5.10 LTS kernel being officially LTS supported for 2 years presents a >> problem: >> why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a >> 6 year LTS. > > Because they want to use all of the latest stuff that 5.10 provides > them. Don't you want faster and more secure kernels for your devices? > >> Yet, various unofficial reports indicate it will be supported for 6 years. > > Rumors are nice, aren't they :) > >> And AOSP has already declared the use >> of 5.10 kernel in their Android S and T releases. > > Publically? Where? And is that really the name of the new Android > releases, I thought they switched to numbers now (hence the naming of > the current android-common kernel branches, marketing is fun...) > >> Is there some way we could make the LTS support more clear. >> A 2 year declaration is not LTS any more. > > Not true at all, a "normal" stable kernel is dropped after the next > release happens, making their lifespan about 4 months long. 2 years is > much longer than 4 months, so it still is a "long term supported" kernel > in contrast, correct? > >> If 5.10 is "actually" going to be supported for 6 years it would be quite >> valuable to make such a declaration. >> https://www.kernel.org/category/releases.html > > Why? What would that change? I believe what Scott is looking for is a form of official statement that indicates that 5.10 is indeed a 6 years Long Term Stable kernel, as opposed to an informal piece of information like this: https://twitter.com/kernellogger/status/1320731458311491587 (personally that is how I got to hear about 5.10 being selected, taking this with a grain of salt). > > Ok, seriously, this happens every year, and every year we go through the > same thing, it's not like this is somehow new, right? It is not new indeed but there is a recurring pattern, and I have been told by some of our customers who use the kernel.org's releases.html page that this is how they know which kernel is going to be LTS and plan ahead accordingly. If you feel making such a statement actually hurts the kernel project as a whole, or sort of steers people into cramming all of their desired features/bug fixes towards particular versions of the kernel that may, or may not be 6 years LTS, that may be an argument against announcing before, but maybe not after the kernel is released? > > I want to see companies _using_ the kernel, and most importantly, > _updating_ their devices with it, to know if it is worth to keep around > for longer than 2 years. I also, hopefully, want to see how those > companies will help me out in the testing and maintenance of that kernel > version in order to make supporting it for 6 years actually possible. This really depends on which market you are in we have (Set-top-box and cable modem) customers that do *want* to use newer kernels but are so slow in adopting new ones that by the time they start the kernel has already been phased out 2 years ago. We have worked (and are still working) with a customer who wanted so badly 4.1 they rattled our whole organization and had my team on call to get there, only to start working on it seriously in 2019. The problem was not necessarily the SoC vendor but the various OEMs and their bigger pile of out of tree patches and the fact that the platforms were so old that the essential knowledge of all of their little details was gone (fired, retired, moved on) already. 6 years provides a great window of time for people to get to a quality production with the kernel while offering enough time for slow users to transition over and yet still have a comfortable window of updates. 2 years is a bit too short and will result in the SoC vendor or whoever gets paid for to either continue support and maybe endorse the responsibility of doing critical back ports themselves (meltdown, spectre, etc.) with all the implied liability, or completely drop support for the unmaintained kernel. In both situations, the device's security is not as great as if a 6 years LTS kernel had been chosen. People who can jump on the 2 year stable kernel will, and those who cannot will not, so having 6 years is a good way to avoid skipping too much yet try to live with what is/was current at the time. > > So, are you planning on using 5.10? Will you will be willing to help > out in testing the -rc releases I make to let me know if there are any > problems, and to help in pointing out and backporting any specific > patches that your platforms need for that kernel release? > > When I get this kind of promises and support from companies, then I am > glad to bump up the length of the kernel support from 2 to 6 years, and > I mark it on the web site. Traditionally this happens in Febuary/March > once I hear from enough companies. Can I count on your support in this > endeavor? I
Re: 5.10 LTS Kernel: 2 or 6 years?
On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote: > Hi All, > > The 5.10 LTS kernel being officially LTS supported for 2 years presents a > problem: > why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 > year LTS. Because they want to use all of the latest stuff that 5.10 provides them. Don't you want faster and more secure kernels for your devices? > Yet, various unofficial reports indicate it will be supported for 6 years. Rumors are nice, aren't they :) > And AOSP has already declared the use > of 5.10 kernel in their Android S and T releases. Publically? Where? And is that really the name of the new Android releases, I thought they switched to numbers now (hence the naming of the current android-common kernel branches, marketing is fun...) > Is there some way we could make the LTS support more clear. > A 2 year declaration is not LTS any more. Not true at all, a "normal" stable kernel is dropped after the next release happens, making their lifespan about 4 months long. 2 years is much longer than 4 months, so it still is a "long term supported" kernel in contrast, correct? > If 5.10 is "actually" going to be supported for 6 years it would be quite > valuable to make such a declaration. > https://www.kernel.org/category/releases.html Why? What would that change? Ok, seriously, this happens every year, and every year we go through the same thing, it's not like this is somehow new, right? I want to see companies _using_ the kernel, and most importantly, _updating_ their devices with it, to know if it is worth to keep around for longer than 2 years. I also, hopefully, want to see how those companies will help me out in the testing and maintenance of that kernel version in order to make supporting it for 6 years actually possible. So, are you planning on using 5.10? Will you will be willing to help out in testing the -rc releases I make to let me know if there are any problems, and to help in pointing out and backporting any specific patches that your platforms need for that kernel release? When I get this kind of promises and support from companies, then I am glad to bump up the length of the kernel support from 2 to 6 years, and I mark it on the web site. Traditionally this happens in Febuary/March once I hear from enough companies. Can I count on your support in this endeavor? Also, a meta-comment. Please reconsider using a single kernel version for longer than 2 years on systems that you actively support and maintain. It's generally a bad idea unless you are stuck with millions of out-of-tree code that something like a customer-unfriendly SoC vendor provides. If you are stuck in that type of situation, well they have decided to spend extra money to keep their out-of-tree code alive, so why are they forcing you to also spend extra money and energy? I can go on about this topic at length if you want me to, I have lots of examples of how to, and not to, maintain a kernel for a device for a long period of time... thanks, greg k-h
Re: 5.10 LTS Kernel: 2 or 6 years?
On Mon, Jan 25, 2021 at 11:55:11AM -0800, Scott Branden wrote: > The 5.10 LTS kernel being officially LTS supported for 2 years presents a > problem: > why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 > year LTS. > > Yet, various unofficial reports indicate it will be supported for 6 > years. And AOSP has already declared the use of 5.10 kernel in their > Android S and T releases. 5.10 will also be used for Debian Bullseye. -- ⢀⣴⠾⠻⢶⣦⠀ .--[ Makefile ] ⣾⠁⢠⠒⠀⣿⡁ # beware of races ⢿⡄⠘⠷⠚⠋⠀ all: pillage burn ⠈⠳⣄ `
5.10 LTS Kernel: 2 or 6 years?
Hi All, The 5.10 LTS kernel being officially LTS supported for 2 years presents a problem: why would anyone select a 5.10 kernel with 2 year LTS when 5.4 kernel has a 6 year LTS. Yet, various unofficial reports indicate it will be supported for 6 years. And AOSP has already declared the use of 5.10 kernel in their Android S and T releases. Is there some way we could make the LTS support more clear. A 2 year declaration is not LTS any more. If 5.10 is "actually" going to be supported for 6 years it would be quite valuable to make such a declaration. https://www.kernel.org/category/releases.html Regards, Scott