Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
Hi Andy, On 24 June 2017 at 10:13, Greg Kroah-Hartman wrote: > On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote: >> Greg, for context, the issue here is that we made what was arguably a >> design error in seccomp's interaction with ptrace. After determining >> that fixing it solved a bunch of problems and didn't break any user >> programs, we fixed it. There might be new code that relies on the fix >> being present in the sense that it would be insecure without the fix. >> >> The problem is that the fix is moderately intrusive and doesn't seem >> like a great candidate for backporting, although we could plausibly do >> it. > > That's fine, not all bugfixes that tests are created to find, should be > backported. That's up to the stable maintainers, or someone who has a > device/vendor tree based on that kernel if they want to do that or not. > > That has nothing to do with the fact that the test should fail or > gracefully degrade. Tests should fail if the action that they are > testing fails. They should degrade and not run if the _feature_ they > are testing is not present. So, any updates on this yet - getting the seccomp tests to degrade gracefully? I realise you mentioned that the fix could be intrusive; just wanted to know if it was on your radar still. > > Yes, sometimes this is hard, like with the seccomp stuff, and will not > always work, but that's the rule for our userspace api independant of > any testing framework or code. > > Look at xfstests, no one gets mad when it adds a new test that old > kernels fail at. It's up to someone else to either backport the kernel > change, if they want it fixed in an old kernel, not to have xfstests > just not run it at all! There's nothing different here either. > > thanks, > > greg k-h Thanks much, Sumit.
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
On Sat, Jun 24, 2017 at 06:45:37AM +0200, Greg Kroah-Hartman wrote: > On Sat, Jun 24, 2017 at 02:34:07AM +0200, Luis R. Rodriguez wrote: > > So taking the position that any kselftest script on linux-next or a future > > kernel should never break stable implicate that *any* fix going upstream for > > which there is a respective ksefltest test *must* have a stable upstream > > fix. > > What, no! If it's a bug, that kselftest points out, great, it's up to > stable or a vendor to backport that fix if they want it. Makes sense! > Again, it's just like any other test suite, look at xfstests, no one > gets mad when it adds new tests that fail on old kernels, due to bugs > there, right? Correct but that means there is an information gap of which test cases fit into this category. An option to run kselftest with the ability to only run tests designed to also include stable would be good, otherwise Sumit Semwal or others would be sending reports or issues for things folks already designed the fix to *not* be a stable candidate. This actually would also prove useful to distributions to ensure to then run kselftest as part of their distribution test suite with all bells and whistles enabled so that they can make a determination of what questionable "fixes" things to fold in somehow. Unless of course we want these discussions to purposely bubble up as an alternative to this kselftest option. It seems rather wasteful though. > > Its not obvious to me that everyone is aware of this. What do we do about > > those cases where we *don't* want a stable fix due to the complexity? > > That's up to the stable maintainers or the vendors that maintain their > own kernel trees. But if the above is already decided having folks send emails about it seems pointless. Luis
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
On Sat, Jun 24, 2017 at 02:34:07AM +0200, Luis R. Rodriguez wrote: > So taking the position that any kselftest script on linux-next or a future > kernel should never break stable implicate that *any* fix going upstream for > which there is a respective ksefltest test *must* have a stable upstream fix. What, no! If it's a bug, that kselftest points out, great, it's up to stable or a vendor to backport that fix if they want it. Again, it's just like any other test suite, look at xfstests, no one gets mad when it adds new tests that fail on old kernels, due to bugs there, right? > Its not obvious to me that everyone is aware of this. What do we do about > those cases where we *don't* want a stable fix due to the complexity? That's up to the stable maintainers or the vendors that maintain their own kernel trees. thanks, greg k-h
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote: > Greg, for context, the issue here is that we made what was arguably a > design error in seccomp's interaction with ptrace. After determining > that fixing it solved a bunch of problems and didn't break any user > programs, we fixed it. There might be new code that relies on the fix > being present in the sense that it would be insecure without the fix. > > The problem is that the fix is moderately intrusive and doesn't seem > like a great candidate for backporting, although we could plausibly do > it. That's fine, not all bugfixes that tests are created to find, should be backported. That's up to the stable maintainers, or someone who has a device/vendor tree based on that kernel if they want to do that or not. That has nothing to do with the fact that the test should fail or gracefully degrade. Tests should fail if the action that they are testing fails. They should degrade and not run if the _feature_ they are testing is not present. Yes, sometimes this is hard, like with the seccomp stuff, and will not always work, but that's the rule for our userspace api independant of any testing framework or code. Look at xfstests, no one gets mad when it adds a new test that old kernels fail at. It's up to someone else to either backport the kernel change, if they want it fixed in an old kernel, not to have xfstests just not run it at all! There's nothing different here either. thanks, greg k-h
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
On Thu, Jun 22, 2017 at 07:40:49PM -0700, Andy Lutomirski wrote: > On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman > wrote: > > On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote: > >> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski wrote: > >> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan wrote: > >> >> On 06/22/2017 10:53 AM, Kees Cook wrote: > >> Right. I really think stable kernels should be tested with their own > >> selftests. If some test is needed in a stable kernel it should be > >> backported to that stable kernel. > > > > Well, ideally all new features added to the kernel should be able to be > > detected by userspace somehow if they are present or not. > > > > How do you expect a program to know if a feature has "failed" or is just > > "not enabled/present in this kernel"? Normally with syscalls this is > > easy, same for sysfs changes. Is seccomp in the bad state where there > > is no way to detect the two different states here? How is userspace > > supposed to deal with that? > > > > We make fun of glibc having a zillion crazy tests to determine kernel > > features, and recently, just not wrapping new syscalls at all because > > they are just frustrated at the compatibility issues over time. Let's > > not make their life any harder than it has to be please. > > > > I don't see how any of the kselftest programs are any different than any > > other userspace program that wants to use our kernel api, and as such, > > any version of kselftest should be able to successfully run on any > > kernel release. If not, then we messed up in how we either wrote the > > test, or how we added a new kernel api. Neither is acceptable. > > That's a fair point. I agreed with it as well just a few threads ago due to similar issues, however, thinking this over I'm afraid this has some interesting side consequences for fixes and what code goes upstream into kselftest. <-- snip --> > The problem is that the fix is moderately intrusive and doesn't seem > like a great candidate for backporting, although we could plausibly do > it. Such is the case often actually. So taking the position that any kselftest script on linux-next or a future kernel should never break stable implicate that *any* fix going upstream for which there is a respective ksefltest test *must* have a stable upstream fix. Its not obvious to me that everyone is aware of this. What do we do about those cases where we *don't* want a stable fix due to the complexity? Luis
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
Hi Shuah, On Fri, Jun 23, 2017 at 2:03 PM, Shuah Khan wrote: > On 06/22/2017 01:48 PM, Tom Gall wrote: >> Hi >> >> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan wrote: >>> On 06/22/2017 11:50 AM, Kees Cook wrote: On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski wrote: > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan wrote: >> On 06/22/2017 10:53 AM, Kees Cook wrote: >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal >>> wrote: Hi Kees, Andy, On 15 June 2017 at 23:26, Sumit Semwal wrote: > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - > feature and test together. > - This one also seems like a security hole being closed, and the > 'feature' could be a candidate for stable backports, but Arnd tried > that, and it was quite non-trivial. So perhaps we'll need some help > from the subsystem developers here. Could you please help us sort this out? Our goal is to help Greg with testing stable kernels, and currently the seccomp tests fail due to missing feature (seccomp ptrace hole closure) getting tested via latest kselftest. If you feel the feature isn't a stable candidate, then could you please help make the test degrade gracefully in its absence? >>> >>> In some cases, it is not easy to degrade and/or check for a feature. >>> Probably several security features could fall in this bucket. >>> >>> >>> I don't really want to have that change be a backport -- it's quite >>> invasive across multiple architectures. >>> >>> Agreed. The same test for kernel applies to tests as well. If a kernel >>> feature can't be back-ported, the test for that feature will fall in the >>> same bucket. It shouldn't be back-ported. >>> >>> >>> I would say just add a kernel version check to the test. This is >>> probably not the only selftest that will need such things. :) >> >> Adding release checks to selftests is going to problematic for >> maintenance. >> Tests should fail gracefully if feature isn't supported in older kernels. >> >> Several tests do that now and please find a way to check for dependencies >> and feature availability and fail the test gracefully. If there is a test >> that can't do that for some reason, we can discuss it, but as a general >> rule, I don't want to see kselftest patches that check release. > > If a future kernel inadvertently loses the new feature and degrades to > the behavior of old kernels, that would be a serious bug and should be > caught. >>> >>> Agreed. If I understand you correctly, by not testing stable kernels >>> with their own selftests, some serious bugs could go undetected. >> >> Personally I'm a bit skeptical. I think the reasoning is more that the >> latest selftests provide more coverage, and therefore should be better >> tests, even on older kernels. >> Right. I really think stable kernels should be tested with their own selftests. If some test is needed in a stable kernel it should be backported to that stable kernel. >>> >>> Correct. This is always a safe option. There might be cases that even >>> prevent tests being built, especially if a new feature adds new fields >>> to an existing structure. >>> >>> It appears in some cases, users want to run newer tests on older kernels. >>> Some tests can clearly detect feature support using module presence and/or >>> Kconfig enabled or disabled. These are conditions even on a kernel that >>> supports a new module or new config option. The kernel the test is running >>> on might not have the feature enabled or module might not be present. In >>> these cases, it would be easier to detect and skip the test. >>> >>> However, some features aren't so easy. For example: >>> >>> - a new flag is added to a syscall, and new test is added. It might not >>> be easy to detect that. >>> - We might have some tests that can't detect and skip. >>> >>> Based on this discussion, it is probably accurate to say: >>> >>> 1. It is recommended that selftests from the same release be run on the >>>kernel. >>> 2. Selftests from newer kernels will run on older kernels, user should >>>understand the risks such as some tests might fail and might not >>>detect feature degradation related bugs. >>> 3. Selftests will fail gracefully on older releases if at all possible. >> >> How about gracefully be skipped instead of fail? >> >> The later suggests the test case in some situations can detect it's >> pointless to run something and say as much instead of emitting a >> failure that would be a waste of time to look into. >> >> As another example take tools/testing/selftests/net/psock_fanout.c >> On 4.9 it'll fail to compile (using master's selftests) because >> PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for >> that symbol and the psock_fanout test will compile
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
On 06/22/2017 01:48 PM, Tom Gall wrote: > Hi > > On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan wrote: >> On 06/22/2017 11:50 AM, Kees Cook wrote: >>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski wrote: On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan wrote: > On 06/22/2017 10:53 AM, Kees Cook wrote: >> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal >> wrote: >>> Hi Kees, Andy, >>> >>> On 15 June 2017 at 23:26, Sumit Semwal wrote: 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - feature and test together. - This one also seems like a security hole being closed, and the 'feature' could be a candidate for stable backports, but Arnd tried that, and it was quite non-trivial. So perhaps we'll need some help from the subsystem developers here. >>> >>> Could you please help us sort this out? Our goal is to help Greg with >>> testing stable kernels, and currently the seccomp tests fail due to >>> missing feature (seccomp ptrace hole closure) getting tested via >>> latest kselftest. >>> >>> If you feel the feature isn't a stable candidate, then could you >>> please help make the test degrade gracefully in its absence? >> >> In some cases, it is not easy to degrade and/or check for a feature. >> Probably several security features could fall in this bucket. >> >> >> I don't really want to have that change be a backport -- it's quite >> invasive across multiple architectures. >> >> Agreed. The same test for kernel applies to tests as well. If a kernel >> feature can't be back-ported, the test for that feature will fall in the >> same bucket. It shouldn't be back-ported. >> >> >> I would say just add a kernel version check to the test. This is >> probably not the only selftest that will need such things. :) > > Adding release checks to selftests is going to problematic for > maintenance. > Tests should fail gracefully if feature isn't supported in older kernels. > > Several tests do that now and please find a way to check for dependencies > and feature availability and fail the test gracefully. If there is a test > that can't do that for some reason, we can discuss it, but as a general > rule, I don't want to see kselftest patches that check release. If a future kernel inadvertently loses the new feature and degrades to the behavior of old kernels, that would be a serious bug and should be caught. >> >> Agreed. If I understand you correctly, by not testing stable kernels >> with their own selftests, some serious bugs could go undetected. > > Personally I'm a bit skeptical. I think the reasoning is more that the > latest selftests provide more coverage, and therefore should be better > tests, even on older kernels. > >>> >>> Right. I really think stable kernels should be tested with their own >>> selftests. If some test is needed in a stable kernel it should be >>> backported to that stable kernel. >> >> Correct. This is always a safe option. There might be cases that even >> prevent tests being built, especially if a new feature adds new fields >> to an existing structure. >> >> It appears in some cases, users want to run newer tests on older kernels. >> Some tests can clearly detect feature support using module presence and/or >> Kconfig enabled or disabled. These are conditions even on a kernel that >> supports a new module or new config option. The kernel the test is running >> on might not have the feature enabled or module might not be present. In >> these cases, it would be easier to detect and skip the test. >> >> However, some features aren't so easy. For example: >> >> - a new flag is added to a syscall, and new test is added. It might not >> be easy to detect that. >> - We might have some tests that can't detect and skip. >> >> Based on this discussion, it is probably accurate to say: >> >> 1. It is recommended that selftests from the same release be run on the >>kernel. >> 2. Selftests from newer kernels will run on older kernels, user should >>understand the risks such as some tests might fail and might not >>detect feature degradation related bugs. >> 3. Selftests will fail gracefully on older releases if at all possible. > > How about gracefully be skipped instead of fail? > > The later suggests the test case in some situations can detect it's > pointless to run something and say as much instead of emitting a > failure that would be a waste of time to look into. > > As another example take tools/testing/selftests/net/psock_fanout.c > On 4.9 it'll fail to compile (using master's selftests) because > PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for > that symbol and the psock_fanout test will compile and run just fine. Unfortunately adding PACKET_FANOUT_FLAG_UNIQUEID isn't correct. This is a new feature that went into 4.12. You don't want to define this for
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
On 06/22/2017 10:02 PM, Sumit Semwal wrote: > Hi Shuah, > > On 23 June 2017 at 01:53, Shuah Khan wrote: >> Hi Tom, >> >> On 06/22/2017 01:48 PM, Tom Gall wrote: >>> Hi >>> >>> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan wrote: On 06/22/2017 11:50 AM, Kees Cook wrote: > On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski wrote: >> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan wrote: >>> On 06/22/2017 10:53 AM, Kees Cook wrote: On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal wrote: > Hi Kees, Andy, > > On 15 June 2017 at 23:26, Sumit Semwal > wrote: >> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >> feature and test together. >> - This one also seems like a security hole being closed, and the >> 'feature' could be a candidate for stable backports, but Arnd tried >> that, and it was quite non-trivial. So perhaps we'll need some help >> from the subsystem developers here. > > Could you please help us sort this out? Our goal is to help Greg with > testing stable kernels, and currently the seccomp tests fail due to > missing feature (seccomp ptrace hole closure) getting tested via > latest kselftest. > > If you feel the feature isn't a stable candidate, then could you > please help make the test degrade gracefully in its absence? In some cases, it is not easy to degrade and/or check for a feature. Probably several security features could fall in this bucket. I don't really want to have that change be a backport -- it's quite invasive across multiple architectures. Agreed. The same test for kernel applies to tests as well. If a kernel feature can't be back-ported, the test for that feature will fall in the same bucket. It shouldn't be back-ported. I would say just add a kernel version check to the test. This is probably not the only selftest that will need such things. :) >>> >>> Adding release checks to selftests is going to problematic for >>> maintenance. >>> Tests should fail gracefully if feature isn't supported in older >>> kernels. >>> >>> Several tests do that now and please find a way to check for >>> dependencies >>> and feature availability and fail the test gracefully. If there is a >>> test >>> that can't do that for some reason, we can discuss it, but as a general >>> rule, I don't want to see kselftest patches that check release. >> >> If a future kernel inadvertently loses the new feature and degrades to >> the behavior of old kernels, that would be a serious bug and should be >> caught. Agreed. If I understand you correctly, by not testing stable kernels with their own selftests, some serious bugs could go undetected. >>> >>> Personally I'm a bit skeptical. I think the reasoning is more that the >>> latest selftests provide more coverage, and therefore should be better >>> tests, even on older kernels. >> >> The assumption that "the latest selftests provide more coverage, and >> therefore should be better tests, even on older kernels." is incorrect. >> >> Selftests in general track the kernel features. In some cases, new >> tests could be added that provide better coverage on older kernels, >> however, it is more likely that new tests are added to test new kernel >> features and enhancements to existing features. Based on the second >> "enhancements to existing features" it is more important to test newer >> kernels with older selftests. This does happen in kernel integration >> cycles during development. >> >> As a general rule, testing stable kernels with their own selftests will >> yield the best results. >> > I would have agreed totally, if the selftests and the kernel were in > sync since forever. But since the kselftests are a comparatively > recent addition, the number of tests available for features existing > in LTS kernels is really quite small. Just as a comparison, 4.4-LTS > misses tests for bpf, cpufreq, gpio, media_tests, networking, prctl, > to name a few. Yes. Several tests have been added since 4.4. Unfortunately these new additions don't qualify as candidates for stable releases. > > Also, while trying to run kselftests from later kernels with 4.4, we > only had a few failures for existing features, while most other tests > ran ok. Just another data point. That is good to hear. Let's fix the problems as you find them and make the selftests from newer releases work well (if not perfect) on older kernels. 4.13 is going to see TAP13 changes as well, so if selftests from newer kernels work for you, you will see TAB13 benefits. thanks, -- Shuah
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
On Thu, Jun 22, 2017 at 7:40 PM, Andy Lutomirski wrote: > On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman > wrote: >> On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote: >>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski wrote: >>> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan wrote: >>> >> On 06/22/2017 10:53 AM, Kees Cook wrote: >>> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal >>> >>> wrote: >>> Hi Kees, Andy, >>> >>> On 15 June 2017 at 23:26, Sumit Semwal wrote: >>> > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>> > feature and test together. >>> > - This one also seems like a security hole being closed, and the >>> > 'feature' could be a candidate for stable backports, but Arnd tried >>> > that, and it was quite non-trivial. So perhaps we'll need some help >>> > from the subsystem developers here. >>> >>> Could you please help us sort this out? Our goal is to help Greg with >>> testing stable kernels, and currently the seccomp tests fail due to >>> missing feature (seccomp ptrace hole closure) getting tested via >>> latest kselftest. >>> >>> If you feel the feature isn't a stable candidate, then could you >>> please help make the test degrade gracefully in its absence? >>> >>> >>> >>> I don't really want to have that change be a backport -- it's quite >>> >>> invasive across multiple architectures. >>> >>> >>> >>> I would say just add a kernel version check to the test. This is >>> >>> probably not the only selftest that will need such things. :) >>> >> >>> >> Adding release checks to selftests is going to problematic for >>> >> maintenance. >>> >> Tests should fail gracefully if feature isn't supported in older kernels. >>> >> >>> >> Several tests do that now and please find a way to check for dependencies >>> >> and feature availability and fail the test gracefully. If there is a test >>> >> that can't do that for some reason, we can discuss it, but as a general >>> >> rule, I don't want to see kselftest patches that check release. >>> > >>> > If a future kernel inadvertently loses the new feature and degrades to >>> > the behavior of old kernels, that would be a serious bug and should be >>> > caught. >>> >>> Right. I really think stable kernels should be tested with their own >>> selftests. If some test is needed in a stable kernel it should be >>> backported to that stable kernel. >> >> Well, ideally all new features added to the kernel should be able to be >> detected by userspace somehow if they are present or not. >> >> How do you expect a program to know if a feature has "failed" or is just >> "not enabled/present in this kernel"? Normally with syscalls this is >> easy, same for sysfs changes. Is seccomp in the bad state where there >> is no way to detect the two different states here? How is userspace >> supposed to deal with that? >> >> We make fun of glibc having a zillion crazy tests to determine kernel >> features, and recently, just not wrapping new syscalls at all because >> they are just frustrated at the compatibility issues over time. Let's >> not make their life any harder than it has to be please. >> >> I don't see how any of the kselftest programs are any different than any >> other userspace program that wants to use our kernel api, and as such, >> any version of kselftest should be able to successfully run on any >> kernel release. If not, then we messed up in how we either wrote the >> test, or how we added a new kernel api. Neither is acceptable. > > That's a fair point. We could add SECCOMP_GET_FEATURES that returns a > mask of otherwise-difficult-to-probe features. But that's silly. The self tests includes all kinds of bug fix tests, and adding each of those to some features list seems like crazy overkill. And every API in the kernel is going to do this? > Greg, for context, the issue here is that we made what was arguably a > design error in seccomp's interaction with ptrace. After determining > that fixing it solved a bunch of problems and didn't break any user > programs, we fixed it. There might be new code that relies on the fix > being present in the sense that it would be insecure without the fix. > > The problem is that the fix is moderately intrusive and doesn't seem > like a great candidate for backporting, although we could plausibly do > it. Even if we did all this for seccomp, we're left with the general case. This is not a situation unique to seccomp. Behaviors and bug fix tests are added to selftests over time, and not all of those things are backport-worthy. If a certain test is desired for a stable kernel, we should backport the test. -Kees -- Kees Cook Pixel Security
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
Hi Shuah, On 23 June 2017 at 01:53, Shuah Khan wrote: > Hi Tom, > > On 06/22/2017 01:48 PM, Tom Gall wrote: >> Hi >> >> On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan wrote: >>> On 06/22/2017 11:50 AM, Kees Cook wrote: On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski wrote: > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan wrote: >> On 06/22/2017 10:53 AM, Kees Cook wrote: >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal >>> wrote: Hi Kees, Andy, On 15 June 2017 at 23:26, Sumit Semwal wrote: > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - > feature and test together. > - This one also seems like a security hole being closed, and the > 'feature' could be a candidate for stable backports, but Arnd tried > that, and it was quite non-trivial. So perhaps we'll need some help > from the subsystem developers here. Could you please help us sort this out? Our goal is to help Greg with testing stable kernels, and currently the seccomp tests fail due to missing feature (seccomp ptrace hole closure) getting tested via latest kselftest. If you feel the feature isn't a stable candidate, then could you please help make the test degrade gracefully in its absence? >>> >>> In some cases, it is not easy to degrade and/or check for a feature. >>> Probably several security features could fall in this bucket. >>> >>> >>> I don't really want to have that change be a backport -- it's quite >>> invasive across multiple architectures. >>> >>> Agreed. The same test for kernel applies to tests as well. If a kernel >>> feature can't be back-ported, the test for that feature will fall in the >>> same bucket. It shouldn't be back-ported. >>> >>> >>> I would say just add a kernel version check to the test. This is >>> probably not the only selftest that will need such things. :) >> >> Adding release checks to selftests is going to problematic for >> maintenance. >> Tests should fail gracefully if feature isn't supported in older kernels. >> >> Several tests do that now and please find a way to check for dependencies >> and feature availability and fail the test gracefully. If there is a test >> that can't do that for some reason, we can discuss it, but as a general >> rule, I don't want to see kselftest patches that check release. > > If a future kernel inadvertently loses the new feature and degrades to > the behavior of old kernels, that would be a serious bug and should be > caught. >>> >>> Agreed. If I understand you correctly, by not testing stable kernels >>> with their own selftests, some serious bugs could go undetected. >> >> Personally I'm a bit skeptical. I think the reasoning is more that the >> latest selftests provide more coverage, and therefore should be better >> tests, even on older kernels. > > The assumption that "the latest selftests provide more coverage, and > therefore should be better tests, even on older kernels." is incorrect. > > Selftests in general track the kernel features. In some cases, new > tests could be added that provide better coverage on older kernels, > however, it is more likely that new tests are added to test new kernel > features and enhancements to existing features. Based on the second > "enhancements to existing features" it is more important to test newer > kernels with older selftests. This does happen in kernel integration > cycles during development. > > As a general rule, testing stable kernels with their own selftests will > yield the best results. > I would have agreed totally, if the selftests and the kernel were in sync since forever. But since the kselftests are a comparatively recent addition, the number of tests available for features existing in LTS kernels is really quite small. Just as a comparison, 4.4-LTS misses tests for bpf, cpufreq, gpio, media_tests, networking, prctl, to name a few. Also, while trying to run kselftests from later kernels with 4.4, we only had a few failures for existing features, while most other tests ran ok. Just another data point. >> Right. I really think stable kernels should be tested with their own selftests. If some test is needed in a stable kernel it should be backported to that stable kernel. >>> >>> Correct. This is always a safe option. There might be cases that even >>> prevent tests being built, especially if a new feature adds new fields >>> to an existing structure. >>> >>> It appears in some cases, users want to run newer tests on older kernels. >>> Some tests can clearly detect feature support using module presence and/or >>> Kconfig enabled or disabled. These are conditions even on a kernel that >>> supports a new module or new config option. The kernel the test is running >>> on might not have the feature enabled or module might not be present. In
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
On Thu, Jun 22, 2017 at 6:52 PM, Greg Kroah-Hartman wrote: > On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote: >> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski wrote: >> > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan wrote: >> >> On 06/22/2017 10:53 AM, Kees Cook wrote: >> >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal >> >>> wrote: >> Hi Kees, Andy, >> >> On 15 June 2017 at 23:26, Sumit Semwal wrote: >> > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >> > feature and test together. >> > - This one also seems like a security hole being closed, and the >> > 'feature' could be a candidate for stable backports, but Arnd tried >> > that, and it was quite non-trivial. So perhaps we'll need some help >> > from the subsystem developers here. >> >> Could you please help us sort this out? Our goal is to help Greg with >> testing stable kernels, and currently the seccomp tests fail due to >> missing feature (seccomp ptrace hole closure) getting tested via >> latest kselftest. >> >> If you feel the feature isn't a stable candidate, then could you >> please help make the test degrade gracefully in its absence? >> >>> >> >>> I don't really want to have that change be a backport -- it's quite >> >>> invasive across multiple architectures. >> >>> >> >>> I would say just add a kernel version check to the test. This is >> >>> probably not the only selftest that will need such things. :) >> >> >> >> Adding release checks to selftests is going to problematic for >> >> maintenance. >> >> Tests should fail gracefully if feature isn't supported in older kernels. >> >> >> >> Several tests do that now and please find a way to check for dependencies >> >> and feature availability and fail the test gracefully. If there is a test >> >> that can't do that for some reason, we can discuss it, but as a general >> >> rule, I don't want to see kselftest patches that check release. >> > >> > If a future kernel inadvertently loses the new feature and degrades to >> > the behavior of old kernels, that would be a serious bug and should be >> > caught. >> >> Right. I really think stable kernels should be tested with their own >> selftests. If some test is needed in a stable kernel it should be >> backported to that stable kernel. > > Well, ideally all new features added to the kernel should be able to be > detected by userspace somehow if they are present or not. > > How do you expect a program to know if a feature has "failed" or is just > "not enabled/present in this kernel"? Normally with syscalls this is > easy, same for sysfs changes. Is seccomp in the bad state where there > is no way to detect the two different states here? How is userspace > supposed to deal with that? > > We make fun of glibc having a zillion crazy tests to determine kernel > features, and recently, just not wrapping new syscalls at all because > they are just frustrated at the compatibility issues over time. Let's > not make their life any harder than it has to be please. > > I don't see how any of the kselftest programs are any different than any > other userspace program that wants to use our kernel api, and as such, > any version of kselftest should be able to successfully run on any > kernel release. If not, then we messed up in how we either wrote the > test, or how we added a new kernel api. Neither is acceptable. That's a fair point. We could add SECCOMP_GET_FEATURES that returns a mask of otherwise-difficult-to-probe features. Greg, for context, the issue here is that we made what was arguably a design error in seccomp's interaction with ptrace. After determining that fixing it solved a bunch of problems and didn't break any user programs, we fixed it. There might be new code that relies on the fix being present in the sense that it would be insecure without the fix. The problem is that the fix is moderately intrusive and doesn't seem like a great candidate for backporting, although we could plausibly do it. --Andy
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
On Thu, Jun 22, 2017 at 10:50:43AM -0700, Kees Cook wrote: > On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski wrote: > > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan wrote: > >> On 06/22/2017 10:53 AM, Kees Cook wrote: > >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal > >>> wrote: > Hi Kees, Andy, > > On 15 June 2017 at 23:26, Sumit Semwal wrote: > > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - > > feature and test together. > > - This one also seems like a security hole being closed, and the > > 'feature' could be a candidate for stable backports, but Arnd tried > > that, and it was quite non-trivial. So perhaps we'll need some help > > from the subsystem developers here. > > Could you please help us sort this out? Our goal is to help Greg with > testing stable kernels, and currently the seccomp tests fail due to > missing feature (seccomp ptrace hole closure) getting tested via > latest kselftest. > > If you feel the feature isn't a stable candidate, then could you > please help make the test degrade gracefully in its absence? > >>> > >>> I don't really want to have that change be a backport -- it's quite > >>> invasive across multiple architectures. > >>> > >>> I would say just add a kernel version check to the test. This is > >>> probably not the only selftest that will need such things. :) > >> > >> Adding release checks to selftests is going to problematic for maintenance. > >> Tests should fail gracefully if feature isn't supported in older kernels. > >> > >> Several tests do that now and please find a way to check for dependencies > >> and feature availability and fail the test gracefully. If there is a test > >> that can't do that for some reason, we can discuss it, but as a general > >> rule, I don't want to see kselftest patches that check release. > > > > If a future kernel inadvertently loses the new feature and degrades to > > the behavior of old kernels, that would be a serious bug and should be > > caught. > > Right. I really think stable kernels should be tested with their own > selftests. If some test is needed in a stable kernel it should be > backported to that stable kernel. Well, ideally all new features added to the kernel should be able to be detected by userspace somehow if they are present or not. How do you expect a program to know if a feature has "failed" or is just "not enabled/present in this kernel"? Normally with syscalls this is easy, same for sysfs changes. Is seccomp in the bad state where there is no way to detect the two different states here? How is userspace supposed to deal with that? We make fun of glibc having a zillion crazy tests to determine kernel features, and recently, just not wrapping new syscalls at all because they are just frustrated at the compatibility issues over time. Let's not make their life any harder than it has to be please. I don't see how any of the kselftest programs are any different than any other userspace program that wants to use our kernel api, and as such, any version of kselftest should be able to successfully run on any kernel release. If not, then we messed up in how we either wrote the test, or how we added a new kernel api. Neither is acceptable. thanks, greg k-h
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
Hi Tom, On 06/22/2017 01:48 PM, Tom Gall wrote: > Hi > > On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan wrote: >> On 06/22/2017 11:50 AM, Kees Cook wrote: >>> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski wrote: On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan wrote: > On 06/22/2017 10:53 AM, Kees Cook wrote: >> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal >> wrote: >>> Hi Kees, Andy, >>> >>> On 15 June 2017 at 23:26, Sumit Semwal wrote: 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - feature and test together. - This one also seems like a security hole being closed, and the 'feature' could be a candidate for stable backports, but Arnd tried that, and it was quite non-trivial. So perhaps we'll need some help from the subsystem developers here. >>> >>> Could you please help us sort this out? Our goal is to help Greg with >>> testing stable kernels, and currently the seccomp tests fail due to >>> missing feature (seccomp ptrace hole closure) getting tested via >>> latest kselftest. >>> >>> If you feel the feature isn't a stable candidate, then could you >>> please help make the test degrade gracefully in its absence? >> >> In some cases, it is not easy to degrade and/or check for a feature. >> Probably several security features could fall in this bucket. >> >> >> I don't really want to have that change be a backport -- it's quite >> invasive across multiple architectures. >> >> Agreed. The same test for kernel applies to tests as well. If a kernel >> feature can't be back-ported, the test for that feature will fall in the >> same bucket. It shouldn't be back-ported. >> >> >> I would say just add a kernel version check to the test. This is >> probably not the only selftest that will need such things. :) > > Adding release checks to selftests is going to problematic for > maintenance. > Tests should fail gracefully if feature isn't supported in older kernels. > > Several tests do that now and please find a way to check for dependencies > and feature availability and fail the test gracefully. If there is a test > that can't do that for some reason, we can discuss it, but as a general > rule, I don't want to see kselftest patches that check release. If a future kernel inadvertently loses the new feature and degrades to the behavior of old kernels, that would be a serious bug and should be caught. >> >> Agreed. If I understand you correctly, by not testing stable kernels >> with their own selftests, some serious bugs could go undetected. > > Personally I'm a bit skeptical. I think the reasoning is more that the > latest selftests provide more coverage, and therefore should be better > tests, even on older kernels. The assumption that "the latest selftests provide more coverage, and therefore should be better tests, even on older kernels." is incorrect. Selftests in general track the kernel features. In some cases, new tests could be added that provide better coverage on older kernels, however, it is more likely that new tests are added to test new kernel features and enhancements to existing features. Based on the second "enhancements to existing features" it is more important to test newer kernels with older selftests. This does happen in kernel integration cycles during development. As a general rule, testing stable kernels with their own selftests will yield the best results. > >>> >>> Right. I really think stable kernels should be tested with their own >>> selftests. If some test is needed in a stable kernel it should be >>> backported to that stable kernel. >> >> Correct. This is always a safe option. There might be cases that even >> prevent tests being built, especially if a new feature adds new fields >> to an existing structure. >> >> It appears in some cases, users want to run newer tests on older kernels. >> Some tests can clearly detect feature support using module presence and/or >> Kconfig enabled or disabled. These are conditions even on a kernel that >> supports a new module or new config option. The kernel the test is running >> on might not have the feature enabled or module might not be present. In >> these cases, it would be easier to detect and skip the test. >> >> However, some features aren't so easy. For example: >> >> - a new flag is added to a syscall, and new test is added. It might not >> be easy to detect that. >> - We might have some tests that can't detect and skip. >> >> Based on this discussion, it is probably accurate to say: >> >> 1. It is recommended that selftests from the same release be run on the >>kernel. >> 2. Selftests from newer kernels will run on older kernels, user should >>understand the risks such as some tests might fail and might not >>detect feature degradation related bugs. >> 3. Selftests will fail gracefully on older release
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
Hi On Thu, Jun 22, 2017 at 2:06 PM, Shuah Khan wrote: > On 06/22/2017 11:50 AM, Kees Cook wrote: >> On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski wrote: >>> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan wrote: On 06/22/2017 10:53 AM, Kees Cook wrote: > On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal > wrote: >> Hi Kees, Andy, >> >> On 15 June 2017 at 23:26, Sumit Semwal wrote: >>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>> feature and test together. >>> - This one also seems like a security hole being closed, and the >>> 'feature' could be a candidate for stable backports, but Arnd tried >>> that, and it was quite non-trivial. So perhaps we'll need some help >>> from the subsystem developers here. >> >> Could you please help us sort this out? Our goal is to help Greg with >> testing stable kernels, and currently the seccomp tests fail due to >> missing feature (seccomp ptrace hole closure) getting tested via >> latest kselftest. >> >> If you feel the feature isn't a stable candidate, then could you >> please help make the test degrade gracefully in its absence? > > In some cases, it is not easy to degrade and/or check for a feature. > Probably several security features could fall in this bucket. > > > I don't really want to have that change be a backport -- it's quite > invasive across multiple architectures. > > Agreed. The same test for kernel applies to tests as well. If a kernel > feature can't be back-ported, the test for that feature will fall in the > same bucket. It shouldn't be back-ported. > > > I would say just add a kernel version check to the test. This is > probably not the only selftest that will need such things. :) Adding release checks to selftests is going to problematic for maintenance. Tests should fail gracefully if feature isn't supported in older kernels. Several tests do that now and please find a way to check for dependencies and feature availability and fail the test gracefully. If there is a test that can't do that for some reason, we can discuss it, but as a general rule, I don't want to see kselftest patches that check release. >>> >>> If a future kernel inadvertently loses the new feature and degrades to >>> the behavior of old kernels, that would be a serious bug and should be >>> caught. > > Agreed. If I understand you correctly, by not testing stable kernels > with their own selftests, some serious bugs could go undetected. Personally I'm a bit skeptical. I think the reasoning is more that the latest selftests provide more coverage, and therefore should be better tests, even on older kernels. >> >> Right. I really think stable kernels should be tested with their own >> selftests. If some test is needed in a stable kernel it should be >> backported to that stable kernel. > > Correct. This is always a safe option. There might be cases that even > prevent tests being built, especially if a new feature adds new fields > to an existing structure. > > It appears in some cases, users want to run newer tests on older kernels. > Some tests can clearly detect feature support using module presence and/or > Kconfig enabled or disabled. These are conditions even on a kernel that > supports a new module or new config option. The kernel the test is running > on might not have the feature enabled or module might not be present. In > these cases, it would be easier to detect and skip the test. > > However, some features aren't so easy. For example: > > - a new flag is added to a syscall, and new test is added. It might not > be easy to detect that. > - We might have some tests that can't detect and skip. > > Based on this discussion, it is probably accurate to say: > > 1. It is recommended that selftests from the same release be run on the >kernel. > 2. Selftests from newer kernels will run on older kernels, user should >understand the risks such as some tests might fail and might not >detect feature degradation related bugs. > 3. Selftests will fail gracefully on older releases if at all possible. How about gracefully be skipped instead of fail? The later suggests the test case in some situations can detect it's pointless to run something and say as much instead of emitting a failure that would be a waste of time to look into. As another example take tools/testing/selftests/net/psock_fanout.c On 4.9 it'll fail to compile (using master's selftests) because PACKET_FANOUT_FLAG_UNIQUEID isn't defined. Add a simple #ifdef for that symbol and the psock_fanout test will compile and run just fine. > Sumit! > > 1. What are the reasons for testing older kernel with selftests from >newer kernels? What are the benefits you see for doing so? I think the presumption is the latest greatest collection of selftests are the best, most complete. >I am looking to understand the need/reasons for this use
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
On 06/22/2017 11:50 AM, Kees Cook wrote: > On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski wrote: >> On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan wrote: >>> On 06/22/2017 10:53 AM, Kees Cook wrote: On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal wrote: > Hi Kees, Andy, > > On 15 June 2017 at 23:26, Sumit Semwal wrote: >> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >> feature and test together. >> - This one also seems like a security hole being closed, and the >> 'feature' could be a candidate for stable backports, but Arnd tried >> that, and it was quite non-trivial. So perhaps we'll need some help >> from the subsystem developers here. > > Could you please help us sort this out? Our goal is to help Greg with > testing stable kernels, and currently the seccomp tests fail due to > missing feature (seccomp ptrace hole closure) getting tested via > latest kselftest. > > If you feel the feature isn't a stable candidate, then could you > please help make the test degrade gracefully in its absence? In some cases, it is not easy to degrade and/or check for a feature. Probably several security features could fall in this bucket. I don't really want to have that change be a backport -- it's quite invasive across multiple architectures. Agreed. The same test for kernel applies to tests as well. If a kernel feature can't be back-ported, the test for that feature will fall in the same bucket. It shouldn't be back-ported. I would say just add a kernel version check to the test. This is probably not the only selftest that will need such things. :) >>> >>> Adding release checks to selftests is going to problematic for maintenance. >>> Tests should fail gracefully if feature isn't supported in older kernels. >>> >>> Several tests do that now and please find a way to check for dependencies >>> and feature availability and fail the test gracefully. If there is a test >>> that can't do that for some reason, we can discuss it, but as a general >>> rule, I don't want to see kselftest patches that check release. >> >> If a future kernel inadvertently loses the new feature and degrades to >> the behavior of old kernels, that would be a serious bug and should be >> caught. Agreed. If I understand you correctly, by not testing stable kernels with their own selftests, some serious bugs could go undetected. > > Right. I really think stable kernels should be tested with their own > selftests. If some test is needed in a stable kernel it should be > backported to that stable kernel. Correct. This is always a safe option. There might be cases that even prevent tests being built, especially if a new feature adds new fields to an existing structure. It appears in some cases, users want to run newer tests on older kernels. Some tests can clearly detect feature support using module presence and/or Kconfig enabled or disabled. These are conditions even on a kernel that supports a new module or new config option. The kernel the test is running on might not have the feature enabled or module might not be present. In these cases, it would be easier to detect and skip the test. However, some features aren't so easy. For example: - a new flag is added to a syscall, and new test is added. It might not be easy to detect that. - We might have some tests that can't detect and skip. Based on this discussion, it is probably accurate to say: 1. It is recommended that selftests from the same release be run on the kernel. 2. Selftests from newer kernels will run on older kernels, user should understand the risks such as some tests might fail and might not detect feature degradation related bugs. 3. Selftests will fail gracefully on older releases if at all possible. Sumit! 1. What are the reasons for testing older kernel with selftests from newer kernels? What are the benefits you see for doing so? I am looking to understand the need/reasons for this use-case. In our previous discussion on this subject, I did say, you should be able to do so with some exceptions. 2. Do you test kernels with the selftests from the same release? 3. Do you find testing with newer selftests to be useful? thanks, -- Shuah
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
On Thu, Jun 22, 2017 at 10:49 AM, Andy Lutomirski wrote: > On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan wrote: >> On 06/22/2017 10:53 AM, Kees Cook wrote: >>> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal >>> wrote: Hi Kees, Andy, On 15 June 2017 at 23:26, Sumit Semwal wrote: > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - > feature and test together. > - This one also seems like a security hole being closed, and the > 'feature' could be a candidate for stable backports, but Arnd tried > that, and it was quite non-trivial. So perhaps we'll need some help > from the subsystem developers here. Could you please help us sort this out? Our goal is to help Greg with testing stable kernels, and currently the seccomp tests fail due to missing feature (seccomp ptrace hole closure) getting tested via latest kselftest. If you feel the feature isn't a stable candidate, then could you please help make the test degrade gracefully in its absence? >>> >>> I don't really want to have that change be a backport -- it's quite >>> invasive across multiple architectures. >>> >>> I would say just add a kernel version check to the test. This is >>> probably not the only selftest that will need such things. :) >> >> Adding release checks to selftests is going to problematic for maintenance. >> Tests should fail gracefully if feature isn't supported in older kernels. >> >> Several tests do that now and please find a way to check for dependencies >> and feature availability and fail the test gracefully. If there is a test >> that can't do that for some reason, we can discuss it, but as a general >> rule, I don't want to see kselftest patches that check release. > > If a future kernel inadvertently loses the new feature and degrades to > the behavior of old kernels, that would be a serious bug and should be > caught. Right. I really think stable kernels should be tested with their own selftests. If some test is needed in a stable kernel it should be backported to that stable kernel. -Kees -- Kees Cook Pixel Security
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
On Thu, Jun 22, 2017 at 10:09 AM, Shuah Khan wrote: > On 06/22/2017 10:53 AM, Kees Cook wrote: >> On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal >> wrote: >>> Hi Kees, Andy, >>> >>> On 15 June 2017 at 23:26, Sumit Semwal wrote: 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - feature and test together. - This one also seems like a security hole being closed, and the 'feature' could be a candidate for stable backports, but Arnd tried that, and it was quite non-trivial. So perhaps we'll need some help from the subsystem developers here. >>> >>> Could you please help us sort this out? Our goal is to help Greg with >>> testing stable kernels, and currently the seccomp tests fail due to >>> missing feature (seccomp ptrace hole closure) getting tested via >>> latest kselftest. >>> >>> If you feel the feature isn't a stable candidate, then could you >>> please help make the test degrade gracefully in its absence? >> >> I don't really want to have that change be a backport -- it's quite >> invasive across multiple architectures. >> >> I would say just add a kernel version check to the test. This is >> probably not the only selftest that will need such things. :) > > Adding release checks to selftests is going to problematic for maintenance. > Tests should fail gracefully if feature isn't supported in older kernels. > > Several tests do that now and please find a way to check for dependencies > and feature availability and fail the test gracefully. If there is a test > that can't do that for some reason, we can discuss it, but as a general > rule, I don't want to see kselftest patches that check release. If a future kernel inadvertently loses the new feature and degrades to the behavior of old kernels, that would be a serious bug and should be caught. --Andy
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
On 06/22/2017 10:53 AM, Kees Cook wrote: > On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal wrote: >> Hi Kees, Andy, >> >> On 15 June 2017 at 23:26, Sumit Semwal wrote: >>> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >>> feature and test together. >>> - This one also seems like a security hole being closed, and the >>> 'feature' could be a candidate for stable backports, but Arnd tried >>> that, and it was quite non-trivial. So perhaps we'll need some help >>> from the subsystem developers here. >> >> Could you please help us sort this out? Our goal is to help Greg with >> testing stable kernels, and currently the seccomp tests fail due to >> missing feature (seccomp ptrace hole closure) getting tested via >> latest kselftest. >> >> If you feel the feature isn't a stable candidate, then could you >> please help make the test degrade gracefully in its absence? > > I don't really want to have that change be a backport -- it's quite > invasive across multiple architectures. > > I would say just add a kernel version check to the test. This is > probably not the only selftest that will need such things. :) Adding release checks to selftests is going to problematic for maintenance. Tests should fail gracefully if feature isn't supported in older kernels. Several tests do that now and please find a way to check for dependencies and feature availability and fail the test gracefully. If there is a test that can't do that for some reason, we can discuss it, but as a general rule, I don't want to see kselftest patches that check release. thanks, -- Shuah
Re: seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
On Thu, Jun 22, 2017 at 9:18 AM, Sumit Semwal wrote: > Hi Kees, Andy, > > On 15 June 2017 at 23:26, Sumit Semwal wrote: >> 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - >> feature and test together. >> - This one also seems like a security hole being closed, and the >> 'feature' could be a candidate for stable backports, but Arnd tried >> that, and it was quite non-trivial. So perhaps we'll need some help >> from the subsystem developers here. > > Could you please help us sort this out? Our goal is to help Greg with > testing stable kernels, and currently the seccomp tests fail due to > missing feature (seccomp ptrace hole closure) getting tested via > latest kselftest. > > If you feel the feature isn't a stable candidate, then could you > please help make the test degrade gracefully in its absence? I don't really want to have that change be a backport -- it's quite invasive across multiple architectures. I would say just add a kernel version check to the test. This is probably not the only selftest that will need such things. :) I'd be happy to review such changes! -Kees -- Kees Cook Pixel Security
seccomp ptrace selftest failures with 4.4-stable [Was: Re: LTS testing with latest kselftests - some failures]
Hi Kees, Andy, On 15 June 2017 at 23:26, Sumit Semwal wrote: > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - > feature and test together. > - This one also seems like a security hole being closed, and the > 'feature' could be a candidate for stable backports, but Arnd tried > that, and it was quite non-trivial. So perhaps we'll need some help > from the subsystem developers here. Could you please help us sort this out? Our goal is to help Greg with testing stable kernels, and currently the seccomp tests fail due to missing feature (seccomp ptrace hole closure) getting tested via latest kselftest. If you feel the feature isn't a stable candidate, then could you please help make the test degrade gracefully in its absence? Greatly appreciated! Thanks and best regards, Sumit
Re: LTS testing with latest kselftests - some failures
On Fri, Jun 16, 2017 at 9:46 AM, Luis R. Rodriguez wrote: >> 47e0bbb7fa98 (test: firmware_class: report errors properly on failure) > > Hrm, come to think of it, this *might* have been a stable fix, however the > fix did not mention any specific about real issue with this. Kees? This was mostly a cosmetic fix, though it does fix the return code. It can certainly go to stable, but I try to only push more critical things to -stable. -Kees -- Kees Cook Pixel Security
Re: LTS testing with latest kselftests - some failures
On Mon, Jun 19, 2017 at 10:55:01PM +0800, Greg Kroah-Hartman wrote: > On Mon, Jun 19, 2017 at 04:48:05PM +0200, Luis R. Rodriguez wrote: > > On Sat, Jun 17, 2017 at 06:16:35AM +0200, Greg Kroah-Hartman wrote: > > > On Fri, Jun 16, 2017 at 09:47:21PM +0200, Luis R. Rodriguez wrote: > > > > Some of the knobs however are for extending tests for > > > > existing APIs in older kernels, the async and custom fallback one are an > > > > example. There are a series of test cases later added which could help > > > > test LTS kernels. Would Linaro pick these test driver enhancements to > > > > help > > > > increase coverage of tests? Or is it not worth it? If its worth it then > > > > what I was curious was how to help make this easier for this process to > > > > bloom. > > > > > > I don't understand, what do you mean by "pick these test driver > > > enhancements"? What kind of "knobs" are there in tests? Shouldn't the > > > tests "just work" with no kind of special configuration of the tests be > > > needed? No user is going to know to enable something special. > > > > Test driver knobs, so for instance the async and custom patches referenced > > enable the shell script to use the async api and the custom API. > > Ah, testing kernel code, that makes more sense. I don't really know, if > the apis are present in the older kernel trees, I don't have a problem > having them be backported to stable kernel releases, as this isn't code > that people are actually running on a "normal" system. Wonderful, will peg test-driver changes as stable then when this fits. I really do think this will make test coverage better. Luis
Re: LTS testing with latest kselftests - some failures
On Mon, Jun 19, 2017 at 04:48:05PM +0200, Luis R. Rodriguez wrote: > On Sat, Jun 17, 2017 at 06:16:35AM +0200, Greg Kroah-Hartman wrote: > > On Fri, Jun 16, 2017 at 09:47:21PM +0200, Luis R. Rodriguez wrote: > > > Some of the knobs however are for extending tests for > > > existing APIs in older kernels, the async and custom fallback one are an > > > example. There are a series of test cases later added which could help > > > test LTS kernels. Would Linaro pick these test driver enhancements to help > > > increase coverage of tests? Or is it not worth it? If its worth it then > > > what I was curious was how to help make this easier for this process to > > > bloom. > > > > I don't understand, what do you mean by "pick these test driver > > enhancements"? What kind of "knobs" are there in tests? Shouldn't the > > tests "just work" with no kind of special configuration of the tests be > > needed? No user is going to know to enable something special. > > Test driver knobs, so for instance the async and custom patches referenced > enable the shell script to use the async api and the custom API. Ah, testing kernel code, that makes more sense. I don't really know, if the apis are present in the older kernel trees, I don't have a problem having them be backported to stable kernel releases, as this isn't code that people are actually running on a "normal" system. thanks, greg k-h
Re: LTS testing with latest kselftests - some failures
On Sat, Jun 17, 2017 at 06:16:35AM +0200, Greg Kroah-Hartman wrote: > On Fri, Jun 16, 2017 at 09:47:21PM +0200, Luis R. Rodriguez wrote: > > Some of the knobs however are for extending tests for > > existing APIs in older kernels, the async and custom fallback one are an > > example. There are a series of test cases later added which could help > > test LTS kernels. Would Linaro pick these test driver enhancements to help > > increase coverage of tests? Or is it not worth it? If its worth it then > > what I was curious was how to help make this easier for this process to > > bloom. > > I don't understand, what do you mean by "pick these test driver > enhancements"? What kind of "knobs" are there in tests? Shouldn't the > tests "just work" with no kind of special configuration of the tests be > needed? No user is going to know to enable something special. Test driver knobs, so for instance the async and custom patches referenced enable the shell script to use the async api and the custom API. Luis
Re: LTS testing with latest kselftests - some failures
On Fri, Jun 16, 2017 at 09:47:21PM +0200, Luis R. Rodriguez wrote: > Some of the knobs however are for extending tests for > existing APIs in older kernels, the async and custom fallback one are an > example. There are a series of test cases later added which could help > test LTS kernels. Would Linaro pick these test driver enhancements to help > increase coverage of tests? Or is it not worth it? If its worth it then > what I was curious was how to help make this easier for this process to > bloom. I don't understand, what do you mean by "pick these test driver enhancements"? What kind of "knobs" are there in tests? Shouldn't the tests "just work" with no kind of special configuration of the tests be needed? No user is going to know to enable something special. Make the tests "just work" please, because given the large number of them, no one is going to know to look for special things. thanks, greg k-h
Re: LTS testing with latest kselftests - some failures
On Fri, Jun 16, 2017 at 06:46:51PM +0200, Luis R. Rodriguez wrote: Kees, please review 47e0bbb7fa98 below. Brian, please review be4a1326d12c below. On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote: Hello Greg, Shuah, While testing 4.4.y and 4.9.y LTS kernels with latest kselftest, To be clear it seems like you are taking the latest upstream ksefltest and run it against older stable kernels. Furthermore you seem to only run the shell script tests but are using older kselftests drivers? Is this all correct? Otherwise it is unclear how you are running into the issues below. Does 0-day so the same? I thought 0-day takes just the kselftest from each tree submitted. That *seemed* to me like the way it was designed. Shuah ? Yes in 0-day, we run the kselftest code corresponding to the current kernel. Thanks, Fengguang
Re: LTS testing with latest kselftests - some failures
On Fri, Jun 16, 2017 at 09:29:52PM +0200, Greg Kroah-Hartman wrote: > On Fri, Jun 16, 2017 at 06:46:51PM +0200, Luis R. Rodriguez wrote: > > Kees, please review 47e0bbb7fa98 below. > > Brian, please review be4a1326d12c below. > > > > On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote: > > > Hello Greg, Shuah, > > > > > > While testing 4.4.y and 4.9.y LTS kernels with latest kselftest, > > > > To be clear it seems like you are taking the latest upstream ksefltest and > > run > > it against older stable kernels. Furthermore you seem to only run the shell > > script tests but are using older kselftests drivers? Is this all correct? > > Otherwise it is unclear how you are running into the issues below. > > > > Does 0-day so the same? I thought 0-day takes just the kselftest from each > > tree > > submitted. That *seemed* to me like the way it was designed. Shuah ? > > > > What's the name of *this* testing effort BTW? Is this part of the overall > > kselftest ? Or is this something Linaro does for LTS kernels ? If there > > is a name to your effort can you document it here so that others are aware: > > It's a "test LTS kernels to make sure Greg didn't break anything" type > of testing effort that Linaro is helping out with. OK so its "standard" :) > This could also be called, "it's about time someone did this..." :) Good to know! > > > we found a couple more test failures due to test-kernel mismatch: > > > > > > 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to > > > tests, and related updates to lib/test_firmware.c to improve the > > > tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable > > > 4.9 misses the second update. > > > > <-- snip, skipped 2. and 3. --> > > > > > For all the 3 listed above, we will try and update the tests to > > > gracefully exit. > > > > Hmm, this actually raises a good kselftest question: > > > > I *though* kselftests were running tests on par with the kernels, so we > > would > > *not* take latest upstream kselftests to test against older kernels. Is this > > incorrect? > > That is incorrect. Your test should always degrade gracefully if the > feature is not present in the kernel under test. OK perfect, now I know to look for knobs in the shell tests to ensure this doesn't happen again. Some of the knobs however are for extending tests for existing APIs in older kernels, the async and custom fallback one are an example. There are a series of test cases later added which could help test LTS kernels. Would Linaro pick these test driver enhancements to help increase coverage of tests? Or is it not worth it? If its worth it then what I was curious was how to help make this easier for this process to bloom. > If the test is for a > bug that was fixed, then that fix should also go to a stable kernel > release. Indeed, that was perfectly clear. Luis
Re: LTS testing with latest kselftests - some failures
On Fri, Jun 16, 2017 at 06:46:51PM +0200, Luis R. Rodriguez wrote: > Kees, please review 47e0bbb7fa98 below. > Brian, please review be4a1326d12c below. > > On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote: > > Hello Greg, Shuah, > > > > While testing 4.4.y and 4.9.y LTS kernels with latest kselftest, > > To be clear it seems like you are taking the latest upstream ksefltest and run > it against older stable kernels. Furthermore you seem to only run the shell > script tests but are using older kselftests drivers? Is this all correct? > Otherwise it is unclear how you are running into the issues below. > > Does 0-day so the same? I thought 0-day takes just the kselftest from each > tree > submitted. That *seemed* to me like the way it was designed. Shuah ? > > What's the name of *this* testing effort BTW? Is this part of the overall > kselftest ? Or is this something Linaro does for LTS kernels ? If there > is a name to your effort can you document it here so that others are aware: It's a "test LTS kernels to make sure Greg didn't break anything" type of testing effort that Linaro is helping out with. This could also be called, "it's about time someone did this..." :) > > we found a couple more test failures due to test-kernel mismatch: > > > > 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to > > tests, and related updates to lib/test_firmware.c to improve the > > tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable > > 4.9 misses the second update. > > <-- snip, skipped 2. and 3. --> > > > For all the 3 listed above, we will try and update the tests to gracefully > > exit. > > Hmm, this actually raises a good kselftest question: > > I *though* kselftests were running tests on par with the kernels, so we would > *not* take latest upstream kselftests to test against older kernels. Is this > incorrect? That is incorrect. Your test should always degrade gracefully if the feature is not present in the kernel under test. If the test is for a bug that was fixed, then that fix should also go to a stable kernel release. thanks, greg k-h
Re: LTS testing with latest kselftests - some failures
On Fri, Jun 16, 2017 at 01:08:04PM +0530, Sumit Semwal wrote: > > Thanks, this was quite helpful, and so now bpf tests build on x86_64 > with current mainline for me. Perhaps we should document these > somewhere, as dependencies? > There is already some documentation available[0], but something in the kernel tree would be nice. Please send the patch(es) to netdev. Thanks. [0]: http://docs.cilium.io/en/latest/bpf/#development-environment -- Mit freundlichen Grüßen Alexander Alemayhu
Re: LTS testing with latest kselftests - some failures
Kees, please review 47e0bbb7fa98 below. Brian, please review be4a1326d12c below. On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote: > Hello Greg, Shuah, > > While testing 4.4.y and 4.9.y LTS kernels with latest kselftest, To be clear it seems like you are taking the latest upstream ksefltest and run it against older stable kernels. Furthermore you seem to only run the shell script tests but are using older kselftests drivers? Is this all correct? Otherwise it is unclear how you are running into the issues below. Does 0-day so the same? I thought 0-day takes just the kselftest from each tree submitted. That *seemed* to me like the way it was designed. Shuah ? What's the name of *this* testing effort BTW? Is this part of the overall kselftest ? Or is this something Linaro does for LTS kernels ? If there is a name to your effort can you document it here so that others are aware: https://bottest.wiki.kernel.org/ Replying below only the firmware stuff. > we found a couple more test failures due to test-kernel mismatch: > > 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to > tests, and related updates to lib/test_firmware.c to improve the > tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable > 4.9 misses the second update. <-- snip, skipped 2. and 3. --> > For all the 3 listed above, we will try and update the tests to gracefully > exit. Hmm, this actually raises a good kselftest question: I *though* kselftests were running tests on par with the kernels, so we would *not* take latest upstream kselftests to test against older kernels. Is this incorrect? If this is indeed incorrect then indeed you have a problem and then I understand this email, however this manual approach seems rather fragile. If I would have understood this practice was expected I would have tried to design tests cases a bit differently, but *also* it does beg the question about what to do when the latest kselftest shell script does require some new knob form a test driver which *is* fine to backport to the respective ksefltest C test-driver for an older kernel. What makes this hard is that C test-drivers may depend on new API, so you might have to do some manual work to backport some fancy new API in old ways. This makes me question the value to this mismatch between shell and C test-drivers on kselftests. Your effort seems to be all manual and empirical ? Did we design kselftests with this in mind? Even though using the latest kselftest shell tests against older stable kernels with older kselftest C drivers seems to be like a good idea (provided the above is resolved) your current suggestion to just drop some tests seems worrisome and seems to *invalidate* the gains of such effort and all the pains you are going through. If you are just dropping patches / tests loosely your approach could be missing out on valid tests which *may* have missed out on respective stable patches. The test-firmware async knobs are a good example, and so is the firmware custom fallback trigger. These patches are just extending test coverage, so they help test the existing old kernel API. Its not worthy to Cc stable on those as they are not fixing a stable issue, however their test at times may reveal an issue which a subsequent patch *does fix* which *is* Cc stable. An alternative to the way you are doing things, if I understood it correctly, would be for us to consider evaluating pegging as stable candidates not only ksefltest shell but kselftest C driver extensions, then instead of using the latest ksefltests against older kernels you could just use the kselftest on the respective old stable kernel, the *backport* effort becomes part of the stable pipeline. Note I think this is very debatable and I would not be surprised if Greg does not like it, but its worth *considering* if there is indeed value to your current attempted approach. The alternative of course, is to only use ksefltest from each respective kernel, under the assumption each stable fix does makes its way through. So -- where is the metric value of your current approach? Do we have stats? > I will also individually request subsystem authors / mailing lists for > each of these towards help in improving these tests if required, but > wanted to use this thread as a converging point. > > Thanks and Best regards, > Sumit. > > > [1]: https://lkml.org/lkml/2015/12/8/816 > Patches added via [1]: > eb910947c82f (test: firmware_class: add asynchronous request trigger) This is an example extension to the C test driver which could be useful for kselftest. > be4a1326d12c (test: firmware_class: use kstrndup() where appropriate) I can't see this being a stable candidate, its unclear why this has come up on this thread ? > 47e0bbb7fa98 (test: firmware_class: report errors properly on failure) Hrm, come to think of it, this *might* have been a stable fix, however the fix did not mention any specific about real issue with this. Kees
Re: LTS testing with latest kselftests - some failures
Hi Alexander, On 16 June 2017 at 12:44, Alexander Alemayhu wrote: > Last time I saw similar kinds of errors gcc libraries were missing. > Can you try rerunning after > > apt-get install -y gcc-multilib g++-multilib Thanks, this was quite helpful, and so now bpf tests build on x86_64 with current mainline for me. Perhaps we should document these somewhere, as dependencies? Best, Sumit.
Re: LTS testing with latest kselftests - some failures
On Fri, Jun 16, 2017 at 10:01:37AM +0530, Sumit Semwal wrote: > > make headers_install was missing, but it still didn't improve the > build - here's the pastebin: https://paste.debian.net/971652/ > Last time I saw similar kinds of errors gcc libraries were missing. Can you try rerunning after apt-get install -y gcc-multilib g++-multilib You probably don't need or have most of them, but if the above doesn't help, try apt-get install -y make gcc libssl-dev bc libelf-dev libcap-dev \ clang gcc-multilib llvm libncurses5-dev git -- Mit freundlichen Grüßen Alexander Alemayhu
Re: LTS testing with latest kselftests - some failures
Hi Alexander, On 16 June 2017 at 04:35, Alexander Alemayhu wrote: > On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote: >> >> 4. bpf tests: These seem to have build failures in mainline as well - >> I also tried to build kselftest-next, but a simple 'make -C >> tools/testing/selftests/bpf' seems to error out. Are there any special >> instructions to build these? [I tried x86_64, arm64 cross-compile on x86_64] >> > Do you have the full failure output? If you haven't already you might > also want to run 'make headers_install' in the top level directory. make headers_install was missing, but it still didn't improve the build - here's the pastebin: https://paste.debian.net/971652/ > > -- > Mit freundlichen Grüßen > > Alexander Alemayhu Best, Sumit.
Re: LTS testing with latest kselftests - some failures
On Thu, Jun 15, 2017 at 11:26:53PM +0530, Sumit Semwal wrote: > > 4. bpf tests: These seem to have build failures in mainline as well - > I also tried to build kselftest-next, but a simple 'make -C > tools/testing/selftests/bpf' seems to error out. Are there any special > instructions to build these? [I tried x86_64, arm64 cross-compile on x86_64] > Do you have the full failure output? If you haven't already you might also want to run 'make headers_install' in the top level directory. -- Mit freundlichen Grüßen Alexander Alemayhu
Re: LTS testing with latest kselftests - some failures
Hi Sumit, On 06/15/2017 11:56 AM, Sumit Semwal wrote: > Hello Greg, Shuah, > > While testing 4.4.y and 4.9.y LTS kernels with latest kselftest, we > found a couple more test failures due to test-kernel mismatch: > > 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to > tests, and related updates to lib/test_firmware.c to improve the > tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable > 4.9 misses the second update. I will take a look the commit list your provided and see if it makes sense to back-port. As we discussed earlier in this thread, fixes are something that can be back-ported to include in the stable releases. Updates will not be. > 2. Bitmap test - test got added in 4.5, fails if test_bitmap.ko isn't present. If you can send a patch to have it exit gracefully and change the message to say unsupported feature, I can pull that into mainline can mark it for stable inclusion if it qualifies. > > 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - > feature and test together. > - This one also seems like a security hole being closed, and the > 'feature' could be a candidate for stable backports, but Arnd tried > that, and it was quite non-trivial. So perhaps we'll need some help > from the subsystem developers here. This is something I need to look at and consult with Security maintainers. > > For all the 3 listed above, we will try and update the tests to gracefully > exit. That is great. > > > 4. bpf tests: These seem to have build failures in mainline as well - > I also tried to build kselftest-next, but a simple 'make -C > tools/testing/selftests/bpf' seems to error out. Are there any special > instructions to build these? [I tried x86_64, arm64 cross-compile on x86_64] Hmm. I didn't notice this one. I will check and see what's going on there. > > > I will also individually request subsystem authors / mailing lists for > each of these towards help in improving these tests if required, but > wanted to use this thread as a converging point. Please cc me and linux-kselftest mailing list. Thanks for reporting the problems. > > > [1]: https://lkml.org/lkml/2015/12/8/816 > Patches added via [1]: > eb910947c82f (test: firmware_class: add asynchronous request trigger) > be4a1326d12c (test: firmware_class: use kstrndup() where appropriate) > 47e0bbb7fa98 (test: firmware_class: report errors properly on failure) > > [2]: https://lkml.org/lkml/2017/1/23/440 > Patch added via [2]: > 061132d2b9c9 (test_firmware: add test custom fallback trigger) > > [3]: https://lkml.org/lkml/2016/6/9/627 > > thanks, -- Shuah
LTS testing with latest kselftests - some failures
Hello Greg, Shuah, While testing 4.4.y and 4.9.y LTS kernels with latest kselftest, we found a couple more test failures due to test-kernel mismatch: 1. firmware tests: - linux 4.5 [1] and 4.10 [2] added a few updates to tests, and related updates to lib/test_firmware.c to improve the tests. Stable-4.4 misses these patches to lib/test_firmware.c. Stable 4.9 misses the second update. 2. Bitmap test - test got added in 4.5, fails if test_bitmap.ko isn't present. 3. 'seccomp ptrace hole closure' patches got added in 4.7 [3] - feature and test together. - This one also seems like a security hole being closed, and the 'feature' could be a candidate for stable backports, but Arnd tried that, and it was quite non-trivial. So perhaps we'll need some help from the subsystem developers here. For all the 3 listed above, we will try and update the tests to gracefully exit. 4. bpf tests: These seem to have build failures in mainline as well - I also tried to build kselftest-next, but a simple 'make -C tools/testing/selftests/bpf' seems to error out. Are there any special instructions to build these? [I tried x86_64, arm64 cross-compile on x86_64] I will also individually request subsystem authors / mailing lists for each of these towards help in improving these tests if required, but wanted to use this thread as a converging point. Thanks and Best regards, Sumit. [1]: https://lkml.org/lkml/2015/12/8/816 Patches added via [1]: eb910947c82f (test: firmware_class: add asynchronous request trigger) be4a1326d12c (test: firmware_class: use kstrndup() where appropriate) 47e0bbb7fa98 (test: firmware_class: report errors properly on failure) [2]: https://lkml.org/lkml/2017/1/23/440 Patch added via [2]: 061132d2b9c9 (test_firmware: add test custom fallback trigger) [3]: https://lkml.org/lkml/2016/6/9/627
