On 11/07/2018 12:10 PM, Mauro Carvalho Chehab wrote: > Em Wed, 07 Nov 2018 12:06:55 +0200 > Laurent Pinchart <laurent.pinch...@ideasonboard.com> escreveu: > >> Hi Hans, >> >> On Wednesday, 7 November 2018 10:05:12 EET Hans Verkuil wrote: >>> On 11/06/2018 08:58 PM, Laurent Pinchart wrote: >>>> On Tuesday, 6 November 2018 15:56:34 EET Hans Verkuil wrote: >>>>> On 11/06/18 14:12, Laurent Pinchart wrote: >>>>>> On Tuesday, 6 November 2018 13:36:55 EET Sakari Ailus wrote: >>>>>>> On Tue, Nov 06, 2018 at 09:37:07AM +0100, Hans Verkuil wrote: >>>>>>>> Hi all, >>>>>>>> >>>>>>>> After the media summit (heavy on test discussions) and the V4L2 event >>>>>>>> regression we just found it is clear we need to do a better job with >>>>>>>> testing. >>>>>>>> >>>>>>>> All the pieces are in place, so what is needed is to combine it and >>>>>>>> create a script that anyone of us as core developers can run to check >>>>>>>> for regressions. The same script can be run as part of the kernelci >>>>>>>> regression testing. >>>>>>> >>>>>>> I'd say that *some* pieces are in place. Of course, the more there is, >>>>>>> the better. >>>>>>> >>>>>>> The more there are tests, the more important it would be they're >>>>>>> automated, preferrably without the developer having to run them on his/ >>>>>>> her own machine. >>>>>> >>>>>> From my experience with testing, it's important to have both a core set >>>>>> of tests (a.k.a. smoke tests) that can easily be run on developers' >>>>>> machines, and extended tests that can be offloaded to a shared testing >>>>>> infrastructure (but possibly also run locally if desired). >>>>> >>>>> That was my idea as well for the longer term. First step is to do the >>>>> basic smoke tests (i.e. run compliance tests, do some (limited) streaming >>>>> test). >>>>> >>>>> There are more extensive (and longer running) tests that can be done, but >>>>> that's something to look at later. >>>>> >>>>>>>> We have four virtual drivers: vivid, vim2m, vimc and vicodec. The last >>>>>>>> one is IMHO not quite good enough yet for testing: it is not fully >>>>>>>> compliant to the upcoming stateful codec spec. Work for that is >>>>>>>> planned as part of an Outreachy project. >>>>>>>> >>>>>>>> My idea is to create a script that is maintained as part of v4l-utils >>>>>>>> that loads the drivers and runs v4l2-compliance and possibly other >>>>>>>> tests against the virtual drivers. > > (adding Shuah) > > IMO, the best would be to have something like that as part of Kernel > self test, as this could give a broader covering than just Kernel CI. >
I agree with the broader coverage benefit that comes with adding tests to kselftest. It makes it easier for making changes to tests/tools coupled with kernel/driver changes. Common TAP13 reporting can be taken advantage of without doing any additional work in the tests if author chooses to do so. Tests can be added such that they don't get run by default if there is a reason do so and Kernel CI and other rings can invoke it as a special case if necessary. There are very clear advantages to making these tests part of the kernel source tree. We can discuss at the Kernel Summit next week if you are interested. thanks, -- Shuah
