Re: RFC: kernelCI media subsystem pilot (Test results for gtucker/kernelci-media - gtucker-kernelci-media-001-6-g1b2c6e5844d8)
On Fri, 2018-10-19 at 09:06 +0200, Hans Verkuil wrote: > On 10/18/2018 09:23 PM, Ezequiel Garcia wrote: > > Hi everyone, > > > > In Collabora, and as part of our kernelci work, we are doing > > research on kernel functional testing with kernelci. > > > > For those new to kernelci, see > > https://github.com/kernelci/kernelci-doc/wiki/KernelCI > > and https://kernelci.org/. > > > > The goal is to lay down the infrastructure required to make > > automated test coverage an integral part of our feature > > and bugfix development process. > > > > So, as a first attempt, we've decided to extend kernelci test > > v4l2 plan support, leading the way to extending > > other subsystems' test plans. > > > > Currently, kernelci looks for a list of branches every hour and > > see if anything changed. For any branch that has changed, it triggers > > builds, boots, tests and reports for each branch that had some changes > > since last time it ran. > > > > For this pilot, we've decided to target just a few devices: > > qemu with vivid, rk3399-gru-kevin and rk3288-veyron-jaq > > with uvc. > > It's running v4l2-compliance, right? > Exactly. > Looking at the test cases, they appear in the reverse order that > v4l2-compliance > performs them, that's a bit odd. > That's something to check in the parser. I'm sure it's fixable if you find it annoying. > And if we include uvc in the testing, then I need to prioritize the work I > started for uvc to remove the last FAILs. > Well, we can run anything. We decided to go for uvc and vivid, just too pick two popular examples, but there are really no restrictions. Thanks, Ezequiel
Re: RFC: kernelCI media subsystem pilot (Test results for gtucker/kernelci-media - gtucker-kernelci-media-001-6-g1b2c6e5844d8)
On 10/18/2018 09:23 PM, Ezequiel Garcia wrote: > Hi everyone, > > In Collabora, and as part of our kernelci work, we are doing > research on kernel functional testing with kernelci. > > For those new to kernelci, see > https://github.com/kernelci/kernelci-doc/wiki/KernelCI > and https://kernelci.org/. > > The goal is to lay down the infrastructure required to make > automated test coverage an integral part of our feature > and bugfix development process. > > So, as a first attempt, we've decided to extend kernelci test > v4l2 plan support, leading the way to extending > other subsystems' test plans. > > Currently, kernelci looks for a list of branches every hour and > see if anything changed. For any branch that has changed, it triggers > builds, boots, tests and reports for each branch that had some changes > since last time it ran. > > For this pilot, we've decided to target just a few devices: > qemu with vivid, rk3399-gru-kevin and rk3288-veyron-jaq > with uvc. It's running v4l2-compliance, right? Looking at the test cases, they appear in the reverse order that v4l2-compliance performs them, that's a bit odd. And if we include uvc in the testing, then I need to prioritize the work I started for uvc to remove the last FAILs. Regards, Hans > > We'd like to get some early feedback on this, so we are sending > an example of how a kernelci report would look like, to trigger > some discussion around > the direction this should take. > > Thanks! > > === > Test results for: > Tree:gtucker > Branch: kernelci-media > Kernel: gtucker-kernelci-media-002-2-gaa27eb0392c7 > URL: https://gitlab.collabora.com/gtucker/linux.git > Commit: aa27eb0392c70adec713e911a9b5267a1d853624 > Test plans: v4l2 > > Summary > --- > 3 test groups results > > 1 | v4l2 | rk3399-gru-kevin | arm64 | 49 total: 17 PASS 4 > FAIL 28 SKIP > 2 | v4l2 | rk3288-veyron-jaq | arm | 49 total: 17 PASS 4 > FAIL 28 SKIP > 3 | v4l2 | qemu | arm64 | 168 total: 102 PASS 0 > FAIL 66 SKIP > > > Tests > - > > 1 | v4l2 | rk3399-gru-kevin | arm64 | 49 total: 17 PASS 4 > FAIL 28 SKIP > > Config: defconfig > Lab Name:lab-collabora-dev > Date:2018-10-18 18:48:52.426000 > TXT log: > http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm64/defconfig/lab-collabora-dev > /v4l2-rk3399-gru-kevin.txt > HTML log: > http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm64/defconfig/lab-collabora-dev > /v4l2-rk3399-gru-kevin.html > Rootfs: > http://staging-storage.kernelci.org/images/rootfs/debian/stretchv4l2/20181017.1//arm64/rootfs.cpio.gz > Test Git:git://linuxtv.org/v4l-utils.git > Test Commit: e889b0d56757b9a74eb8c4c8cc2ebd5b18e228b2 > > > Test cases: > > * MMAP: FAIL > * blocking-wait: PASS > * read/write: SKIP > * read/write: SKIP > * read/write: SKIP > * VIDIOC_EXPBUF: PASS > * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: PASS > * VIDIOC_TRY_DECODER_CMD: SKIP > * VIDIOC_G_ENC_INDEX: SKIP > * VIDIOC_TRY_ENCODER_CMD: SKIP > * Scaling: SKIP > * Composing: SKIP > * Cropping: SKIP > * VIDIOC_G_SLICED_VBI_CAP: SKIP > * VIDIOC_S_FMT: PASS > * VIDIOC_TRY_FMT: PASS > * VIDIOC_G_FMT: PASS > * VIDIOC_G_FBUF: SKIP > * VIDIOC_G/S_PARM: FAIL > * VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: PASS > * VIDIOC_G/S_JPEGCOMP: SKIP > * VIDIOC_UNSUBSCRIBE_EVENT/DQEVENT: PASS > * VIDIOC_G/S/TRY_EXT_CTRLS: FAIL > * VIDIOC_G/S_CTRL: PASS > * VIDIOC_QUERYCTRL: PASS > * VIDIOC_QUERY_EXT_CTRL/QUERYMENU: FAIL > * VIDIOC_G/S_EDID: SKIP > * VIDIOC_DV_TIMINGS_CAP: SKIP > * VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: SKIP > * VIDIOC_ENUM/G/S/QUERY_STD: SKIP > * VIDIOC_G/S_AUDOUT: SKIP > * VIDIOC_G/S/ENUMOUTPUT: SKIP > * VIDIOC_ENUMAUDOUT: SKIP > * VIDIOC_G/S_FREQUENCY: SKIP > * VIDIOC_G/S_MODULATOR: SKIP > * VIDIOC_G/S_AUDIO: SKIP > * VIDIOC_G/S/ENUMINPUT: PASS > * VIDIOC_ENUMAUDIO: SKIP > * VIDIOC_S_HW_FREQ_SEEK: SKIP > * VIDIOC_G/S_FREQUENCY: SKIP > * VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: SKIP > * VIDIOC_LOG_STATUS: SKIP > * VIDIOC_DBG_G/S_REGISTER: SKIP > * for-unlimited-opens: PASS > * VIDIOC_G/S_PRIORITY: PASS > * VIDIOC_QUERYCAP: PASS > * second-/dev/video0-open: PASS > * VIDIOC_QUERYCAP: PASS > * MC-information-see-Media-Driver-Info-above: PASS > > > > 2 | v4l2 | rk3288-veyron-jaq | arm | 49 total: 17 PASS 4 > FAIL 28 SKIP > > Config: multi_v7_defconfig > Lab Name:lab-collabora-dev > Date:2018-10-18 17:00:41.724000 > TXT log: > http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm/multi_v7_defconfig/lab-collab >
RFC: kernelCI media subsystem pilot (Test results for gtucker/kernelci-media - gtucker-kernelci-media-001-6-g1b2c6e5844d8)
Hi everyone, In Collabora, and as part of our kernelci work, we are doing research on kernel functional testing with kernelci. For those new to kernelci, see https://github.com/kernelci/kernelci-doc/wiki/KernelCI and https://kernelci.org/. The goal is to lay down the infrastructure required to make automated test coverage an integral part of our feature and bugfix development process. So, as a first attempt, we've decided to extend kernelci test v4l2 plan support, leading the way to extending other subsystems' test plans. Currently, kernelci looks for a list of branches every hour and see if anything changed. For any branch that has changed, it triggers builds, boots, tests and reports for each branch that had some changes since last time it ran. For this pilot, we've decided to target just a few devices: qemu with vivid, rk3399-gru-kevin and rk3288-veyron-jaq with uvc. We'd like to get some early feedback on this, so we are sending an example of how a kernelci report would look like, to trigger some discussion around the direction this should take. Thanks! === Test results for: Tree:gtucker Branch: kernelci-media Kernel: gtucker-kernelci-media-002-2-gaa27eb0392c7 URL: https://gitlab.collabora.com/gtucker/linux.git Commit: aa27eb0392c70adec713e911a9b5267a1d853624 Test plans: v4l2 Summary --- 3 test groups results 1 | v4l2 | rk3399-gru-kevin | arm64 | 49 total: 17 PASS 4 FAIL 28 SKIP 2 | v4l2 | rk3288-veyron-jaq | arm | 49 total: 17 PASS 4 FAIL 28 SKIP 3 | v4l2 | qemu | arm64 | 168 total: 102 PASS 0 FAIL 66 SKIP Tests - 1 | v4l2 | rk3399-gru-kevin | arm64 | 49 total: 17 PASS 4 FAIL 28 SKIP Config: defconfig Lab Name:lab-collabora-dev Date:2018-10-18 18:48:52.426000 TXT log: http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm64/defconfig/lab-collabora-dev /v4l2-rk3399-gru-kevin.txt HTML log: http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm64/defconfig/lab-collabora-dev /v4l2-rk3399-gru-kevin.html Rootfs: http://staging-storage.kernelci.org/images/rootfs/debian/stretchv4l2/20181017.1//arm64/rootfs.cpio.gz Test Git:git://linuxtv.org/v4l-utils.git Test Commit: e889b0d56757b9a74eb8c4c8cc2ebd5b18e228b2 Test cases: * MMAP: FAIL * blocking-wait: PASS * read/write: SKIP * read/write: SKIP * read/write: SKIP * VIDIOC_EXPBUF: PASS * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: PASS * VIDIOC_TRY_DECODER_CMD: SKIP * VIDIOC_G_ENC_INDEX: SKIP * VIDIOC_TRY_ENCODER_CMD: SKIP * Scaling: SKIP * Composing: SKIP * Cropping: SKIP * VIDIOC_G_SLICED_VBI_CAP: SKIP * VIDIOC_S_FMT: PASS * VIDIOC_TRY_FMT: PASS * VIDIOC_G_FMT: PASS * VIDIOC_G_FBUF: SKIP * VIDIOC_G/S_PARM: FAIL * VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: PASS * VIDIOC_G/S_JPEGCOMP: SKIP * VIDIOC_UNSUBSCRIBE_EVENT/DQEVENT: PASS * VIDIOC_G/S/TRY_EXT_CTRLS: FAIL * VIDIOC_G/S_CTRL: PASS * VIDIOC_QUERYCTRL: PASS * VIDIOC_QUERY_EXT_CTRL/QUERYMENU: FAIL * VIDIOC_G/S_EDID: SKIP * VIDIOC_DV_TIMINGS_CAP: SKIP * VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: SKIP * VIDIOC_ENUM/G/S/QUERY_STD: SKIP * VIDIOC_G/S_AUDOUT: SKIP * VIDIOC_G/S/ENUMOUTPUT: SKIP * VIDIOC_ENUMAUDOUT: SKIP * VIDIOC_G/S_FREQUENCY: SKIP * VIDIOC_G/S_MODULATOR: SKIP * VIDIOC_G/S_AUDIO: SKIP * VIDIOC_G/S/ENUMINPUT: PASS * VIDIOC_ENUMAUDIO: SKIP * VIDIOC_S_HW_FREQ_SEEK: SKIP * VIDIOC_G/S_FREQUENCY: SKIP * VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: SKIP * VIDIOC_LOG_STATUS: SKIP * VIDIOC_DBG_G/S_REGISTER: SKIP * for-unlimited-opens: PASS * VIDIOC_G/S_PRIORITY: PASS * VIDIOC_QUERYCAP: PASS * second-/dev/video0-open: PASS * VIDIOC_QUERYCAP: PASS * MC-information-see-Media-Driver-Info-above: PASS 2 | v4l2 | rk3288-veyron-jaq | arm | 49 total: 17 PASS 4 FAIL 28 SKIP Config: multi_v7_defconfig Lab Name:lab-collabora-dev Date:2018-10-18 17:00:41.724000 TXT log: http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm/multi_v7_defconfig/lab-collab ora-dev/v4l2-rk3288-veyron-jaq.txt HTML log: http://staging-storage.kernelci.org/gtucker/kernelci-media/gtucker-kernelci-media-002-2-gaa27eb0392c7/arm/multi_v7_defconfig/lab-collab ora-dev/v4l2-rk3288-veyron-jaq.html Rootfs: http://staging-storage.kernelci.org/images/rootfs/debian/stretchv4l2/20181017.1//armhf/rootfs.cpio.gz Test Git:git://linuxtv.org/v4l-utils.git Test Commit: e889b0d56757b9a74eb8c4c8cc2ebd5b18e228b2 Test cases: * MMAP: FAIL * blocking-wait: PASS * read/write: SKIP * read/write: SKIP * read/write: SKIP * VIDIOC_EXPBUF: PASS *
