Re: [Intel-gfx] [PATCH i-g-t RFC] tests: Add explicit test lists for CI use.

[Daniel Vetter]

On Thu, Oct 06, 2016 at 09:31:32AM +0300, Petri Latvala wrote:
> See tests/intel-ci/README for rationale and explanation.
> 
> Signed-off-by: Petri Latvala 
> ---
> 
> 
> The plans for CI are to use explicit test lists in the future, as laid
> out in this commit. Comments, acks, nacks, flames, etc would be very
> welcome.
> 
> The only list so far is a fattened version of the currently run set,
> also known as the BAT set. The tests added are such that they all pass
> in preliminary test runs, and do it sufficiently quickly to still meet
> the time requirement of 15 minute runs.
> 
> Having said that, the limit is getting a bit tight and some of these
> should be dropped before deploying to production. Comments on test
> overlap and such also appreciated.

I think we should split this into 2 patches: First switch the BAT test
list over to this file, then in a 2nd patch extend it.
-Daniel


> 
> 
> 
> 
> 
>  tests/intel-ci/README |  34 ++
>  tests/intel-ci/fast-feedback.testlist | 611 
> ++
>  2 files changed, 645 insertions(+)
>  create mode 100644 tests/intel-ci/README
>  create mode 100644 tests/intel-ci/fast-feedback.testlist
> 
> diff --git a/tests/intel-ci/README b/tests/intel-ci/README
> new file mode 100644
> index 000..dc7e5c8
> --- /dev/null
> +++ b/tests/intel-ci/README
> @@ -0,0 +1,34 @@
> +This directory contains test lists that are used by Intel's CI. The
> +files are passed to piglit with the --test-list parameter directly.
> +
> +The test lists are contained in the IGT repository for several
> +reasons:
> +
> +- The lists stay synchronized with the IGT codebase.
> +- Public availability. Kernel developers can see what tests are run,
> +  and can see what changes are done to the set, when, and why.
> +- Explicit test lists in general make it possible to implement a new
> +  test without having it run in CI (accidentally or on purpose) before
> +  CI is ready for it, and for CI to stop running a test without
> +  removing the test or changing its name. The previous method of test
> +  selection was based on name patterns.
> +
> +Changing the test lists should only happen with approval from the CI
> +people. If you make a new test that should be tested by CI regularly,
> +ask a maintainer (see MAINTAINERS).
> +
> +==
> +fast-feedback.testlist
> +==
> +
> +Fast-feedback is the testlist that is run for Patchwork and Trybot
> +testing. It is also run on various drm-intel-* branches.
> +
> +Time limit: Full run must complete in 15 minutes on all tested
> +platforms.
> +
> +Feature coverage: Should test all working features, with at least 1
> +test per feature.
> +
> +The string "basic" in a test name means the test probably belongs in
> +this list.
> diff --git a/tests/intel-ci/fast-feedback.testlist 
> b/tests/intel-ci/fast-feedback.testlist
> new file mode 100644
> index 000..6f9349c
> --- /dev/null
> +++ b/tests/intel-ci/fast-feedback.testlist
> @@ -0,0 +1,611 @@
> +igt@core_auth@basic-auth
> +igt@core_auth@many-magics
> +igt@core_getclient
> +igt@core_get_client_auth@master-drop
> +igt@core_get_client_auth@simple
> +igt@core_getstats
> +igt@core_getversion
> +igt@core_prop_blob@basic
> +igt@core_prop_blob@blob-multiple
> +igt@core_prop_blob@blob-prop-core
> +igt@core_prop_blob@blob-prop-lifetime
> +igt@core_prop_blob@blob-prop-validate
> +igt@core_prop_blob@invalid-get-prop
> +igt@core_prop_blob@invalid-get-prop-any
> +igt@core_prop_blob@invalid-set-prop
> +igt@core_prop_blob@invalid-set-prop-any
> +igt@core_setmaster_vs_auth
> +igt@debugfs_emon_crash
> +igt@drm_import_export@import-close-race-flink
> +igt@drm_import_export@import-close-race-prime
> +igt@drm_read@empty-nonblock
> +igt@drm_read@fault-buffer
> +igt@drm_read@invalid-buffer
> +igt@drm_read@short-buffer-block
> +igt@drm_read@short-buffer-nonblock
> +igt@drm_vma_limiter
> +igt@drm_vma_limiter_cpu
> +igt@drv_debugfs_reader
> +igt@drv_getparams_basic@basic-eu-total
> +igt@drv_getparams_basic@basic-subslice-total
> +igt@drv_hangman@error-state-basic
> +igt@drv_hangman@error-state-debugfs-entry
> +igt@drv_hangman@error-state-sysfs-entry
> +igt@drv_module_reload_basic
> +igt@gem_bad_length
> +igt@gem_bad_reloc@negative-reloc-blt
> +igt@gem_bad_reloc@negative-reloc-bltcopy
> +igt@gem_bad_reloc@negative-reloc-bsd
> +igt@gem_bad_reloc@negative-reloc-bsd1
> +igt@gem_bad_reloc@negative-reloc-bsd2
> +igt@gem_bad_reloc@negative-reloc-default
> +igt@gem_bad_reloc@negative-reloc-lut-blt
> +igt@gem_bad_reloc@negative-reloc-lut-bsd
> +igt@gem_bad_reloc@negative-reloc-lut-bsd1
> +igt@gem_bad_reloc@negative-reloc-lut-bsd2
> +igt@gem_bad_reloc@negative-reloc-lut-default
> +igt@gem_bad_reloc@negative-reloc-lut-render
> +igt@gem_bad_reloc@negative-reloc-lut-vebox
> +igt@gem_bad_reloc@negative-reloc-render
> +igt@gem_bad_reloc@negative-reloc-vebox
> +igt@gem_basic@bad-close
> +igt@gem_basic@create-close
> +igt@gem_basic@create-fd-close

Re: [Intel-gfx] [PATCH i-g-t RFC] tests: Add explicit test lists for CI use.

[Daniel Vetter]

On Thu, Oct 06, 2016 at 07:51:46AM +0100, Chris Wilson wrote:
> On Thu, Oct 06, 2016 at 09:31:32AM +0300, Petri Latvala wrote:
> > See tests/intel-ci/README for rationale and explanation.
> 
> A lot of those tests are garbage and not future proof making them
> useless for regression testing. Including them just adds noise and us
> ignoring the results.

We need to nuke bogus tests, carrying them around in igt.git seems
pointless to me.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
___
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Intel-gfx] [PATCH i-g-t RFC] tests: Add explicit test lists for CI use.

[Chris Wilson]

On Thu, Oct 06, 2016 at 09:31:32AM +0300, Petri Latvala wrote:
> See tests/intel-ci/README for rationale and explanation.

A lot of those tests are garbage and not future proof making them
useless for regression testing. Including them just adds noise and us
ignoring the results.
-Chris

-- 
Chris Wilson, Intel Open Source Technology Centre
___
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

[Intel-gfx] [PATCH i-g-t RFC] tests: Add explicit test lists for CI use.

[Petri Latvala]

See tests/intel-ci/README for rationale and explanation.

Signed-off-by: Petri Latvala 
---


The plans for CI are to use explicit test lists in the future, as laid
out in this commit. Comments, acks, nacks, flames, etc would be very
welcome.

The only list so far is a fattened version of the currently run set,
also known as the BAT set. The tests added are such that they all pass
in preliminary test runs, and do it sufficiently quickly to still meet
the time requirement of 15 minute runs.

Having said that, the limit is getting a bit tight and some of these
should be dropped before deploying to production. Comments on test
overlap and such also appreciated.





 tests/intel-ci/README |  34 ++
 tests/intel-ci/fast-feedback.testlist | 611 ++
 2 files changed, 645 insertions(+)
 create mode 100644 tests/intel-ci/README
 create mode 100644 tests/intel-ci/fast-feedback.testlist

diff --git a/tests/intel-ci/README b/tests/intel-ci/README
new file mode 100644
index 000..dc7e5c8
--- /dev/null
+++ b/tests/intel-ci/README
@@ -0,0 +1,34 @@
+This directory contains test lists that are used by Intel's CI. The
+files are passed to piglit with the --test-list parameter directly.
+
+The test lists are contained in the IGT repository for several
+reasons:
+
+- The lists stay synchronized with the IGT codebase.
+- Public availability. Kernel developers can see what tests are run,
+  and can see what changes are done to the set, when, and why.
+- Explicit test lists in general make it possible to implement a new
+  test without having it run in CI (accidentally or on purpose) before
+  CI is ready for it, and for CI to stop running a test without
+  removing the test or changing its name. The previous method of test
+  selection was based on name patterns.
+
+Changing the test lists should only happen with approval from the CI
+people. If you make a new test that should be tested by CI regularly,
+ask a maintainer (see MAINTAINERS).
+
+==
+fast-feedback.testlist
+==
+
+Fast-feedback is the testlist that is run for Patchwork and Trybot
+testing. It is also run on various drm-intel-* branches.
+
+Time limit: Full run must complete in 15 minutes on all tested
+platforms.
+
+Feature coverage: Should test all working features, with at least 1
+test per feature.
+
+The string "basic" in a test name means the test probably belongs in
+this list.
diff --git a/tests/intel-ci/fast-feedback.testlist 
b/tests/intel-ci/fast-feedback.testlist
new file mode 100644
index 000..6f9349c
--- /dev/null
+++ b/tests/intel-ci/fast-feedback.testlist
@@ -0,0 +1,611 @@
+igt@core_auth@basic-auth
+igt@core_auth@many-magics
+igt@core_getclient
+igt@core_get_client_auth@master-drop
+igt@core_get_client_auth@simple
+igt@core_getstats
+igt@core_getversion
+igt@core_prop_blob@basic
+igt@core_prop_blob@blob-multiple
+igt@core_prop_blob@blob-prop-core
+igt@core_prop_blob@blob-prop-lifetime
+igt@core_prop_blob@blob-prop-validate
+igt@core_prop_blob@invalid-get-prop
+igt@core_prop_blob@invalid-get-prop-any
+igt@core_prop_blob@invalid-set-prop
+igt@core_prop_blob@invalid-set-prop-any
+igt@core_setmaster_vs_auth
+igt@debugfs_emon_crash
+igt@drm_import_export@import-close-race-flink
+igt@drm_import_export@import-close-race-prime
+igt@drm_read@empty-nonblock
+igt@drm_read@fault-buffer
+igt@drm_read@invalid-buffer
+igt@drm_read@short-buffer-block
+igt@drm_read@short-buffer-nonblock
+igt@drm_vma_limiter
+igt@drm_vma_limiter_cpu
+igt@drv_debugfs_reader
+igt@drv_getparams_basic@basic-eu-total
+igt@drv_getparams_basic@basic-subslice-total
+igt@drv_hangman@error-state-basic
+igt@drv_hangman@error-state-debugfs-entry
+igt@drv_hangman@error-state-sysfs-entry
+igt@drv_module_reload_basic
+igt@gem_bad_length
+igt@gem_bad_reloc@negative-reloc-blt
+igt@gem_bad_reloc@negative-reloc-bltcopy
+igt@gem_bad_reloc@negative-reloc-bsd
+igt@gem_bad_reloc@negative-reloc-bsd1
+igt@gem_bad_reloc@negative-reloc-bsd2
+igt@gem_bad_reloc@negative-reloc-default
+igt@gem_bad_reloc@negative-reloc-lut-blt
+igt@gem_bad_reloc@negative-reloc-lut-bsd
+igt@gem_bad_reloc@negative-reloc-lut-bsd1
+igt@gem_bad_reloc@negative-reloc-lut-bsd2
+igt@gem_bad_reloc@negative-reloc-lut-default
+igt@gem_bad_reloc@negative-reloc-lut-render
+igt@gem_bad_reloc@negative-reloc-lut-vebox
+igt@gem_bad_reloc@negative-reloc-render
+igt@gem_bad_reloc@negative-reloc-vebox
+igt@gem_basic@bad-close
+igt@gem_basic@create-close
+igt@gem_basic@create-fd-close
+igt@gem_busy@basic-busy-default
+igt@gem_busy@basic-hang-default
+igt@gem_caching@reads
+igt@gem_close_race@basic-process
+igt@gem_close_race@basic-threads
+igt@gem_close_race@process-exit
+igt@gem_cpu_reloc@basic
+igt@gem_create@create-invalid-nonaligned
+igt@gem_create@create-invalid-size
+igt@gem_create@create-valid-nonaligned
+igt@gem_cs_tlb@basic-default
+igt@gem_cs_tlb@blt
+igt@gem_cs_tlb@bsd
+igt@gem_cs_tlb@bsd1
+igt@gem_cs_tlb@bsd2