Re: [Piglit] [PATCH] framework/run.py: allow additional excluded tests on resume
OK, I'm abandoning this patch in favor of this one: [Piglit] [PATCH] framework/run.py: add option to not retry incomplete tests on resume Mike -Original Message- From: Dylan Baker [mailto:baker.dyla...@gmail.com] Sent: Tuesday, May 19, 2015 3:05 PM To: Mason, Michael W Cc: Daniel Vetter; piglit@lists.freedesktop.org Subject: Re: [Piglit] [PATCH] framework/run.py: allow additional excluded tests on resume On Tue, May 19, 2015 at 04:45:59PM +, Mason, Michael W wrote: How about something like this instead of my original patch? I think we should preserve the fact that some tests didn't complete. I'll submit this in a separate email if you all agree. --- a/framework/programs/run.py +++ b/framework/programs/run.py @@ -313,6 +313,9 @@ def resume(input_): type=argparse.FileType(r), help=Optionally specify a piglit config file to use. Default is piglit.conf) +parser.add_argument(-n, --no-retry, dest='no_retry', Drop dest down to a new line like the others please. +action='store_true', +help=Do not retry incomplete tests) args = parser.parse_args(input_) _disable_windows_exception_messages() @@ -342,7 +345,7 @@ def resume(input_): # Don't re-run tests that have already completed, incomplete status tests # have obviously not completed. for name, result in results.tests.iteritems(): -if result['result'] != 'incomplete': +if result['result'] != 'incomplete' or args.no_retry: It really doesn't matter but, should we check args.no_retry first? Since python is lazy I think it will be optimal in most cases. opts.exclude_tests.add(name) This looks reasonable to me. I left a couple of nits above. Dylan ___ Piglit mailing list Piglit@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/piglit
Re: [Piglit] [PATCH] framework/run.py: add option to not retry incomplete tests on resume
-Original Message- From: Dylan Baker [mailto:baker.dyla...@gmail.com] Sent: Wednesday, May 20, 2015 5:27 PM To: Mason, Michael W Cc: piglit@lists.freedesktop.org Subject: Re: [Piglit] [PATCH] framework/run.py: add option to not retry incomplete tests on resume Looks good to me. Reviewed-by: Dylan Baker baker.dyla...@gmail.com btw, do you have push access? I don't think so. I've never tried. On Wed, May 20, 2015 at 02:17:04PM -0700, Mike Mason wrote: This patch adds an option to not retry incomplete tests when resuming a test run. This is especially useful when a failing test causes a crash or reboot. Currently, that same test runs again when attempting to resume the test run, resulting in the same crash or reboot. Signed-off-by: Mike Mason michael.w.ma...@intel.com --- framework/programs/run.py | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/framework/programs/run.py b/framework/programs/run.py index 6053074..c6d7afe 100644 --- a/framework/programs/run.py +++ b/framework/programs/run.py @@ -313,6 +313,10 @@ def resume(input_): type=argparse.FileType(r), help=Optionally specify a piglit config file to use. Default is piglit.conf) +parser.add_argument(-n, --no-retry, +dest=no_retry, +action=store_true, +help=Do not retry incomplete tests) args = parser.parse_args(input_) _disable_windows_exception_messages() @@ -342,7 +346,7 @@ def resume(input_): # Don't re-run tests that have already completed, incomplete status tests # have obviously not completed. for name, result in results.tests.iteritems(): -if result['result'] != 'incomplete': +if args.no_retry or result['result'] != 'incomplete': opts.exclude_tests.add(name) profile = framework.profile.merge_test_profiles(results.options['profile']) -- 1.9.1 ___ Piglit mailing list Piglit@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/piglit ___ Piglit mailing list Piglit@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/piglit
Re: [Piglit] [PATCH] framework/run.py: allow additional excluded tests on resume
-Original Message- From: Dylan Baker [mailto:baker.dyla...@gmail.com] Sent: Tuesday, May 19, 2015 9:42 AM To: Daniel Vetter Cc: Mason, Michael W; piglit@lists.freedesktop.org Subject: Re: [Piglit] [PATCH] framework/run.py: allow additional excluded tests on resume On Tue, May 19, 2015 at 12:06:54PM +0200, Daniel Vetter wrote: On Mon, May 18, 2015 at 05:12:17PM -0700, Dylan Baker wrote: I'm not sure this is a good idea, Ken Graunke and I talked this to death when I did the refactor that split the run and resume code into separate paths, and we agreed that changing the tests during a resume would make it impossible to reproduce a run, since the regex you pass might exclude tests that have already completed. That said, I'm opened to being convinced it is a good idea. With your patches to mark a test as incomplete, should we instead just have an option on resume to mark all incomplete tests as failed and not try to restart them? Or is that already what happens (tbh I haven't tried). -Daniel Currently they're always retried. It seems reasonable to me to either change the behavior or add a switch to change the behavior to either mark them as failed or just leave them as incomplete. How about something like this instead of my original patch? I think we should preserve the fact that some tests didn't complete. I'll submit this in a separate email if you all agree. --- a/framework/programs/run.py +++ b/framework/programs/run.py @@ -313,6 +313,9 @@ def resume(input_): type=argparse.FileType(r), help=Optionally specify a piglit config file to use. Default is piglit.conf) +parser.add_argument(-n, --no-retry, dest='no_retry', +action='store_true', +help=Do not retry incomplete tests) args = parser.parse_args(input_) _disable_windows_exception_messages() @@ -342,7 +345,7 @@ def resume(input_): # Don't re-run tests that have already completed, incomplete status tests # have obviously not completed. for name, result in results.tests.iteritems(): -if result['result'] != 'incomplete': +if result['result'] != 'incomplete' or args.no_retry: opts.exclude_tests.add(name) [snip] ___ Piglit mailing list Piglit@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/piglit
Re: [Piglit] [PATCH] deqp: Add test profile for external dEQP-GLES3 tests (v4)
Reviewed-by: Michael W Mason michael.w.ma...@intel.com I ran this on link-freon and it worked as expected. Thanks Chad! -Original Message- From: Chad Versace [mailto:chad.vers...@linux.intel.com] Sent: Tuesday, December 16, 2014 1:49 PM To: piglit@lists.freedesktop.org Cc: Chad Versace; Baker, DylanX C; Mason, Michael W Subject: [PATCH] deqp: Add test profile for external dEQP-GLES3 tests (v4) Google opensourced the drawElements Quality Product Testsuite (dEQP) as part of the Android Lollipop release. It's git repo lives in the Android tree at [https://android.googlesource.com/platform/external/deqp/]. This patch adds a new module, deqp_gles3.py, that runs the dEQP-GLES3 tests. It queries the 'deqp-gles3' executable at runtime for the list of testcases. The module attempts queries the 'deqp-gles3' executable only if the environment variable PIGLIT_DEQP_GLES3_EXE or the piglit.conf option deqp-gles3.exe is set. Why do we need to use Pigit as the testrunner for dEQP? (After all, dEQP has its own testrunner). Because dEQP runs all tests in a single process. If test 17530 of 55409 crashes, then the dEQP testrunner crashes and the remaining tests never run. Piglit doesn't suffer from that problem, because it runs each test as a separate process. Tested on Intel Ivybridge by running the command piglit run -t dEQP-GLES3/info/vendor \ tests/deqp_gles3.py tests/quick.py results with and without PIGLIT_DEQP_GLES3_EXE set and with and without piglit.conf:deqp-gles.exe set. Also tested with various combinations of valid and invalid values for PIGLIT_DEQP_GLES_EXTRA_ARGS and piglit.conf:deqp-gles.extra_args. Cc: Dylan Baker dylanx.c.ba...@intel.com Cc: Michael W Mason michael.w.ma...@intel.com Signed-off-by: Chad Versace chad.vers...@linux.intel.com --- v2: - Remove unused imports (dcbaker) - Remove extra newlines (dcbaker) - Don't use 'global' keyword (dcbaker) - Use global constant for DEQP_GLES3_EXE (dcbaker) - Fix DEQPTest.interpret_result to correctly parse test output. (mwmason) - Use 'super' keyword (dcbaker) v3: - Fix misnamed variable s/caselist_txt/caselist_path/. (mwmason) v4: - Add option piglit.conf:extra_args and env var PIGLIT_DEQP_GLES3_EXTRA_ARGS. (requested by mwmason) - Remove '--deqp-visibility=hidden'. (mwmason) - Fixed exception handling in func get_option. piglit.conf.example | 12 + tests/deqp_gles3.py | 139 2 files changed, 151 insertions(+) create mode 100644 tests/deqp_gles3.py diff --git a/piglit.conf.example b/piglit.conf.example index a864c01..b3869c2 100644 --- a/piglit.conf.example +++ b/piglit.conf.example @@ -33,6 +33,18 @@ bindir=/home/usr/oclconform testA testB +;[deqp-gles3] +; +; Path to the deqp-gles3 executable. You can also set this with the +environment ; variable PIGLIT_DEQP_GLES3_EXE. Piglit will run the +dEQP-GLES3 tests if and ; only if this option is set. +;exe=/home/knuth/deqp/modules/gles3/deqp-gles3 +; +; Space-separated list of extra command line arguments for deqp-gles3. +The ; option is not required. The environment variable +PIGLIT_DEQP_GLES3_EXTRA_ARGS ; overrides the value set here. +;extra_args=--deqp-visibility hidden + ; Section for specific oclconform test. One of these sections is required for ; each test list in the oclconform section and must be called: ; oclconform-$testname diff --git a/tests/deqp_gles3.py b/tests/deqp_gles3.py new file mode 100644 index 000..b1bb33b --- /dev/null +++ b/tests/deqp_gles3.py @@ -0,0 +1,139 @@ +# Copyright 2014 Intel Corporation +# +# Permission is hereby granted, free of charge, to any person obtaining +a copy # of this software and associated documentation files (the +Software), to deal # in the Software without restriction, including +without limitation the rights # to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell # copies of the Software, and to +permit persons to whom the Software is # furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be +included in # all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS +BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE # SOFTWARE. + +import ConfigParser +import os +import subprocess + +# Piglit modules +import framework +from framework.profile import Test, TestProfile + +__all__ = ['profile
Re: [Piglit] [PATCH] deqp: Add test profile for external dEQP-GLES3 tests (v3)
-Original Message- From: Dylan Baker [mailto:baker.dyla...@gmail.com] Sent: Friday, December 12, 2014 6:04 PM To: piglit@lists.freedesktop.org Cc: Mason, Michael W; Chad Versace Subject: Re: [Piglit] [PATCH] deqp: Add test profile for external dEQP-GLES3 tests (v3) On Saturday, December 13, 2014 12:53:42 AM Mason, Michael W wrote: A couple questions... First, deqp-gles3 has many command line options. Is there any way in piglit to pass them to deqp-gles3 from the piglit command line? I don't see a way. One option in particular is important when testing on Freon. I have to pass --deqp-gl- config-name=rgbad24s8 to get the images to display full screen. No doubt we'll want to use other options in the future. There is no way currently. It wouldn't be that hard to implement the feature however. The other option is to do what we do for several of the integrated suites and use a mixture of environment variables and piglit.conf settings, depending on how often you want to change those settings. Using the env variable/piglit.conf method to pass options to deqp-gles3 works great. See my suggestions below. Second, how do you kill a piglit test run once it's started? I tried multiple ctrl-C's without success. I couldn't get it to stop. Is this a piglit limitation or something to do with the piglit/deqp combo? BTW, the latest patch does work, but I'd like to understand if the above limitations are expected or not before giving it my blessing. As Matt pointed out ctrl+\ will kill all tests. ctrl+c will kill a single running test. Mike -Original Message- From: Chad Versace [mailto:chad.vers...@linux.intel.com] Sent: Wednesday, December 10, 2014 9:39 AM To: piglit@lists.freedesktop.org Cc: Chad Versace; Baker, DylanX C; Mason, Michael W Subject: [PATCH] deqp: Add test profile for external dEQP-GLES3 tests (v3) Google opensourced the drawElements Quality Product Testsuite (dEQP) as part of the Android Lollipop release. It's git repo lives in the Android tree at [https://android.googlesource.com/platform/external/deqp/]. This patch adds a new module, deqp_gles3.py, that runs the dEQP-GLES3 tests. It queries the 'deqp-gles3' executable at runtime for the list of testcases. The module attempts queries the 'deqp-gles3' executable only if the environment variable PIGLIT_DEQP_GLES3_EXE or the piglit.conf option deqp-gles3.exe is set. Why do we need to use Pigit as the testrunner for dEQP? (After all, dEQP has its own testrunner). Because dEQP runs all tests in a single process. If test 17530 of 55409 crashes, then the dEQP testrunner crashes and the remaining tests never run. Piglit doesn't suffer from that problem, because it runs each test as a separate process. Tested on Intel Ivybridge by running the command piglit run -t dEQP-GLES3/info/vendor \ tests/deqp_gles3.py tests/quick.py results with and without PIGLIT_DEQP_GLES3_EXE set and with and without piglit.conf:deqp-gles.exe set. Cc: Dylan Baker dylanx.c.ba...@intel.com Cc: Michael W Mason michael.w.ma...@intel.com Signed-off-by: Chad Versace chad.vers...@linux.intel.com --- v2: - Remove unused imports (dcbaker) - Remove extra newlines (dcbaker) - Don't use 'global' keyword (dcbaker) - Use global constant for DEQP_GLES3_EXE (dcbaker) - Fix DEQPTest.interpret_result to correctly parse test output. (mwmason) - Use 'super' keyword (dcbaker) v3: - Fix misnamed variable s/caselist_txt/caselist_path/. (mwmason) piglit.conf.example | 7 +++ tests/deqp_gles3.py | 136 2 files changed, 143 insertions(+) create mode 100644 tests/deqp_gles3.py diff --git a/piglit.conf.example b/piglit.conf.example index a864c01..0ac0f4f 100644 --- a/piglit.conf.example +++ b/piglit.conf.example @@ -33,6 +33,13 @@ bindir=/home/usr/oclconform testA testB +;[deqp-gles3] +; +; Path to the deqp-gles3 executable. You can also set this with the +environment ; variable PIGLIT_DEQP_GLES3_EXE. Piglit will run the +dEQP-GLES3 tests if and ; only if this option is set. +;exe=/home/knuth/deqp/modules/gles3/deqp-gles3 + ; Section for specific oclconform test. One of these sections is required for ; each test list in the oclconform section and must be called: ; oclconform-$testname diff --git a/tests/deqp_gles3.py b/tests/deqp_gles3.py new file mode 100644 index 000..24035c6 --- /dev/null +++ b/tests/deqp_gles3.py @@ -0,0 +1,136 @@ +# Copyright 2014 Intel Corporation +# +# Permission is hereby granted, free of charge, to any person +obtaining a copy # of this software and associated documentation +files (the Software), to deal # in the Software without +restriction, including without limitation the rights # to use, copy, +modify, merge, publish
Re: [Piglit] [PATCH] deqp: Add test profile for external dEQP-GLES3 tests (v3)
A couple questions... First, deqp-gles3 has many command line options. Is there any way in piglit to pass them to deqp-gles3 from the piglit command line? I don't see a way. One option in particular is important when testing on Freon. I have to pass --deqp-gl-config-name=rgbad24s8 to get the images to display full screen. No doubt we'll want to use other options in the future. Second, how do you kill a piglit test run once it's started? I tried multiple ctrl-C's without success. I couldn't get it to stop. Is this a piglit limitation or something to do with the piglit/deqp combo? BTW, the latest patch does work, but I'd like to understand if the above limitations are expected or not before giving it my blessing. Mike -Original Message- From: Chad Versace [mailto:chad.vers...@linux.intel.com] Sent: Wednesday, December 10, 2014 9:39 AM To: piglit@lists.freedesktop.org Cc: Chad Versace; Baker, DylanX C; Mason, Michael W Subject: [PATCH] deqp: Add test profile for external dEQP-GLES3 tests (v3) Google opensourced the drawElements Quality Product Testsuite (dEQP) as part of the Android Lollipop release. It's git repo lives in the Android tree at [https://android.googlesource.com/platform/external/deqp/]. This patch adds a new module, deqp_gles3.py, that runs the dEQP-GLES3 tests. It queries the 'deqp-gles3' executable at runtime for the list of testcases. The module attempts queries the 'deqp-gles3' executable only if the environment variable PIGLIT_DEQP_GLES3_EXE or the piglit.conf option deqp-gles3.exe is set. Why do we need to use Pigit as the testrunner for dEQP? (After all, dEQP has its own testrunner). Because dEQP runs all tests in a single process. If test 17530 of 55409 crashes, then the dEQP testrunner crashes and the remaining tests never run. Piglit doesn't suffer from that problem, because it runs each test as a separate process. Tested on Intel Ivybridge by running the command piglit run -t dEQP-GLES3/info/vendor \ tests/deqp_gles3.py tests/quick.py results with and without PIGLIT_DEQP_GLES3_EXE set and with and without piglit.conf:deqp-gles.exe set. Cc: Dylan Baker dylanx.c.ba...@intel.com Cc: Michael W Mason michael.w.ma...@intel.com Signed-off-by: Chad Versace chad.vers...@linux.intel.com --- v2: - Remove unused imports (dcbaker) - Remove extra newlines (dcbaker) - Don't use 'global' keyword (dcbaker) - Use global constant for DEQP_GLES3_EXE (dcbaker) - Fix DEQPTest.interpret_result to correctly parse test output. (mwmason) - Use 'super' keyword (dcbaker) v3: - Fix misnamed variable s/caselist_txt/caselist_path/. (mwmason) piglit.conf.example | 7 +++ tests/deqp_gles3.py | 136 2 files changed, 143 insertions(+) create mode 100644 tests/deqp_gles3.py diff --git a/piglit.conf.example b/piglit.conf.example index a864c01..0ac0f4f 100644 --- a/piglit.conf.example +++ b/piglit.conf.example @@ -33,6 +33,13 @@ bindir=/home/usr/oclconform testA testB +;[deqp-gles3] +; +; Path to the deqp-gles3 executable. You can also set this with the +environment ; variable PIGLIT_DEQP_GLES3_EXE. Piglit will run the +dEQP-GLES3 tests if and ; only if this option is set. +;exe=/home/knuth/deqp/modules/gles3/deqp-gles3 + ; Section for specific oclconform test. One of these sections is required for ; each test list in the oclconform section and must be called: ; oclconform-$testname diff --git a/tests/deqp_gles3.py b/tests/deqp_gles3.py new file mode 100644 index 000..24035c6 --- /dev/null +++ b/tests/deqp_gles3.py @@ -0,0 +1,136 @@ +# Copyright 2014 Intel Corporation +# +# Permission is hereby granted, free of charge, to any person obtaining +a copy # of this software and associated documentation files (the +Software), to deal # in the Software without restriction, including +without limitation the rights # to use, copy, modify, merge, publish, +distribute, sublicense, and/or sell # copies of the Software, and to +permit persons to whom the Software is # furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be +included in # all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND +NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS +BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN +ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE # SOFTWARE. + +import ConfigParser +import os +import subprocess + +# Piglit modules +import framework +from framework.profile import Test, TestProfile + +__all__ = ['profile'] + +def get_option(env_varname, config_option): +Query
