Re: [Piglit] Newbie question: command to run tests that passed before?
On Wed, Sep 7, 2016 at 6:59 PM, Dan Kegel wrote: > Hi! > I'd like to use piglit as a regression test for my opengl test rigs. > My plan is to run piglit a few times on each OS/card combo I need to support, > get a list of tests that reliably pass on each platform, and then > forevermore run just those tests. > > I know about the -t and -x options, and the --test-list option, and > about writing test profiles, > but am still not sure what the best way to generate a list of passing > tests to pass to e.g. --test-list. > Test names are a little hard for this newbie to identify in the output > of piglit summary console. > > After looking at the source a bit, I tried generating a list of passed > tests like this: > ./piglit summary console results/quick | grep ': pass$' | sed > 's/:.*//' | tr / @ > tests.list > but that includes output > spec@!opengl 1.1@max-texture-size-level > spec@!opengl 1.1@max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16 > > which leads to errors from --test-list tests.list like > Fatal Error: Cannot reorder test: "spec@!opengl > 1.1@max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16", it is not in the > profile. > > Suggestions? not sure if it helps at all, but I'm using this locally: https://paste.fedoraproject.org/423693/32953711/raw/ prints cmdline of tests split up by result, (ie. with single src to ./piglit-summary.py it will just print cmdlines split up by category of pass/fail/crash/etc, with two srcs it will print pass->fail, crash->pass, etc).. I typically use it when I'm making some change in driver to just cut/paste cmdlines for pass->fail or pass->crash into a script to re-run a subset of tests while I'm debugging whatever it is that I'm working on at the time, to avoid full piglit runs or time-consuming running of regressed tests one by one as I debug. It was suggested to integrate this w/ piglit-print-commands.py instead, which seems pretty sensible but I haven't gotten around to yet. BR, -R > p.s. Mesa Haswell is behaving very well on stock ubuntu 16.04, only 1% > of quick tests fail, and there are no hangs or kernel complaints. > ___ > Piglit mailing list > Piglit@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/piglit ___ Piglit mailing list Piglit@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/piglit
Re: [Piglit] Newbie question: command to run tests that passed before?
Quoting Dylan Baker (2016-09-07 16:23:43)
> Quoting Dan Kegel (2016-09-07 15:59:21)
> > Hi!
> > I'd like to use piglit as a regression test for my opengl test rigs.
> > My plan is to run piglit a few times on each OS/card combo I need to
> > support,
> > get a list of tests that reliably pass on each platform, and then
> > forevermore run just those tests.
> >
> > I know about the -t and -x options, and the --test-list option, and
> > about writing test profiles,
> > but am still not sure what the best way to generate a list of passing
> > tests to pass to e.g. --test-list.
> > Test names are a little hard for this newbie to identify in the output
> > of piglit summary console.
> >
> > After looking at the source a bit, I tried generating a list of passed
> > tests like this:
> > ./piglit summary console results/quick | grep ': pass$' | sed
> > 's/:.*//' | tr / @ > tests.list
> > but that includes output
> > spec@!opengl 1.1@max-texture-size-level
> > spec@!opengl 1.1@max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16
> >
> > which leads to errors from --test-list tests.list like
> > Fatal Error: Cannot reorder test: "spec@!opengl
> > 1.1@max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16", it is not in the
> > profile.
> >
> > Suggestions?
>
> That's an interesting corner you've found. What's happening is that
> GL_PROXY_TEXTURE_1D-GL_RGBA16 is a subtest of
> spec@!opengl 1.1@max-texture-size, and thus isn't in the test list at
> all.
>
> This is actually an odd corner of a known deficiency, the inability to
> filter on subtests. I've been thinking a lot about how to fix this, but
> there isn't a straight forward way to fix it (that I've been able to
> come up with). I guess I should bump this up on my list of things to do.
>
> The easiest thing I can com up with off the top my head would be to
> write a little script to parse the output and output what you want.
>
> Something like this should work (untested):
>
> """
> import argparse
>
> from framework import backends
>
> def main():
> parser = argparse.ArgumentParser()
> parser.add_argument('file')
> args = parser.parse_args()
>
> results = backends.load(args._file)
Oops, that should be 'args.file'. Just my auto-completion being
"helpful".
> with open('myfile', 'w') as f:
> for name, value in results.tests.items():
> if value.result == 'pass':
> f.write(name)
> f.write('\n')
>
>
> if __name__ == '__main__':
> main()
> """
>
> Obviously you'll need to be in the piglit source directory or add it to
> your PYTHON_PATH to make this work, since it relies on using the piglit
> framework.
>
> Dylan
>
>
> ___
> Piglit mailing list
> Piglit@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/piglit
signature.asc
Description: signature
___
Piglit mailing list
Piglit@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/piglit
Re: [Piglit] Newbie question: command to run tests that passed before?
Quoting Mark Janes (2016-09-07 16:24:11) > Piglit converts "known failing" tests to a "skip" status. This is only true of the JUnit backend. I've never bothered to get that working with the default JSON backend. Its on my list of things to do, just not very high up on that list. Dylan signature.asc Description: signature ___ Piglit mailing list Piglit@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/piglit
Re: [Piglit] Newbie question: command to run tests that passed before?
Quoting Dan Kegel (2016-09-07 16:10:46) > Hrm. A solution that involved Jenkins would be way overkill. > I'm just looking for the little core idiom for running just the > test of tests that passed in the, um, past. > > Maybe that's not the way people do it? Does everyone just compare > logs against old logs? The Intel CI system is set up to do regression testing, tests that failed before the CI system was created or are new and fail immediately are marked as expected failures and ignored, only tests that previously passed but now fail are considered bugs. We also blacklist tests (per platform) that pass or fail sporadically. Dylan signature.asc Description: signature ___ Piglit mailing list Piglit@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/piglit
Re: [Piglit] Newbie question: command to run tests that passed before?
Dan Kegel writes: > Hrm. A solution that involved Jenkins would be way overkill. > I'm just looking for the little core idiom for running just the > test of tests that passed in the, um, past. Even if you don't want to set up a CI with jenkins, you can refer to the automation that I've written at https://github.com/janesma/mesa_jenkins Our approach is to list all failing/crashing tests in a config file, and pass that to piglit. Piglit converts "known failing" tests to a "skip" status. The only tests we can't run are the ones that have flaky results or cause gpu hang. For example: https://github.com/janesma/mesa_jenkins/blob/master/piglit-test/bdw.conf > Maybe that's not the way people do it? Does everyone just compare > logs against old logs? > > Thanks, and say hi to Ethyl for me. > - Dan > > On Wed, Sep 7, 2016 at 4:08 PM, Ilia Mirkin wrote: >> [+janesma] >> >> Mark has set up a CI system at Intel, and I'm fairly sure did a >> writeup about it and shared code, but I can't for the life of me >> remember the details. Maybe a few Guinesses ago, but definitely not >> now... hopefully he can share. [It involves Jenkins.] >> >> -ilia >> >> On Wed, Sep 7, 2016 at 6:59 PM, Dan Kegel wrote: >>> Hi! >>> I'd like to use piglit as a regression test for my opengl test rigs. >>> My plan is to run piglit a few times on each OS/card combo I need to >>> support, >>> get a list of tests that reliably pass on each platform, and then >>> forevermore run just those tests. >>> >>> I know about the -t and -x options, and the --test-list option, and >>> about writing test profiles, >>> but am still not sure what the best way to generate a list of passing >>> tests to pass to e.g. --test-list. >>> Test names are a little hard for this newbie to identify in the output >>> of piglit summary console. >>> >>> After looking at the source a bit, I tried generating a list of passed >>> tests like this: >>> ./piglit summary console results/quick | grep ': pass$' | sed >>> 's/:.*//' | tr / @ > tests.list >>> but that includes output >>> spec@!opengl 1.1@max-texture-size-level >>> spec@!opengl 1.1@max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16 >>> >>> which leads to errors from --test-list tests.list like >>> Fatal Error: Cannot reorder test: "spec@!opengl >>> 1.1@max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16", it is not in the >>> profile. >>> >>> Suggestions? >>> >>> p.s. Mesa Haswell is behaving very well on stock ubuntu 16.04, only 1% >>> of quick tests fail, and there are no hangs or kernel complaints. >>> ___ >>> Piglit mailing list >>> Piglit@lists.freedesktop.org >>> https://lists.freedesktop.org/mailman/listinfo/piglit ___ Piglit mailing list Piglit@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/piglit
Re: [Piglit] Newbie question: command to run tests that passed before?
Quoting Dan Kegel (2016-09-07 15:59:21)
> Hi!
> I'd like to use piglit as a regression test for my opengl test rigs.
> My plan is to run piglit a few times on each OS/card combo I need to support,
> get a list of tests that reliably pass on each platform, and then
> forevermore run just those tests.
>
> I know about the -t and -x options, and the --test-list option, and
> about writing test profiles,
> but am still not sure what the best way to generate a list of passing
> tests to pass to e.g. --test-list.
> Test names are a little hard for this newbie to identify in the output
> of piglit summary console.
>
> After looking at the source a bit, I tried generating a list of passed
> tests like this:
> ./piglit summary console results/quick | grep ': pass$' | sed
> 's/:.*//' | tr / @ > tests.list
> but that includes output
> spec@!opengl 1.1@max-texture-size-level
> spec@!opengl 1.1@max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16
>
> which leads to errors from --test-list tests.list like
> Fatal Error: Cannot reorder test: "spec@!opengl
> 1.1@max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16", it is not in the
> profile.
>
> Suggestions?
That's an interesting corner you've found. What's happening is that
GL_PROXY_TEXTURE_1D-GL_RGBA16 is a subtest of
spec@!opengl 1.1@max-texture-size, and thus isn't in the test list at
all.
This is actually an odd corner of a known deficiency, the inability to
filter on subtests. I've been thinking a lot about how to fix this, but
there isn't a straight forward way to fix it (that I've been able to
come up with). I guess I should bump this up on my list of things to do.
The easiest thing I can com up with off the top my head would be to
write a little script to parse the output and output what you want.
Something like this should work (untested):
"""
import argparse
from framework import backends
def main():
parser = argparse.ArgumentParser()
parser.add_argument('file')
args = parser.parse_args()
results = backends.load(args._file)
with open('myfile', 'w') as f:
for name, value in results.tests.items():
if value.result == 'pass':
f.write(name)
f.write('\n')
if __name__ == '__main__':
main()
"""
Obviously you'll need to be in the piglit source directory or add it to
your PYTHON_PATH to make this work, since it relies on using the piglit
framework.
Dylan
signature.asc
Description: signature
___
Piglit mailing list
Piglit@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/piglit
Re: [Piglit] Newbie question: command to run tests that passed before?
Hrm. A solution that involved Jenkins would be way overkill. I'm just looking for the little core idiom for running just the test of tests that passed in the, um, past. Maybe that's not the way people do it? Does everyone just compare logs against old logs? Thanks, and say hi to Ethyl for me. - Dan On Wed, Sep 7, 2016 at 4:08 PM, Ilia Mirkin wrote: > [+janesma] > > Mark has set up a CI system at Intel, and I'm fairly sure did a > writeup about it and shared code, but I can't for the life of me > remember the details. Maybe a few Guinesses ago, but definitely not > now... hopefully he can share. [It involves Jenkins.] > > -ilia > > On Wed, Sep 7, 2016 at 6:59 PM, Dan Kegel wrote: >> Hi! >> I'd like to use piglit as a regression test for my opengl test rigs. >> My plan is to run piglit a few times on each OS/card combo I need to support, >> get a list of tests that reliably pass on each platform, and then >> forevermore run just those tests. >> >> I know about the -t and -x options, and the --test-list option, and >> about writing test profiles, >> but am still not sure what the best way to generate a list of passing >> tests to pass to e.g. --test-list. >> Test names are a little hard for this newbie to identify in the output >> of piglit summary console. >> >> After looking at the source a bit, I tried generating a list of passed >> tests like this: >> ./piglit summary console results/quick | grep ': pass$' | sed >> 's/:.*//' | tr / @ > tests.list >> but that includes output >> spec@!opengl 1.1@max-texture-size-level >> spec@!opengl 1.1@max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16 >> >> which leads to errors from --test-list tests.list like >> Fatal Error: Cannot reorder test: "spec@!opengl >> 1.1@max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16", it is not in the >> profile. >> >> Suggestions? >> >> p.s. Mesa Haswell is behaving very well on stock ubuntu 16.04, only 1% >> of quick tests fail, and there are no hangs or kernel complaints. >> ___ >> Piglit mailing list >> Piglit@lists.freedesktop.org >> https://lists.freedesktop.org/mailman/listinfo/piglit ___ Piglit mailing list Piglit@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/piglit
Re: [Piglit] Newbie question: command to run tests that passed before?
[+janesma] Mark has set up a CI system at Intel, and I'm fairly sure did a writeup about it and shared code, but I can't for the life of me remember the details. Maybe a few Guinesses ago, but definitely not now... hopefully he can share. [It involves Jenkins.] -ilia On Wed, Sep 7, 2016 at 6:59 PM, Dan Kegel wrote: > Hi! > I'd like to use piglit as a regression test for my opengl test rigs. > My plan is to run piglit a few times on each OS/card combo I need to support, > get a list of tests that reliably pass on each platform, and then > forevermore run just those tests. > > I know about the -t and -x options, and the --test-list option, and > about writing test profiles, > but am still not sure what the best way to generate a list of passing > tests to pass to e.g. --test-list. > Test names are a little hard for this newbie to identify in the output > of piglit summary console. > > After looking at the source a bit, I tried generating a list of passed > tests like this: > ./piglit summary console results/quick | grep ': pass$' | sed > 's/:.*//' | tr / @ > tests.list > but that includes output > spec@!opengl 1.1@max-texture-size-level > spec@!opengl 1.1@max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16 > > which leads to errors from --test-list tests.list like > Fatal Error: Cannot reorder test: "spec@!opengl > 1.1@max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16", it is not in the > profile. > > Suggestions? > > p.s. Mesa Haswell is behaving very well on stock ubuntu 16.04, only 1% > of quick tests fail, and there are no hangs or kernel complaints. > ___ > Piglit mailing list > Piglit@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/piglit ___ Piglit mailing list Piglit@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/piglit
[Piglit] Newbie question: command to run tests that passed before?
Hi! I'd like to use piglit as a regression test for my opengl test rigs. My plan is to run piglit a few times on each OS/card combo I need to support, get a list of tests that reliably pass on each platform, and then forevermore run just those tests. I know about the -t and -x options, and the --test-list option, and about writing test profiles, but am still not sure what the best way to generate a list of passing tests to pass to e.g. --test-list. Test names are a little hard for this newbie to identify in the output of piglit summary console. After looking at the source a bit, I tried generating a list of passed tests like this: ./piglit summary console results/quick | grep ': pass$' | sed 's/:.*//' | tr / @ > tests.list but that includes output spec@!opengl 1.1@max-texture-size-level spec@!opengl 1.1@max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16 which leads to errors from --test-list tests.list like Fatal Error: Cannot reorder test: "spec@!opengl 1.1@max-texture-size@GL_PROXY_TEXTURE_1D-GL_RGBA16", it is not in the profile. Suggestions? p.s. Mesa Haswell is behaving very well on stock ubuntu 16.04, only 1% of quick tests fail, and there are no hangs or kernel complaints. ___ Piglit mailing list Piglit@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/piglit
