Re: [Piglit] [PATCH] junit.py: Avoid special test names.

2015-02-20 Thread Jose Fonseca 

On 20/02/15 22:18, Mark Janes wrote:

Jose Fonseca  writes:


On 20/02/15 20:50, Mark Janes wrote:

We support using the test names as displayed in jenkins to start bisect
jobs.  If a test fails, you can copy/paste the name into a job, and it
will find the commit that broke it.

With this change, special-cased test names can't be used with
--include-tests anymore.



Couldn't the bisect job easily trim trailing underscores to the test
name that is copy'n'pasted?


Yes, I could do that.  As it turns out, I won't need to.  Because we
use suffixes on all our tests, we will never have a test named "api"

The underscore will interfere with matching test names for
expected-failures in a config file.  Again, it won't matter for us, and
AFAIK we are the only ones using that feature.


  > Instead of adding an underscore, can we change one of the characters to
  > ".", so that the expression matching will still work?

'.' is special for Jenkins -- it's equivalent to / in a path.  So I
suspect springing dots in the name will cause problems.


We currently append ".{hardware_suffix}" to each test name, so we can
run the same tests dozens of times on different platforms and display
all the results together.  The '.' is handled without error.

Since it won't matter to us, and it is such a tiny corner case for
everyone else, I'm fine with it.


Ah, yes, if you use a suffix then testnames with api/search can never 
happen.



Reviewed-by: Mark Janes 


Thanks.

Jose




Another solution is to would be to dump the pristine name (or even
better, a properly quoted regex suitable to be passed in
-include-tests  ) in the junit test result (stdout or stderr). e.g.:

"

To run only this test pass --include-tests '^foo/doo/boo$'"


Jose




-Mark

Dylan Baker  writes:


This looks fine to me, though I haven't tested it.

Mark, does this seem good to you?

Reviewed-by: Dylan Baker 

On Fri, Feb 20, 2015 at 11:53:50AM +, Jose Fonseca wrote:

For example, Jenkins was displaying its REST API page when one navigated
to  .../testReport/piglit.spec/ARB_occlusion_query2/api/ result.
---
   framework/backends/junit.py | 10 +-
   1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/framework/backends/junit.py b/framework/backends/junit.py
index 839c99a..ddaf826 100644
--- a/framework/backends/junit.py
+++ b/framework/backends/junit.py
@@ -166,8 +166,16 @@ class JUnitBackend(FileBackend):
   # set different root names.
   classname = 'piglit.' + classname

+testname += self._test_suffix
+
+# Jenkins will display special pages when the test has certain names.
+# 
https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins-2Dci.org_issue_18062&d=AwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=zfmBZnnVGHeYde45pMKNnVyzeaZbdIqVLprmZCM2zzE&m=6iqiAc1uzBPZFCypdivM6WaoDheu37Q45KJTMIFulgo&s=3OKSPXH947_Gwt5zZ4v0DIJRLWf5-u6P4a2yc3cY1AU&e=
+# 
https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins-2Dci.org_issue_19810&d=AwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=zfmBZnnVGHeYde45pMKNnVyzeaZbdIqVLprmZCM2zzE&m=6iqiAc1uzBPZFCypdivM6WaoDheu37Q45KJTMIFulgo&s=Kr5YcnwFlFnWhiOHklE4CT5QTR7A5PiMJFoNuK4dhaM&e=
+if testname in ('api', 'search'):
+testname += '_'
+
   # Create the root element
-element = etree.Element('testcase', name=testname + self._test_suffix,
+element = etree.Element('testcase', name=testname,
   classname=classname,
   time=str(data['time']),
   status=str(data['result']))
--
2.1.0

___
Piglit mailing list
Piglit@lists.freedesktop.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.freedesktop.org_mailman_listinfo_piglit&d=AwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=zfmBZnnVGHeYde45pMKNnVyzeaZbdIqVLprmZCM2zzE&m=6iqiAc1uzBPZFCypdivM6WaoDheu37Q45KJTMIFulgo&s=KdZJAJSf6EP4bKM8xp2zYCOpqQ8WY14LPCxx90LhrdY&e=


___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH] junit.py: Avoid special test names.

2015-02-20 Thread Mark Janes 
Jose Fonseca  writes:

> On 20/02/15 20:50, Mark Janes wrote:
>> We support using the test names as displayed in jenkins to start bisect
>> jobs.  If a test fails, you can copy/paste the name into a job, and it
>> will find the commit that broke it.
>>
>> With this change, special-cased test names can't be used with
>> --include-tests anymore.
>>
>
> Couldn't the bisect job easily trim trailing underscores to the test 
> name that is copy'n'pasted?

Yes, I could do that.  As it turns out, I won't need to.  Because we
use suffixes on all our tests, we will never have a test named "api"

The underscore will interfere with matching test names for
expected-failures in a config file.  Again, it won't matter for us, and
AFAIK we are the only ones using that feature.

>  > Instead of adding an underscore, can we change one of the characters to
>  > ".", so that the expression matching will still work?
>
> '.' is special for Jenkins -- it's equivalent to / in a path.  So I 
> suspect springing dots in the name will cause problems.

We currently append ".{hardware_suffix}" to each test name, so we can
run the same tests dozens of times on different platforms and display
all the results together.  The '.' is handled without error.

Since it won't matter to us, and it is such a tiny corner case for
everyone else, I'm fine with it.  

Reviewed-by: Mark Janes 

> Another solution is to would be to dump the pristine name (or even 
> better, a properly quoted regex suitable to be passed in 
> -include-tests  ) in the junit test result (stdout or stderr). e.g.:
>
>"
>
>To run only this test pass --include-tests '^foo/doo/boo$'"
>
>
> Jose
>
>
>>
>> -Mark
>>
>> Dylan Baker  writes:
>>
>>> This looks fine to me, though I haven't tested it.
>>>
>>> Mark, does this seem good to you?
>>>
>>> Reviewed-by: Dylan Baker 
>>>
>>> On Fri, Feb 20, 2015 at 11:53:50AM +, Jose Fonseca wrote:
 For example, Jenkins was displaying its REST API page when one navigated
 to  .../testReport/piglit.spec/ARB_occlusion_query2/api/ result.
 ---
   framework/backends/junit.py | 10 +-
   1 file changed, 9 insertions(+), 1 deletion(-)

 diff --git a/framework/backends/junit.py b/framework/backends/junit.py
 index 839c99a..ddaf826 100644
 --- a/framework/backends/junit.py
 +++ b/framework/backends/junit.py
 @@ -166,8 +166,16 @@ class JUnitBackend(FileBackend):
   # set different root names.
   classname = 'piglit.' + classname

 +testname += self._test_suffix
 +
 +# Jenkins will display special pages when the test has certain 
 names.
 +# 
 https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins-2Dci.org_issue_18062&d=AwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=zfmBZnnVGHeYde45pMKNnVyzeaZbdIqVLprmZCM2zzE&m=6iqiAc1uzBPZFCypdivM6WaoDheu37Q45KJTMIFulgo&s=3OKSPXH947_Gwt5zZ4v0DIJRLWf5-u6P4a2yc3cY1AU&e=
 +# 
 https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins-2Dci.org_issue_19810&d=AwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=zfmBZnnVGHeYde45pMKNnVyzeaZbdIqVLprmZCM2zzE&m=6iqiAc1uzBPZFCypdivM6WaoDheu37Q45KJTMIFulgo&s=Kr5YcnwFlFnWhiOHklE4CT5QTR7A5PiMJFoNuK4dhaM&e=
 +if testname in ('api', 'search'):
 +testname += '_'
 +
   # Create the root element
 -element = etree.Element('testcase', name=testname + 
 self._test_suffix,
 +element = etree.Element('testcase', name=testname,
   classname=classname,
   time=str(data['time']),
   status=str(data['result']))
 --
 2.1.0

 ___
 Piglit mailing list
 Piglit@lists.freedesktop.org
 https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.freedesktop.org_mailman_listinfo_piglit&d=AwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=zfmBZnnVGHeYde45pMKNnVyzeaZbdIqVLprmZCM2zzE&m=6iqiAc1uzBPZFCypdivM6WaoDheu37Q45KJTMIFulgo&s=KdZJAJSf6EP4bKM8xp2zYCOpqQ8WY14LPCxx90LhrdY&e=
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH] junit.py: Avoid special test names.

2015-02-20 Thread Jose Fonseca 

On 20/02/15 20:50, Mark Janes wrote:

We support using the test names as displayed in jenkins to start bisect
jobs.  If a test fails, you can copy/paste the name into a job, and it
will find the commit that broke it.

With this change, special-cased test names can't be used with
--include-tests anymore.



Couldn't the bisect job easily trim trailing underscores to the test 
name that is copy'n'pasted?


> Instead of adding an underscore, can we change one of the characters to
> ".", so that the expression matching will still work?

'.' is special for Jenkins -- it's equivalent to / in a path.  So I 
suspect springing dots in the name will cause problems.


Another solution is to would be to dump the pristine name (or even 
better, a properly quoted regex suitable to be passed in 
-include-tests  ) in the junit test result (stdout or stderr). e.g.:


  "

  To run only this test pass --include-tests '^foo/doo/boo$'"


Jose




-Mark

Dylan Baker  writes:


This looks fine to me, though I haven't tested it.

Mark, does this seem good to you?

Reviewed-by: Dylan Baker 

On Fri, Feb 20, 2015 at 11:53:50AM +, Jose Fonseca wrote:

For example, Jenkins was displaying its REST API page when one navigated
to  .../testReport/piglit.spec/ARB_occlusion_query2/api/ result.
---
  framework/backends/junit.py | 10 +-
  1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/framework/backends/junit.py b/framework/backends/junit.py
index 839c99a..ddaf826 100644
--- a/framework/backends/junit.py
+++ b/framework/backends/junit.py
@@ -166,8 +166,16 @@ class JUnitBackend(FileBackend):
  # set different root names.
  classname = 'piglit.' + classname

+testname += self._test_suffix
+
+# Jenkins will display special pages when the test has certain names.
+# 
https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins-2Dci.org_issue_18062&d=AwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=zfmBZnnVGHeYde45pMKNnVyzeaZbdIqVLprmZCM2zzE&m=6iqiAc1uzBPZFCypdivM6WaoDheu37Q45KJTMIFulgo&s=3OKSPXH947_Gwt5zZ4v0DIJRLWf5-u6P4a2yc3cY1AU&e=
+# 
https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins-2Dci.org_issue_19810&d=AwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=zfmBZnnVGHeYde45pMKNnVyzeaZbdIqVLprmZCM2zzE&m=6iqiAc1uzBPZFCypdivM6WaoDheu37Q45KJTMIFulgo&s=Kr5YcnwFlFnWhiOHklE4CT5QTR7A5PiMJFoNuK4dhaM&e=
+if testname in ('api', 'search'):
+testname += '_'
+
  # Create the root element
-element = etree.Element('testcase', name=testname + self._test_suffix,
+element = etree.Element('testcase', name=testname,
  classname=classname,
  time=str(data['time']),
  status=str(data['result']))
--
2.1.0

___
Piglit mailing list
Piglit@lists.freedesktop.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.freedesktop.org_mailman_listinfo_piglit&d=AwIBAg&c=Sqcl0Ez6M0X8aeM67LKIiDJAXVeAw-YihVMNtXt-uEs&r=zfmBZnnVGHeYde45pMKNnVyzeaZbdIqVLprmZCM2zzE&m=6iqiAc1uzBPZFCypdivM6WaoDheu37Q45KJTMIFulgo&s=KdZJAJSf6EP4bKM8xp2zYCOpqQ8WY14LPCxx90LhrdY&e=


___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH] junit.py: Avoid special test names.

2015-02-20 Thread Mark Janes 
We support using the test names as displayed in jenkins to start bisect
jobs.  If a test fails, you can copy/paste the name into a job, and it
will find the commit that broke it.

With this change, special-cased test names can't be used with
--include-tests anymore.

Instead of adding an underscore, can we change one of the characters to
".", so that the expression matching will still work?

-Mark

Dylan Baker  writes:

> This looks fine to me, though I haven't tested it.
>
> Mark, does this seem good to you?
>
> Reviewed-by: Dylan Baker 
>
> On Fri, Feb 20, 2015 at 11:53:50AM +, Jose Fonseca wrote:
>> For example, Jenkins was displaying its REST API page when one navigated
>> to  .../testReport/piglit.spec/ARB_occlusion_query2/api/ result.
>> ---
>>  framework/backends/junit.py | 10 +-
>>  1 file changed, 9 insertions(+), 1 deletion(-)
>> 
>> diff --git a/framework/backends/junit.py b/framework/backends/junit.py
>> index 839c99a..ddaf826 100644
>> --- a/framework/backends/junit.py
>> +++ b/framework/backends/junit.py
>> @@ -166,8 +166,16 @@ class JUnitBackend(FileBackend):
>>  # set different root names.
>>  classname = 'piglit.' + classname
>>  
>> +testname += self._test_suffix
>> +
>> +# Jenkins will display special pages when the test has certain 
>> names.
>> +# https://jenkins-ci.org/issue/18062
>> +# https://jenkins-ci.org/issue/19810
>> +if testname in ('api', 'search'):
>> +testname += '_'
>> +
>>  # Create the root element
>> -element = etree.Element('testcase', name=testname + 
>> self._test_suffix,
>> +element = etree.Element('testcase', name=testname,
>>  classname=classname,
>>  time=str(data['time']),
>>  status=str(data['result']))
>> -- 
>> 2.1.0
>> 
>> ___
>> Piglit mailing list
>> Piglit@lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/piglit
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

Re: [Piglit] [PATCH] junit.py: Avoid special test names.

2015-02-20 Thread Dylan Baker 
This looks fine to me, though I haven't tested it.

Mark, does this seem good to you?

Reviewed-by: Dylan Baker 

On Fri, Feb 20, 2015 at 11:53:50AM +, Jose Fonseca wrote:
> For example, Jenkins was displaying its REST API page when one navigated
> to  .../testReport/piglit.spec/ARB_occlusion_query2/api/ result.
> ---
>  framework/backends/junit.py | 10 +-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/framework/backends/junit.py b/framework/backends/junit.py
> index 839c99a..ddaf826 100644
> --- a/framework/backends/junit.py
> +++ b/framework/backends/junit.py
> @@ -166,8 +166,16 @@ class JUnitBackend(FileBackend):
>  # set different root names.
>  classname = 'piglit.' + classname
>  
> +testname += self._test_suffix
> +
> +# Jenkins will display special pages when the test has certain names.
> +# https://jenkins-ci.org/issue/18062
> +# https://jenkins-ci.org/issue/19810
> +if testname in ('api', 'search'):
> +testname += '_'
> +
>  # Create the root element
> -element = etree.Element('testcase', name=testname + 
> self._test_suffix,
> +element = etree.Element('testcase', name=testname,
>  classname=classname,
>  time=str(data['time']),
>  status=str(data['result']))
> -- 
> 2.1.0
> 
> ___
> Piglit mailing list
> Piglit@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/piglit


signature.asc
Description: Digital signature
___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit

[Piglit] [PATCH] junit.py: Avoid special test names.

2015-02-20 Thread Jose Fonseca 
For example, Jenkins was displaying its REST API page when one navigated
to  .../testReport/piglit.spec/ARB_occlusion_query2/api/ result.
---
 framework/backends/junit.py | 10 +-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/framework/backends/junit.py b/framework/backends/junit.py
index 839c99a..ddaf826 100644
--- a/framework/backends/junit.py
+++ b/framework/backends/junit.py
@@ -166,8 +166,16 @@ class JUnitBackend(FileBackend):
 # set different root names.
 classname = 'piglit.' + classname
 
+testname += self._test_suffix
+
+# Jenkins will display special pages when the test has certain names.
+# https://jenkins-ci.org/issue/18062
+# https://jenkins-ci.org/issue/19810
+if testname in ('api', 'search'):
+testname += '_'
+
 # Create the root element
-element = etree.Element('testcase', name=testname + self._test_suffix,
+element = etree.Element('testcase', name=testname,
 classname=classname,
 time=str(data['time']),
 status=str(data['result']))
-- 
2.1.0

___
Piglit mailing list
Piglit@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/piglit