Re: [Xen-devel] [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree
Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
> On 04/08/16 12:49, Ian Jackson wrote:
> > Andrew, do you want to have osstest run its xtf push gate between
> > `staging' and `master' branches of xtf, or do you want to just push to
> > xtf master, and have a separate `osstest/tested' or some such branch
> > for the osstest push gate output ?
>
> The latter please.
NP.
> We should also clarify the force push criteria. It is moderately likely
> that we get a fix or extension to an existing test which starts showing
> up a new bug in the code under test.
Can this not be made into a new test ?
osstest would like such an extension to an existing test to be a new
`step' (in osstest terminology). That would mean that the fact that
it fails with old versions of Xen can be seen not to be a regression.
Ie, osstest would like to be able to distinguish:
* this version of xtf didn't have the new part of the test
* old part of the test failed (and presumably new part was not run)
* old part succeeded and new part failed
* both parts were run and succeeded
> In this case, OSSTest will identify a regression, but it isn't a
> regression isn't in XTF, nor is it reasonable to call it a regression in
> Xen. We would absolutely want to fix it in upstream, but there are no
> guarentees that the bugfix would be suitable for backport to older trees.
Indeed.
Ian.
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
Re: [Xen-devel] [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree
On 04/08/16 16:21, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf
> tree"):
>> On 04/08/16 12:49, Ian Jackson wrote:
>>> Andrew, do you want to have osstest run its xtf push gate between
>>> `staging' and `master' branches of xtf, or do you want to just push to
>>> xtf master, and have a separate `osstest/tested' or some such branch
>>> for the osstest push gate output ?
>> The latter please.
> NP.
>
>> We should also clarify the force push criteria. It is moderately likely
>> that we get a fix or extension to an existing test which starts showing
>> up a new bug in the code under test.
> Can this not be made into a new test ?
That very much depends, and probably needs deciding on a case by case basis.
What absolutely shouldn't happen is ending up with test-$FOO,
test-$FOO-2, test-more-$FOO, test-$FOO-again because that will result in
the same logical test being split up in ad-hoc ways. While XTF is
useful for automation, it is first and foremost a tool for humans.
~Andrew
>
> osstest would like such an extension to an existing test to be a new
> `step' (in osstest terminology). That would mean that the fact that
> it fails with old versions of Xen can be seen not to be a regression.
>
> Ie, osstest would like to be able to distinguish:
> * this version of xtf didn't have the new part of the test
> * old part of the test failed (and presumably new part was not run)
> * old part succeeded and new part failed
> * both parts were run and succeeded
>
>> In this case, OSSTest will identify a regression, but it isn't a
>> regression isn't in XTF, nor is it reasonable to call it a regression in
>> Xen. We would absolutely want to fix it in upstream, but there are no
>> guarentees that the bugfix would be suitable for backport to older trees.
> Indeed.
>
> Ian.
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
Re: [Xen-devel] [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree
Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
> On 04/08/16 16:21, Ian Jackson wrote:
> > Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf
> > tree"):
> >> We should also clarify the force push criteria. It is moderately likely
> >> that we get a fix or extension to an existing test which starts showing
> >> up a new bug in the code under test.
> > Can this not be made into a new test ?
>
> That very much depends, and probably needs deciding on a case by case basis.
>
> What absolutely shouldn't happen is ending up with test-$FOO,
> test-$FOO-2, test-more-$FOO, test-$FOO-again because that will result in
> the same logical test being split up in ad-hoc ways. While XTF is
> useful for automation, it is first and foremost a tool for humans.
You mean that this would contort the code for the test ?
Here is another approach that could be used to satisfy osstest's
desire for stability in the meaning of test names: you could
explicitly copy the entire source of the test to a new test name, and
deprecate the old test. Later, if the old versions of Xen are fixed,
or after they are retired you would delete the old test.
The result would be that the old, less-good, test would still exist
and still detect regressions. The new, better test, would be seen to
have never passed on old branches.
Force pushing XTF is not an answer to this problem, for two reasons:
Firstly, if the improved test correctly fails on old versions of Xen,
this will appear as a regression in each old version of Xen. Each old
Xen tree would need to be force pushed, every time this happened.
Secondly, in any case, force pushing XTF might not be necessary. In
the usual case, one would expect xen-unstable (which is what would
drive the XTF push gate) to be fixed so that the improved test
passes. So osstest wouldn't spot the issue until it ran the new XTF
with old Xen.
Another possibility would be to have the capability for an single XTF
test execution to return multiple statuses.
Ian.
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
Re: [Xen-devel] [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree
On 04/08/16 17:13, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf
> tree"):
>> On 04/08/16 16:21, Ian Jackson wrote:
>>> Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf
>>> tree"):
We should also clarify the force push criteria. It is moderately likely
that we get a fix or extension to an existing test which starts showing
up a new bug in the code under test.
>>> Can this not be made into a new test ?
>> That very much depends, and probably needs deciding on a case by case basis.
>>
>> What absolutely shouldn't happen is ending up with test-$FOO,
>> test-$FOO-2, test-more-$FOO, test-$FOO-again because that will result in
>> the same logical test being split up in ad-hoc ways. While XTF is
>> useful for automation, it is first and foremost a tool for humans.
> You mean that this would contort the code for the test ?
>
> Here is another approach that could be used to satisfy osstest's
> desire for stability in the meaning of test names: you could
> explicitly copy the entire source of the test to a new test name, and
> deprecate the old test. Later, if the old versions of Xen are fixed,
> or after they are retired you would delete the old test.
>
> The result would be that the old, less-good, test would still exist
> and still detect regressions. The new, better test, would be seen to
> have never passed on old branches.
>
> Force pushing XTF is not an answer to this problem, for two reasons:
>
> Firstly, if the improved test correctly fails on old versions of Xen,
> this will appear as a regression in each old version of Xen. Each old
> Xen tree would need to be force pushed, every time this happened.
>
> Secondly, in any case, force pushing XTF might not be necessary. In
> the usual case, one would expect xen-unstable (which is what would
> drive the XTF push gate) to be fixed so that the improved test
> passes. So osstest wouldn't spot the issue until it ran the new XTF
> with old Xen.
>
> Another possibility would be to have the capability for an single XTF
> test execution to return multiple statuses.
After an IRL discussion, the following has been proposed.
Each test can have a revision number associated with it. The revision
gets bumped every time there is a change to the test which adds new
functionality (including bugfixes which cause it to spot a preexisting
error).
By default, a human will just want the latest and greatest. Test
systems on the other hand will want to test "the same thing as before".
The XTF build system will support a way to build older revisions of the
tests. It is likely that this will involve recompiling the current code
with an older revision id, and leave the onus on the programmer to
ensure that the result is functionally equivalent to the older test.
This way, test automation systems can internal treat different revisions
of the test as logically separate, for the purpose of identifying
regressions.
~Andrew
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
Re: [Xen-devel] [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree
Wei Liu writes ("[OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
> Signed-off-by: Wei Liu
...
> +: ${TREE_XTF:=git://xenbits.xen.org/xtf.git}
> +: ${PUSH_TREE_XTF:=$XENBITS:/home/xen/git/osstest/xtf.git}
> +: ${BASE_TREE_XTF:=git://xenbits.xen.org/osstest/xtf.git}
We should probably have the osstest tested version in the same repo.
Andrew, do you want to have osstest run its xtf push gate between
`staging' and `master' branches of xtf, or do you want to just push to
xtf master, and have a separate `osstest/tested' or some such branch
for the osstest push gate output ?
Thanks,
Ian.
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
Re: [Xen-devel] [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree
On 04/08/16 12:49, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
>> Signed-off-by: Wei Liu
> ...
>> +: ${TREE_XTF:=git://xenbits.xen.org/xtf.git}
>> +: ${PUSH_TREE_XTF:=$XENBITS:/home/xen/git/osstest/xtf.git}
>> +: ${BASE_TREE_XTF:=git://xenbits.xen.org/osstest/xtf.git}
> We should probably have the osstest tested version in the same repo.
>
> Andrew, do you want to have osstest run its xtf push gate between
> `staging' and `master' branches of xtf, or do you want to just push to
> xtf master, and have a separate `osstest/tested' or some such branch
> for the osstest push gate output ?
The latter please.
We should also clarify the force push criteria. It is moderately likely
that we get a fix or extension to an existing test which starts showing
up a new bug in the code under test.
In this case, OSSTest will identify a regression, but it isn't a
regression isn't in XTF, nor is it reasonable to call it a regression in
Xen. We would absolutely want to fix it in upstream, but there are no
guarentees that the bugfix would be suitable for backport to older trees.
~Andrew
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
Re: [Xen-devel] [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree
On Thu, Aug 04, 2016 at 12:49:08PM +0100, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
> > Signed-off-by: Wei Liu
> ...
> > +: ${TREE_XTF:=git://xenbits.xen.org/xtf.git}
> > +: ${PUSH_TREE_XTF:=$XENBITS:/home/xen/git/osstest/xtf.git}
> > +: ${BASE_TREE_XTF:=git://xenbits.xen.org/osstest/xtf.git}
>
> We should probably have the osstest tested version in the same repo.
>
> Andrew, do you want to have osstest run its xtf push gate between
> `staging' and `master' branches of xtf, or do you want to just push to
> xtf master, and have a separate `osstest/tested' or some such branch
> for the osstest push gate output ?
>
It is currently arranged in latter way.
See "Create XTF branch" patch. OSSTest will push to xen-tested-master.
Wei.
> Thanks,
> Ian.
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
