[openstack-dev] False Positive testing for 3rd party CI

2014-02-21 Thread Aaron Rosen 
Hi,

Yesterday, I pushed a patch to review and was surprised that several of the
third party CI systems reported back that the patch-set worked where it
definitely shouldn't have. Anyways, I tested out my theory a little more
and it turns out a few of the 3rd party CI systems for neutron are just
returning  SUCCESS even if the patch set didn't run successfully (
https://review.openstack.org/#/c/75304/).

Here's a short summery of what I found.

Hyper-V CI -- This seems like an easy fix as it's posting "build succeeded"
but also puts to the side "test run failed". Would probably be a good idea
to remove the "build succeeded" message to avoid any confusion.


Brocade CI - From the log files it posts it shows that it tries to apply my
patch but fails:

2014-02-20 20:23:48 + cd /opt/stack/neutron
2014-02-20 20:23:48 + git fetch
https://review.openstack.org/openstack/neutron.git
refs/changes/04/75304/1
2014-02-20 20:24:00 From https://review.openstack.org/openstack/neutron
2014-02-20 20:24:00  * branchrefs/changes/04/75304/1 -> FETCH_HEAD
2014-02-20 20:24:00 + git checkout FETCH_HEAD
2014-02-20 20:24:00 error: Your local changes to the following files
would be overwritten by checkout:
2014-02-20 20:24:00 etc/neutron/plugins/ml2/ml2_conf_brocade.ini
2014-02-20 20:24:00 neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py
2014-02-20 20:24:00 Please, commit your changes or stash them before
you can switch branches.
2014-02-20 20:24:00 Aborting
2014-02-20 20:24:00 + cd /opt/stack/neutron

but still continues running (without my patchset) and reports success. --
This actually looks like a devstack bug  (i'll check it out).

PLUMgrid CI - Seems to always vote +1 without a failure (
https://review.openstack.org/#/dashboard/10117) though the logs are private
so we can't really tell whats going on.

I was thinking it might be worth while or helpful to have a job that tests
that CI is actually fails when we expect it to.

Best,

Aaron
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] False Positive testing for 3rd party CI

2014-02-21 Thread Octavian Ciuhandu 
Hi,

On 21 Feb 2014, at 20:34, Aaron Rosen 
mailto:aaronoro...@gmail.com>> wrote:

Hi,

Yesterday, I pushed a patch to review and was surprised that several of the 
third party CI systems reported back that the patch-set worked where it 
definitely shouldn't have. Anyways, I tested out my theory a little more and it 
turns out a few of the 3rd party CI systems for neutron are just returning  
SUCCESS even if the patch set didn't run successfully 
(https://review.openstack.org/#/c/75304/).

Here's a short summery of what I found.

Hyper-V CI -- This seems like an easy fix as it's posting "build succeeded" but 
also puts to the side "test run failed". Would probably be a good idea to 
remove the "build succeeded" message to avoid any confusion.

The Hyper-V CI is non-voting (as required for new third-party CIs) and this is 
the reason why any post from it will show “build succeeded”. As published in 
other threads, AFAIK the only way to get rid of this issue is to have the CI as 
voting.

Brocade CI - From the log files it posts it shows that it tries to apply my 
patch but fails:


2014-02-20 20:23:48 + cd /opt/stack/neutron
2014-02-20 20:23:48 + git fetch 
https://review.openstack.org/openstack/neutron.git refs/changes/04/75304/1
2014-02-20 20:24:00 From https://review.openstack.org/openstack/neutron
2014-02-20 20:24:00  
* branchrefs/changes/04/75304/1 -> FETCH_HEAD
2014-02-20 20:24:00 + git checkout FETCH_HEAD
2014-02-20 20:24:00 error: Your local changes to the following files would be 
overwritten by checkout:
2014-02-20 20:24:00 etc/neutron/plugins/ml2/ml2_conf_brocade.ini
2014-02-20 20:24:00 neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py
2014-02-20 20:24:00 Please, commit your changes or stash them before you can 
switch branches.
2014-02-20 20:24:00 Aborting
2014-02-20 20:24:00 + cd /opt/stack/neutron

but still continues running (without my patchset) and reports success. -- This 
actually looks like a devstack bug  (i'll check it out).

PLUMgrid CI - Seems to always vote +1 without a failure 
(https://review.openstack.org/#/dashboard/10117) though the logs are private so 
we can't really tell whats going on.

I was thinking it might be worth while or helpful to have a job that tests that 
CI is actually fails when we expect it to.

Best,

Aaron

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Thanks,

Octavian.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] False Positive testing for 3rd party CI

2014-02-21 Thread Aaron Rosen 
This should fix the false positive for brocade:
https://review.openstack.org/#/c/75486/

Aaron


On Fri, Feb 21, 2014 at 10:34 AM, Aaron Rosen  wrote:

> Hi,
>
> Yesterday, I pushed a patch to review and was surprised that several of
> the third party CI systems reported back that the patch-set worked where it
> definitely shouldn't have. Anyways, I tested out my theory a little more
> and it turns out a few of the 3rd party CI systems for neutron are just
> returning  SUCCESS even if the patch set didn't run successfully (
> https://review.openstack.org/#/c/75304/).
>
> Here's a short summery of what I found.
>
> Hyper-V CI -- This seems like an easy fix as it's posting "build
> succeeded" but also puts to the side "test run failed". Would probably be a
> good idea to remove the "build succeeded" message to avoid any confusion.
>
>
> Brocade CI - From the log files it posts it shows that it tries to apply
> my patch but fails:
>
> 2014-02-20 20:23:48 + cd /opt/stack/neutron
> 2014-02-20 20:23:48 + git fetch 
> https://review.openstack.org/openstack/neutron.git refs/changes/04/75304/1
> 2014-02-20 20:24:00 From https://review.openstack.org/openstack/neutron
> 2014-02-20  
> 20:24:00  * branchrefs/changes/04/75304/1 -> FETCH_HEAD
> 2014-02-20 20:24:00 + git checkout FETCH_HEAD
> 2014-02-20 20:24:00 error: Your local changes to the following files would be 
> overwritten by checkout:
> 2014-02-20 20:24:00   etc/neutron/plugins/ml2/ml2_conf_brocade.ini
> 2014-02-20 20:24:00   neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py
> 2014-02-20 20:24:00 Please, commit your changes or stash them before you can 
> switch branches.
> 2014-02-20 20:24:00 Aborting
> 2014-02-20 20:24:00 + cd /opt/stack/neutron
>
> but still continues running (without my patchset) and reports success. --
> This actually looks like a devstack bug  (i'll check it out).
>
> PLUMgrid CI - Seems to always vote +1 without a failure (
> https://review.openstack.org/#/dashboard/10117) though the logs are
> private so we can't really tell whats going on.
>
> I was thinking it might be worth while or helpful to have a job that tests
> that CI is actually fails when we expect it to.
>
> Best,
>
> Aaron
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] False Positive testing for 3rd party CI

2014-02-21 Thread Armando M. 
Nice one!

On 21 February 2014 11:22, Aaron Rosen  wrote:
> This should fix the false positive for brocade:
> https://review.openstack.org/#/c/75486/
>
> Aaron
>
>
> On Fri, Feb 21, 2014 at 10:34 AM, Aaron Rosen  wrote:
>>
>> Hi,
>>
>> Yesterday, I pushed a patch to review and was surprised that several of
>> the third party CI systems reported back that the patch-set worked where it
>> definitely shouldn't have. Anyways, I tested out my theory a little more and
>> it turns out a few of the 3rd party CI systems for neutron are just
>> returning  SUCCESS even if the patch set didn't run successfully
>> (https://review.openstack.org/#/c/75304/).
>>
>> Here's a short summery of what I found.
>>
>> Hyper-V CI -- This seems like an easy fix as it's posting "build
>> succeeded" but also puts to the side "test run failed". Would probably be a
>> good idea to remove the "build succeeded" message to avoid any confusion.
>>
>>
>> Brocade CI - From the log files it posts it shows that it tries to apply
>> my patch but fails:
>>
>> 2014-02-20 20:23:48 + cd /opt/stack/neutron
>> 2014-02-20 20:23:48 + git fetch
>> https://review.openstack.org/openstack/neutron.git refs/changes/04/75304/1
>> 2014-02-20 20:24:00 From https://review.openstack.org/openstack/neutron
>> 2014-02-20 20:24:00  * branchrefs/changes/04/75304/1 ->
>> FETCH_HEAD
>> 2014-02-20 20:24:00 + git checkout FETCH_HEAD
>> 2014-02-20 20:24:00 error: Your local changes to the following files would
>> be overwritten by checkout:
>> 2014-02-20 20:24:00  etc/neutron/plugins/ml2/ml2_conf_brocade.ini
>> 2014-02-20 20:24:00
>>  neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py
>> 2014-02-20 20:24:00 Please, commit your changes or stash them before you
>> can switch branches.
>> 2014-02-20 20:24:00 Aborting
>> 2014-02-20 20:24:00 + cd /opt/stack/neutron
>>
>> but still continues running (without my patchset) and reports success. --
>> This actually looks like a devstack bug  (i'll check it out).
>>
>> PLUMgrid CI - Seems to always vote +1 without a failure
>> (https://review.openstack.org/#/dashboard/10117) though the logs are private
>> so we can't really tell whats going on.
>>
>> I was thinking it might be worth while or helpful to have a job that tests
>> that CI is actually fails when we expect it to.
>>
>> Best,
>>
>> Aaron
>>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] False Positive testing for 3rd party CI

2014-02-21 Thread Carl Baldwin 
Aaron,

I was thinking the same thing recently with this patch [1].  Patch
sets 1-5 should have failed for any plugin besides ml2 yet some passed
and I wondered how that could happen.  Kudos to those patches that
failed my patch sets correctly.

Carl

[1] https://review.openstack.org/#/c/72565/

On Fri, Feb 21, 2014 at 11:34 AM, Aaron Rosen  wrote:
> Hi,
>
> Yesterday, I pushed a patch to review and was surprised that several of the
> third party CI systems reported back that the patch-set worked where it
> definitely shouldn't have. Anyways, I tested out my theory a little more and
> it turns out a few of the 3rd party CI systems for neutron are just
> returning  SUCCESS even if the patch set didn't run successfully
> (https://review.openstack.org/#/c/75304/).
>
> Here's a short summery of what I found.
>
> Hyper-V CI -- This seems like an easy fix as it's posting "build succeeded"
> but also puts to the side "test run failed". Would probably be a good idea
> to remove the "build succeeded" message to avoid any confusion.
>
>
> Brocade CI - From the log files it posts it shows that it tries to apply my
> patch but fails:
>
> 2014-02-20 20:23:48 + cd /opt/stack/neutron
> 2014-02-20 20:23:48 + git fetch
> https://review.openstack.org/openstack/neutron.git refs/changes/04/75304/1
> 2014-02-20 20:24:00 From https://review.openstack.org/openstack/neutron
> 2014-02-20 20:24:00  * branchrefs/changes/04/75304/1 ->
> FETCH_HEAD
> 2014-02-20 20:24:00 + git checkout FETCH_HEAD
> 2014-02-20 20:24:00 error: Your local changes to the following files would
> be overwritten by checkout:
> 2014-02-20 20:24:00   etc/neutron/plugins/ml2/ml2_conf_brocade.ini
> 2014-02-20 20:24:00
>   neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py
> 2014-02-20 20:24:00 Please, commit your changes or stash them before you can
> switch branches.
> 2014-02-20 20:24:00 Aborting
> 2014-02-20 20:24:00 + cd /opt/stack/neutron
>
> but still continues running (without my patchset) and reports success. --
> This actually looks like a devstack bug  (i'll check it out).
>
> PLUMgrid CI - Seems to always vote +1 without a failure
> (https://review.openstack.org/#/dashboard/10117) though the logs are private
> so we can't really tell whats going on.
>
> I was thinking it might be worth while or helpful to have a job that tests
> that CI is actually fails when we expect it to.
>
> Best,
>
> Aaron
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] False Positive testing for 3rd party CI

2014-02-23 Thread Hemanth Ravi 
One Convergence CI did report the third party tests as failed, but is
configured not to vote. Should we turn on the voting?

Thanks,
-hemanth


On Fri, Feb 21, 2014 at 1:29 PM, Carl Baldwin  wrote:

> Aaron,
>
> I was thinking the same thing recently with this patch [1].  Patch
> sets 1-5 should have failed for any plugin besides ml2 yet some passed
> and I wondered how that could happen.  Kudos to those patches that
> failed my patch sets correctly.
>
> Carl
>
> [1] https://review.openstack.org/#/c/72565/
>
> On Fri, Feb 21, 2014 at 11:34 AM, Aaron Rosen 
> wrote:
> > Hi,
> >
> > Yesterday, I pushed a patch to review and was surprised that several of
> the
> > third party CI systems reported back that the patch-set worked where it
> > definitely shouldn't have. Anyways, I tested out my theory a little more
> and
> > it turns out a few of the 3rd party CI systems for neutron are just
> > returning  SUCCESS even if the patch set didn't run successfully
> > (https://review.openstack.org/#/c/75304/).
> >
> > Here's a short summery of what I found.
> >
> > Hyper-V CI -- This seems like an easy fix as it's posting "build
> succeeded"
> > but also puts to the side "test run failed". Would probably be a good
> idea
> > to remove the "build succeeded" message to avoid any confusion.
> >
> >
> > Brocade CI - From the log files it posts it shows that it tries to apply
> my
> > patch but fails:
> >
> > 2014-02-20 20:23:48 + cd /opt/stack/neutron
> > 2014-02-20 20:23:48 + git fetch
> > https://review.openstack.org/openstack/neutron.gitrefs/changes/04/75304/1
> > 2014-02-20 20:24:00 From https://review.openstack.org/openstack/neutron
> > 2014-02-20 20:24:00  * branchrefs/changes/04/75304/1 ->
> > FETCH_HEAD
> > 2014-02-20 20:24:00 + git checkout FETCH_HEAD
> > 2014-02-20 20:24:00 error: Your local changes to the following files
> would
> > be overwritten by checkout:
> > 2014-02-20 20:24:00   etc/neutron/plugins/ml2/ml2_conf_brocade.ini
> > 2014-02-20 20:24:00
> >   neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py
> > 2014-02-20 20:24:00 Please, commit your changes or stash them before you
> can
> > switch branches.
> > 2014-02-20 20:24:00 Aborting
> > 2014-02-20 20:24:00 + cd /opt/stack/neutron
> >
> > but still continues running (without my patchset) and reports success. --
> > This actually looks like a devstack bug  (i'll check it out).
> >
> > PLUMgrid CI - Seems to always vote +1 without a failure
> > (https://review.openstack.org/#/dashboard/10117) though the logs are
> private
> > so we can't really tell whats going on.
> >
> > I was thinking it might be worth while or helpful to have a job that
> tests
> > that CI is actually fails when we expect it to.
> >
> > Best,
> >
> > Aaron
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] False Positive testing for 3rd party CI

2014-02-23 Thread Akihiro Motoki 
Thanks Aaron, it is really interesting.

 > The Hyper-V CI is non-voting (as required for new third-party CIs) and this 
 > is the reason why any post from it will show “build succeeded”. As published 
 > in other threads, AFAIK the only way to get rid
 > of this issue is to have the CI as voting.

Results of zuul non-voting job is ignored.
In Hyper-V CI, only one testing job is run, so I think it is a good start
to run the testing job as voting and post a verified score with "0"
for both success and failure. If so, we have a correct message
which reflects test results and it does not affects any verified score.

Thanks,
Akihiro

(2014/02/22 4:13), Octavian Ciuhandu wrote:
> Hi,
>
> On 21 Feb 2014, at 20:34, Aaron Rosen  > wrote:
>
>> Hi,
>>
>> Yesterday, I pushed a patch to review and was surprised that several of the 
>> third party CI systems reported back that the patch-set worked where it 
>> definitely shouldn't have. Anyways, I tested out
>> my theory a little more and it turns out a few of the 3rd party CI systems 
>> for neutron are just returning  SUCCESS even if the patch set didn't run 
>> successfully
>> (https://review.openstack.org/#/c/75304/).
>>
>> Here's a short summery of what I found.
>>
>> Hyper-V CI -- This seems like an easy fix as it's posting "build succeeded" 
>> but also puts to the side "test run failed". Would probably be a good idea 
>> to remove the "build succeeded" message to
>> avoid any confusion.
>>
> The Hyper-V CI is non-voting (as required for new third-party CIs) and this 
> is the reason why any post from it will show “build succeeded”. As published 
> in other threads, AFAIK the only way to get rid
> of this issue is to have the CI as voting.
>>
>> Brocade CI - From the log files it posts it shows that it tries to apply my 
>> patch but fails:
>>
>> 2014-02-20 20:23:48 + cd /opt/stack/neutron
>> 2014-02-20 20:23:48 + git 
>> fetchhttps://review.openstack.org/openstack/neutron.git  
>> refs/changes/04/75304/1
>> 2014-02-20 20:24:00 Fromhttps://review.openstack.org/openstack/neutron
>> 2014-02-20    
>> 20:24:00  * branchrefs/changes/04/75304/1 -> FETCH_HEAD
>> 2014-02-20 20:24:00 + git checkout FETCH_HEAD
>> 2014-02-20 20:24:00 error: Your local changes to the following files would 
>> be overwritten by checkout:
>> 2014-02-20 20:24:00  etc/neutron/plugins/ml2/ml2_conf_brocade.ini
>> 2014-02-20 20:24:00  neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py
>> 2014-02-20 20:24:00 Please, commit your changes or stash them before you can 
>> switch branches.
>> 2014-02-20 20:24:00 Aborting
>> 2014-02-20 20:24:00 + cd /opt/stack/neutron
>> but still continues running (without my patchset) and reports success. -- 
>> This actually looks like a devstack bug  (i'll check it out).
>>
>> PLUMgrid CI - Seems to always vote +1 without a failure 
>> (https://review.openstack.org/#/dashboard/10117) though the logs are private 
>> so we can't really tell whats going on.
>>
>> I was thinking it might be worth while or helpful to have a job that tests 
>> that CI is actually fails when we expect it to.
>>
>> Best,
>>
>> Aaron
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> Thanks,
>
> Octavian.
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev