Thanks Aaron, it is really interesting. > The Hyper-V CI is non-voting (as required for new third-party CIs) and this > is the reason why any post from it will show “build succeeded”. As published > in other threads, AFAIK the only way to get rid > of this issue is to have the CI as voting.
Results of zuul non-voting job is ignored. In Hyper-V CI, only one testing job is run, so I think it is a good start to run the testing job as voting and post a verified score with "0" for both success and failure. If so, we have a correct message which reflects test results and it does not affects any verified score. Thanks, Akihiro (2014/02/22 4:13), Octavian Ciuhandu wrote: > Hi, > > On 21 Feb 2014, at 20:34, Aaron Rosen <aaronoro...@gmail.com > <mailto:aaronoro...@gmail.com>> wrote: > >> Hi, >> >> Yesterday, I pushed a patch to review and was surprised that several of the >> third party CI systems reported back that the patch-set worked where it >> definitely shouldn't have. Anyways, I tested out >> my theory a little more and it turns out a few of the 3rd party CI systems >> for neutron are just returning SUCCESS even if the patch set didn't run >> successfully >> (https://review.openstack.org/#/c/75304/). >> >> Here's a short summery of what I found. >> >> Hyper-V CI -- This seems like an easy fix as it's posting "build succeeded" >> but also puts to the side "test run failed". Would probably be a good idea >> to remove the "build succeeded" message to >> avoid any confusion. >> > The Hyper-V CI is non-voting (as required for new third-party CIs) and this > is the reason why any post from it will show “build succeeded”. As published > in other threads, AFAIK the only way to get rid > of this issue is to have the CI as voting. >> >> Brocade CI - From the log files it posts it shows that it tries to apply my >> patch but fails: >> >> 2014-02-20 20:23:48 + cd /opt/stack/neutron >> 2014-02-20 20:23:48 + git >> fetchhttps://review.openstack.org/openstack/neutron.git >> refs/changes/04/75304/1 >> 2014-02-20 20:24:00 Fromhttps://review.openstack.org/openstack/neutron >> 2014-02-20 <https://review.openstack.org/openstack/neutron2014-02-20> >> 20:24:00 * branch refs/changes/04/75304/1 -> FETCH_HEAD >> 2014-02-20 20:24:00 + git checkout FETCH_HEAD >> 2014-02-20 20:24:00 error: Your local changes to the following files would >> be overwritten by checkout: >> 2014-02-20 20:24:00 etc/neutron/plugins/ml2/ml2_conf_brocade.ini >> 2014-02-20 20:24:00 neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py >> 2014-02-20 20:24:00 Please, commit your changes or stash them before you can >> switch branches. >> 2014-02-20 20:24:00 Aborting >> 2014-02-20 20:24:00 + cd /opt/stack/neutron >> but still continues running (without my patchset) and reports success. -- >> This actually looks like a devstack bug (i'll check it out). >> >> PLUMgrid CI - Seems to always vote +1 without a failure >> (https://review.openstack.org/#/dashboard/10117) though the logs are private >> so we can't really tell whats going on. >> >> I was thinking it might be worth while or helpful to have a job that tests >> that CI is actually fails when we expect it to. >> >> Best, >> >> Aaron >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org <mailto:OpenStack-dev@lists.openstack.org> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > Thanks, > > Octavian. > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev