Re: [openstack-dev] False Positive testing for 3rd party CI
One Convergence CI did report the third party tests as failed, but is configured not to vote. Should we turn on the voting? Thanks, -hemanth On Fri, Feb 21, 2014 at 1:29 PM, Carl Baldwin c...@ecbaldwin.net wrote: Aaron, I was thinking the same thing recently with this patch [1]. Patch sets 1-5 should have failed for any plugin besides ml2 yet some passed and I wondered how that could happen. Kudos to those patches that failed my patch sets correctly. Carl [1] https://review.openstack.org/#/c/72565/ On Fri, Feb 21, 2014 at 11:34 AM, Aaron Rosen aaronoro...@gmail.com wrote: Hi, Yesterday, I pushed a patch to review and was surprised that several of the third party CI systems reported back that the patch-set worked where it definitely shouldn't have. Anyways, I tested out my theory a little more and it turns out a few of the 3rd party CI systems for neutron are just returning SUCCESS even if the patch set didn't run successfully (https://review.openstack.org/#/c/75304/). Here's a short summery of what I found. Hyper-V CI -- This seems like an easy fix as it's posting build succeeded but also puts to the side test run failed. Would probably be a good idea to remove the build succeeded message to avoid any confusion. Brocade CI - From the log files it posts it shows that it tries to apply my patch but fails: 2014-02-20 20:23:48 + cd /opt/stack/neutron 2014-02-20 20:23:48 + git fetch https://review.openstack.org/openstack/neutron.gitrefs/changes/04/75304/1 2014-02-20 20:24:00 From https://review.openstack.org/openstack/neutron 2014-02-20 20:24:00 * branchrefs/changes/04/75304/1 - FETCH_HEAD 2014-02-20 20:24:00 + git checkout FETCH_HEAD 2014-02-20 20:24:00 error: Your local changes to the following files would be overwritten by checkout: 2014-02-20 20:24:00 etc/neutron/plugins/ml2/ml2_conf_brocade.ini 2014-02-20 20:24:00 neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py 2014-02-20 20:24:00 Please, commit your changes or stash them before you can switch branches. 2014-02-20 20:24:00 Aborting 2014-02-20 20:24:00 + cd /opt/stack/neutron but still continues running (without my patchset) and reports success. -- This actually looks like a devstack bug (i'll check it out). PLUMgrid CI - Seems to always vote +1 without a failure (https://review.openstack.org/#/dashboard/10117) though the logs are private so we can't really tell whats going on. I was thinking it might be worth while or helpful to have a job that tests that CI is actually fails when we expect it to. Best, Aaron ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] False Positive testing for 3rd party CI
Thanks Aaron, it is really interesting. The Hyper-V CI is non-voting (as required for new third-party CIs) and this is the reason why any post from it will show “build succeeded”. As published in other threads, AFAIK the only way to get rid of this issue is to have the CI as voting. Results of zuul non-voting job is ignored. In Hyper-V CI, only one testing job is run, so I think it is a good start to run the testing job as voting and post a verified score with 0 for both success and failure. If so, we have a correct message which reflects test results and it does not affects any verified score. Thanks, Akihiro (2014/02/22 4:13), Octavian Ciuhandu wrote: Hi, On 21 Feb 2014, at 20:34, Aaron Rosen aaronoro...@gmail.com mailto:aaronoro...@gmail.com wrote: Hi, Yesterday, I pushed a patch to review and was surprised that several of the third party CI systems reported back that the patch-set worked where it definitely shouldn't have. Anyways, I tested out my theory a little more and it turns out a few of the 3rd party CI systems for neutron are just returning SUCCESS even if the patch set didn't run successfully (https://review.openstack.org/#/c/75304/). Here's a short summery of what I found. Hyper-V CI -- This seems like an easy fix as it's posting build succeeded but also puts to the side test run failed. Would probably be a good idea to remove the build succeeded message to avoid any confusion. The Hyper-V CI is non-voting (as required for new third-party CIs) and this is the reason why any post from it will show “build succeeded”. As published in other threads, AFAIK the only way to get rid of this issue is to have the CI as voting. Brocade CI - From the log files it posts it shows that it tries to apply my patch but fails: 2014-02-20 20:23:48 + cd /opt/stack/neutron 2014-02-20 20:23:48 + git fetchhttps://review.openstack.org/openstack/neutron.git refs/changes/04/75304/1 2014-02-20 20:24:00 Fromhttps://review.openstack.org/openstack/neutron 2014-02-20 https://review.openstack.org/openstack/neutron2014-02-20 20:24:00 * branchrefs/changes/04/75304/1 - FETCH_HEAD 2014-02-20 20:24:00 + git checkout FETCH_HEAD 2014-02-20 20:24:00 error: Your local changes to the following files would be overwritten by checkout: 2014-02-20 20:24:00 etc/neutron/plugins/ml2/ml2_conf_brocade.ini 2014-02-20 20:24:00 neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py 2014-02-20 20:24:00 Please, commit your changes or stash them before you can switch branches. 2014-02-20 20:24:00 Aborting 2014-02-20 20:24:00 + cd /opt/stack/neutron but still continues running (without my patchset) and reports success. -- This actually looks like a devstack bug (i'll check it out). PLUMgrid CI - Seems to always vote +1 without a failure (https://review.openstack.org/#/dashboard/10117) though the logs are private so we can't really tell whats going on. I was thinking it might be worth while or helpful to have a job that tests that CI is actually fails when we expect it to. Best, Aaron ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org mailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Thanks, Octavian. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] False Positive testing for 3rd party CI
Hi, Yesterday, I pushed a patch to review and was surprised that several of the third party CI systems reported back that the patch-set worked where it definitely shouldn't have. Anyways, I tested out my theory a little more and it turns out a few of the 3rd party CI systems for neutron are just returning SUCCESS even if the patch set didn't run successfully ( https://review.openstack.org/#/c/75304/). Here's a short summery of what I found. Hyper-V CI -- This seems like an easy fix as it's posting build succeeded but also puts to the side test run failed. Would probably be a good idea to remove the build succeeded message to avoid any confusion. Brocade CI - From the log files it posts it shows that it tries to apply my patch but fails: 2014-02-20 20:23:48 + cd /opt/stack/neutron 2014-02-20 20:23:48 + git fetch https://review.openstack.org/openstack/neutron.git refs/changes/04/75304/1 2014-02-20 20:24:00 From https://review.openstack.org/openstack/neutron 2014-02-20 20:24:00 * branchrefs/changes/04/75304/1 - FETCH_HEAD 2014-02-20 20:24:00 + git checkout FETCH_HEAD 2014-02-20 20:24:00 error: Your local changes to the following files would be overwritten by checkout: 2014-02-20 20:24:00 etc/neutron/plugins/ml2/ml2_conf_brocade.ini 2014-02-20 20:24:00 neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py 2014-02-20 20:24:00 Please, commit your changes or stash them before you can switch branches. 2014-02-20 20:24:00 Aborting 2014-02-20 20:24:00 + cd /opt/stack/neutron but still continues running (without my patchset) and reports success. -- This actually looks like a devstack bug (i'll check it out). PLUMgrid CI - Seems to always vote +1 without a failure ( https://review.openstack.org/#/dashboard/10117) though the logs are private so we can't really tell whats going on. I was thinking it might be worth while or helpful to have a job that tests that CI is actually fails when we expect it to. Best, Aaron ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] False Positive testing for 3rd party CI
Hi, On 21 Feb 2014, at 20:34, Aaron Rosen aaronoro...@gmail.commailto:aaronoro...@gmail.com wrote: Hi, Yesterday, I pushed a patch to review and was surprised that several of the third party CI systems reported back that the patch-set worked where it definitely shouldn't have. Anyways, I tested out my theory a little more and it turns out a few of the 3rd party CI systems for neutron are just returning SUCCESS even if the patch set didn't run successfully (https://review.openstack.org/#/c/75304/). Here's a short summery of what I found. Hyper-V CI -- This seems like an easy fix as it's posting build succeeded but also puts to the side test run failed. Would probably be a good idea to remove the build succeeded message to avoid any confusion. The Hyper-V CI is non-voting (as required for new third-party CIs) and this is the reason why any post from it will show “build succeeded”. As published in other threads, AFAIK the only way to get rid of this issue is to have the CI as voting. Brocade CI - From the log files it posts it shows that it tries to apply my patch but fails: 2014-02-20 20:23:48 + cd /opt/stack/neutron 2014-02-20 20:23:48 + git fetch https://review.openstack.org/openstack/neutron.git refs/changes/04/75304/1 2014-02-20 20:24:00 From https://review.openstack.org/openstack/neutron 2014-02-20https://review.openstack.org/openstack/neutron2014-02-20 20:24:00 * branchrefs/changes/04/75304/1 - FETCH_HEAD 2014-02-20 20:24:00 + git checkout FETCH_HEAD 2014-02-20 20:24:00 error: Your local changes to the following files would be overwritten by checkout: 2014-02-20 20:24:00 etc/neutron/plugins/ml2/ml2_conf_brocade.ini 2014-02-20 20:24:00 neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py 2014-02-20 20:24:00 Please, commit your changes or stash them before you can switch branches. 2014-02-20 20:24:00 Aborting 2014-02-20 20:24:00 + cd /opt/stack/neutron but still continues running (without my patchset) and reports success. -- This actually looks like a devstack bug (i'll check it out). PLUMgrid CI - Seems to always vote +1 without a failure (https://review.openstack.org/#/dashboard/10117) though the logs are private so we can't really tell whats going on. I was thinking it might be worth while or helpful to have a job that tests that CI is actually fails when we expect it to. Best, Aaron ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Thanks, Octavian. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] False Positive testing for 3rd party CI
This should fix the false positive for brocade: https://review.openstack.org/#/c/75486/ Aaron On Fri, Feb 21, 2014 at 10:34 AM, Aaron Rosen aaronoro...@gmail.com wrote: Hi, Yesterday, I pushed a patch to review and was surprised that several of the third party CI systems reported back that the patch-set worked where it definitely shouldn't have. Anyways, I tested out my theory a little more and it turns out a few of the 3rd party CI systems for neutron are just returning SUCCESS even if the patch set didn't run successfully ( https://review.openstack.org/#/c/75304/). Here's a short summery of what I found. Hyper-V CI -- This seems like an easy fix as it's posting build succeeded but also puts to the side test run failed. Would probably be a good idea to remove the build succeeded message to avoid any confusion. Brocade CI - From the log files it posts it shows that it tries to apply my patch but fails: 2014-02-20 20:23:48 + cd /opt/stack/neutron 2014-02-20 20:23:48 + git fetch https://review.openstack.org/openstack/neutron.git refs/changes/04/75304/1 2014-02-20 20:24:00 From https://review.openstack.org/openstack/neutron 2014-02-20 https://review.openstack.org/openstack/neutron2014-02-20 20:24:00 * branchrefs/changes/04/75304/1 - FETCH_HEAD 2014-02-20 20:24:00 + git checkout FETCH_HEAD 2014-02-20 20:24:00 error: Your local changes to the following files would be overwritten by checkout: 2014-02-20 20:24:00 etc/neutron/plugins/ml2/ml2_conf_brocade.ini 2014-02-20 20:24:00 neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py 2014-02-20 20:24:00 Please, commit your changes or stash them before you can switch branches. 2014-02-20 20:24:00 Aborting 2014-02-20 20:24:00 + cd /opt/stack/neutron but still continues running (without my patchset) and reports success. -- This actually looks like a devstack bug (i'll check it out). PLUMgrid CI - Seems to always vote +1 without a failure ( https://review.openstack.org/#/dashboard/10117) though the logs are private so we can't really tell whats going on. I was thinking it might be worth while or helpful to have a job that tests that CI is actually fails when we expect it to. Best, Aaron ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] False Positive testing for 3rd party CI
Nice one! On 21 February 2014 11:22, Aaron Rosen aaronoro...@gmail.com wrote: This should fix the false positive for brocade: https://review.openstack.org/#/c/75486/ Aaron On Fri, Feb 21, 2014 at 10:34 AM, Aaron Rosen aaronoro...@gmail.com wrote: Hi, Yesterday, I pushed a patch to review and was surprised that several of the third party CI systems reported back that the patch-set worked where it definitely shouldn't have. Anyways, I tested out my theory a little more and it turns out a few of the 3rd party CI systems for neutron are just returning SUCCESS even if the patch set didn't run successfully (https://review.openstack.org/#/c/75304/). Here's a short summery of what I found. Hyper-V CI -- This seems like an easy fix as it's posting build succeeded but also puts to the side test run failed. Would probably be a good idea to remove the build succeeded message to avoid any confusion. Brocade CI - From the log files it posts it shows that it tries to apply my patch but fails: 2014-02-20 20:23:48 + cd /opt/stack/neutron 2014-02-20 20:23:48 + git fetch https://review.openstack.org/openstack/neutron.git refs/changes/04/75304/1 2014-02-20 20:24:00 From https://review.openstack.org/openstack/neutron 2014-02-20 20:24:00 * branchrefs/changes/04/75304/1 - FETCH_HEAD 2014-02-20 20:24:00 + git checkout FETCH_HEAD 2014-02-20 20:24:00 error: Your local changes to the following files would be overwritten by checkout: 2014-02-20 20:24:00 etc/neutron/plugins/ml2/ml2_conf_brocade.ini 2014-02-20 20:24:00 neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py 2014-02-20 20:24:00 Please, commit your changes or stash them before you can switch branches. 2014-02-20 20:24:00 Aborting 2014-02-20 20:24:00 + cd /opt/stack/neutron but still continues running (without my patchset) and reports success. -- This actually looks like a devstack bug (i'll check it out). PLUMgrid CI - Seems to always vote +1 without a failure (https://review.openstack.org/#/dashboard/10117) though the logs are private so we can't really tell whats going on. I was thinking it might be worth while or helpful to have a job that tests that CI is actually fails when we expect it to. Best, Aaron ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] False Positive testing for 3rd party CI
Aaron, I was thinking the same thing recently with this patch [1]. Patch sets 1-5 should have failed for any plugin besides ml2 yet some passed and I wondered how that could happen. Kudos to those patches that failed my patch sets correctly. Carl [1] https://review.openstack.org/#/c/72565/ On Fri, Feb 21, 2014 at 11:34 AM, Aaron Rosen aaronoro...@gmail.com wrote: Hi, Yesterday, I pushed a patch to review and was surprised that several of the third party CI systems reported back that the patch-set worked where it definitely shouldn't have. Anyways, I tested out my theory a little more and it turns out a few of the 3rd party CI systems for neutron are just returning SUCCESS even if the patch set didn't run successfully (https://review.openstack.org/#/c/75304/). Here's a short summery of what I found. Hyper-V CI -- This seems like an easy fix as it's posting build succeeded but also puts to the side test run failed. Would probably be a good idea to remove the build succeeded message to avoid any confusion. Brocade CI - From the log files it posts it shows that it tries to apply my patch but fails: 2014-02-20 20:23:48 + cd /opt/stack/neutron 2014-02-20 20:23:48 + git fetch https://review.openstack.org/openstack/neutron.git refs/changes/04/75304/1 2014-02-20 20:24:00 From https://review.openstack.org/openstack/neutron 2014-02-20 20:24:00 * branchrefs/changes/04/75304/1 - FETCH_HEAD 2014-02-20 20:24:00 + git checkout FETCH_HEAD 2014-02-20 20:24:00 error: Your local changes to the following files would be overwritten by checkout: 2014-02-20 20:24:00 etc/neutron/plugins/ml2/ml2_conf_brocade.ini 2014-02-20 20:24:00 neutron/plugins/ml2/drivers/brocade/mechanism_brocade.py 2014-02-20 20:24:00 Please, commit your changes or stash them before you can switch branches. 2014-02-20 20:24:00 Aborting 2014-02-20 20:24:00 + cd /opt/stack/neutron but still continues running (without my patchset) and reports success. -- This actually looks like a devstack bug (i'll check it out). PLUMgrid CI - Seems to always vote +1 without a failure (https://review.openstack.org/#/dashboard/10117) though the logs are private so we can't really tell whats going on. I was thinking it might be worth while or helpful to have a job that tests that CI is actually fails when we expect it to. Best, Aaron ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
