Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
On 17 June 2014 09:55, Luke Gorrie l...@tail-f.com wrote: I have a problem that appeared at the same time and may be related? testr list-tests in the tempest directory is failing with an obscure error message. Seems to be exactly the situation described here: https://bugs.launchpad.net/subunit/+bug/1278539 Any tips? How should I go about getting help with this? Mailing list + IRC is not getting anybody's attention and I want to get my CI back online. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
Hi Luke, That kind of message usually shows up in unit tests job when there is some syntax error or circular import. But I think that it's not your case. Usually you see an import error message towards the end of the garbage. If you can point me to a failing log of your CI I can have a look at it and see if I can help you. Salvatore On 18 June 2014 11:09, Luke Gorrie l...@tail-f.com wrote: On 17 June 2014 09:55, Luke Gorrie l...@tail-f.com wrote: I have a problem that appeared at the same time and may be related? testr list-tests in the tempest directory is failing with an obscure error message. Seems to be exactly the situation described here: https://bugs.launchpad.net/subunit/+bug/1278539 Any tips? How should I go about getting help with this? Mailing list + IRC is not getting anybody's attention and I want to get my CI back online. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
On 18 June 2014 15:48, Salvatore Orlando sorla...@nicira.com wrote: Hi Luke, That kind of message usually shows up in unit tests job when there is some syntax error or circular import. But I think that it's not your case. Usually you see an import error message towards the end of the garbage. If you can point me to a failing log of your CI I can have a look at it and see if I can help you. Thanks, Salvatore! I have a log here: http://88.198.8.227:81/html/ci-logs/problem-1.log and on this machine I can reproduce the problem using the steps in the bug that I referenced above: ci@egg:/tmp$ mkdir bug ci@egg:/tmp$ cd !$ cd bug ci@egg:/tmp/bug$ git clone https://github.com/openstack/tempest.git Cloning into 'tempest'... remote: Reusing existing pack: 35264, done. remote: Counting objects: 229, done. remote: Compressing objects: 100% (221/221), done. remote: Total 35493 (delta 105), reused 25 (delta 8) Receiving objects: 100% (35493/35493), 8.51 MiB | 1.61 MiB/s, done. Resolving deltas: 100% (25835/25835), done. Checking connectivity... done. ci@egg:/tmp/bug$ cd tempest ci@egg:/tmp/bug/tempest$ sudo pip install -r requirements.txt Requirement already satisfied (use --upgrade to upgrade): pbr=0.6,!=0.7,1.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): anyjson=0.3.3 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): httplib2=0.7.5 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): jsonschema=2.0.0,3.0.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): testtools=0.9.34 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 5)) Requirement already satisfied (use --upgrade to upgrade): lxml=2.3 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 6)) Requirement already satisfied (use --upgrade to upgrade): boto=2.12.0,!=2.13.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 7)) Requirement already satisfied (use --upgrade to upgrade): paramiko=1.13.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 8)) Requirement already satisfied (use --upgrade to upgrade): netaddr=0.7.6 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 9)) Requirement already satisfied (use --upgrade to upgrade): python-glanceclient=0.9.0 in /opt/stack/python-glanceclient (from -r requirements.txt (line 10)) Requirement already satisfied (use --upgrade to upgrade): python-keystoneclient=0.8.0 in /opt/stack/python-keystoneclient (from -r requirements.txt (line 11)) Requirement already satisfied (use --upgrade to upgrade): python-novaclient=2.17.0 in /opt/stack/python-novaclient (from -r requirements.txt (line 12)) Requirement already satisfied (use --upgrade to upgrade): python-neutronclient=2.3.4,3 in /opt/stack/python-neutronclient (from -r requirements.txt (line 13)) Requirement already satisfied (use --upgrade to upgrade): python-cinderclient=1.0.6 in /opt/stack/python-cinderclient (from -r requirements.txt (line 14)) Requirement already satisfied (use --upgrade to upgrade): python-heatclient=0.2.9 in /opt/stack/python-heatclient (from -r requirements.txt (line 15)) Requirement already satisfied (use --upgrade to upgrade): python-ironicclient in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 16)) Requirement already satisfied (use --upgrade to upgrade): python-saharaclient=0.6.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 17)) Requirement already satisfied (use --upgrade to upgrade): python-swiftclient=2.0.2 in /opt/stack/python-swiftclient (from -r requirements.txt (line 18)) Requirement already satisfied (use --upgrade to upgrade): testresources=0.2.4 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 19)) Requirement already satisfied (use --upgrade to upgrade): testrepository=0.0.18 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 20)) Requirement already satisfied (use --upgrade to upgrade): oslo.config=1.2.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 21)) Requirement already satisfied (use --upgrade to upgrade): six=1.6.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 22)) Requirement already satisfied (use --upgrade to upgrade): iso8601=0.1.9 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 23)) Requirement already satisfied (use --upgrade to upgrade): fixtures=0.3.14 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 24)) Requirement already satisfied (use --upgrade to upgrade): testscenarios=0.4 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 25))
Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
Hi Luke, it seems something is not quite right with your tempest environment - you have import errors at startup [1] This might be happening because of missing dependencies, or, if you have applied some custom patches to tempest trunk, possibly those are causing some problems. Salvatore [1] http://paste.openstack.org/show/84406/ On 18 June 2014 18:01, Luke Gorrie l...@snabb.co wrote: On 18 June 2014 15:48, Salvatore Orlando sorla...@nicira.com wrote: Hi Luke, That kind of message usually shows up in unit tests job when there is some syntax error or circular import. But I think that it's not your case. Usually you see an import error message towards the end of the garbage. If you can point me to a failing log of your CI I can have a look at it and see if I can help you. Thanks, Salvatore! I have a log here: http://88.198.8.227:81/html/ci-logs/problem-1.log and on this machine I can reproduce the problem using the steps in the bug that I referenced above: ci@egg:/tmp$ mkdir bug ci@egg:/tmp$ cd !$ cd bug ci@egg:/tmp/bug$ git clone https://github.com/openstack/tempest.git Cloning into 'tempest'... remote: Reusing existing pack: 35264, done. remote: Counting objects: 229, done. remote: Compressing objects: 100% (221/221), done. remote: Total 35493 (delta 105), reused 25 (delta 8) Receiving objects: 100% (35493/35493), 8.51 MiB | 1.61 MiB/s, done. Resolving deltas: 100% (25835/25835), done. Checking connectivity... done. ci@egg:/tmp/bug$ cd tempest ci@egg:/tmp/bug/tempest$ sudo pip install -r requirements.txt Requirement already satisfied (use --upgrade to upgrade): pbr=0.6,!=0.7,1.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): anyjson=0.3.3 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): httplib2=0.7.5 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): jsonschema=2.0.0,3.0.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): testtools=0.9.34 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 5)) Requirement already satisfied (use --upgrade to upgrade): lxml=2.3 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 6)) Requirement already satisfied (use --upgrade to upgrade): boto=2.12.0,!=2.13.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 7)) Requirement already satisfied (use --upgrade to upgrade): paramiko=1.13.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 8)) Requirement already satisfied (use --upgrade to upgrade): netaddr=0.7.6 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 9)) Requirement already satisfied (use --upgrade to upgrade): python-glanceclient=0.9.0 in /opt/stack/python-glanceclient (from -r requirements.txt (line 10)) Requirement already satisfied (use --upgrade to upgrade): python-keystoneclient=0.8.0 in /opt/stack/python-keystoneclient (from -r requirements.txt (line 11)) Requirement already satisfied (use --upgrade to upgrade): python-novaclient=2.17.0 in /opt/stack/python-novaclient (from -r requirements.txt (line 12)) Requirement already satisfied (use --upgrade to upgrade): python-neutronclient=2.3.4,3 in /opt/stack/python-neutronclient (from -r requirements.txt (line 13)) Requirement already satisfied (use --upgrade to upgrade): python-cinderclient=1.0.6 in /opt/stack/python-cinderclient (from -r requirements.txt (line 14)) Requirement already satisfied (use --upgrade to upgrade): python-heatclient=0.2.9 in /opt/stack/python-heatclient (from -r requirements.txt (line 15)) Requirement already satisfied (use --upgrade to upgrade): python-ironicclient in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 16)) Requirement already satisfied (use --upgrade to upgrade): python-saharaclient=0.6.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 17)) Requirement already satisfied (use --upgrade to upgrade): python-swiftclient=2.0.2 in /opt/stack/python-swiftclient (from -r requirements.txt (line 18)) Requirement already satisfied (use --upgrade to upgrade): testresources=0.2.4 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 19)) Requirement already satisfied (use --upgrade to upgrade): testrepository=0.0.18 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 20)) Requirement already satisfied (use --upgrade to upgrade): oslo.config=1.2.0 in /usr/local/lib/python2.7/dist-packages (from -r requirements.txt (line 21)) Requirement already satisfied (use --upgrade to upgrade): six=1.6.0 in /usr/local/lib/python2.7/dist-packages (from -r
Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
On 18 June 2014 18:24, Salvatore Orlando sorla...@nicira.com wrote: it seems something is not quite right with your tempest environment - you have import errors at startup [1] This might be happening because of missing dependencies, or, if you have applied some custom patches to tempest trunk, possibly those are causing some problems. Interesting. I'm using a fresh checkout of tempest from master on github and running pip install -r requirements.txt in tempest/. Any ideas on what I should check next? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
When I get these errors I try to import the modules for which I get an import error in a python shell. This sometimes give a fairly explicative error message. That will hopefully help. Also - consider submitting your question to ask.openstack.org, where a lot more experienced operators and developers might have an answer to your problem. Salvatore On 18 June 2014 18:54, Luke Gorrie l...@snabb.co wrote: On 18 June 2014 18:24, Salvatore Orlando sorla...@nicira.com wrote: it seems something is not quite right with your tempest environment - you have import errors at startup [1] This might be happening because of missing dependencies, or, if you have applied some custom patches to tempest trunk, possibly those are causing some problems. Interesting. I'm using a fresh checkout of tempest from master on github and running pip install -r requirements.txt in tempest/. Any ideas on what I should check next? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
On Sat, Jun 14, 2014 at 09:39:50PM -0700, Sukhdev Kapur wrote: Oppss...sorry wrong link... please use this http://paste.openstack.org/show/84073/. Just a friendly note -- pastebins usually expire, and if someone looks at the archives years later, there'll be nothing to look at. It's more useful to reproduce the content in plain text in email. -- /kashyap ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
On 15 June 2014 02:45, Sukhdev Kapur sukhdevka...@gmail.com wrote: I thought I send out this message in case other CI maintainers are investigating this issue. I have a problem that appeared at the same time and may be related? testr list-tests in the tempest directory is failing with an obscure error message. Seems to be exactly the situation described here: https://bugs.launchpad.net/subunit/+bug/1278539 Any tips? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
On 06/14/2014 09:39 PM, Sukhdev Kapur wrote: Oppss...sorry wrong link... please use this http://paste.openstack.org/show/84073/. If anybody needs help, please ping me or go to #openstack-infra. The relevant patch has been merged into upstream setuptools and a new setuptools release, 5.0.2 has been cut. On Sat, Jun 14, 2014 at 9:34 PM, Sukhdev Kapur sukhdevka...@gmail.com wrote: Fellow Stackers, I have an update on the issue. Kudos to the Infra folks, a huge thanks to Monty for coming up with patch for this setuptools issue, and Anita for for being on top of this. Please follow the steps in http://paste.openstack.org/show/84076/ to pull this patch on your local systems to get past the issue - until the fix in the upstream is merged. Note that you have to install mercurial to pull this patch. Hope this helps. regards.. -Sukhdev On Sat, Jun 14, 2014 at 5:45 PM, Sukhdev Kapur sukhdevka...@gmail.com wrote: I noticed this afternoon (Saturday PST 1:18pm) that most of the Third Party test systems started to fail because of the seuptools bug because of dependency in python-swiftclient. I further noticed that some of the CI's are voting +1, but, when I look through the logs, they seem to be hitting this issue as well. I have been on #openstack-infra most of the afternoon discussing various options suggested by folks. Infra folks have confirmed this issue and are looking for solution. I tried fixes suggested in [1] and [2] below and removed the setuptools and reinstalled version 3.8. This did not help. I have opened the bug[3] to track this issue. I thought I send out this message in case other CI maintainers are investigating this issue. Please share ideas/thoughts so that we can get the CIs fixed as soon as possible. Thanks -Sukhdev [1] https://bugs.launchpad.net/python-swiftclient/+bug/1326972 [2] https://mail.python.org/pipermail/distutils-sig/2014-June/024478.html [3] https://bugs.launchpad.net/python-swiftclient/+bug/1330140 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
Fellow Stackers, I have an update on the issue. Kudos to the Infra folks, a huge thanks to Monty for coming up with patch for this setuptools issue, and Anita for for being on top of this. Please follow the steps in http://paste.openstack.org/show/84076/ to pull this patch on your local systems to get past the issue - until the fix in the upstream is merged. Note that you have to install mercurial to pull this patch. Hope this helps. regards.. -Sukhdev On Sat, Jun 14, 2014 at 5:45 PM, Sukhdev Kapur sukhdevka...@gmail.com wrote: I noticed this afternoon (Saturday PST 1:18pm) that most of the Third Party test systems started to fail because of the seuptools bug because of dependency in python-swiftclient. I further noticed that some of the CI's are voting +1, but, when I look through the logs, they seem to be hitting this issue as well. I have been on #openstack-infra most of the afternoon discussing various options suggested by folks. Infra folks have confirmed this issue and are looking for solution. I tried fixes suggested in [1] and [2] below and removed the setuptools and reinstalled version 3.8. This did not help. I have opened the bug[3] to track this issue. I thought I send out this message in case other CI maintainers are investigating this issue. Please share ideas/thoughts so that we can get the CIs fixed as soon as possible. Thanks -Sukhdev [1] https://bugs.launchpad.net/python-swiftclient/+bug/1326972 [2] https://mail.python.org/pipermail/distutils-sig/2014-June/024478.html [3] https://bugs.launchpad.net/python-swiftclient/+bug/1330140 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][Swift][third-party] Most Third Party CI's failing
Oppss...sorry wrong link... please use this http://paste.openstack.org/show/84073/. If anybody needs help, please ping me or go to #openstack-infra. regards.. -Sukhdev On Sat, Jun 14, 2014 at 9:34 PM, Sukhdev Kapur sukhdevka...@gmail.com wrote: Fellow Stackers, I have an update on the issue. Kudos to the Infra folks, a huge thanks to Monty for coming up with patch for this setuptools issue, and Anita for for being on top of this. Please follow the steps in http://paste.openstack.org/show/84076/ to pull this patch on your local systems to get past the issue - until the fix in the upstream is merged. Note that you have to install mercurial to pull this patch. Hope this helps. regards.. -Sukhdev On Sat, Jun 14, 2014 at 5:45 PM, Sukhdev Kapur sukhdevka...@gmail.com wrote: I noticed this afternoon (Saturday PST 1:18pm) that most of the Third Party test systems started to fail because of the seuptools bug because of dependency in python-swiftclient. I further noticed that some of the CI's are voting +1, but, when I look through the logs, they seem to be hitting this issue as well. I have been on #openstack-infra most of the afternoon discussing various options suggested by folks. Infra folks have confirmed this issue and are looking for solution. I tried fixes suggested in [1] and [2] below and removed the setuptools and reinstalled version 3.8. This did not help. I have opened the bug[3] to track this issue. I thought I send out this message in case other CI maintainers are investigating this issue. Please share ideas/thoughts so that we can get the CIs fixed as soon as possible. Thanks -Sukhdev [1] https://bugs.launchpad.net/python-swiftclient/+bug/1326972 [2] https://mail.python.org/pipermail/distutils-sig/2014-June/024478.html [3] https://bugs.launchpad.net/python-swiftclient/+bug/1330140 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
