[Yahoo-eng-team] [Bug 1586979] Re: AMQP 2.0 prevents services from starting
Confirmed this also affects Ironic.
Searched and found this open bug in oslo.messaging, addressing the underlying
issue:
https://bugs.launchpad.net/oslo.messaging/+bug/1586840
** Also affects: ironic
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1586979
Title:
AMQP 2.0 prevents services from starting
Status in Ironic:
New
Status in OpenStack Identity (keystone):
New
Status in neutron:
New
Status in OpenStack Compute (nova):
New
Bug description:
We are running nova from the stable/mitaka branch (not using
packages).
Our build process has flagged this. Previously, when we've built,
we've used the requirements.txt file to pull in dependencies, and one
of the dependencies pulls in AMQP.
Initial investigation leads us to believe that oslo.messaging pulls in
AMQP as a dependency. It looks to be pulling in the latest version
(2.0). Our previous successful builds show AMQP of 1.4.9, which is
fully functional.
2.0 completely breaks, for example, the nova-compute service with the
following trace:
2016-05-30 01:04:58.192 11335 CRITICAL nova
[req-992ce9e3-fab8-47f9-8879-36b42594fae8 - - - - -] AttributeError:
'Connection' object has no attribute '_frame_writer'
2016-05-30 01:04:58.192 11335 ERROR nova Traceback (most recent call last):
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/bin/nova-compute", line 10, in
2016-05-30 01:04:58.192 11335 ERROR nova sys.exit(main())
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/cmd/compute.py",
line 73, in main
2016-05-30 01:04:58.192 11335 ERROR nova
db_allowed=CONF.conductor.use_local)
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py",
line 218, in create
2016-05-30 01:04:58.192 11335 ERROR nova db_allowed=db_allowed)
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/service.py",
line 101, in init
2016-05-30 01:04:58.192 11335 ERROR nova
self.conductor_api.wait_until_ready(context.get_admin_context())
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/conductor/api.py",
line 157, in wait_until_ready
2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout)
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/nova/baserpc.py",
line 58, in ping
2016-05-30 01:04:58.192 11335 ERROR nova return cctxt.call(context,
'ping', arg=arg_p)
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
line 157, in call
2016-05-30 01:04:58.192 11335 ERROR nova retry=self.retry)
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/transport.py",
line 91, in _send
2016-05-30 01:04:58.192 11335 ERROR nova timeout=timeout, retry=retry)
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 466, in send
2016-05-30 01:04:58.192 11335 ERROR nova retry=retry)
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 410, in _send
2016-05-30 01:04:58.192 11335 ERROR nova msg.update({'_reply_q':
self._get_reply_q()})
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 382, in _get_reply_q
2016-05-30 01:04:58.192 11335 ERROR nova conn =
self._get_connection(rpc_common.PURPOSE_LISTEN)
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
line 373, in _get_connection
2016-05-30 01:04:58.192 11335 ERROR nova purpose=purpose)
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py",
line 396, in init
2016-05-30 01:04:58.192 11335 ERROR nova self.connection =
connection_pool.create(purpose)
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py",
line 110, in create
2016-05-30 01:04:58.192 11335 ERROR nova return
self.connection_cls(self.conf, self.url, purpose)
2016-05-30 01:04:58.192 11335 ERROR nova File
"/opt/mhos/openstack/nova/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py",
line 556, in
[Yahoo-eng-team] [Bug 1561796] [NEW] ironic driver does not support ssl cafile
Public bug reported: Even though Ironic's python client supports SSL encrypted connections to the ironic service, and securing intra-service connections is a recommended practice, the nova.virt.Ironic driver currently lacks an option to specify a custom CA Certificate for validating the SSL connection to the Ironic service. On the other hand, other OpenStack services which Nova connects to (eg, Glance, Neutron...) have support for this via a service-specific "cafile" config option. ** Affects: nova Importance: Undecided Assignee: Devananda van der Veen (devananda) Status: In Progress ** Tags: ironic security ** Tags added: ironic ** Tags added: security -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1561796 Title: ironic driver does not support ssl cafile Status in OpenStack Compute (nova): In Progress Bug description: Even though Ironic's python client supports SSL encrypted connections to the ironic service, and securing intra-service connections is a recommended practice, the nova.virt.Ironic driver currently lacks an option to specify a custom CA Certificate for validating the SSL connection to the Ironic service. On the other hand, other OpenStack services which Nova connects to (eg, Glance, Neutron...) have support for this via a service-specific "cafile" config option. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1561796/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1544195] Re: User can not provision ironic node via nova when providing pre-created port
** Also affects: magnum Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1544195 Title: User can not provision ironic node via nova when providing pre-created port Status in Magnum: New Status in OpenStack Compute (nova): Confirmed Bug description: When booting a nova instance with baremetal flavor, one can not provide a pre-created neutron port to "nova boot" command. The reason is obvious - to successfully deploy, mac address of the port must be the same as mac address of the ironic port corresponding to the ironic node where provisioning will happen, and although it is possible to specify a mac address during port create, a user can not know to which exactly ironic node provisioning will be assigned to by nova compute (more over, ordinary user has no access to list of ironic ports/macs whatsoever). This is most probably a known limitation, but the big problem is that it breaks many sorts of cloud orchestration attempts. For example, the most flexible in terms of usage approach in Heat is to pre-create a port, and create the server with this port provided (actually this is the only way if one needs to assign a floating IP to the instance via Neutron). Some other consumers of Heat extensively use this approach. So this limitation precludes Murano or Sahara to provision their instances on bare metal via Nova/Ironic. The solution might be to update the mac of the port to correct one (mac address update is possible with admin context) when working with baremetal nodes/Ironic. To manage notifications about this bug go to: https://bugs.launchpad.net/magnum/+bug/1544195/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1412197] [NEW] cells assumes compute nodes are subdivisible
Public bug reported: In nova/cells/starte.py _update_our_capacity(), the calculations of free units of compute are based on optimistic packing of each instance type on each compute node. This does not account for hypervisors which do not allow more than one instance per node, such as ironic. ** Affects: nova Importance: Undecided Status: New ** Tags: cells ironic ** Tags added: cells ironic -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1412197 Title: cells assumes compute nodes are subdivisible Status in OpenStack Compute (Nova): New Bug description: In nova/cells/starte.py _update_our_capacity(), the calculations of free units of compute are based on optimistic packing of each instance type on each compute node. This does not account for hypervisors which do not allow more than one instance per node, such as ironic. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1412197/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1404331] [NEW] ironic driver logs incorrect error message when node in unexpected state
Public bug reported:
When an Ironic node is not in the expected state (eg, it somehow is out
of sync with the nova driver), an incorrect error message is logged in
Nova.
This showed up while testing changes to Ironic's state machine (so the
node being in the wrong state is not Nova's fault; I broke something in
Ironic to cause that). Regardless of the cause of the InvalidState
error, our Nova driver should handle it better.
Here is a copy of the trace from this test run:
http://logs.openstack.org/83/140883/6/check/check-tempest-dsvm-ironic-pxe_ssh/369aebc/logs/screen-n-cpu.txt.gz?#_2014-12-19_16_52_57_030
2014-12-19 16:52:57.030 WARNING ironicclient.common.http
[req-7059788d-3695-4b22-851a-bec30922e823 demo demo] Request returned failure
status.
2014-12-19 16:52:57.030 WARNING nova.virt.ironic.client_wrapper
[req-7059788d-3695-4b22-851a-bec30922e823 demo demo] Error contacting Ironic
server for 'node.update'. Attempt 59 of 60
...
{error_message: {\debuginfo\: null, \faultcode\: \Client\,
\faultstring\: \Node 07a3ce7c-0726-4fc2-a94b-a707d0450b5a can not be updated
while a state transition is in progress.\}}
log_http_response
/usr/local/lib/python2.7/dist-packages/ironicclient/common/http.py:119
2014-12-19 16:52:59.196 WARNING ironicclient.common.http
[req-7059788d-3695-4b22-851a-bec30922e823 demo demo] Request returned failure
status.
2014-12-19 16:52:59.196 ERROR nova.virt.ironic.client_wrapper
[req-7059788d-3695-4b22-851a-bec30922e823 demo demo] Error contacting Ironic
server for 'node.update'. Attempt 60 of 60
2014-12-19 16:52:59.197 ERROR nova.compute.manager
[req-7059788d-3695-4b22-851a-bec30922e823 demo demo] [instance:
604b621c-2103-4343-85f4-acaef2b0eb18] Setting instance vm_state to ERROR
2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance:
604b621c-2103-4343-85f4-acaef2b0eb18] Traceback (most recent call last):
2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance:
604b621c-2103-4343-85f4-acaef2b0eb18] File
/opt/stack/new/nova/nova/compute/manager.py, line 6148, in
_error_out_instance_on_exception
2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance:
604b621c-2103-4343-85f4-acaef2b0eb18] yield
2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance:
604b621c-2103-4343-85f4-acaef2b0eb18] File
/opt/stack/new/nova/nova/compute/manager.py, line 2865, in rebuild_instance
2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance:
604b621c-2103-4343-85f4-acaef2b0eb18] self.driver.rebuild(**kwargs)
2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance:
604b621c-2103-4343-85f4-acaef2b0eb18] File
/opt/stack/new/nova/nova/virt/ironic/driver.py, line 1007, in rebuild
2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance:
604b621c-2103-4343-85f4-acaef2b0eb18] preserve_ephemeral)
2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance:
604b621c-2103-4343-85f4-acaef2b0eb18] File
/opt/stack/new/nova/nova/virt/ironic/driver.py, line 297, in
_add_driver_fields
2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance:
604b621c-2103-4343-85f4-acaef2b0eb18] ironicclient.call('node.update',
node.uuid, patch)
2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance:
604b621c-2103-4343-85f4-acaef2b0eb18] File
/opt/stack/new/nova/nova/virt/ironic/client_wrapper.py, line 118, in call
2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance:
604b621c-2103-4343-85f4-acaef2b0eb18] raise exception.NovaException(msg)
2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance:
604b621c-2103-4343-85f4-acaef2b0eb18] NovaException: Error contacting Ironic
server for 'node.update'. Attempt 60 of 60
2014-12-19 16:52:59.197 31679 TRACE nova.compute.manager [instance:
604b621c-2103-4343-85f4-acaef2b0eb18]
** Affects: nova
Importance: Medium
Status: Triaged
** Tags: ironic
** Changed in: nova
Status: New = Triaged
** Changed in: nova
Importance: Undecided = Medium
** Tags added: ironic
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1404331
Title:
ironic driver logs incorrect error message when node in unexpected
state
Status in OpenStack Compute (Nova):
Triaged
Bug description:
When an Ironic node is not in the expected state (eg, it somehow is
out of sync with the nova driver), an incorrect error message is
logged in Nova.
This showed up while testing changes to Ironic's state machine (so the
node being in the wrong state is not Nova's fault; I broke something
in Ironic to cause that). Regardless of the cause of the InvalidState
error, our Nova driver should handle it better.
Here is a copy of the trace from this test run:
[Yahoo-eng-team] [Bug 1379952] [NEW] API accepts tenant name for TenantId, fails, and provides not helpful message
Public bug reported: When authenticating with Keystone's REST API, if I happen to provide my tenant name in the TenantId field, the resulting error tells me that I am not authorized for that tenant, even though all the information (user, pass, tenant) are correct. It *should* tell me that I just passed invalid data into a field that expects a UUID, which, as a user, would tell me exactly what was wrong. For what it's worth, in debugging my auth problem, I ended up tcpdump'ing keystone which led to this gem of a demonstration of the problem: http://paste.openstack.org/show/4v5JtwbGNu6QhQ3K5oQ1/ ** Affects: keystone Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Keystone. https://bugs.launchpad.net/bugs/1379952 Title: API accepts tenant name for TenantId, fails, and provides not helpful message Status in OpenStack Identity (Keystone): New Bug description: When authenticating with Keystone's REST API, if I happen to provide my tenant name in the TenantId field, the resulting error tells me that I am not authorized for that tenant, even though all the information (user, pass, tenant) are correct. It *should* tell me that I just passed invalid data into a field that expects a UUID, which, as a user, would tell me exactly what was wrong. For what it's worth, in debugging my auth problem, I ended up tcpdump'ing keystone which led to this gem of a demonstration of the problem: http://paste.openstack.org/show/4v5JtwbGNu6QhQ3K5oQ1/ To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1379952/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1310135] Re: Stopping an instance via the Nova API when using the Nova Ironic driver incorrectly reports powerstate
Digging further after proposing a fix to the Nova driver, there is *also* a race inside of ironic/conductor/manager.py and ironic/conductor/utils.py -- I am posting a fix for those now. ** Changed in: ironic Status: Invalid = In Progress ** Changed in: ironic Assignee: Rakesh H S (rh-s) = Devananda van der Veen (devananda) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1310135 Title: Stopping an instance via the Nova API when using the Nova Ironic driver incorrectly reports powerstate Status in OpenStack Bare Metal Provisioning Service (Ironic): In Progress Status in OpenStack Compute (Nova): In Progress Bug description: When using the Ironic Nova driver, a stopped server is still presented as Running even when the server is stopped. Checking via the Ironic API correctly shows the instance as powered down: stack@ironic:~/logs/screen$ nova list +--+-+++-+---+ | ID | Name| Status | Task State | Power State | Networks | +--+-+++-+---+ | 5b43d631-91e1-4384-9b87-93283b3ae958 | testing | ACTIVE | - | Running | private=10.1.0.10 | +--+-+++-+---+ stack@ironic:~/logs/screen$ nova stop 5b43d631-91e1-4384-9b87-93283b3ae958 stack@ironic:~/logs/screen$ nova list +--+-+-++-+---+ | ID | Name| Status | Task State | Power State | Networks | +--+-+-++-+---+ | 5b43d631-91e1-4384-9b87-93283b3ae958 | testing | SHUTOFF | - | Running | private=10.1.0.10 | +--+-+-++-+---+ stack@ironic:~/logs/screen$ ping 10.1.0.10 PING 10.1.0.10 (10.1.0.10) 56(84) bytes of data. From 172.24.4.2 icmp_seq=1 Destination Host Unreachable From 172.24.4.2 icmp_seq=5 Destination Host Unreachable From 172.24.4.2 icmp_seq=6 Destination Host Unreachable From 172.24.4.2 icmp_seq=7 Destination Host Unreachable From 172.24.4.2 icmp_seq=8 Destination Host Unreachable --- 10.1.0.10 ping statistics --- 9 packets transmitted, 0 received, +5 errors, 100% packet loss, time 8000ms stack@ironic:~/logs/screen$ ironic node-list +--+--+-++-+ | UUID | Instance UUID | Power State | Provisioning State | Maintenance | +--+--+-++-+ | 91e81c38-4dce-412b-8a1b-a914d28943e4 | 5b43d631-91e1-4384-9b87-93283b3ae958 | power off | active | False | +--+--+-++-+ To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1310135/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1263790] Re: ipmitool does not support OPERATOR priv level
** Changed in: nova Status: Triaged = Won't Fix -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1263790 Title: ipmitool does not support OPERATOR priv level Status in OpenStack Bare Metal Provisioning Service (Ironic): Fix Released Status in OpenStack Compute (Nova): Won't Fix Bug description: If the BMC / IPMI credentials being used for management of hardware were only granted OPERATOR privileges, there is no way to inform Nova's baremetal driver or Ironic's ipmitool driver to use this non- default privilege level. These will issue ipmitool commands with no -L parameter, resulting in privilege errors, because the default ipmitool privlvl is ADMINISTRATOR. This could be fixed by adding an optional field to store the privilege level. To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1263790/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1373671] [NEW] SSH driver does not work with non-english locale
Public bug reported:
The SSH driver's vmware and virsh command sets contain a grep clause
which depends upon localized strings, causing this driver to be unusable
on systems with a non-english locale setting.
For vmware:
136 'list_running': (
137 vmsvc/power.getstate {_NodeName_} |
138 grep 'Powered on' /dev/null
139 echo '\{_NodeName_}\' || true),
For virsh:
159 'list_running': (list --all|grep running |
160 awk -v qc='\' -F\ \ '{print qc$2qc}'),
Below is the output when the system locale is changed to de_DE. The
translation of the unexpected error is Error: Domain is already active.
However, any attempt to turn the node off is a no-op, because the SSH driver
does not believe the node is running.
2014-09-24 16:55:14.587 DEBUG ironic.drivers.modules.ssh [-] Checking Node:
baremetalbrbm_2's Mac address. from (pid=19417) _get_hosts_name_for_node
/opt/stack/ironic/ironic/drivers/modules/ssh.py:404
2014-09-24 16:55:14.684 DEBUG ironic.drivers.modules.ssh [-] Found Mac address:
52:54:00:de:30:e3 from (pid=19417) _get_hosts_name_for_node
/opt/stack/ironic/ironic/drivers/modules/ssh.py:417
2014-09-24 16:55:14.782 DEBUG ironic.drivers.modules.ssh [-] Cannot execute SSH
cmd /usr/bin/virsh --connect qemu:///system start baremetalbrbm_2. Reason:
Unexpected error while running command.
Command: /usr/bin/virsh --connect qemu:///system start baremetalbrbm_2
Exit code: 1
Stdout: '\n'
Stderr: 'Fehler: Domain ist bereits aktiv\n'. from (pid=19417) _ssh_execute
/opt/stack/ironic/ironic/drivers/modules/ssh.py:277
2014-09-24 16:55:14.794 WARNING ironic.conductor.manager [-] Error in deploy of
node 20770fde-b719-413d-9170-b257a76064a7: Failed to execute command via SSH:
/usr/bin/virsh --connect qemu:///system start baremetalbrbm_2.
Traceback (most recent call last):
File /usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py, line 455,
in fire_timers
timer()
File /usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py, line
58, in __call__
cb(*args, **kw)
File /usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py, line
212, in main
result = function(*args, **kwargs)
File /opt/stack/ironic/ironic/conductor/manager.py, line 504, in
_do_node_deploy
node.target_provision_state = states.NOSTATE
File /usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py, line
82, in __exit__
six.reraise(self.type_, self.value, self.tb)
File /opt/stack/ironic/ironic/conductor/manager.py, line 497, in
_do_node_deploy
new_state = task.driver.deploy.deploy(task)
File /opt/stack/ironic/ironic/conductor/task_manager.py, line 116, in
wrapper
return f(*args, **kwargs)
File /opt/stack/ironic/ironic/drivers/modules/pxe.py, line 344, in deploy
manager_utils.node_power_action(task, states.REBOOT)
File /opt/stack/ironic/ironic/conductor/task_manager.py, line 116, in
wrapper
return f(*args, **kwargs)
File /opt/stack/ironic/ironic/conductor/utils.py, line 118, in
node_power_action
'target': new_state, 'error': e}
File /usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py, line
82, in __exit__
six.reraise(self.type_, self.value, self.tb)
File /opt/stack/ironic/ironic/conductor/utils.py, line 112, in
node_power_action
task.driver.power.reboot(task)
File /opt/stack/ironic/ironic/conductor/task_manager.py, line 116, in
wrapper
return f(*args, **kwargs)
File /opt/stack/ironic/ironic/drivers/modules/ssh.py, line 586, in reboot
state = _power_on(ssh_obj, driver_info)
File /opt/stack/ironic/ironic/drivers/modules/ssh.py, line 446, in _power_on
_ssh_execute(ssh_obj, cmd_to_power_on)
File /opt/stack/ironic/ironic/drivers/modules/ssh.py, line 278, in
_ssh_execute
raise exception.SSHCommandFailed(cmd=cmd_to_exec)
SSHCommandFailed: Failed to execute command via SSH: /usr/bin/virsh --connect
qemu:///system start baremetalbrbm_2.
** Affects: ironic
Importance: High
Status: Confirmed
** Project changed: nova = ironic
** Changed in: ironic
Status: New = Confirmed
** Changed in: ironic
Importance: Undecided = High
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1373671
Title:
SSH driver does not work with non-english locale
Status in OpenStack Bare Metal Provisioning Service (Ironic):
Confirmed
Bug description:
The SSH driver's vmware and virsh command sets contain a grep
clause which depends upon localized strings, causing this driver to be
unusable on systems with a non-english locale setting.
For vmware:
136 'list_running': (
137 vmsvc/power.getstate {_NodeName_} |
138 grep 'Powered on' /dev/null
139 echo '\{_NodeName_}\' || true),
For virsh:
159 'list_running': (list
[Yahoo-eng-team] [Bug 1310135] Re: Stopping an instance via the Nova API when using the Nova Ironic driver incorrectly reports powerstate
** Tags added: ironic ** Also affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1310135 Title: Stopping an instance via the Nova API when using the Nova Ironic driver incorrectly reports powerstate Status in OpenStack Bare Metal Provisioning Service (Ironic): Confirmed Status in OpenStack Compute (Nova): New Bug description: When using the Ironic Nova driver, a stopped server is still presented as Running even when the server is stopped. Checking via the Ironic API correctly shows the instance as powered down: stack@ironic:~/logs/screen$ nova list +--+-+++-+---+ | ID | Name| Status | Task State | Power State | Networks | +--+-+++-+---+ | 5b43d631-91e1-4384-9b87-93283b3ae958 | testing | ACTIVE | - | Running | private=10.1.0.10 | +--+-+++-+---+ stack@ironic:~/logs/screen$ nova stop 5b43d631-91e1-4384-9b87-93283b3ae958 stack@ironic:~/logs/screen$ nova list +--+-+-++-+---+ | ID | Name| Status | Task State | Power State | Networks | +--+-+-++-+---+ | 5b43d631-91e1-4384-9b87-93283b3ae958 | testing | SHUTOFF | - | Running | private=10.1.0.10 | +--+-+-++-+---+ stack@ironic:~/logs/screen$ ping 10.1.0.10 PING 10.1.0.10 (10.1.0.10) 56(84) bytes of data. From 172.24.4.2 icmp_seq=1 Destination Host Unreachable From 172.24.4.2 icmp_seq=5 Destination Host Unreachable From 172.24.4.2 icmp_seq=6 Destination Host Unreachable From 172.24.4.2 icmp_seq=7 Destination Host Unreachable From 172.24.4.2 icmp_seq=8 Destination Host Unreachable --- 10.1.0.10 ping statistics --- 9 packets transmitted, 0 received, +5 errors, 100% packet loss, time 8000ms stack@ironic:~/logs/screen$ ironic node-list +--+--+-++-+ | UUID | Instance UUID | Power State | Provisioning State | Maintenance | +--+--+-++-+ | 91e81c38-4dce-412b-8a1b-a914d28943e4 | 5b43d631-91e1-4384-9b87-93283b3ae958 | power off | active | False | +--+--+-++-+ To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1310135/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1362733] Re: Rebuilding a node in ERROR state should set status to REBUILD
** Changed in: ironic Status: Confirmed = Won't Fix -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1362733 Title: Rebuilding a node in ERROR state should set status to REBUILD Status in OpenStack Bare Metal Provisioning Service (Ironic): Won't Fix Status in OpenStack Compute (Nova): Confirmed Bug description: I recently had a few nova-driven ironic nodes fail to deploy, and resurrected them by issuing another nova rebuild. This worked quite nicely, but the Status stayed as ERROR, when I would have expected it to change back to REBUILD To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1362733/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1313779] Re: ResourceTracker auditing the wrong amount of free resources for Ironic
** Tags removed: nova-driver ** Changed in: ironic Status: Triaged = Won't Fix -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1313779 Title: ResourceTracker auditing the wrong amount of free resources for Ironic Status in OpenStack Bare Metal Provisioning Service (Ironic): Won't Fix Status in OpenStack Compute (Nova): Confirmed Bug description: I've two nodes avaiable in Ironic, they both have cpus=1, memory_mb=512, local_gb=10, cpu_arch=x86_64 but when you look at the audit logs it seems to be reporting the amount of resources of only one of the nodes: N-cpu: 2014-04-28 16:09:47.200 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 512 2014-04-28 16:09:47.200 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 10 2014-04-28 16:09:47.200 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 1 If I update the first of the nodes of the list and let's say double the ram, the audit will report it: N-cpu: 2014-04-28 16:11:26.885 AUDIT nova.compute.resource_tracker [req-8a8a5d53-8cf1-4b9e-9420-5f0e3a6f9b27 None None] Free ram (MB): 1024 But if I update the second node, no changes are reported back to the resource tracker... ... Worst, if I delete the properties from the first node, now the Resource Tracker will report: $ ironic node-update $NODE remove properties N-cpu: 2014-04-28 16:13:07.735 AUDIT nova.compute.resource_tracker [req-c3211bd1-768d-40ea-b2cf-6e73c69e39b1 None None] Free ram (MB): 0 2014-04-28 16:13:07.735 AUDIT nova.compute.resource_tracker [req-c3211bd1-768d-40ea-b2cf-6e73c69e39b1 None None] Free disk (GB): 0 2014-04-28 16:13:07.735 AUDIT nova.compute.resource_tracker [req-c3211bd1-768d-40ea-b2cf-6e73c69e39b1 None None] Free VCPU information unavailable UPD from comment: We need to change Nova to understand the Ironic use case better. For nova each n-cpu is responsable for managing a X number of machines, but when the Ironic driver is loaded the n-cpu is just a small thin layer that talks to the Ironic api, and every n-cpu is managing _all_ the machines in the cluster. So in the Ironic use case different n-cpus would share the same machines and that's what making nova confused when auditing the resources. To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1313779/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1313779] Re: ResourceTracker auditing the wrong amount of free resources for Ironic
Hi Lucas, I have checked this on a local devstack install with 3 ironic nodes, and I could not reproduce your results. n-cpu is logging the available resources for each node individually, and upon updating a node, I see the change to that node in n-cpu's log at the next periodic interval. Updates to additional nodes also are logged appropraitely for each additional node. As such, I'm closing this bug. Please re-open and add more details if you can reproduce with the current code. ** Changed in: nova Status: Confirmed = Won't Fix ** Changed in: nova Status: Won't Fix = Fix Committed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1313779 Title: ResourceTracker auditing the wrong amount of free resources for Ironic Status in OpenStack Bare Metal Provisioning Service (Ironic): Won't Fix Status in OpenStack Compute (Nova): Fix Committed Bug description: I've two nodes avaiable in Ironic, they both have cpus=1, memory_mb=512, local_gb=10, cpu_arch=x86_64 but when you look at the audit logs it seems to be reporting the amount of resources of only one of the nodes: N-cpu: 2014-04-28 16:09:47.200 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 512 2014-04-28 16:09:47.200 AUDIT nova.compute.resource_tracker [-] Free disk (GB): 10 2014-04-28 16:09:47.200 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 1 If I update the first of the nodes of the list and let's say double the ram, the audit will report it: N-cpu: 2014-04-28 16:11:26.885 AUDIT nova.compute.resource_tracker [req-8a8a5d53-8cf1-4b9e-9420-5f0e3a6f9b27 None None] Free ram (MB): 1024 But if I update the second node, no changes are reported back to the resource tracker... ... Worst, if I delete the properties from the first node, now the Resource Tracker will report: $ ironic node-update $NODE remove properties N-cpu: 2014-04-28 16:13:07.735 AUDIT nova.compute.resource_tracker [req-c3211bd1-768d-40ea-b2cf-6e73c69e39b1 None None] Free ram (MB): 0 2014-04-28 16:13:07.735 AUDIT nova.compute.resource_tracker [req-c3211bd1-768d-40ea-b2cf-6e73c69e39b1 None None] Free disk (GB): 0 2014-04-28 16:13:07.735 AUDIT nova.compute.resource_tracker [req-c3211bd1-768d-40ea-b2cf-6e73c69e39b1 None None] Free VCPU information unavailable UPD from comment: We need to change Nova to understand the Ironic use case better. For nova each n-cpu is responsable for managing a X number of machines, but when the Ironic driver is loaded the n-cpu is just a small thin layer that talks to the Ironic api, and every n-cpu is managing _all_ the machines in the cluster. So in the Ironic use case different n-cpus would share the same machines and that's what making nova confused when auditing the resources. To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1313779/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1369317] [NEW] ironic_host_manager's consume_from_instance() takes exactly 2 arguments (3 given)
Public bug reported: https://review.openstack.org/#/c/118391/ Change-id I012ee5c118265e044ff41fb58b732728946ee85a included a change to the definition of nova/scheduler/host_manager.py HostState.consume_from_instance(), changing the method from requiring 2 to 3 parameters. This change did not update the ironic_host_manager's HostState class, thus breaking it. This has broken Ironic's tempest tests, resulting in all our CI tests failing. ** Affects: nova Importance: Undecided Assignee: Devananda van der Veen (devananda) Status: New ** Tags: ironic ** Tags added: ironic ** Description changed: - Change-id I012ee5c118265e044ff41fb58b732728946ee85a included a change - to the definition of nova/scheduler/host_manager.py + https://review.openstack.org/#/c/118391/ + Change-id I012ee5c118265e044ff41fb58b732728946ee85a + + included a change to the definition of nova/scheduler/host_manager.py HostState.consume_from_instance(), changing the method from requiring 2 to 3 parameters. This change did not update the ironic_host_manager's HostState class, thus breaking it. This has broken Ironic's tempest tests, resulting in all our CI tests failing. ** Changed in: nova Assignee: (unassigned) = Devananda van der Veen (devananda) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1369317 Title: ironic_host_manager's consume_from_instance() takes exactly 2 arguments (3 given) Status in OpenStack Compute (Nova): New Bug description: https://review.openstack.org/#/c/118391/ Change-id I012ee5c118265e044ff41fb58b732728946ee85a included a change to the definition of nova/scheduler/host_manager.py HostState.consume_from_instance(), changing the method from requiring 2 to 3 parameters. This change did not update the ironic_host_manager's HostState class, thus breaking it. This has broken Ironic's tempest tests, resulting in all our CI tests failing. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1369317/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1367007] [NEW] Ironic driver requires extra_specs
Public bug reported: Comments on review https://review.openstack.org/#/c/111429/ suggested that the Ironic driver should use flavor = instance.get_flavor() instead of flavor = flavor_obj.Flavor.get_by_id(context, instance['instance_type_id']) During the crunch to land things before feature freeze, these were integrated in the proposal to the Nova tree prior to being landed in the Ironic tree (the only place where they would have been tested). These changes actually broke the driver, since it requires access to flavor['extra_specs'] -- which is not present in the instance's cached copy of the flavor. This problem was discovered when attempting to update the devstack config and begin testing with the driver from the Nova tree (rather than the copy of the driver in the Ironic tree). That patch is here: https://review.openstack.org/#/c/119844/ The error being encountered can be seen both on the devstack patch (eg, in the Nova code) http://logs.openstack.org/44/119844/2/check/check-tempest-dsvm-virtual- ironic-nv/ce443f8/logs/screen-n-cpu.txt.gz and in the back-port of the same code to Ironic here: http://logs.openstack.org/65/119165/3/check/check-tempest-dsvm-virtual- ironic/c161a89/logs/screen-n-cpu.txt.gz#_2014-09-08_08_41_06_821 == Proposed fix == Fetch flavor['extra_specs'] on demand, when needed by the Ironic driver. ** Affects: nova Importance: Undecided Assignee: Devananda van der Veen (devananda) Status: New ** Changed in: nova Assignee: (unassigned) = Devananda van der Veen (devananda) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1367007 Title: Ironic driver requires extra_specs Status in OpenStack Compute (Nova): New Bug description: Comments on review https://review.openstack.org/#/c/111429/ suggested that the Ironic driver should use flavor = instance.get_flavor() instead of flavor = flavor_obj.Flavor.get_by_id(context, instance['instance_type_id']) During the crunch to land things before feature freeze, these were integrated in the proposal to the Nova tree prior to being landed in the Ironic tree (the only place where they would have been tested). These changes actually broke the driver, since it requires access to flavor['extra_specs'] -- which is not present in the instance's cached copy of the flavor. This problem was discovered when attempting to update the devstack config and begin testing with the driver from the Nova tree (rather than the copy of the driver in the Ironic tree). That patch is here: https://review.openstack.org/#/c/119844/ The error being encountered can be seen both on the devstack patch (eg, in the Nova code) http://logs.openstack.org/44/119844/2/check/check-tempest-dsvm- virtual-ironic-nv/ce443f8/logs/screen-n-cpu.txt.gz and in the back-port of the same code to Ironic here: http://logs.openstack.org/65/119165/3/check/check-tempest-dsvm- virtual- ironic/c161a89/logs/screen-n-cpu.txt.gz#_2014-09-08_08_41_06_821 == Proposed fix == Fetch flavor['extra_specs'] on demand, when needed by the Ironic driver. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1367007/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1347795] Re: All baremetal instance going to ERROR state
** Also affects: ironic Importance: Undecided Status: New ** Changed in: ironic Status: New = In Progress ** Changed in: ironic Importance: Undecided = Critical -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1347795 Title: All baremetal instance going to ERROR state Status in OpenStack Bare Metal Provisioning Service (Ironic): In Progress Status in OpenStack Compute (Nova): In Progress Status in tripleo - openstack on openstack: Triaged Bug description: As of 1300 UTC approce all tripleo CI is failing to bring up instances looks like the commit that caused it is https://review.openstack.org/#/c/71557/ just waiting for some CI to finish to confirm. To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1347795/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1342274] Re: auth_token middleware in keystoneclient is deprecated
** Also affects: ironic Importance: Undecided Status: New ** Changed in: ironic Assignee: (unassigned) = Devananda van der Veen (devananda) ** Changed in: ironic Status: New = In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1342274 Title: auth_token middleware in keystoneclient is deprecated Status in OpenStack Telemetry (Ceilometer): In Progress Status in Cinder: In Progress Status in OpenStack Image Registry and Delivery Service (Glance): In Progress Status in Orchestration API (Heat): In Progress Status in OpenStack Bare Metal Provisioning Service (Ironic): In Progress Status in OpenStack Neutron (virtual network service): Fix Committed Status in OpenStack Compute (Nova): Fix Committed Status in Python client library for Keystone: In Progress Status in OpenStack Data Processing (Sahara, ex. Savanna): Triaged Bug description: The auth_token middleware in keystoneclient is deprecated and will only get security updates. Projects should use the auth_token middleware in keystonemiddleware. To manage notifications about this bug go to: https://bugs.launchpad.net/ceilometer/+bug/1342274/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1328997] Re: Unit test failure: openstack_citest is being accessed by other users\nDETAIL: There are 1 other session(s) using the database.
Adding Ironic since we inherited the db migration test code from Nova, and while it turns out that we're not testing db migrations in the gate due to a misconfiguration, if we were to enable them, we'd face the same race conditions in our unit tests. ** Tags added: db ** Also affects: ironic Importance: Undecided Status: New ** Changed in: ironic Importance: Undecided = High ** Changed in: ironic Status: New = Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1328997 Title: Unit test failure: openstack_citest is being accessed by other users\nDETAIL: There are 1 other session(s) using the database. Status in OpenStack Bare Metal Provisioning Service (Ironic): Confirmed Status in OpenStack Compute (Nova): Confirmed Bug description: We are periodically seeing this nova unit test failure in our CI system: openstack_citest is being accessed by other users\nDETAIL: There are 1 other session(s) using the database. http://logs.openstack.org/76/98376/1/gate/gate-nova- python27/d2a0593/console.html 2014-06-11 06:26:40.002 | FAIL: nova.tests.db.test_migrations.TestNovaMigrations.test_postgresql_opportunistically 2014-06-11 06:26:40.002 | tags: worker-6 2014-06-11 06:26:40.003 | -- 2014-06-11 06:26:40.003 | Empty attachments: 2014-06-11 06:26:40.003 | pythonlogging:'' 2014-06-11 06:26:40.003 | stderr 2014-06-11 06:26:40.003 | stdout 2014-06-11 06:26:40.003 | 2014-06-11 06:26:40.004 | Traceback (most recent call last): 2014-06-11 06:26:40.004 | File nova/tests/db/test_migrations.py, line 139, in test_postgresql_opportunistically 2014-06-11 06:26:40.004 | self._test_postgresql_opportunistically() 2014-06-11 06:26:40.004 | File nova/tests/db/test_migrations.py, line 428, in _test_postgresql_opportunistically 2014-06-11 06:26:40.004 | self._reset_database(database) 2014-06-11 06:26:40.004 | File nova/tests/db/test_migrations.py, line 335, in _reset_database 2014-06-11 06:26:40.004 | self._reset_pg(conn_pieces) 2014-06-11 06:26:40.005 | File nova/openstack/common/lockutils.py, line 249, in inner 2014-06-11 06:26:40.005 | return f(*args, **kwargs) 2014-06-11 06:26:40.005 | File nova/tests/db/test_migrations.py, line 244, in _reset_pg 2014-06-11 06:26:40.005 | self.execute_cmd(droptable) 2014-06-11 06:26:40.005 | File nova/tests/db/test_migrations.py, line 227, in execute_cmd 2014-06-11 06:26:40.005 | Failed to run: %s\n%s % (cmd, output)) 2014-06-11 06:26:40.005 | File /home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py, line 321, in assertEqual 2014-06-11 06:26:40.006 | self.assertThat(observed, matcher, message) 2014-06-11 06:26:40.006 | File /home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/testtools/testcase.py, line 406, in assertThat 2014-06-11 06:26:40.006 | raise mismatch_error 2014-06-11 06:26:40.006 | MismatchError: !=: 2014-06-11 06:26:40.006 | reference = '' 2014-06-11 06:26:40.006 | actual= '''\ 2014-06-11 06:26:40.007 | Unexpected error while running command. 2014-06-11 06:26:40.007 | Command: psql -w -U openstack_citest -h localhost -c 'drop database if exists openstack_citest;' -d template1 2014-06-11 06:26:40.007 | Exit code: 1 2014-06-11 06:26:40.007 | Stdout: '' 2014-06-11 06:26:40.007 | Stderr: 'ERROR: database openstack_citest is being accessed by other users\\nDETAIL: There are 1 other session(s) using the database.\\n\ 2014-06-11 06:26:40.007 | : Failed to run: psql -w -U openstack_citest -h localhost -c 'drop database if exists openstack_citest;' -d template1 elastic-search query: message:Stderr: \'ERROR: database \openstack_citest\ is being accessed by other users\\nDETAIL: There are 1 other session(s) using the database.\\n\' AND project:openstack/nova To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1328997/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1301036] Re: openstack.common.db.sqlalchemy.migration utf8 table check issue on initial migration
Does not affect Ironic at the moment. ironic.cmd.dbsync imports ironic.db.migration, which is lazy-loading ironic.db.sqlalchemy.migration, which is importing alembic.migration directly. We'll need to resync oslo.openstack.common.db.migration to get the fix for this, but presumably we'll do that before switching to use it anyway. ** Changed in: ironic Status: New = Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Keystone. https://bugs.launchpad.net/bugs/1301036 Title: openstack.common.db.sqlalchemy.migration utf8 table check issue on initial migration Status in Ironic (Bare Metal Provisioning): Invalid Status in OpenStack Identity (Keystone): New Status in Oslo - a Library of Common OpenStack Code: New Bug description: The code in openstack.common.db.sqlachemy.migration that does the sanity checking on the UTF8 table is a bit overzealous and checks the migration_version (and alembic equivalent) table for utf8 status. This will cause migrations to fail in any case where the db isn't forcing a default character set of utf8, the db engine isn't forced to utf8, or the migration_version table isn't fixed before migrations occur. This was duplicated by doing a clean Ubuntu 12.04 install with mysql, using the default latin1 character set and simply creating the DB with ``create database keystone;`` The result is migrations fail at migration 0 unless the db sanity check is disabled (e.g. glance). root@precise64:~/keystone# keystone-manage --config-file /etc/keystone.conf db_sync 2014-04-01 14:03:23.858 19840 CRITICAL keystone [-] ValueError: Tables migrate_version have non utf8 collation, please make sure all tables are CHARSET=utf8 This is unaffected by the character set in the connection string. The solution is to explicitly ignore the migrate_version (and alembic equivalent) table in the sanity check. Code in question is here: http://git.openstack.org/cgit/openstack/keystone/tree/keystone/openstack/common/db/sqlalchemy/migration.py?id=51772e1addf8ab6f7483df44b7243fd7842eba4a#n200 This will affect any project using this code for migrations when using the mysql engine. To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1301036/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1251525] Re: baremetal deploy helper should abort immediately if target block devices are not available
** Changed in: ironic Status: In Progress = Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1251525 Title: baremetal deploy helper should abort immediately if target block devices are not available Status in Ironic (Bare Metal Provisioning): Fix Released Status in OpenStack Compute (Nova): Fix Released Bug description: nova-baremetal-deploy-helper does not stop a deployment even if a target block device or partitions are not available. In such case, the instance will fail to boot but nova will mark the instance as ACTIVE. So the user can know the failure only by the fact that the instance does not become accessible after a long time has passed. The deployment should be stopped immediately and the instance should be marked as ERROR. To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1251525/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1178103] Re: can't disable file injection for bare metal
Marking as Invalid for Ironic -- file injection is not implemented in the PXE driver and thus doesn't need to be disabled. ** Changed in: ironic Status: Triaged = Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1178103 Title: can't disable file injection for bare metal Status in Ironic (Bare Metal Provisioning): Invalid Status in OpenStack Compute (Nova): Fix Released Status in tripleo - openstack on openstack: Fix Released Bug description: For two reasons : a) until we have quantum-pxe done, it won't work, and b) file injection always happens. One of the reasons to want to disable file injection is to work with hardware that gets a ethernet interface other than 'eth0' - e.g. if only eth1 is plugged in on the hardware, file injection with it's hardcoded parameters interferes with network bringup. A workaround for homogeneous environments is to change the template to hardcode the interface name (s/iface.name/eth2/) To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1178103/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1275301] [NEW] Error during file injection if libguestfs is installed
Public bug reported:
Since devstack patch 43d95084376913 landed, which installed and enabled
libguestfs on Ubuntu, several tempest tests in the ironic and neutron
pipelines have been failing with Error mounting
/opt/stack/data/nova/instances/{UUID}/disk with libguestfs. This seems
to be happening because Nova is trying to use file injection with
libguestfs instead of nbd now.
Traceback:
http://paste.openstack.org/show/62297/
Code:
https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n2564
Links to a few failures:
http://logs.openstack.org/96/66796/10/check/check-tempest-dsvm-ironic-nv/88c3fff/logs/screen-n-cpu.txt.gz?level=TRACE
http://logs.openstack.org/98/69798/2/check/check-tempest-dsvm-ironic-nv/103f6d8/logs/screen-n-cpu.txt.gz?level=TRACE
** Affects: nova
Importance: Undecided
Status: New
** Summary changed:
- Error mounting libguestfs
+ Error during file injection if libguestfs is installed
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1275301
Title:
Error during file injection if libguestfs is installed
Status in OpenStack Compute (Nova):
New
Bug description:
Since devstack patch 43d95084376913 landed, which installed and
enabled libguestfs on Ubuntu, several tempest tests in the ironic and
neutron pipelines have been failing with Error mounting
/opt/stack/data/nova/instances/{UUID}/disk with libguestfs. This
seems to be happening because Nova is trying to use file injection
with libguestfs instead of nbd now.
Traceback:
http://paste.openstack.org/show/62297/
Code:
https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n2564
Links to a few failures:
http://logs.openstack.org/96/66796/10/check/check-tempest-dsvm-ironic-nv/88c3fff/logs/screen-n-cpu.txt.gz?level=TRACE
http://logs.openstack.org/98/69798/2/check/check-tempest-dsvm-ironic-nv/103f6d8/logs/screen-n-cpu.txt.gz?level=TRACE
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1275301/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1273455] Re: stevedore 0.14 changes _load_plugins parameter list, mocking breaks
Fix proposed to Ironic, but tagged with the duplicate bug and already in our gate pipe. Review: https://review.openstack.org/69495 ** Also affects: ironic Importance: Undecided Status: New ** Changed in: ironic Status: New = In Progress ** Changed in: ironic Importance: Undecided = Critical ** Changed in: ironic Assignee: (unassigned) = Devananda van der Veen (devananda) ** Changed in: ironic Milestone: None = icehouse-3 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1273455 Title: stevedore 0.14 changes _load_plugins parameter list, mocking breaks Status in OpenStack Telemetry (Ceilometer): In Progress Status in Ironic (Bare Metal Provisioning): In Progress Status in OpenStack Compute (Nova): Fix Committed Status in OpenStack Compute (nova) grizzly series: In Progress Status in OpenStack Compute (nova) havana series: Fix Committed Status in Messaging API for OpenStack: In Progress Status in Manage plugins for Python applications: Fix Released Bug description: In stevedore 0.14 the signature on _load_plugins changed. It now takes an extra parameter. The nova and ceilometer unit tests mocked to the old signature, which is causing breaks in the gate. To manage notifications about this bug go to: https://bugs.launchpad.net/ceilometer/+bug/1273455/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1208638] Re: baremetal PXE timeout interrupts active deploys
This doesn't affect Ironic as we don't (currently) have a timeout mechanism for deploys (which is another issue unto itself) and our state tracking is different than Nova's, so once we add operation-timeouts at a higher level, it'll be accounted for. ** Changed in: ironic Status: In Progress = Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1208638 Title: baremetal PXE timeout interrupts active deploys Status in Ironic (Bare Metal Provisioning): Invalid Status in OpenStack Compute (Nova): Triaged Bug description: When the DD of an image takes an unexpectedly long time (e.g. due to network congestion), the PXE deploy timeout may interrupt the deploy by powering off the node, which then causes it to be rescheduled and exacerbates the problem. If we monitor dd and check it is making progress, we could use this as a heartbeat to prevent inappropriate interrupts - and have the timeout look for a period of no progress (vs just absolute time). To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1208638/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1263790] [NEW] ipmitool does not support OPERATOR priv level
Public bug reported: If the BMC / IPMI credentials being used for management of hardware were only granted OPERATOR privileges, there is no way to inform Nova's baremetal driver or Ironic's ipmitool driver to use this non-default privilege level. These will issue ipmitool commands with no -L parameter, resulting in privilege errors, because the default ipmitool privlvl is ADMINISTRATOR. This could be fixed by adding an optional field to store the privilege level. ** Affects: ironic Importance: Medium Status: Triaged ** Affects: nova Importance: Medium Status: Triaged ** Tags: baremetal ** Changed in: ironic Status: New = Triaged ** Changed in: ironic Importance: Undecided = Medium ** Also affects: nova Importance: Undecided Status: New ** Changed in: nova Status: New = Triaged ** Changed in: nova Importance: Undecided = Medium ** Tags added: baremetal -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1263790 Title: ipmitool does not support OPERATOR priv level Status in Ironic (Bare Metal Provisioning): Triaged Status in OpenStack Compute (Nova): Triaged Bug description: If the BMC / IPMI credentials being used for management of hardware were only granted OPERATOR privileges, there is no way to inform Nova's baremetal driver or Ironic's ipmitool driver to use this non- default privilege level. These will issue ipmitool commands with no -L parameter, resulting in privilege errors, because the default ipmitool privlvl is ADMINISTRATOR. This could be fixed by adding an optional field to store the privilege level. To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1263790/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1260102] [NEW] Removal of a baremetal node does not propagate to scheduler quickly
Public bug reported: With the baremetal driver, if a baremetal node is deleted, it is not removed from pool of available resources until the next run of update_available_resource(). During this window, the scheduler may continue to attempt to schedule instances on it, leading to unnecessary failures and scheduling retries. This bug will also apply to the ironic driver, which will rely on the same mechanism(s) for listing and scheduling compute resources. ** Affects: nova Importance: Medium Status: Triaged ** Tags: baremetal ironic scheduler -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1260102 Title: Removal of a baremetal node does not propagate to scheduler quickly Status in OpenStack Compute (Nova): Triaged Bug description: With the baremetal driver, if a baremetal node is deleted, it is not removed from pool of available resources until the next run of update_available_resource(). During this window, the scheduler may continue to attempt to schedule instances on it, leading to unnecessary failures and scheduling retries. This bug will also apply to the ironic driver, which will rely on the same mechanism(s) for listing and scheduling compute resources. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1260102/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1238595] Re: baremetal allows to register an interface having an empty mac address
I believe this does not affect Ironic, as there is validation of MAC addresses done within the db/api layer, which was not done by Nova. I will re-open if it is demonstrated that one can create an invalid or empty mac address in Ironic, leading to similar failures. ** Changed in: ironic Status: New = Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1238595 Title: baremetal allows to register an interface having an empty mac address Status in Ironic (Bare Metal Provisioning): Invalid Status in OpenStack Compute (Nova): In Progress Bug description: It leads a network manager to create an invalid entry in dnsmasq's hostsfile, like: ,bm.novalocal,10.0.0.2 (The first column is a mac address. In this case, the instance never receive 10.0.0.2) In addition, once such a interface is added, we cannot delete it. $ nova baremetal-interface-remove 1 '' ERROR: Must specify id or address (HTTP 400) (Request-ID: req-d0631fc4-361d-43fa-870a-82e077cd454b) To manage notifications about this bug go to: https://bugs.launchpad.net/ironic/+bug/1238595/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1155323] Re: bm_pxe_ips table is not getting populated - Grizzly G3
for reference: https://bugs.launchpad.net/nova/+bug/1156745 ** Changed in: nova Status: New = Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1155323 Title: bm_pxe_ips table is not getting populated - Grizzly G3 Status in OpenStack Compute (Nova): Invalid Bug description: Using Grizzly G3. After provisioning a baremetal with an image, bm_pxe_ips table is not getting populated. This table remains empty. Only way to know which mac addresse got which ip address is by looking at /var/lib/misc/dnsmasq.leases To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1155323/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1142846] Re: Nova baremetal compute does not start when IPTables firewalll is used
This is correct -- firewall does not work with baremetal, and really
doesn't make sense. The wiki (updated just recently) now mentions that
you must use NoopFirewallDriver with baremetal driver.
** Tags added: baremetal
** Changed in: nova
Status: Incomplete = Won't Fix
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1142846
Title:
Nova baremetal compute does not start when IPTables firewalll is used
Status in OpenStack Compute (Nova):
Won't Fix
Bug description:
When nova compute is configured with IPTables firewall, nova-compute
fails to start
[Use firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
fails. Changing it to firewall_driver :
nova.virt.firewall.NoopFirewallDriver works ]
root# /usr/local/bin/nova-compute --debug
2013-02-11 15:38:30 19935 INFO nova.virt.driver [-] Loading compute driver
'nova.virt.baremetal.driver.BareMetalDriver'
2013-02-11 15:38:30 19935 INFO nova.manager [-] Skipping periodic task
_periodic_update_dns because its interval is negative
2013-02-11 15:38:30 19935 CRITICAL nova [-] __init__() takes at least 2
arguments (1 given)
2013-02-11 15:38:30 19935 TRACE nova Traceback (most recent call last):
2013-02-11 15:38:30 19935 TRACE nova File /usr/local/bin/nova-compute,
line 5, in module
2013-02-11 15:38:30 19935 TRACE nova
pkg_resources.run_script('nova==2013.1', 'nova-compute')
2013-02-11 15:38:30 19935 TRACE nova File
/usr/lib/python2.7/dist-packages/pkg_resources.py, line 499, in run_script
2013-02-11 15:38:30 19935 TRACE nova
self.require(requires)[0].run_script(script_name, ns)
2013-02-11 15:38:30 19935 TRACE nova File
/usr/lib/python2.7/dist-packages/pkg_resources.py, line 1235, in run_script
2013-02-11 15:38:30 19935 TRACE nova execfile(script_filename, namespace,
namespace)
2013-02-11 15:38:30 19935 TRACE nova File
/usr/local/lib/python2.7/dist-packages/nova-2013.1-py2.7.egg/EGG-INFO/scripts/nova-compute,
line 58, in module
2013-02-11 15:38:30 19935 TRACE nova topic=CONF.compute_topic)
2013-02-11 15:38:30 19935 TRACE nova File
/usr/local/lib/python2.7/dist-packages/nova-2013.1-py2.7.egg/nova/service.py,
line 511, in create
2013-02-11 15:38:30 19935 TRACE nova
periodic_interval_max=periodic_interval_max)
2013-02-11 15:38:30 19935 TRACE nova File
/usr/local/lib/python2.7/dist-packages/nova-2013.1-py2.7.egg/nova/service.py,
line 396, in __init__
2013-02-11 15:38:30 19935 TRACE nova self.manager =
manager_class(host=self.host, *args, **kwargs)
2013-02-11 15:38:30 19935 TRACE nova File
/usr/local/lib/python2.7/dist-packages/nova-2013.1-py2.7.egg/nova/compute/manager.py,
line 307, in __init__
2013-02-11 15:38:30 19935 TRACE nova self.driver =
driver.load_compute_driver(self.virtapi, compute_driver)
2013-02-11 15:38:30 19935 TRACE nova File
/usr/local/lib/python2.7/dist-packages/nova-2013.1-py2.7.egg/nova/virt/driver.py,
line 835, in load_compute_driver
2013-02-11 15:38:30 19935 TRACE nova virtapi)
2013-02-11 15:38:30 19935 TRACE nova File
/usr/local/lib/python2.7/dist-packages/nova-2013.1-py2.7.egg/nova/openstack/common/importutils.py,
line 53, in import_object_ns
2013-02-11 15:38:30 19935 TRACE nova return
import_class(import_str)(*args, **kwargs)
2013-02-11 15:38:30 19935 TRACE nova File
/usr/local/lib/python2.7/dist-packages/nova-2013.1-py2.7.egg/nova/virt/baremetal/driver.py,
line 130, in __init__
2013-02-11 15:38:30 19935 TRACE nova default=DEFAULT_FIREWALL_DRIVER)
2013-02-11 15:38:30 19935 TRACE nova File
/usr/local/lib/python2.7/dist-packages/nova-2013.1-py2.7.egg/nova/virt/firewall.py,
line 49, in load_driver
2013-02-11 15:38:30 19935 TRACE nova return fw_class(*args, **kwargs)
2013-02-11 15:38:30 19935 TRACE nova TypeError: __init__() takes at least 2
arguments (1 given)
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1142846/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1098694] Re: Baremetal driver does not set instance state to ERROR for all error cases
*** This bug is a duplicate of bug 1088655 *** https://bugs.launchpad.net/bugs/1088655 This was fixed in https://review.openstack.org/21564 ** This bug has been marked a duplicate of bug 1088655 bare metal node partitioning does not handle errors well -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1098694 Title: Baremetal driver does not set instance state to ERROR for all error cases Status in OpenStack Compute (Nova): New Bug description: Nova Baremetal driver does not always report an accurate state. Expected: Upon `nova boot`, nova should report the instance state as BUILD, transitioning to ACTIVE once instance is ready, or ERROR on failure. Actual: Upon `nova boot`, nova reports instance state as ACTIVE before instance is ready, and does not always set state to ERROR on failure. To Reproduce: - Set up nova/bin/nova-baremetal-deploy-helper to fail, e.g. by using an invalid disk image, or by raising an exception in method deploy - `nova boot` a new baremetal node - instance will be set to ACTIVE before its disk is copied over. - boot the baremetal node, causing nova-baremetal-deploy-helper to fail. - instance state remains ACTIVE after the error. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1098694/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
