nova-no-nd-reflection is one of the static network filters in libvirt, which
only needs to be created once. In this case, there was a race condition between
2 instance creation workflows both of which tried to setup these static
filters, so the latter failed with 'nwfilter already exists' error.
The nova libvirt NWFilterFirewall driver has no locking around the
static_filters_configured flag used to determine if this one-time setup has
been done. Moreover, libvirt driver doesn't look at the exact error code in
exception being thrown during lookup; it should only create new filter if
lookup fails with error VIR_ERR_NO_NWFILTER = 62.
** Changed in: tripleo
Assignee: (unassigned) => Pooja Ghumre (pooja-9)
** Project changed: tripleo => nova
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1632041
Title:
Concurrent nova instance creation fails with nova-no-nd-reflection
filter already exists error
Status in OpenStack Compute (nova):
New
Bug description:
Immediately after starting nova-compute service on the host, if
concurrent instance creation operations are triggered on the same
host, both the workflows try to define the static network filters and
the latter one fails with 'nova-no-nd-reflection filter already
exists'.
Log snippet:
2016-08-07 05:40:57.628 17601 ERROR nova.scheduler.utils
[req-137df929-3b97-4437-b26d-3280dcbcd8e3 b5dc09cde3ac440d9060d6cc89f7ba71
87841dbf785d4ff3a43d5fd5c53ead50 - - -] [insta
nce: b8d573aa-1aae-4dec-a275-ae8c82f4a274] Error from last host:
dc5e6fb9-bcd1-4548-ad2e-4e571b783fe8 (node
cinder-glance-centos7-10-4-252-239.platform9.sys): [u'Traceback (most
recent call last):\n', u' File
"/opt/pf9/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2128,
in _do_build_and_run_instance\n filter_properties)\n', u' File
"/opt/pf9/venv/lib/python2.7/site-packages/nova/compute/manager.py", line
2284, in _build_and_run_instance\n instance_uuid=instance.uuid,
reason=six.text_type(e))\n', u"Resch
eduledException: Build of instance b8d573aa-1aae-4dec-a275-ae8c82f4a274 was
re-scheduled: operation failed: filter 'nova-no-nd-reflection' already exists
with uuid 60ba4736-64c7-
42c3-8efe-e6919dec44d2\n"]
2016-08-07 05:40:57.865 17601 WARNING nova.scheduler.utils
[req-137df929-3b97-4437-b26d-3280dcbcd8e3 b5dc09cde3ac440d9060d6cc89f7ba71
87841dbf785d4ff3a43d5fd5c53ead50 - - -] Fail
ed to compute_task_build_instances: No valid host was found.
Traceback (most recent call last):
File
"/opt/pf9/nova/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line
150, in inner
return func(*args, **kwargs)
File "/opt/pf9/nova/lib/python2.7/site-packages/nova/scheduler/manager.py",
line 84, in select_destinations
filter_properties)
File
"/opt/pf9/nova/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py",
line 96, in select_destinations
raise exception.NoValidHost(reason=msg)
NoValidHost: No valid host was found.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1632041/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
