Reviewed: https://review.openstack.org/292207
Committed:
https://git.openstack.org/cgit/openstack/neutron/commit/?id=dcb2a931b5b84fb7aa41f08b37a5148bf6e987bc
Submitter: Jenkins
Branch: master
commit dcb2a931b5b84fb7aa41f08b37a5148bf6e987bc
Author: Ryan Tidwell <ryan.tidw...@hpe.com>
Date: Fri Apr 8 14:21:03 2016 -0700
Compute IPAvailabilityRanges in memory during IP allocation
This patch computes IP availability in memory without locking on
IPAvailabilityRanges. IP availability is generated in memory, and
to avoid contention an IP address is selected by randomly
selecting from within the first 10 available IP addresses on a
subnet. Raises IPAddressGenerationFailure if unable to allocate an
IP address from within the window.
Change-Id: I52e4485e832cbe6798de6b4afb6a7cfd88db11e2
Depends-On: I84195b0eb63b7ca6a4e00becbe09e579ff8b718e
Closes-Bug: #1543094
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1543094
Title:
[Pluggable IPAM] DB exceeded retry limit (RetryRequest) on
create_router call
Status in neutron:
Fix Released
Bug description:
Observed errors "DB exceeded retry limit" [1] in cases where pluggable ipam
is enabled, observed on master branch.
Each time retest is done different tests are failed, so looks like concurency
issue.
4 errors 'DB exceeded retry limit' are observed in [1].
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api
[req-7ad8b69e-a851-4b6c-8c2c-33258c53bb54 admin -] DB exceeded retry limit.
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api Traceback (most recent call
last):
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 137, in wrapper
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api return f(*args, **kwargs)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/api/v2/base.py", line 519, in _create
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api obj = do_create(body)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/api/v2/base.py", line 501, in do_create
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api request.context,
reservation.reservation_id)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in
__exit__
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api six.reraise(self.type_,
self.value, self.tb)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/api/v2/base.py", line 494, in do_create
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api return
obj_creator(request.context, **kwargs)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/l3_hamode_db.py", line 411, in create_router
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api
self).create_router(context, router)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/l3_db.py", line 200, in create_router
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api
self.delete_router(context, router_db.id)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in
__exit__
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api six.reraise(self.type_,
self.value, self.tb)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/l3_db.py", line 196, in create_router
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api gw_info, router=router_db)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/l3_gwmode_db.py", line 69, in
_update_router_gw_info
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api context, router_id, info,
router=router)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/l3_db.py", line 429, in
_update_router_gw_info
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api ext_ips)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/l3_dvr_db.py", line 185, in _create_gw_port
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api ext_ips)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/l3_db.py", line 399, in _create_gw_port
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api new_network_id, ext_ips)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/l3_db.py", line 310, in
_create_router_gw_port
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api context.elevated(),
{'port': port_data})
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/plugins/common/utils.py", line 149, in
create_port
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api return
core_plugin.create_port(context, {'port': port_data})
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1069, in
create_port
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api result, mech_context =
self._create_port_db(context, port)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1045, in
_create_port_db
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api result = super(Ml2Plugin,
self).create_port(context, port)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1193, in
create_port
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api port_id)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 172, in
allocate_ips_for_port_and_store
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api revert_on_fail=False)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in
__exit__
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api six.reraise(self.type_,
self.value, self.tb)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 156, in
allocate_ips_for_port_and_store
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api ips =
self._allocate_ips_for_port(context, port)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 228, in
_allocate_ips_for_port
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api return
self._ipam_allocate_ips(context, ipam_driver, p, ips)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 135, in
_ipam_allocate_ips
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api "external system for
%s"), addresses)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in
__exit__
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api six.reraise(self.type_,
self.value, self.tb)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 120, in
_ipam_allocate_ips
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api context, ipam_driver,
port, ip_list)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 91, in
_ipam_allocate_single_ip
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api port, subnet),
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 80, in
_ipam_try_allocate_ip
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api return
ipam_subnet.allocate(ip_request)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line
350, in allocate
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api auto_generated)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line
219, in _allocate_specific_ip
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api session, db_range,
first_ip=first_ip, last_ip=last_ip)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api File
"/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line
167, in update_range
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api raise
db_exc.RetryRequest(ipam_exc.IPAllocationFailed)
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api RetryRequest
2016-02-04 11:55:59.944 15476 ERROR oslo_db.api
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource
[req-7ad8b69e-a851-4b6c-8c2c-33258c53bb54 admin -] create failed
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource Traceback (most
recent call last):
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource File
"/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource result =
method(request=request, **args)
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource File
"/opt/stack/new/neutron/neutron/api/v2/base.py", line 408, in create
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource return
self._create(request, body, **kwargs)
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource ectxt.value =
e.inner_exc
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in
__exit__
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource
six.reraise(self.type_, self.value, self.tb)
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource File
"<string>", line 2, in reraise
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource TypeError:
exceptions must be old-style classes or derived from BaseException, not type
2016-02-04 11:55:59.946 15476 ERROR neutron.api.v2.resource
[1]
http://logs.openstack.org/23/181023/54/check/gate-neutron-dsvm-api/054071e/logs/screen-q-svc.txt.gz#_2016-02-04_11_55_59_944
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1543094/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
