** Changed in: manila (Ubuntu Groovy) Importance: Undecided => High ** Changed in: manila (Ubuntu Groovy) Status: New => Triaged
** Also affects: masakari Importance: Undecided Status: New ** Also affects: masakari (Ubuntu) Importance: Undecided Status: New ** Changed in: masakari (Ubuntu Focal) Importance: Undecided => High ** Changed in: masakari (Ubuntu Focal) Status: New => Triaged -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1863021 Title: [SRU] eventlet monkey patch results in assert len(_active) == 1 AssertionError Status in Cinder: In Progress Status in Designate: In Progress Status in Glance: Fix Released Status in OpenStack Shared File Systems Service (Manila): In Progress Status in masakari: In Progress Status in Mistral: New Status in BaGPipe: In Progress Status in networking-hyperv: In Progress Status in networking-l2gw: In Progress Status in Mellanox backend integration with Neutron (networking-mlnx): In Progress Status in networking-sfc: In Progress Status in neutron: In Progress Status in OpenStack Compute (nova): Fix Released Status in oslo.service: In Progress Status in OpenStack Object Storage (swift): In Progress Status in barbican package in Ubuntu: Triaged Status in cinder package in Ubuntu: Fix Released Status in designate package in Ubuntu: Triaged Status in glance package in Ubuntu: Fix Released Status in heat package in Ubuntu: Triaged Status in ironic package in Ubuntu: Triaged Status in ironic-inspector package in Ubuntu: Triaged Status in magnum package in Ubuntu: Triaged Status in manila package in Ubuntu: Triaged Status in masakari package in Ubuntu: Triaged Status in mistral package in Ubuntu: Triaged Status in networking-bagpipe package in Ubuntu: Triaged Status in networking-hyperv package in Ubuntu: Triaged Status in networking-l2gw package in Ubuntu: Triaged Status in networking-mlnx package in Ubuntu: Triaged Status in networking-sfc package in Ubuntu: New Status in neutron package in Ubuntu: Fix Released Status in nova package in Ubuntu: Fix Released Status in python-oslo.service package in Ubuntu: Triaged Status in swift package in Ubuntu: Triaged Status in barbican source package in Focal: Triaged Status in cinder source package in Focal: Triaged Status in designate source package in Focal: Triaged Status in glance source package in Focal: Fix Released Status in heat source package in Focal: Triaged Status in ironic source package in Focal: Triaged Status in ironic-inspector source package in Focal: Triaged Status in magnum source package in Focal: Triaged Status in manila source package in Focal: Triaged Status in masakari source package in Focal: Triaged Status in mistral source package in Focal: Triaged Status in networking-bagpipe source package in Focal: Triaged Status in networking-hyperv source package in Focal: Triaged Status in networking-l2gw source package in Focal: Triaged Status in networking-mlnx source package in Focal: Triaged Status in networking-sfc source package in Focal: Triaged Status in neutron source package in Focal: Triaged Status in nova source package in Focal: Fix Released Status in python-oslo.service source package in Focal: Triaged Status in swift source package in Focal: Triaged Status in barbican source package in Groovy: Triaged Status in cinder source package in Groovy: Fix Released Status in designate source package in Groovy: Triaged Status in glance source package in Groovy: Fix Released Status in heat source package in Groovy: Triaged Status in ironic source package in Groovy: Triaged Status in ironic-inspector source package in Groovy: Triaged Status in magnum source package in Groovy: Triaged Status in manila source package in Groovy: Triaged Status in masakari source package in Groovy: Triaged Status in mistral source package in Groovy: Triaged Status in networking-bagpipe source package in Groovy: Triaged Status in networking-hyperv source package in Groovy: Triaged Status in networking-l2gw source package in Groovy: Triaged Status in networking-mlnx source package in Groovy: Triaged Status in networking-sfc source package in Groovy: New Status in neutron source package in Groovy: Fix Released Status in nova source package in Groovy: Fix Released Status in python-oslo.service source package in Groovy: Triaged Status in swift source package in Groovy: Triaged Bug description: [Impact] This appears to be the same issue documented here: https://github.com/eventlet/eventlet/issues/592 However I seem to only hit this with python3.8. Basically nova, glance, neutron, and cinder services fail with: Exception ignored in: <function _after_fork at 0x7f1100d075e0> Traceback (most recent call last): File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork assert len(_active) == 1 AssertionError: Exception ignored in: <function _after_fork at 0x7f1100d075e0> Traceback (most recent call last): File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork assert len(_active) == 1 AssertionError: Patching nova/monkey_patch.py with the following appears to fix this: diff --git a/nova/monkey_patch.py b/nova/monkey_patch.py index a07ff91dac..bb7252c643 100644 --- a/nova/monkey_patch.py +++ b/nova/monkey_patch.py @@ -59,6 +59,9 @@ def _monkey_patch(): else: eventlet.monkey_patch() + import __original_module_threading + import threading + __original_module_threading.current_thread.__globals__['_active'] = threading._active # NOTE(rpodolyaka): import oslo_service first, so that it makes eventlet # hub use a monotonic clock to avoid issues with drifts of system time (see Similar patches to glance/cmd/api.py, glance/cmd/scrubber.py and glance/cmd/registry.py appears to fix it for glance. [Test Case] 'systemctl status <service>' should not show the failure reported in [Impact]. [Regression Potential] Should be low. The fix provided is a well-known solution to the problem. Openstack deployment + tempest regression testing will be performed to help reduce regression potential. To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1863021/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp