Public bug reported:
In 2025.1 (deployed with OpenStack-Ansible 31.0.0), when requesting the
user_data for an instance that is gzip-encoded, the client gets a 502
Bad Gateway response. This worked previously in our 2024.2 deployment
(and rolling back just the neutron-ovn-metadata-agent to 2024.2 fixes
the issue).
This is the error found in the neutron-ovn-metadata-agent logs:
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
[-] Error while receiving data.: UnicodeDecodeError: 'utf-8' codec can't decode
byte 0x8b in position 1: invalid start byte
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
Traceback (most recent call last):
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 144, in handle
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
res = self._proxy_request(instance_id, project_id, req)
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 103, in _proxy_request
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
return self._http_response(resp, req)
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 49, in _http_response
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
out = request.http_version + ' ' + str(_res)
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py",
line 403, in __str__
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
parts += ['', self.body if PY2 else self.text]
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py",
line 622, in _text__get
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
return body.decode(decoding, self.unicode_errors)
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid
start byte
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
----------------------------------------
Exception occurred during processing of request from
Traceback (most recent call last):
File "/usr/lib/python3.11/socketserver.py", line 691, in
process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python3.11/socketserver.py", line 361, in finish_request
self.RequestHandlerClass(request, client_address, self)
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 133, in __init__
super().__init__(self._conf, has_cache=False, request=request,
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/metadata/proxy_base.py",
line 48, in __init__
super().__init__(**kwargs)
File "/usr/lib/python3.11/socketserver.py", line 755, in __init__
self.handle()
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 165, in handle
raise exc
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 144, in handle
res = self._proxy_request(instance_id, project_id, req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 103, in _proxy_request
return self._http_response(resp, req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 49, in _http_response
out = request.http_version + ' ' + str(_res)
^^^^^^^^^
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py",
line 403, in __str__
parts += ['', self.body if PY2 else self.text]
^^^^^^^^^
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py",
line 622, in _text__get
return body.decode(decoding, self.unicode_errors)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid
start byte
----------------------------------------
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2120723
Title:
[OVN] Metadata agent errors when returning gzip-encoded data
Status in neutron:
New
Bug description:
In 2025.1 (deployed with OpenStack-Ansible 31.0.0), when requesting
the user_data for an instance that is gzip-encoded, the client gets a
502 Bad Gateway response. This worked previously in our 2024.2
deployment (and rolling back just the neutron-ovn-metadata-agent to
2024.2 fixes the issue).
This is the error found in the neutron-ovn-metadata-agent logs:
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
[-] Error while receiving data.: UnicodeDecodeError: 'utf-8' codec can't decode
byte 0x8b in position 1: invalid start byte
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
Traceback (most recent call last):
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 144, in handle
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
res = self._proxy_request(instance_id, project_id, req)
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 103, in _proxy_request
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
return self._http_response(resp, req)
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 49, in _http_response
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
out = request.http_version + ' ' + str(_res)
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py",
line 403, in __str__
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
parts += ['', self.body if PY2 else self.text]
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py",
line 622, in _text__get
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
return body.decode(decoding, self.unicode_errors)
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid
start byte
2025-08-15 14:20:07.225 657820 ERROR neutron.agent.ovn.metadata.server_socket
----------------------------------------
Exception occurred during processing of request from
Traceback (most recent call last):
File "/usr/lib/python3.11/socketserver.py", line 691, in
process_request_thread
self.finish_request(request, client_address)
File "/usr/lib/python3.11/socketserver.py", line 361, in finish_request
self.RequestHandlerClass(request, client_address, self)
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 133, in __init__
super().__init__(self._conf, has_cache=False, request=request,
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/metadata/proxy_base.py",
line 48, in __init__
super().__init__(**kwargs)
File "/usr/lib/python3.11/socketserver.py", line 755, in __init__
self.handle()
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 165, in handle
raise exc
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 144, in handle
res = self._proxy_request(instance_id, project_id, req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 103, in _proxy_request
return self._http_response(resp, req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/neutron/agent/ovn/metadata/server_socket.py",
line 49, in _http_response
out = request.http_version + ' ' + str(_res)
^^^^^^^^^
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py",
line 403, in __str__
parts += ['', self.body if PY2 else self.text]
^^^^^^^^^
File
"/openstack/venvs/neutron-31.0.0/lib/python3.11/site-packages/webob/response.py",
line 622, in _text__get
return body.decode(decoding, self.unicode_errors)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1:
invalid start byte
----------------------------------------
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2120723/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
