** Description changed:
[ Impact ]
heat-engine can permanently exhaust its database connection pool when stack
snapshots are listed or when a stack is deleted. Once the pool is exhausted,
every database call in heat-engine blocks for 30 seconds and then fails with
sqlalchemy.exc.TimeoutError. All subsequent Heat API requests return "ERROR:
Internal Error" until heat-engine is restarted.
snapshot_get_all_by_stack() in heat/db/api.py is decorated with
@context_manager.reader but returns an unexecuted SQLAlchemy Query object.
The decorator closes its session when the function returns, so the query
only runs later, when the caller iterates over it. At that point SQLAlchemy
silently begins a new transaction on the closed session and checks out a new
connection that nothing ever returns to the pool.
This affects releases in which heat-engine runs under eventlet. The pool
slots are only lost permanently when eventlet is in use. The connection is
only reclaimed when Python's garbage collector frees the session. When the
garbage collector happens to run inside the eventlet hub, resetting the
connection requires blocking socket I/O, which eventlet refuses with
"AssertionError: do not call blocking functions from the mainloop". The
connection is never returned and its pool slot is lost permanently. Under
concurrent load the pool is exhausted quickly.
The fix adds .all() so that the query runs inside the session scope and the
decorator returns the connection to the pool. This fix has already been
accepted upstream [0].
[ Test Plan ]
** Setup OpenStack Environment **
1. Deploy a test cloud in your reproducer bastion using stsstack-bundles:
- $ cd stsstack-bundles/openstack
- $ ./generate-bundle.sh --series <UBUNTU_RELEASE> --release
<OPENSTACK_RELEASE> --name heat-test --run
- # wait until all units are active/idle (some will be blocked/waiting)
- $ ./tools/vault-unseal-and-authorise.sh
- # wait until all units are active/idle
- $ ./configure
- $ source novarc
+ $ cd stsstack-bundles/openstack
+ $ ./generate-bundle.sh --series <UBUNTU_RELEASE> --release
<OPENSTACK_RELEASE> --name heat-test --run
+ # wait until all units are active/idle (some will be blocked/waiting)
+ $ ./tools/vault-unseal-and-authorise.sh
+ # wait until all units are active/idle
+ $ ./configure
+ $ source novarc
2. Reduce heat-engine to a single worker so that all requests share one
- connection pool:
- $ juju config heat worker-multiplier=0.5 # adjust based on your env
- # wait until heat/0 is active/idle
- $ juju ssh heat/0 "sudo grep num_engine_workers /etc/heat/heat.conf"
- num_engine_workers = 1
+ connection pool:
+ $ juju config heat worker-multiplier=0.5 # adjust based on your env
+ # wait until heat/0 is active/idle
+ $ juju ssh heat/0 "sudo grep num_engine_workers /etc/heat/heat.conf"
+ num_engine_workers = 1
** Verify the Bug **
3. Create a file test-heat.yaml with the following content:
- heat_template_version: 2015-04-30
- resources:
- test_resource:
- type: OS::Heat::TestResource
+ heat_template_version: 2015-04-30
+ resources:
+ test_resource:
+ type: OS::Heat::TestResource
4. Create a test stack and a snapshot:
- $ STACK_NAME="leak-test-stack"
- $ openstack stack create -t test-heat.yaml "$STACK_NAME" --wait
- $ openstack stack snapshot create "$STACK_NAME"
+ $ STACK_NAME="leak-test-stack"
+ $ openstack stack create -t test-heat.yaml "$STACK_NAME" --wait
+ $ openstack stack snapshot create "$STACK_NAME"
5. List the snapshots concurrently:
- $ seq 1 50 | xargs -P 20 -I {} openstack stack snapshot list
"$STACK_NAME" > /dev/null
+ $ seq 1 50 | xargs -P 20 -I {} openstack stack snapshot list
"$STACK_NAME" > /dev/null
6. Confirm the failure:
- Some of the requests above fail with:
- ERROR: Internal Error
+ Some of the requests above fail with:
+ ERROR: Internal Error
- After the load has finished, Heat remains unusable:
- $ openstack stack list
- ERROR: Internal Error
+ After the load has finished, Heat remains unusable:
+ $ openstack stack list
+ ERROR: Internal Error
- Verify the errors in the logs:
- $ juju ssh heat/0 "sudo grep 'do not call blocking functions'
/var/log/heat/heat-engine.log"
- $ juju ssh heat/0 "sudo grep -E 'Exception during reset|TimeoutError'
/var/log/heat/heat-engine.log | tail -n 5"
+ Verify the errors in the logs:
+ $ juju ssh heat/0 "sudo grep 'do not call blocking functions'
/var/log/heat/heat-engine.log"
+ $ juju ssh heat/0 "sudo grep -E 'Exception during reset|TimeoutError'
/var/log/heat/heat-engine.log | tail -n 5"
- Observed results:
- - "openstack stack list" keeps failing until heat-engine is restarted
- - Log shows a connection that could not be returned to the pool:
+ Observed results:
+ - "openstack stack list" keeps failing until heat-engine is restarted
+ - Log shows a connection that could not be returned to the pool:
2026-09-25 12:36:46.462 517667 ERROR sqlalchemy.pool.impl.QueuePool [-]
Exception during reset or similar: AssertionError: do not call blocking
functions from the mainloop
- - Followed by pool exhaustion:
+ - Followed by pool exhaustion:
2026-09-25 12:53:47.028 517667 ERROR heat.engine.service [None
req-9046d18e-5da7-409c-94e5-0526ef170d75 - - - - - -] Service
133afc47-b5a4-4a48-817b-7366a171c5aa update failed: QueuePool limit of size 5
overflow 64 reached, connection timed out, timeout 30.00 (Background on this
error at: https://sqlalche.me/e/14/3o7r): sqlalchemy.exc.TimeoutError:
QueuePool limit of size 5 overflow 64 reached, connection timed out, timeout
30.00 (Background on this error at: https://sqlalche.me/e/14/3o7r)
-
** Verify the Fix **
7. Upgrade to patched heat, then restart heat-engine to clear the exhausted
- pool from step 6:
- $ juju ssh heat/0 "sudo systemctl restart heat-engine"
+ pool from step 6:
+ $ juju ssh heat/0 "sudo systemctl restart heat-engine"
8. Repeat the concurrent snapshot listing:
- $ seq 1 50 | xargs -P 20 -I {} openstack stack snapshot list
"$STACK_NAME" > /dev/null
+ $ seq 1 50 | xargs -P 20 -I {} openstack stack snapshot list
"$STACK_NAME" > /dev/null
9. Confirm the fix:
- $ openstack stack list
- $ juju ssh heat/0 "sudo grep -E 'Exception during reset|TimeoutError'
/var/log/heat/heat-engine.log | tail -n 5"
+ $ openstack stack list
+ $ juju ssh heat/0 "sudo grep -E 'Exception during reset|TimeoutError'
/var/log/heat/heat-engine.log | tail -n 5"
- Observed results:
- - No "ERROR: Internal Error" from any request
- - "openstack stack list" succeeds
- - No new "Exception during reset" or TimeoutError entries in the log
- after the restart in step 7
+ Observed results:
+ - No "ERROR: Internal Error" from any request
+ - "openstack stack list" succeeds
+ - No new "Exception during reset" or TimeoutError entries in the log
+ after the restart in step 7
[ Where problems could occur ]
The function now returns a list instead of a Query. Any code that calls
db_api.snapshot_get_all_by_stack() and uses Query methods on the result,
e.g. .filter() or .count(), would fail. Its only caller in Heat is
Snapshot.get_all_by_stack() in heat/objects/snapshot.py, which just iterates
over the result.
The returned rows are detached from their session by the time the caller
uses them. Accessing a lazy-loaded relationship on them would raise
DetachedInstanceError. The only caller copies plain column attributes,
which the query has already loaded.
Any regression would show up as errors from "openstack stack snapshot list"
or from "openstack stack delete".
[ Other Info ]
This bug was introduced in Caracal by the migration of the Heat DB API to the
oslo.db enginefacade pattern [1]. Earlier releases therefore do not need to be
fixed.
The fix is merged upstream and backported until stable/2025.1. Ubuntu
packages in
Resolute onward already have the fix. Similarly, in the UCA, packages Flamingo
onward have the fix too.
+ The upstream bug report can be found in [2].
+
[0] https://review.opendev.org/c/openstack/heat/+/1003916
[1]
https://opendev.org/openstack/heat/commit/bb6b5ed5c7e559f11e570e94edda0e160d907d5f
+ [2] https://storyboard.openstack.org/#!/story/2011298
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2168828
Title:
heat-engine DB connection pool exhaustion after stack delete or
snapshot list
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2168828/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
