Public bug reported:
[ Impact ]
heat-engine can permanently exhaust its database connection pool when stack
snapshots are listed or when a stack is deleted. Once the pool is exhausted,
every database call in heat-engine blocks for 30 seconds and then fails with
sqlalchemy.exc.TimeoutError. All subsequent Heat API requests return "ERROR:
Internal Error" until heat-engine is restarted.
snapshot_get_all_by_stack() in heat/db/api.py is decorated with
@context_manager.reader but returns an unexecuted SQLAlchemy Query object.
The decorator closes its session when the function returns, so the query
only runs later, when the caller iterates over it. At that point SQLAlchemy
silently begins a new transaction on the closed session and checks out a new
connection that nothing ever returns to the pool.
This affects releases in which heat-engine runs under eventlet. The pool
slots are only lost permanently when eventlet is in use. The connection is
only reclaimed when Python's garbage collector frees the session. When the
garbage collector happens to run inside the eventlet hub, resetting the
connection requires blocking socket I/O, which eventlet refuses with
"AssertionError: do not call blocking functions from the mainloop". The
connection is never returned and its pool slot is lost permanently. Under
concurrent load the pool is exhausted quickly.
The fix adds .all() so that the query runs inside the session scope and the
decorator returns the connection to the pool. This fix has already been
accepted upstream [0].
[ Test Plan ]
** Setup OpenStack Environment **
1. Deploy a test cloud in your reproducer bastion using stsstack-bundles:
$ cd stsstack-bundles/openstack
$ ./generate-bundle.sh --series <UBUNTU_RELEASE> --release
<OPENSTACK_RELEASE> --name heat-test --run
# wait until all units are active/idle (some will be blocked/waiting)
$ ./tools/vault-unseal-and-authorise.sh
# wait until all units are active/idle
$ ./configure
$ source novarc
2. Reduce heat-engine to a single worker so that all requests share one
connection pool:
$ juju config heat worker-multiplier=0.5 # adjust based on your env
# wait until heat/0 is active/idle
$ juju ssh heat/0 "sudo grep num_engine_workers /etc/heat/heat.conf"
num_engine_workers = 1
** Verify the Bug **
3. Create a file test-heat.yaml with the following content:
heat_template_version: 2015-04-30
resources:
test_resource:
type: OS::Heat::TestResource
4. Create a test stack and a snapshot:
$ STACK_NAME="leak-test-stack"
$ openstack stack create -t test-heat.yaml "$STACK_NAME" --wait
$ openstack stack snapshot create "$STACK_NAME"
5. List the snapshots concurrently:
$ seq 1 50 | xargs -P 20 -I {} openstack stack snapshot list "$STACK_NAME"
> /dev/null
6. Confirm the failure:
Some of the requests above fail with:
ERROR: Internal Error
After the load has finished, Heat remains unusable:
$ openstack stack list
ERROR: Internal Error
Verify the errors in the logs:
$ juju ssh heat/0 "sudo grep 'do not call blocking functions'
/var/log/heat/heat-engine.log"
$ juju ssh heat/0 "sudo grep -E 'Exception during reset|TimeoutError'
/var/log/heat/heat-engine.log | tail -n 5"
Observed results:
- "openstack stack list" keeps failing until heat-engine is restarted
- Log shows a connection that could not be returned to the pool:
2026-09-25 12:36:46.462 517667 ERROR sqlalchemy.pool.impl.QueuePool [-]
Exception during reset or similar: AssertionError: do not call blocking
functions from the mainloop
- Followed by pool exhaustion:
2026-09-25 12:53:47.028 517667 ERROR heat.engine.service [None
req-9046d18e-5da7-409c-94e5-0526ef170d75 - - - - - -] Service
133afc47-b5a4-4a48-817b-7366a171c5aa update failed: QueuePool limit of size 5
overflow 64 reached, connection timed out, timeout 30.00 (Background on this
error at: https://sqlalche.me/e/14/3o7r): sqlalchemy.exc.TimeoutError:
QueuePool limit of size 5 overflow 64 reached, connection timed out, timeout
30.00 (Background on this error at: https://sqlalche.me/e/14/3o7r)
** Verify the Fix **
7. Upgrade to patched heat, then restart heat-engine to clear the exhausted
pool from step 6:
$ juju ssh heat/0 "sudo systemctl restart heat-engine"
8. Repeat the concurrent snapshot listing:
$ seq 1 50 | xargs -P 20 -I {} openstack stack snapshot list "$STACK_NAME"
> /dev/null
9. Confirm the fix:
$ openstack stack list
$ juju ssh heat/0 "sudo grep -E 'Exception during reset|TimeoutError'
/var/log/heat/heat-engine.log | tail -n 5"
Observed results:
- No "ERROR: Internal Error" from any request
- "openstack stack list" succeeds
- No new "Exception during reset" or TimeoutError entries in the log
after the restart in step 7
[ Where problems could occur ]
The function now returns a list instead of a Query. Any code that calls
db_api.snapshot_get_all_by_stack() and uses Query methods on the result,
e.g. .filter() or .count(), would fail. Its only caller in Heat is
Snapshot.get_all_by_stack() in heat/objects/snapshot.py, which just iterates
over the result.
The returned rows are detached from their session by the time the caller
uses them. Accessing a lazy-loaded relationship on them would raise
DetachedInstanceError. The only caller copies plain column attributes,
which the query has already loaded.
Any regression would show up as errors from "openstack stack snapshot list"
or from "openstack stack delete".
[ Other Info ]
This bug was introduced in Caracal by the migration of the Heat DB API to the
oslo.db enginefacade pattern [1]. Earlier releases therefore do not need to be
fixed.
The fix is merged upstream and backported until stable/2025.1. Ubuntu packages
in
Resolute onward already have the fix. Similarly, in the UCA, packages Flamingo
onward have the fix too.
[0] https://review.opendev.org/c/openstack/heat/+/1003916
[1]
https://opendev.org/openstack/heat/commit/bb6b5ed5c7e559f11e570e94edda0e160d907d5f
** Affects: cloud-archive
Importance: Undecided
Assignee: Munir Siddiqui (munirsid)
Status: In Progress
** Affects: cloud-archive/epoxy
Importance: Undecided
Assignee: Munir Siddiqui (munirsid)
Status: In Progress
** Affects: cloud-archive/flamingo
Importance: Undecided
Status: Fix Released
** Affects: cloud-archive/gazpacho
Importance: Undecided
Status: Fix Released
** Affects: heat (Ubuntu)
Importance: Undecided
Status: Fix Released
** Affects: heat (Ubuntu Noble)
Importance: Undecided
Assignee: Munir Siddiqui (munirsid)
Status: In Progress
** Affects: heat (Ubuntu Resolute)
Importance: Undecided
Status: Fix Released
** Affects: heat (Ubuntu Stonking)
Importance: Undecided
Status: Fix Released
** Also affects: cloud-archive
Importance: Undecided
Status: New
** Changed in: heat (Ubuntu)
Status: New => In Progress
** Changed in: cloud-archive
Status: New => In Progress
** Changed in: heat (Ubuntu)
Assignee: (unassigned) => Munir Siddiqui (munirsid)
** Changed in: cloud-archive
Assignee: (unassigned) => Munir Siddiqui (munirsid)
** Also affects: cloud-archive/epoxy
Importance: Undecided
Status: New
** Also affects: cloud-archive/flamingo
Importance: Undecided
Status: New
** Also affects: cloud-archive/gazpacho
Importance: Undecided
Status: New
** Changed in: cloud-archive/epoxy
Status: New => In Progress
** Changed in: cloud-archive/epoxy
Assignee: (unassigned) => Munir Siddiqui (munirsid)
** Changed in: cloud-archive/flamingo
Status: New => Fix Released
** Changed in: cloud-archive/gazpacho
Status: New => Fix Released
** Changed in: heat (Ubuntu)
Status: In Progress => Invalid
** Changed in: heat (Ubuntu)
Status: Invalid => In Progress
** Also affects: heat (Ubuntu Stonking)
Importance: Undecided
Assignee: Munir Siddiqui (munirsid)
Status: In Progress
** Also affects: heat (Ubuntu Noble)
Importance: Undecided
Status: New
** Also affects: heat (Ubuntu Resolute)
Importance: Undecided
Status: New
** Changed in: heat (Ubuntu Noble)
Status: New => In Progress
** Changed in: heat (Ubuntu Stonking)
Status: In Progress => Fix Released
** Changed in: heat (Ubuntu Stonking)
Assignee: Munir Siddiqui (munirsid) => (unassigned)
** Changed in: heat (Ubuntu Resolute)
Status: New => Fix Released
** Changed in: heat (Ubuntu Noble)
Assignee: (unassigned) => Munir Siddiqui (munirsid)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2168828
Title:
heat-engine DB connection pool exhaustion after stack delete or
snapshot list
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/2168828/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
