+openstack-dev since I believe this is an issue with the Heat source code.
On 06/18/2018 11:19 AM, Spyros Trigazis wrote:
Hello list,
I'm hitting quite easily this [1] exception with heat. The db server is
configured to have 1000
max_connnections and 1000 max_user_connections and in the database
section of heat
conf I have these values set:
max_pool_size = 22
max_overflow = 0
Full config attached.
I ended up with this configuration based on this formula:
num_heat_hosts=4
heat_api_workers=2
heat_api_cfn_workers=2
num_engine_workers=4
max_pool_size=22
max_overflow=0
num_heat_hosts * (max_pool_size + max_overflow) * (heat_api_workers +
num_engine_workers + heat_api_cfn_workers)
704
What I have noticed is that the number of connections I expected with
the above formula is not respected.
Based on this formula each node (every node runs the heat-api,
heat-api-cfn and heat-engine) should
use up to 176 connections but they even reach 400 connections.
Has anyone noticed a similar behavior?
Looking through the Heat code, I see that there are many methods in the
/heat/db/sqlalchemy/api.py module that use a SQLAlchemy session but
never actually call session.close() [1] which means that the session
will not be released back to the connection pool, which might be the
reason why connections keep piling up.
Not sure if there's any setting in Heat that will fix this problem.
Disabling connection pooling will likely not help since connections are
not properly being closed and returned to the connection pool to begin with.
Best,
-jay
[1] Heat apparently doesn't use the oslo.db enginefacade transaction
context managers either, which would help with this problem since the
transaction context manager would take responsibility for calling
session.flush()/close() appropriately.
https://github.com/openstack/oslo.db/blob/43af1cf08372006aa46d836ec45482dd4b5b5349/oslo_db/sqlalchemy/enginefacade.py#L626
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
