One thing that may not be intuitive is the fact that each web "worker" will create its own pool, so a gunicorn process with 4 workers and a pool of 20 you can expect up to 80 connections.
Max On Fri, Apr 30, 2021 at 6:08 AM Thammaneni, Srinivasa Rao < [email protected]> wrote: > Hi Team, > > > i am using superset0.36 version, have observed superset is creating more > no.of db sessions when dashboards/reports are running continuously. > Sometimes due to high load/no.of schedule jobs are running db sessions are > breaking. I thought to see whether connection pool creation for superset > will improve db sessions failures. i have followed below SIP as a reference. > > https://github.com/apache/superset/issues/8574 > > i have followed the sqlalchemy documentation and used QueuePool to created > connection pool when superset is establishing db session with help of > "create_engine" method. Below are the connection pool parameters and file > locations where i have updated in superset. > > Documentation: > https://docs.sqlalchemy.org/en/13/core/pooling.html#sqlalchemy.pool.QueuePool > > Connection pool parameters: > filepath: superset/models/core.py [ function: get_sqla_engine ] > params["poolclass"] = QueuePool > params["pool_size"] = 20 > params["max_overflow"] = 10 > params["pool_recycle"]= 3600 > params["pool_pre_ping"] = True > params["pool_use_lifo"] = True > > filepath: superset/migrations/env.py > engine = engine_from_config( > config.get_section(config.config_ini_section), > prefix="sqlalchemy.", > poolclass=pool.QueuePool, > ) > > I can these are the main functions where superset initiating for db > sessions creations. there are couple of places where i have override > nullpool parameter to false, since nullpoll is for disabling connection > pool. > > i also tried to see by varying the pool size but i can't see any > difference after applying connection pool code. I need suggestion to > whether i am going in right direction or not OR is there a way to test/see > the connection pool usage at application level to make sure the new code > customizations that i have done is working. > > Please give some suggestions/ if there are any implementation done w.r.to > this requirement in superset before. > > Thanks for the attention and time. > Have created same in Github ticket : > https://github.com/apache/superset/issues/14414 > > Thanks, > Srini T. > > > > This electronic mail (including any attachments) may contain information > that is privileged, confidential, and/or otherwise protected from > disclosure to anyone other than its intended recipient(s). Any > dissemination or use of this electronic mail or its contents (including any > attachments) by persons other than the intended recipient(s) is strictly > prohibited. If you have received this message in error, please notify us > immediately by reply e-mail so that we may correct our internal records. > Please then delete the original message (including any attachments) in its > entirety. Thank you >
