villebro commented on issue #8574: URL: https://github.com/apache/superset/issues/8574#issuecomment-1749752198
Rereading this now, I wonder if the correct design for this should in fact exist downstream of Superset, and not within superset, for the reasons explained by Max in his first post: > There's a challenge here around the fact that each subprocess (gunicorn worker and celery worker) gets its own pool, and that each one of those can connect to multiple databases. Depending on whether you configure these things to use threads or workers (subprocesses), you can end up with a lot of connections very quickly that is vastly bigger than the number of active connections. Especially in the context of horizontal scaling of workers, keeping connections open on the worker processes will quickly lead to lots of idle open connections that are not efficiently utilized. So if someone has seen a generic connection pooler for SQLAlchemy connections, similar to Pgpool-II, that could be an awesome solution to this problem (although I'm sure configuring it would be a challenge in itself..). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
