We're using celery, a job distribution package. On a single machine, there are 20+ celery workers running, each with their own Python process. We had some issues with the processes attempting to use the same SQLAlchemy connections (I think because they are referenced by file descriptors, and so can cross process boundaries?), and so we're now using a NullPool.
But is there a way of doing this more efficiently than a NullPool? There's are two suggestions in the docs <http://docs.sqlalchemy.org/en/latest/core/pooling.html#using-connection-pools-with-multiprocessing> : - Dispose the connection as soon as the other process receives it. This wouldn't work with celery, since the workers are started on their own, not forked from another python process. - Implement a listener that invalidates connections created in another process. I'm unclear how this works and whether that would be practical in this context. Specifically, given the number of processes we have running on each machine, is the pool that a process is selecting & testing from shared between all processes on that machine? If a pool is shared across 20 processes, the chances that a connection chosen at random would have been created in that process is fairly low... It's also possible we're woefully misunderstanding how this works; corrections appreciated. Many thanks, Max -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
