Thank you Mike! That's very useful. I can see which engine & worker is emitting each SQL statement (I had to bypass the usage of 'pyramid_celery` so all logging settings would "stick").
*Is there a way to identify every engine instance?* *Every celery worker is creating its own engines-sessions dual pair but there are 8 concurrent workers.* I see some Pool messages but I am unable to get anything useful from them. Here is the log preceding the TransactionRollbackError: [2018-06-25 12:51:30,373: WARNING/Worker-2] [p2server.celery_tasks.generate_version(dfad02fa-1282-460e-85a2-720436f7f9d8)] p2server.celery_tasks CHECKPOINT 'p2server.celery_tasks.generate_version'['dfad02fa-1282-460e-85a2-720436f7f9d8'] UT:3 'started' [2018-06-25 12:51:30,473: WARNING/Worker-2] celery.redirected /Users/henddher/.pyenv/versions/3.6.3/envs/p2server_py36/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>. """) ... tons of sqlalchemy.mapper logs from same Worker-2 ... [2018-06-25 12:51:31,200: INFO/Worker-2] [p2server.celery_tasks.generate_version(dfad02fa-1282-460e-85a2-720436f7f9d8)] sqlalchemy.engine.base.Engine.checkpoint-engine select version() ... rest of boilerplate SQL at transaction start checkpoint-engine ... [2018-06-25 12:51:31,213: INFO/Worker-2] [p2server.celery_tasks.generate_version(dfad02fa-1282-460e-85a2-720436f7f9d8)] sqlalchemy.engine.base.Engine.checkpoint-engine BEGIN (implicit) [2018-06-25 12:51:31,214: INFO/Worker-2] [p2server.celery_tasks.generate_version(dfad02fa-1282-460e-85a2-720436f7f9d8)] sqlalchemy.engine.base.Engine.checkpoint-engine SELECT usertransactions.id [2018-06-25 12:51:31,218: INFO/Worker-2] [p2server.celery_tasks.generate_version(dfad02fa-1282-460e-85a2-720436f7f9d8)] sqlalchemy.engine.base.Engine.checkpoint-engine UPDATE usertransaction ... [2018-06-25 12:51:31,220: INFO/Worker-2] [p2server.celery_tasks.generate_version(dfad02fa-1282-460e-85a2-720436f7f9d8)] sqlalchemy.engine.base.Engine.checkpoint-engine COMMIT # domain model stuff after checkpoint 'started' is done and commit [2018-06-25 12:51:31,232: INFO/Worker-2] [p2server.celery_tasks.generate_version(dfad02fa-1282-460e-85a2-720436f7f9d8)] sqlalchemy.engine.base.Engine.normal-engine select version() ... rest of boilerplate SQL at transaction start normal-engine (aka DOMAIN MODEL): current_schema, SELECT CAST ... ### Meanwhile, Worker-3 is doing its thing with its own checkpoint-engine and normal-engine about to trigger Concurrent Update # Worker-2 starts to log 'failed' checkpoint [2018-06-25 12:51:35,374: WARNING/Worker-2] [p2server.celery_tasks.generate_version(dfad02fa-1282-460e-85a2-720436f7f9d8)] p2server.celery_tasks CHECKPOINT 'p2server.celery_tasks.generate_version'['dfad02fa-1282-460e-85a2-720436f7f9d8'] UT:3 'failed' ... EXCEPTION STACK TRACE ... sqlalchemy.exc.OperationalError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (psycopg2.extensions.TransactionRollbackError) could not serialize access due to concurrent update [2018-06-25 12:51:35,413: WARNING/Worker-2] celery.redirected 2018-06-25 12:51:35,413 INFO sqlalchemy.pool.QueuePool Pool disposed. Pool size: 5 Connections in pool: 0 Current Overflow: -5 Current Checked out connections: 0 [2018-06-25 12:51:35,413: INFO/Worker-2] [p2server.celery_tasks.generate_version(dfad02fa-1282-460e-85a2-720436f7f9d8)] sqlalchemy.pool.QueuePool Pool disposed. Pool size: 5 Connections in pool: 0 Current Overflow: -5 Current Checked out connections: 0 [2018-06-25 12:51:35,414: WARNING/Worker-2] celery.redirected 2018-06-25 12:51:35,414 INFO sqlalchemy.pool.QueuePool Pool recreating [2018-06-25 12:51:35,414: INFO/Worker-2] [p2server.celery_tasks.generate_version(dfad02fa-1282-460e-85a2-720436f7f9d8)] sqlalchemy.pool.QueuePool Pool recreating Notice that in this log, there is no UPDATE, COMMIT, ROLLBACK for Worker-2. Only pool disposed message after the exception stack trace. This makes me wonder if the previous logs showing ROLLBACK and whatnot came from other workers. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
