On Mon, Jun 25, 2018 at 6:24 PM, HP3 wrote:
> Thank you both
>
> With `before_cursor_execute` I am able to log each Connection, Cursor and
> context instance.
>
> I see that every worker is doing its thing and the Connection instance is
> always different for each SQL statement
>
> Also, each SQL
Thank you both
With `before_cursor_execute` I am able to log each Connection, Cursor and
context instance.
I see that every worker is doing its thing and the Connection instance is
always different for each SQL statement
Also, each SQL has its own distinct
you might want to build out events to log exactly what you need and where:
http://docs.sqlalchemy.org/en/latest/core/events.html#sql-execution-and-connection-events
use your own logger, look at the connection / engine / current thread
/ etc. for whatever you need to keep track of.
On Mon, Jun
>
> @HP3 just to test this, i would try adding a slightly different connection
> string or argument to the celery connection. e.g. create a different user,
> or toss in a config argument that doesn't affect your code. if the error
> stops, that's most-likely the reason why.
>
I'll try that
Thank you Mike!
That's very useful. I can see which engine & worker is emitting each SQL
statement (I had to bypass the usage of 'pyramid_celery` so all logging
settings would "stick").
*Is there a way to identify every engine instance?*
*Every celery worker is creating its own
On Monday, June 25, 2018 at 11:31:07 AM UTC-4, HP3 wrote:
>
> I'm confused about what you said about the underlined connection: I am
> creating 2 different engines. Why would both share the same connection?
>
>
That wasn't clear from the above, however..
looking at the code you've shared, it
On Mon, Jun 25, 2018 at 11:31 AM, HP3 wrote:
> I'm confused about what you said about the underlined connection: I am
> creating 2 different engines. Why would both share the same connection?
just as a note, if you do have two different engines, you can apply an
individual logging name to each:
>
>
> I verified that the failure checkpoint is not being persisted either when
I raise an exception after 'do something' checkpoint.
--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal,
I'm confused about what you said about the underlined connection: I am
creating 2 different engines. Why would both share the same connection?
FYI -
I just tried without zope.sqlalchemy and got a slightly different result:
My concurrent transactions are no longer failing
with
Thank you Jonathan,
This is how I create both engine/session pairs:
from sqlalchemy import engine_from_config
from sqlalchemy.orm import sessionmaker
import zope.sqlalchemy
class DBTask(app.task)
_engine = None
_domain_model_session = None
def checkpoint(self, label):
*#
Can you share/show how/where they engines and connections are created?
This is odd.
FWIW, with the forking bug- the issue isn't in the transaction/session but
in the underlying database connections. The SqlAlchemy connection pool
isn't threadsafe, so all the commits/rollbacks/etc in different
No dice!
I verified that engines and sessions are created after fork.
By hijacking celery logging, verified each worker had its own transaction
and session.
The SQL logs I described above are indeed accurate and belong to the same
worker.
--
SQLAlchemy -
The Python SQL Toolkit and Object
On Friday, June 22, 2018 at 1:32:15 PM UTC-4, HP3 wrote:... but I'll 2x
check!
> (I recall that task-inheritance in celery makes certain things happen
> before and others after the fork - I am using prefork)
>
i don't use pyramid_celery, but my own pyramid and celery integration...
>
> Is there a chance there is a query/connection being made between the
> initialization and worker process?
>
Hmmm ...
I am using celery task inheritance
(http://docs.celeryproject.org/en/latest/userguide/tasks.html#task-inheritance)
with `@property`s for domain model engine and domain
Is there a chance there is a query/connection being made between the
initialization and worker process? If so, that could screw up the
connection pool. To address that, you can try adding an `engine.dispose()`
before celery forks.
I'll take a look at the code later. I was home sick this
I added logging to `session.transaction` and `session.transaction.parent`
(til parent=None) and `session.transaction.nested` and they all seemed to
be different, never nested, never subtransaction. No hint there.
I modified my code slightly by adding a new
Thank you Jonathan!
I checked if my celery_state_session had been mistakenly connected to
zope.sqlalchemy as I am using pyramid_celery. I didn't find anything.
On the other hand, `domain_model_session` is wired to transaction.manager
by zope.sqlalchemy (zope.sqlalchemy.registry()).
I even
On Thursday, June 21, 2018 at 10:02:27 PM UTC-4, HP3 wrote:
>
> What's the correct way to create a session that is not automatically bound
> to any transaction or that can be explicitly bound to an isolated
> transaction that can be committed whenever?
>
That's what `Session()` does by
Thank you Mike!!!
Sorry about that - my example had errors :(
In reality, I am calling `celery_state_session.flush()` and
`celery_state_session.commit()` right after I update any of the `cs.`
values.
@app.task(
base=BaseTask,
bind=True
)
def my_task(self, ...):
# Persist
>From looking at your code example, I don't see you calling
celery_session.flush() or celery_session.commit(), so nothing is going to
persist. Just setting an attribute value does not send any SQL.
On Thu, Jun 21, 2018, 7:22 PM HP3 wrote:
> Hello all:
>
> Within a celery task, I need to have 2
Hello all:
Within a celery task, I need to have 2 unrelated sessions that can
commit/rollback independently of each other:
One session (`domain_model_session`) performs vanilla domain model
operations and the other (`celery_state_session`) is to persist the state
of the celery task itself
Hi,
I am using tornado, sqlalchemy behind ngnix in reverse proxy mode. I run 4+
app servers and each maintaining it's own db session and connection.
I am seeing the same problem as mentioned in the forum. One session creates
new data and other session doesn't get updated information. As I am
Here is a simplified version of my scenario:
My application has two session objects. One is being used for editing certain
data, and the other is reading this data, doing some calculations, and
displaying the results. I've set up a callback kind of system so that whenever
the first session does
On Dec 18, 2011, at 8:25 PM, Jackson, Cameron wrote:
Here is a simplified version of my scenario:
My application has two session objects. One is being used for editing certain
data, and the other is reading this data, doing some calculations, and
displaying the results. I've set up a
...@thalesgroup.com.au |
www.thalesgroup.com.auhttp://www.thalesgroup.com.au
From: sqlalchemy@googlegroups.com [mailto:sqlalchemy@googlegroups.com] On
Behalf Of Michael Bayer
Sent: Monday, 19 December 2011 12:51 PM
To: sqlalchemy@googlegroups.com
Subject: Re: [sqlalchemy] Multiple sessions and data conflicts
I've found a strange behaviour when using multiple sessions: A
committed change to an object in one session does not get reflected in
the other session, even after a session.expire_all() (or expire() or
refresh() on the object. In fact, even an explicit query does not
retrieve the changed data.
On Aug 12, 2010, at 7:36 AM, Erich Eder wrote:
I've found a strange behaviour when using multiple sessions: A
committed change to an object in one session does not get reflected in
the other session, even after a session.expire_all() (or expire() or
refresh() on the object. In fact, even an
27 matches
Mail list logo
