Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?
--On 4. Mai 2007 21:22:49 +0200 Dieter Maurer <[EMAIL PROTECTED]> wrote: Andreas Jung wrote at 2007-5-4 21:13 +0200: --On 4. Mai 2007 21:05:00 +0200 Dieter Maurer <[EMAIL PROTECTED]> wrote: But, the transactions are not concurrent in your original description! Instead, one transaction has been committed and (only!) then you see a transaction with the same id again. What are you trying to tell? The issue would also happen with a concurrent setup. That's why I presented a case where I would see the error in a non-concurrent environment. Got it? No. Not at all. Lets look at the subject which fortunately remained: You assume that some transaction objects are reused for subsequent requests. Transactions from subsequent requests are *NEVER* concurrent. Tim explained to you that it is natural that transactions in subsequent requests can easily have the same id. But, *CONCURRENT* transaction can *NEVER* have the same id. Thus, you can easily use your current caching algorithm: if you see the same transaction id, then either it is indeed the same transaction and you must associate the same connection or it is a different transaction. In this case, it is definitely nonconcurrent and can therefore, too, get the same connection. As I said, I have a working solution meanwhile :-) Andreas pgpFDifJT5DHg.pgp Description: PGP signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?
Andreas Jung wrote at 2007-5-4 21:13 +0200: > > >--On 4. Mai 2007 21:05:00 +0200 Dieter Maurer <[EMAIL PROTECTED]> wrote: > >> But, the transactions are not concurrent in your original description! >> Instead, one transaction has been committed and (only!) then you >> see a transaction with the same id again. > >What are you trying to tell? The issue would also happen with a concurrent >setup. That's why I presented a case where I would see the error in a >non-concurrent environment. Got it? No. Not at all. Lets look at the subject which fortunately remained: You assume that some transaction objects are reused for subsequent requests. Transactions from subsequent requests are *NEVER* concurrent. Tim explained to you that it is natural that transactions in subsequent requests can easily have the same id. But, *CONCURRENT* transaction can *NEVER* have the same id. Thus, you can easily use your current caching algorithm: if you see the same transaction id, then either it is indeed the same transaction and you must associate the same connection or it is a different transaction. In this case, it is definitely nonconcurrent and can therefore, too, get the same connection. >> And if you read carefully you see "provided their lifespans do not >> overlap". Obviously, transactions with non overlapping lifespans are not >> concurrent... > >The transactions were not overlapping. As Tim wrote: the transactions were >distinct but they used the address. Yes, therefore, you do not need to distinguish the two transactions -- it is sufficient to distinguish the transaction ids. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?
--On 4. Mai 2007 21:05:00 +0200 Dieter Maurer <[EMAIL PROTECTED]> wrote: But, the transactions are not concurrent in your original description! Instead, one transaction has been committed and (only!) then you see a transaction with the same id again. What are you trying to tell? The issue would also happen with a concurrent setup. That's why I presented a case where I would see the error in a non-concurrent environment. Got it? And if you read carefully you see "provided their lifespans do not overlap". Obviously, transactions with non overlapping lifespans are not concurrent... The transactions were not overlapping. As Tim wrote: the transactions were distinct but they used the address. Problem solved, end-of-thred, Andreas pgpZwrh1NiYwT.pgp Description: PGP signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?
Andreas Jung wrote at 2007-5-4 20:04 +0200: > > >--On 4. Mai 2007 19:58:47 +0200 Dieter Maurer <[EMAIL PROTECTED]> wrote: > >> Andreas Jung wrote at 2007-5-1 11:23 +0200: >>> ... >>> I think you are right (as always). Then let me rephrase the question: >>> how can one distinguish if two transaction objects represent the same or >>> different transactions in such case where memory address is identical? >> >> Why are you interested in such a distinction? >> >> While you must deliver the same connection in the same transaction, >> there is no harm to deliver a given connection to different transactions >> (provided their lifespans do not overlap). > >You can't use one connection for concurrent transactions in different >threads. Starting a transaction over the same connection would result in >nested transactions instead of concurrent transactions. But, the transactions are not concurrent in your original description! Instead, one transaction has been committed and (only!) then you see a transaction with the same id again. And if you read carefully you see "provided their lifespans do not overlap". Obviously, transactions with non overlapping lifespans are not concurrent... -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?
--On 4. Mai 2007 19:58:47 +0200 Dieter Maurer <[EMAIL PROTECTED]> wrote: Andreas Jung wrote at 2007-5-1 11:23 +0200: ... I think you are right (as always). Then let me rephrase the question: how can one distinguish if two transaction objects represent the same or different transactions in such case where memory address is identical? Why are you interested in such a distinction? While you must deliver the same connection in the same transaction, there is no harm to deliver a given connection to different transactions (provided their lifespans do not overlap). You can't use one connection for concurrent transactions in different threads. Starting a transaction over the same connection would result in nested transactions instead of concurrent transactions. -aj pgpA3zA8VZfG3.pgp Description: PGP signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?
Andreas Jung wrote at 2007-5-1 11:23 +0200: > ... >I think you are right (as always). Then let me rephrase the question: how >can one distinguish if two transaction objects represent the same or >different transactions in such case where memory address is identical? Why are you interested in such a distinction? While you must deliver the same connection in the same transaction, there is no harm to deliver a given connection to different transactions (provided their lifespans do not overlap). -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?
[Tim Peters] ... In any case, yes, the intent is that a new transaction object is used for each transaction; but, no, seeing the same memory address does not mean that isn't happening. [Andreas Jung] I think you are right (as always). Luckily for everyone, I'll die someday ;-) Then let me rephrase the question: how can one distinguish if two transaction objects represent the same or different transactions in such case where memory address is identical? I expect you'd have to add another data attribute, holding a unique (over time) ID. Memory address (id()) is unique only among objects simultaneously alive. A tid (transaction ID) is unique over time, but tids are created by storages (not by transaction objects), and a TO has no idea what tid is associated with it. I'm not sure what you're trying to accomplish. As a hack, you could arrange to make transaction objects immortal (save a reference to one in some global data structure; e.g., a dict mapping the id() of a TO to the TO). Then id() would remain unique over time (at the cost of hanging on to transaction objects forever). ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?
--On 1. Mai 2007 05:15:37 -0400 Tim Peters <[EMAIL PROTECTED]> wrote:
Here's a very simple example using builtins to illustrate this:
x = {1: 2}
id(x)
10603952
del x
x = {3: 4}
id(x)
10603952
Those are obviously entirely different dictionaries, but they live at
the same memory address (and likely so, by design, and thanks to the
quick memory reuse allowed by refcounting).
In any case, yes, the intent is that a new transaction object is used
for each transaction; but, no, seeing the same memory address does not
mean that isn't happening.
I think you are right (as always). Then let me rephrase the question: how
can one distinguish if two transaction objects represent the same or
different transactions in such case where memory address is identical?
Andreas
pgpPCvVtwHAWp.pgp
Description: PGP signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?
[Andreas Jung]
I encountered the following strange behavior with Zope 2.8.8.
I couldn't find "a problem" in the following. Are you having a
problem, or just asking a question?
The following code is used to integrate SQLAlchemy with
Zope. A registered utility subclassing ZopeBaseWrapper provides
a 'connection' property. This property should always return
for a given transaction the same sqlalchemy.Connection object
(which is like a connection from a connection pool within a DA).
Within a thread-local cache the last id of the transaction and the last
connection is stored in order to return the connection from the cache if
the property 'connection' is called/used multiple times within one
request/one
transaction.
A ConnectionDataManger instance is added a data manager to the current
transaction in order to integrate the ZODB transaction with the transaction
system by SQLAlchemy.
class ConnectionDataManager(object):
""" Wraps connection into transaction context of Zope """
implements(IDataManager)
def __init__(self, connection):
self.connection = connection
self.transaction = connection.begin()
def tpc_begin(self, trans):
log('tpc_begin() - %s' % trans)
pass
def abort(self, trans):
self.transaction.rollback()
self.connection.close()
self.connection = None
log('abort() - %s' % trans)
def commit(self, trans):
self.transaction.commit()
log('commit() - %s' % trans)
self.connection.close()
self.connection = None
def tpc_vote(self, trans):
pass
def tpc_finish(self, trans):
log('tcp_finish() - %s' % trans)
pass
def tpc_abort(self, trans):
log('tcp_abort() - %s' % trans)
pass
def sortKey(self):
return str(id(self))
_connection_cache = threading.local() # module-level cache
class ZopeBaseWrapper(BaseWrapper):
@property
def connection(self):
if not hasattr(_connection_cache, 'last_connection'):
_connection_cache.last_transaction = None
_connection_cache.last_connection = None
# get current transaction
txn = transaction.get()
txn_str = str(txn)
log('current thread - %s' % threading.currentThread())
log('checking for transaction - %s' % txn_str)
# return cached connection if we are within the same transaction
# and same thread
if txn_str == _connection_cache.last_transaction:
log('returning cached connection - %s' %
_connection_cache.last_connection)
return _connection_cache.last_connection
# no cached connection, let's create a new one
connection = self.engine.connect()
log('creating new connection - %s' % connection)
# register a DataManager with the current transaction
txn.join(ConnectionDataManager(connection))
# update thread-local cache
_connection_cache.last_transaction = txn_str
_connection_cache.last_connection = connection
# return the connection
return connection
This works almost.
In what specific way (if any) does it /not/ work?
However when I hammer my Zope instance using ab2
(without concurrent request, option -c 1) then in some rare cases I see
that a new request uses a formerly used transaction object. Look at the
output
Request #1:
*** <_DummyThread(Dummy-1, started daemon)> - current thread
<_DummyThread(Dummy-1, started daemon)>
*** <_DummyThread(Dummy-1, started daemon)> - checking for transaction -
*** <_DummyThread(Dummy-1, started daemon)> - creating new connection -
*** <_DummyThread(Dummy-1, started daemon)> - tpc_begin() -
*** <_DummyThread(Dummy-1, started daemon)> - commit() -
*** <_DummyThread(Dummy-1, started daemon)> - tcp_finish() -
Request #2:
*** <_DummyThread(Dummy-1, started daemon)> - current thread -
<_DummyThread(Dummy-1, started daemon)>
*** <_DummyThread(Dummy-1, started daemon)> - checking for transaction -
*** <_DummyThread(Dummy-1, started daemon)> - returning cached connection -
As you can see request #1 commits without a problem.
Are you implying (by silence ;-)) that request #2 does not commit
without problem?
But for the second request handled by the same thread the same transaction
object is re-used.
Bug or feature?
There's no way to know from the above whether or not it's "the same
transaction object". All I can tell from the output is that the
transaction objects (TO) in both blocks of output happen to live at
the same memory address. That's in fact very likely if the TO from
block #1 is freed and then very quickly a new TO is allocated.
Python's size-segregated small-object allocator deliberately works (up
to a point) like a stack, reusing a chunk of memory ASAP (while it's
most likely to still be high in the OS+HW memory hierarchy).
Here's a very simple example using builtins to illustrate this:
x = {1: 2}
id(x)
1060
