Re: [Zope3-dev] Phantom of GadflyAdapter
On Tue, 13 Dec 2005 10:08:17 -0500
[EMAIL PROTECTED] wrote.
On Tuesday 13 December 2005 09:54, Tadashi Matsumoto wrote:
If you write a test for this, I'll check it in.
I have made a test. It works well (perhaps).
This test program is small, so I have attached in this mail.
Tadashi Matsumoto
[EMAIL PROTECTED]
import os, shutil
import tempfile, threading
from unittest import TestCase, TestSuite, main, makeSuite
from zope.app.rdb.gadflyda import GadflyAdapter, setGadflyRoot
class GadflyTestBase(TestCase):
def setUp(self):
TestCase.setUp(self)
self.tempdir = None
def tearDown(self):
TestCase.tearDown(self)
if self.tempdir:
shutil.rmtree(self.tempdir)
setGadflyRoot()
def getGadflyRoot(self):
if not self.tempdir:
self.tempdir = tempfile.mkdtemp('gadfly')
setGadflyRoot(self.tempdir)
return self.tempdir
def _create(self, *args):
return GadflyAdapter(*args)
def exec_sql(adapter, sql, args, fetch=False):
conn = adapter()
cur =conn.cursor()
cur.execute(sql, args)
rows = []
if fetch:
rows = cur.fetchall()
conn.commit()
return rows
class TestPhantom(GadflyTestBase):
def setUp(self):
GadflyTestBase.setUp(self)
dir = self.getGadflyRoot()
os.mkdir(os.path.join(dir, demo))
self.adapter = self._create(dbi://demo)
conn = self.adapter()
cur = conn.cursor()
cur.execute(create table t1 (name varchar))
conn.commit()
def test_Phantom(self):
adapter = self.adapter
insert = insert into t1 values (?)
select = select name from t1
delete = delete from t1
count = 0
for name in ('a', 'b', 'c'):
t = threading.Thread(target=exec_sql,
args=(adapter, insert, (name,)))
t.start()
t.join()
rows = exec_sql(adapter, select, args=(), fetch=True)
count += 1
self.assertEqual(len(rows), count)
exec_sql(adapter, delete, args=())
t = threading.Thread(target=exec_sql,
args=(adapter, delete, ()))
t.start()
t.join()
rows = exec_sql(adapter, select, args=(), fetch=True)
self.assertEqual(len(rows), 0)
def test_suite():
return TestSuite((
makeSuite(TestPhantom),
))
if __name__=='__main__':
main(defaultTest='test_suite')
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Phantom of GadflyAdapter
Phantom of GadflyAdapter GadflyAdapter sometimes causes phantom read. The reason is that many connections are made simultaneously on a gadfly database through the thead local variable _v_connection. Threadsafety level 2 or higher is required to use the thread local variable. To remedy this fault, it is enough to use usual variable as _v_connection. But users need to control lock/unlock of the connection by themselves. Simple patch: --- gadflyda.py.org 2005-12-06 22:19:47.0 +0900 +++ gadflyda.py 2005-12-12 22:41:07.0 +0900 @@ -51,6 +51,8 @@ # The registerable object needs to have a container __name__ = __parent__ = None +_v_connection = None +paramstyle = 'qmark' def _connection_factory(self): Create a Gadfly DBI connection based on the DSN. Tadashi Matsumoto [EMAIL PROTECTED] ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Phantom of GadflyAdapter
On Tue, 13 Dec 2005 10:08:17 -0500 [EMAIL PROTECTED] wrote. On Tuesday 13 December 2005 09:54, Tadashi Matsumoto wrote: Simple patch: If you write a test for this, I'll check it in. Ok, I will try. But I'm not sure I can make a test that always fails for the origial code and succeeds for the patched one. Tadashi Matsumoto [EMAIL PROTECTED] ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] publishTraverse method in viewmeta.py
In zope/app/publisher/browser/viewmeta.py, I found the following comment. # This should go away, but noone seems to remember what to do. :-( I don't know what this comment means, but the publishTraverse methods can be rewritten by natural way. Please test this code and include into the original if it works well. replace line 280-304 to the following. def publishTraverse(self, request, name, pages=pages, getattr=getattr): if name in pages: return getattr(self, pages[name]) return super(self.__class__, self).publishTraverse(request, name) replace line 433-437 to the following class simple(BrowserView): implements(IBrowserPublisher) def publishTraverse(self, request, name): view = zapi.queryMultiAdapter((self, request), name=name) if view is not None: return view raise NotFound(self, name, request) Tadashi Matsumoto [EMAIL PROTECTED] ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Potential bug of intid
Hello I was looking into the following code of intid's __init__.py. 91 def _generateId(self): 92 Generate an id which is not yet taken. 93 94 This tries to allocate sequential ids so they fall into the 95 same BTree bucket, and randomizes if it stumbles upon a 96 used one. 97 98 while True: 99 if self._v_nextid is None: 100 self._v_nextid = random.randint(0, 2**31) 101 uid = self._v_nextid 102 self._v_nextid += 1 103 if uid not in self.refs: 104 return uid 105 self._v_nextid = None 106 107 def register(self, ob): 108 # Note that we'll still need to keep this proxy removal. 109 ob = removeSecurityProxy(ob) 110 key = IKeyReference(ob) 111 112 if key in self.ids: 113 return self.ids[key] 114 uid = self._generateId() 115 self.refs[uid] = key 116 self.ids[key] = uid 117 return uid 118 2**31 used at the line 100 is a long integer, and there is a possibility of generating long integer. long interger can not be used with Btree module. In fact, from BTrees import IOBTree t=IOBTree.IOBTree() t[2**30] = 'abc' t[2**31] = 'efg' Traceback (most recent call last): File stdin, line 1, in ? TypeError: expected integer key I think it is better to use, for example, 2**30 instead of 2**31. By the way, there is a probability (less than 1/2**31 percent) of generating same intids, if thread switching occurs between the line 114 and 115. Tadashi Matsumoto [EMAIL PROTECTED] ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: Re: [Zope3-dev] Potential bug of intid
On Sun, 11 Dec 2005 08:30:06 -0500 [EMAIL PROTECTED] wrote. On Dec 11, 2005, at 4:23 AM, Tadashi Matsumoto wrote: Hello I was looking into the following code of intid's __init__.py. ... I think it is better to use, for example, 2**30 instead of 2**31. Looks like a good catch to me. Also looks like 2**31 -1 would be sufficient (typical sys.maxint), No it wouldn't, because intid will increase. rather than 2**30...or maybe just sys.maxint (though I wouldn't be surprised if the author had consciously rejected maxint for database portability). By the way, there is a probability (less than 1/2**31 percent) of generating same intids, if thread switching occurs between the line 114 and 115. At which point you should get a conflict error, which in Zope would cause one of the transactions to be retried without the user's knowledge and everything would still be good. Gary Generating same intids does not always mean a conflict error. Tadashi Matsumoto [EMAIL PROTECTED] ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Potential bug of intid
On Sun, 11 Dec 2005 12:43:03 -0500 [EMAIL PROTECTED] wrote. On Dec 11, 2005, at 11:00 AM, Tadashi Matsumoto wrote: On Sun, 11 Dec 2005 08:30:06 -0500 [EMAIL PROTECTED] wrote. At which point you should get a conflict error, which in Zope would cause one of the transactions to be retried without the user's knowledge and everything would still be good. Generating same intids does not always mean a conflict error. Why not? AFAIK it will. Gary At first, I was thinking the following situation. 114 uid = self._generateId() 115 self.refs[uid] = key Suppose that one thread have generated an id and is stopping at line 114, another thread happen to genarate same id and finish it's transaction, after that the former finishes the transaction. Then the conflict will not occur. But it seems to be my misunderstanding of zodb's mechanism, I am sorry. Tadashi Matsumoto [EMAIL PROTECTED] ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
