[sqlalchemy] sqlalchemy gc and memory leak
Hi,
inspecting the gc I see a sqlalchemy memory leak in my application,
here is the output from the gc:
class 'sqlalchemy.engine.base.Connection': 2 - 3 (+1)
class 'sqlalchemy.engine.base.RootTransaction': 2 - 3 (+1)
class 'sqlalchemy.util.LRUCache': 1 - 2 (+1)
class 'sqlalchemy.util.PopulateDict': 2 - 3 (+1)
type 'tuple': 5987 - 5990 (+3)
class 'sqlalchemy.sql.expression._BindParamClause': 73 - 80 (+7)
type 'dict': 8943 - 8954 (+11)
class 'sqlalchemy.sql.expression.Insert': 1 - 2 (+1)
class 'sqlalchemy.util.OrderedDict': 180 - 181 (+1)
type 'instancemethod': 490 - 487 (-3)
class 'sqlalchemy.dialects.sqlite.base.SQLiteCompiler': 2 - 3 (+1)
type 'collections.defaultdict': 34 - 35 (+1)
type 'weakref': 3830 - 3831 (+1)
type 'list': 2953 - 2963 (+10)
every time I make some database object a reference is added to
sqlalchemy objects and never released. I'm sure the problem is my
application and not sa, however I would like to know how to force
sqlalchemy to delete objects references.
I'm using these function to query gc:
def gcHistogram(self):
import gc
result = {}
for o in gc.get_objects():
t = type(o)
count = result.get(t, 0)
result[t] = count + 1
print len(result)
return result
def diffHists(self,h1, h2):
for k in h1:
if k in h2:
if h1[k] != h2[k]:
print %s: %d - %d (%s%d) % (
k, h1[k], h2[k], h2[k]
h1[k] and + or , h2[k] - h1[k])
thanks
Nicola
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalch...@googlegroups.com.
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] Re: 0.6 and c extensions
On 21 Mar, 21:11, Gaetan de Menten gdemen...@gmail.com wrote: On Sun, Mar 21, 2010 at 17:13, drakkan drakkan1...@gmail.com wrote: On 21 Mar, 14:31, Antoine Pitrou solip...@pitrou.net wrote: Le dimanche 21 mars 2010 à 02:10 -0700, drakkan a écrit : a really interesting feature in sa 0.6 are the c extensions, however I think they should be implemented using ctypes so if python ctypes extension is available (default in python =2.5 and available even for 2.4) the c extensions are used as default, ctypes cannot compile your own C code. Yes I know my question is why sqlalchemy wrote his own C code and not use ctypes? Correct me if I'm wrong, but AFAIK ctypes is a way to interface your Python code with external C libraries. You cannot create any new functionality with ctypes. And what I did for the C extension was rewrite in C the most speed-critical parts of *SQLAlchemy* (not of an external lib) ! You are right I misunderstood the implementation sorry I could have used cython (and I might actually rewrite what I have done thus far in cython at some point in the future), but ctypes??? -- Gaëtan de Menten -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] 0.6 and c extensions
Hi, a really interesting feature in sa 0.6 are the c extensions, however I think they should be implemented using ctypes so if python ctypes extension is available (default in python =2.5 and available even for 2.4) the c extensions are used as default, using ctypes no compilation is needed you can use the same code on x86,x86_64, arm ecc.. what do you think about? drakkan -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] Re: 0.6 and c extensions
On 21 Mar, 14:31, Antoine Pitrou solip...@pitrou.net wrote: Le dimanche 21 mars 2010 à 02:10 -0700, drakkan a écrit : Hi, a really interesting feature in sa 0.6 are the c extensions, however I think they should be implemented using ctypes so if python ctypes extension is available (default in python =2.5 and available even for 2.4) the c extensions are used as default, ctypes cannot compile your own C code. Yes I know my question is why sqlalchemy wrote his own C code and not use ctypes? -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] Problem with many-to-many relation on the same table
Hi all,
my class Clients inherits from BaseApparati
and a client can have multiple relations with others clients.
I use the class ClientCrossRef as table for the relation.
class BaseApparati(Base, Dictionaryable):
__tablename__ = 'baseapparati'
id = Column(Integer, Sequence((__tablename__+'id_seq')[-30:]),
autoincrement=True, primary_key=True)
realtype = Column(Unicode(30), nullable=False, index=True)
__mapper_args__ = {'polymorphic_on': realtype,
'polymorphic_identity': 'baseapparati'
class Clients(BaseApparati):
__tablename__ = 'clients'
__mapper_args__ = {'polymorphic_identity': 'client'}
id = Column(Integer, ForeignKey('baseapparati.id'), primary_key=True)
class ClientCrossRef(Base):
__tablename__ = 'clientcrossref'
id = Column(Integer, Sequence((__tablename__+'id_seq')[-30:]),
autoincrement=True, primary_key=True)
master_id = Column(Integer, ForeignKey('clients.id'))
slave_id = Column(Integer, ForeignKey('clients.id'))
master = relation(Clients, uselist=False, primaryjoin= master_id ==
Clients.id, foreign_keys = Clients.id)
slave = relation(Clients, uselist=False, primaryjoin= slave_id ==
Clients.id, foreign_keys = Clients.id)
def __init__(self, master, slave):
self.master = master
self.slave = slave
When i try to commit a new ClientCrossRef, SQLAlchemy seems not to
know the objects i passed to the constructor.
Example:
v= sa.Session.query(sa.Clients).all()
v[0]
Client('client_1')
v[1]
Client('client_2')
so v[0] and v[1] are instances of Clients model
sa.Session.add( sa.ClientCrossRef( v[0], v[1] ) )
sa.Session.commit()
this is the sql genrated:
INSERT INTO clientcrossref (id, master_id, slave_id) VALUES
(:id, :master_id, :slave_id)
{'master_id': None, 'id': 3, 'slave_id': None}
as you can see, master_id and slave_id are == None, so it raise an
exception.
If i change the constructor of ClientCrossRef to accept the id of the
object (not the object!)
it obviously works:
def __init__(self, master_id, slave_id):
self.master_id = master_id
self.slave_id = slave_id
sa.Session.add( sa.ClientCrossRef( v[0].id, v[1].id) )
sa.Session.commit()
INSERT INTO clientcrossref (id, master_id, slave_id) VALUES
(:id, :master_id, :slave_id)
{'master_id': 1, 'id': 3, 'slave_id': 2}
So i think the problem is the definition of the relations 'master' and
'slave'.
Can anyone help me? I need to pass objects to the constructor, not the
ids.
Thanks
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalch...@googlegroups.com.
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] Re: sqlite and thread
On 16 Dic, 19:03, Michael Bayer mike...@zzzcomputing.com wrote: drakkan wrote: Hi, connectionstring = 'sqlite:///' + settings.DATABASE_PATH+'? check_same_thread=False' #engine = create_engine(connectionstring, echo=settings.DEBUG, echo_pool=settings.DEBUG) engine = create_engine(connectionstring, echo=settings.DEBUG, this line: pool=NullPool(lambda: sqlite3.connect(settings.DATABASE_PATH))) negates the effect of the check_same_thread flag in this line since the fully constructed pool + creation function you're sending means the URL isn't used: engine = create_engine(connectionstring, echo=settings.DEBUG, you only need pool_class=NullPool in your create_engine here. Example here:http://www.sqlalchemy.org/trac/wiki/DatabaseNotes#SQLite thanks, this way works better but there is another problem: 1) I make a long select in a separate thread 2) before 1) complete I make an insert or a delete in the main thread 3) when I commit the changes in 2) if the select is still running I get The transaction is inactive due to a rollback in a subtransaction. Issue rollback() to cancel the transaction. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] Re: sqlite and thread
On 20 Dic, 00:03, Michael Bayer mike...@zzzcomputing.com wrote: On Dec 19, 2009, at 4:50 PM, drakkan wrote: On 16 Dic, 19:03, Michael Bayer mike...@zzzcomputing.com wrote: drakkan wrote: Hi, connectionstring = 'sqlite:///' + settings.DATABASE_PATH+'? check_same_thread=False' #engine = create_engine(connectionstring, echo=settings.DEBUG, echo_pool=settings.DEBUG) engine = create_engine(connectionstring, echo=settings.DEBUG, this line: pool=NullPool(lambda: sqlite3.connect(settings.DATABASE_PATH))) negates the effect of the check_same_thread flag in this line since the fully constructed pool + creation function you're sending means the URL isn't used: engine = create_engine(connectionstring, echo=settings.DEBUG, you only need pool_class=NullPool in your create_engine here. Example here:http://www.sqlalchemy.org/trac/wiki/DatabaseNotes#SQLite thanks, this way works better but there is another problem: 1) I make a long select in a separate thread 2) before 1) complete I make an insert or a delete in the main thread 3) when I commit the changes in 2) if the select is still running I get The transaction is inactive due to a rollback in a subtransaction. Issue rollback() to cancel the transaction. that means an exception was thrown within flush(), the transaction was rolled back, and the exception rethrown. it should propagate outwards where you can catch it, or otherwise it will halt the application and dump to stderr. if you catch it and ignore it, you get that error when you attempt to continue. Sorry, I don't fully understand you answer: I have a main application that start a new thread to do a long running task, while the thread is running the main app do other things and when it commit the changes the thread has an exception caused by this commit, is the exception propagated to the main app even if I catch it in the thread? Why I have the exception only if the main app do a commit and no expetion if I wait the end of the thread before do other things? -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] Re: sqlite and thread
Thanks I found my error: I defined my session as scoped session but in a global file I was redefining it as session=sess() and session was not more scoped On 20 Dic, 00:37, Michael Bayer mike...@zzzcomputing.com wrote: On Dec 19, 2009, at 6:20 PM, drakkan wrote: Sorry, I don't fully understand you answer: I have a main application that start a new thread to do a long running task, while the thread is running the main app do other things and when it commit the changes the thread has an exception caused by this commit, is the exception propagated to the main app even if I catch it in the thread? Why I have the exception only if the main app do a commit and no expetion if I wait the end of the thread before do other things? Not sure what you have going on here, but if you are sharing the same Session object (not a scoped_session, but the actual session) between threads, you will get many errors - the Session is not threadsafe. That's the only way you could get behavior like the exception is propagated to the main app even if I catch it in the thread. If you are using a different Session in the main app versus the thread, then the only way you can get that error is if the Session has an exception, and you continue to try to use that same Session without rolling it back. I just added some detail to the FAQ section which includes an example athttp://www.sqlalchemy.org/trac/wiki/FAQ#Thetransactionisinactivedueto -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group athttp://groups.google.com/group/sqlalchemy?hl=en. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] sqlite and thread
Hi, I'm using a sqlite file based database, I'm having problem making database calls from two different thread, the error is the following: SQLite objects created in a thread can only be used in that same thread here is my sa configuration: from sqlalchemy.orm import sessionmaker from sqlalchemy.orm import scoped_session from sqlalchemy import create_engine from sqlalchemy.pool import NullPool from myapp import settings import sqlite3 connectionstring = 'sqlite:///' + settings.DATABASE_PATH+'? check_same_thread=False' #engine = create_engine(connectionstring, echo=settings.DEBUG, echo_pool=settings.DEBUG) engine = create_engine(connectionstring, echo=settings.DEBUG, pool=NullPool(lambda: sqlite3.connect(settings.DATABASE_PATH))) #engine = create_engine(connectionstring, echo=True, echo_pool=True) sess = scoped_session(sessionmaker(bind=engine)) #sess = sessionmaker(bind=engine) as you can see I'm using check_same_thread=False, Null connection pool and scoped session, what's wrong? thanks Nicola -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] Oracle timestamp and postgres datetime compatibility
Hi all, I have a sa model working with postgres, here is a code fragment: class Test(Base): __tablename__ = 'test' fileld1= Column(Unicode(40), nullable=False, primary_key=True) date = Column(DateTime, nullable=False) in postgres the sqltypes.DateTime is converted in: date timestamp without time zone NOT NULL I tryed to port my model to oracle and I found the same column has been translated in: DATE DATE NOT NULL I think it should be: DATE TIMESTAMP NOT NULL to have the same type between oracle and postgres, any hints? -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
[sqlalchemy] avoid select nextval bedore insert
Hi, I'm using postgres, sqlalchemy on every insert for the primary key do select nexval , postgres ia able to find itself the id, there is a way to avoid this select? thanks drakkan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] share objects with threads and scoped_session
Hi all,
I'm trying to share an object with a thread, I already tryed to use
Session.merge but I'm doing something wrong,
here is a test case:
from sqlalchemy import
*
from sqlalchemy.orm import
*
e = create_engine('postgres://postgres:postg...@127.0.0.1/test',
echo=True)
m = MetaData(e)
t1 = Table('t1', m, Column('a', Integer, primary_key=True),
Column('b', Integer))
class A(object):
def __init__(self, a, b):
self.a = a
self.b = b
mapper(A, t1)
m.create_all()
Session = scoped_session(sessionmaker())
Session.add(A(1, 1))
Session.commit()
import threading
def testthread(a):
a.a=2
Session.add(a)
Session.commit()
a = Session.query(A).get(1)
t=threading.Thread(target=testthread,args=(a,))
t.start()
and this is the generated expection:
Traceback (most recent call last):
File /usr/lib/python2.6/threading.py, line 525, in
__bootstrap_inner
self.run()
File /usr/lib/python2.6/threading.py, line 477, in run
self.__target(*self.__args, **self.__kwargs)
File testthread.py, line 28, in testthread
Session.add(a)
.
InvalidRequestError: Object 'A at 0x2dbf390' is already attached to
session '47955344' (this is '47969872')
what is the correct way to share object between different threads?
regards
drakkan
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: share objects with threads and scoped_session
Seems that once I defined a scoped session as:
Session = scoped_session(sessionmaker())
I can switch to a non scoped session simply calling:
sess=Session()
infact:
sess=Session
sess
sqlalchemy.orm.scoping.ScopedSession object at 0x1debbd0
sess=Session()
sess
sqlalchemy.orm.session.Session object at 0x2383050
is this the intended behaviuor?
thanks
drakkan
On 5 Ago, 11:27, drakkan drakkan1...@gmail.com wrote:
Hi all,
I'm trying to share an object with a thread, I already tryed to use
Session.merge but I'm doing something wrong,
here is a test case:
from sqlalchemy import
*
from sqlalchemy.orm import
*
e = create_engine('postgres://postgres:postg...@127.0.0.1/test',
echo=True)
m = MetaData(e)
t1 = Table('t1', m, Column('a', Integer, primary_key=True),
Column('b', Integer))
class A(object):
def __init__(self, a, b):
self.a = a
self.b = b
mapper(A, t1)
m.create_all()
Session = scoped_session(sessionmaker())
Session.add(A(1, 1))
Session.commit()
import threading
def testthread(a):
a.a=2
Session.add(a)
Session.commit()
a = Session.query(A).get(1)
t=threading.Thread(target=testthread,args=(a,))
t.start()
and this is the generated expection:
Traceback (most recent call last):
File /usr/lib/python2.6/threading.py, line 525, in
__bootstrap_inner
self.run()
File /usr/lib/python2.6/threading.py, line 477, in run
self.__target(*self.__args, **self.__kwargs)
File testthread.py, line 28, in testthread
Session.add(a)
.
InvalidRequestError: Object 'A at 0x2dbf390' is already attached to
session '47955344' (this is '47969872')
what is the correct way to share object between different threads?
regards
drakkan
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: Thread problem
Thanks I defined scoped session the wrong way:
engine = create_engine(connectionstring, echo=settings.DEBUG,
echo_pool=settings.DEBUG,
pool_size=20, max_overflow=400)
session = scoped_session(sessionmaker(bind=engine))
s=session()
instead of:
engine = create_engine(connectionstring, echo=settings.DEBUG,
echo_pool=settings.DEBUG,
pool_size=20, max_overflow=400)
s = scoped_session(sessionmaker(bind=engine))
so the session was not scoped and give the errors I report, thanks for
your support,
regards
drakkan
On 3 Ago, 06:32, Michael Bayer mike...@zzzcomputing.com wrote:
heres my script, runs fine
from sqlalchemy import *
from sqlalchemy.orm import *
e = create_engine('postgres://scott:ti...@localhost/test', echo=True)
m = MetaData(e)
t1 = Table('t1', m, Column('a', Integer, primary_key=True),
Column('b', Integer))
t2 = Table('t2', m, Column('a', Integer, primary_key=True),
Column('arefid', Integer, ForeignKey('t1.a')))
class A(object):
def __init__(self, a, b):
self.a = a
self.b = b
class B(object):
def __init__(self, b):
self.aref = b
mapper(A, t1)
mapper(B, t2, properties={
'aref':relation(A)
})
m.create_all()
Session = scoped_session(sessionmaker())
Session.add(A(1, 1))
Session.commit()
import threading
def insertalarm():
a = Session.query(A).get(1)
b = B(a)
Session.add(b)
Session.commit()
for i in range(100):
t=threading.Thread(target=insertalarm)
t.start()
it doesnt run 100 concurrent threads since the inserts happen more
quickly than the threads can be started.
On Aug 2, 2009, at 10:28 PM, drakkan wrote:
and here is the django equivalent that works as expeted:
def insertalarmdjango():
print 'qui'
a=Allarmi()
a.tipoallarme=TipAllarmi.objects.get(pk=1)
a.save()
for i in range(100):
t=threading.Thread(target=insertalarmdjango)
t.start()
On 3 Ago, 04:21, drakkan drakkan1...@gmail.com wrote:
The problem is riproducible with this simple script:
import samodels as sa
import threading
def insertalarm():
s=sa.Session
a=sa.Allarmi(s.query(sa.TipoAllarmi).get(1))
s.add(a)
s.commit()
for i in range(100):
t=threading.Thread(target=insertalarm)
t.start()
with range(1) works as expect, if you change the range the script
hang
or give errors, I'm using scoped_session as you suggested
On 3 Ago, 03:34, drakkan drakkan1...@gmail.com wrote:
After the select there is an insert, can three concurrent threads
inserting data in the same table cause the hang?
If so how can I avoid this?
Please note that until now the same application was using django orm
with no deadlock problems, I only changed the query to use
sqlalchemy
thanks
drakkan
On 3 Ago, 03:20, Michael Bayer mike...@zzzcomputing.com wrote:
On Aug 2, 2009, at 9:02 PM, drakkan wrote:
here is a sample sa output when the application hangs:
009-08-03 01:05:20,458 INFO sqlalchemy.engine.base.Engine.0x...
1634
{'param_1': None}
2009-08-03 01:05:33,673 INFO sqlalchemy.engine.base.Engine.0x...
1634
SELECT tipallarmi.id AS tipallarmi_id, tipallarmi.codice AS
tipallarmi_codice, tipallarmi.tipo AS tipallarmi_tipo
FROM tipallarmi
WHERE tipallarmi.codice = %(codice_1)s
LIMIT 1 OFFSET 0
2009-08-03 01:05:33,681 INFO sqlalchemy.engine.base.Engine.0x...
1634
{'codice_1': 1}
2009-08-03 01:05:33,679 INFO sqlalchemy.engine.base.Engine.0x...
1634
SELECT tipallarmi.id AS tipallarmi_id, tipallarmi.codice AS
tipallarmi_codice, tipallarmi.tipo AS tipallarmi_tipo
FROM tipallarmi
WHERE tipallarmi.codice = %(codice_1)s
LIMIT 1 OFFSET 0
2009-08-03 01:05:33,681 INFO sqlalchemy.engine.base.Engine.0x...
1634
{'codice_1': 1}
2009-08-03 01:05:33,680 INFO sqlalchemy.engine.base.Engine.0x...
1634
SELECT tipallarmi.id AS tipallarmi_id, tipallarmi.codice AS
tipallarmi_codice, tipallarmi.tipo AS tipallarmi_tipo
FROM tipallarmi
WHERE tipallarmi.codice = %(codice_1)s
LIMIT 1 OFFSET 0
2009-08-03 01:05:33,682 INFO sqlalchemy.engine.base.Engine.0x...
1634
{'codice_1': 1}
seems there are three concurrent threads making the same query
if you have three threads all calling query.get() then that would
be
the result.
none of the SELECTs should be causing deadlocks so something else
is
causing it to hang.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Thread problem
Hi all, I know sqlalchemy session is not thread safe but in my application I need to use thread, I have my session defined as following: engine = create_engine(connectionstring, echo=settings.DEBUG, echo_pool=settings.DEBUG, pool_size=20, max_overflow=400) session = sessionmaker(bind=engine) sess=session() and I use sess to make database logic, my database is postgres In my app I launch two threads that can work the same object, suppose I have an object named tr i pass tr.id to the thread and in the thread I make a new query to get the object by id, additionally one thread make read operation and the other write operation, this seems not enough the database get locked and the application hang, any hints? thanks drakkan1000 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Thread problem
here is a sample sa output when the application hangs:
009-08-03 01:05:20,458 INFO sqlalchemy.engine.base.Engine.0x...1634
{'param_1': None}
2009-08-03 01:05:33,673 INFO sqlalchemy.engine.base.Engine.0x...1634
SELECT tipallarmi.id AS tipallarmi_id, tipallarmi.codice AS
tipallarmi_codice, tipallarmi.tipo AS tipallarmi_tipo
FROM tipallarmi
WHERE tipallarmi.codice = %(codice_1)s
LIMIT 1 OFFSET 0
2009-08-03 01:05:33,681 INFO sqlalchemy.engine.base.Engine.0x...1634
{'codice_1': 1}
2009-08-03 01:05:33,679 INFO sqlalchemy.engine.base.Engine.0x...1634
SELECT tipallarmi.id AS tipallarmi_id, tipallarmi.codice AS
tipallarmi_codice, tipallarmi.tipo AS tipallarmi_tipo
FROM tipallarmi
WHERE tipallarmi.codice = %(codice_1)s
LIMIT 1 OFFSET 0
2009-08-03 01:05:33,681 INFO sqlalchemy.engine.base.Engine.0x...1634
{'codice_1': 1}
2009-08-03 01:05:33,680 INFO sqlalchemy.engine.base.Engine.0x...1634
SELECT tipallarmi.id AS tipallarmi_id, tipallarmi.codice AS
tipallarmi_codice, tipallarmi.tipo AS tipallarmi_tipo
FROM tipallarmi
WHERE tipallarmi.codice = %(codice_1)s
LIMIT 1 OFFSET 0
2009-08-03 01:05:33,682 INFO sqlalchemy.engine.base.Engine.0x...1634
{'codice_1': 1}
seems there are three concurrent threads making the same query
On 3 Ago, 00:49, drakkan drakkan1...@gmail.com wrote:
In my postgres log I have:
ERROR: duplicate key value violates unique constraint allarmi_pkey
when sqlalchemy give the error but nothing when the application hangs
On 2 Ago, 23:45, Michael Bayer mike...@zzzcomputing.com wrote:
unless theres some detail missing here, you should be using
scoped_session() so that there's one session per thread.
On Aug 2, 2009, at 5:04 PM, drakkan wrote:
Hi all,
I know sqlalchemy session is not thread safe but in my application I
need to use thread,
I have my session defined as following:
engine = create_engine(connectionstring, echo=settings.DEBUG,
echo_pool=settings.DEBUG,
pool_size=20, max_overflow=400)
session = sessionmaker(bind=engine)
sess=session()
and I use sess to make database logic, my database is postgres
In my app I launch two threads that can work the same object, suppose
I have an object named tr i pass tr.id to the thread and in the
thread I make a new query to get the object by id, additionally one
thread make read operation and the other write operation, this seems
not enough the database get locked and the application hang, any
hints?
thanks
drakkan1000
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: Thread problem
After the select there is an insert, can three concurrent threads
inserting data in the same table cause the hang?
If so how can I avoid this?
Please note that until now the same application was using django orm
with no deadlock problems, I only changed the query to use sqlalchemy
thanks
drakkan
On 3 Ago, 03:20, Michael Bayer mike...@zzzcomputing.com wrote:
On Aug 2, 2009, at 9:02 PM, drakkan wrote:
here is a sample sa output when the application hangs:
009-08-03 01:05:20,458 INFO sqlalchemy.engine.base.Engine.0x...1634
{'param_1': None}
2009-08-03 01:05:33,673 INFO sqlalchemy.engine.base.Engine.0x...1634
SELECT tipallarmi.id AS tipallarmi_id, tipallarmi.codice AS
tipallarmi_codice, tipallarmi.tipo AS tipallarmi_tipo
FROM tipallarmi
WHERE tipallarmi.codice = %(codice_1)s
LIMIT 1 OFFSET 0
2009-08-03 01:05:33,681 INFO sqlalchemy.engine.base.Engine.0x...1634
{'codice_1': 1}
2009-08-03 01:05:33,679 INFO sqlalchemy.engine.base.Engine.0x...1634
SELECT tipallarmi.id AS tipallarmi_id, tipallarmi.codice AS
tipallarmi_codice, tipallarmi.tipo AS tipallarmi_tipo
FROM tipallarmi
WHERE tipallarmi.codice = %(codice_1)s
LIMIT 1 OFFSET 0
2009-08-03 01:05:33,681 INFO sqlalchemy.engine.base.Engine.0x...1634
{'codice_1': 1}
2009-08-03 01:05:33,680 INFO sqlalchemy.engine.base.Engine.0x...1634
SELECT tipallarmi.id AS tipallarmi_id, tipallarmi.codice AS
tipallarmi_codice, tipallarmi.tipo AS tipallarmi_tipo
FROM tipallarmi
WHERE tipallarmi.codice = %(codice_1)s
LIMIT 1 OFFSET 0
2009-08-03 01:05:33,682 INFO sqlalchemy.engine.base.Engine.0x...1634
{'codice_1': 1}
seems there are three concurrent threads making the same query
if you have three threads all calling query.get() then that would be
the result.
none of the SELECTs should be causing deadlocks so something else is
causing it to hang.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Re: Thread problem
The problem is riproducible with this simple script:
import samodels as sa
import threading
def insertalarm():
s=sa.Session
a=sa.Allarmi(s.query(sa.TipoAllarmi).get(1))
s.add(a)
s.commit()
for i in range(100):
t=threading.Thread(target=insertalarm)
t.start()
with range(1) works as expect, if you change the range the script hang
or give errors, I'm using scoped_session as you suggested
On 3 Ago, 03:34, drakkan drakkan1...@gmail.com wrote:
After the select there is an insert, can three concurrent threads
inserting data in the same table cause the hang?
If so how can I avoid this?
Please note that until now the same application was using django orm
with no deadlock problems, I only changed the query to use sqlalchemy
thanks
drakkan
On 3 Ago, 03:20, Michael Bayer mike...@zzzcomputing.com wrote:
On Aug 2, 2009, at 9:02 PM, drakkan wrote:
here is a sample sa output when the application hangs:
009-08-03 01:05:20,458 INFO sqlalchemy.engine.base.Engine.0x...1634
{'param_1': None}
2009-08-03 01:05:33,673 INFO sqlalchemy.engine.base.Engine.0x...1634
SELECT tipallarmi.id AS tipallarmi_id, tipallarmi.codice AS
tipallarmi_codice, tipallarmi.tipo AS tipallarmi_tipo
FROM tipallarmi
WHERE tipallarmi.codice = %(codice_1)s
LIMIT 1 OFFSET 0
2009-08-03 01:05:33,681 INFO sqlalchemy.engine.base.Engine.0x...1634
{'codice_1': 1}
2009-08-03 01:05:33,679 INFO sqlalchemy.engine.base.Engine.0x...1634
SELECT tipallarmi.id AS tipallarmi_id, tipallarmi.codice AS
tipallarmi_codice, tipallarmi.tipo AS tipallarmi_tipo
FROM tipallarmi
WHERE tipallarmi.codice = %(codice_1)s
LIMIT 1 OFFSET 0
2009-08-03 01:05:33,681 INFO sqlalchemy.engine.base.Engine.0x...1634
{'codice_1': 1}
2009-08-03 01:05:33,680 INFO sqlalchemy.engine.base.Engine.0x...1634
SELECT tipallarmi.id AS tipallarmi_id, tipallarmi.codice AS
tipallarmi_codice, tipallarmi.tipo AS tipallarmi_tipo
FROM tipallarmi
WHERE tipallarmi.codice = %(codice_1)s
LIMIT 1 OFFSET 0
2009-08-03 01:05:33,682 INFO sqlalchemy.engine.base.Engine.0x...1634
{'codice_1': 1}
seems there are three concurrent threads making the same query
if you have three threads all calling query.get() then that would be
the result.
none of the SELECTs should be causing deadlocks so something else is
causing it to hang.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] Migration from django orm: override save method?
Hi all, I'm migrating from django orm to sqlalchemy, in django was very simple to override save method to do own stuff before save the value to database, for example: class TestModel(models.Model): field1=models.CharField(max_length=255) def save(): self.field1=self.field1.upper() super(TestModel,self).save() so when I add or modify an object it is ever converted to uppercase. In sa this simple model become: class TestModel(Base) __tablename__='testtable' field1=Column(Unicode(255)) def __init__(field1): self.field1=field1.upper() this way if I call the init method field1 is converted to upper but if i modify the field I have to manually convert to upper. There is some way to override save method as in django orm? regards drakkan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups sqlalchemy group. To post to this group, send email to sqlalchemy@googlegroups.com To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en -~--~~~~--~~--~--~---
[sqlalchemy] Re: Migration from django orm: override save method?
Thanks,
I did it using synonym
class TestModel(Base)
__tablename__='testtable'
_field1=Column('field1',Unicode(255))
def _set_field1(self, field1):
self._field1 = self.field1.upper()
def _get_field1(self):
return self._fiedl1
field1 = synonym('_field1', descriptor=property
(_get_field1,_set_field1))
thanks
drakkan
On 15 Giu, 17:52, Laurent Rahuel laurent.rah...@gmail.com wrote:
Hi,
You should take a look at MapperExtension. Then you'll be able to add
method which would be called depending on the orm actions.
For example, you'll be able to add your own custom before_insert or
after_insert or after_delete methods.
Regards,
Laurent
Le 15/06/2009 17:36, Didip Kerabat a écrit :
Quick note about __init__ method. SA select(..) or query(...) does not
call __init__(), so your to upper logic won't be executed then. If you
want it to be called every object construction you need to do this:
from sqlalchemy import orm
@orm.reconstructor
def some_function():
self.field1=field1.upper()
# Call that method inside __init__ as well
def __init__(self):
some_function()
Sorry for not answering the problem.
- Didip -
On Mon, Jun 15, 2009 at 1:43 AM, drakkan drakkan1...@gmail.com
mailto:drakkan1...@gmail.com wrote:
Hi all,
I'm migrating from django orm to sqlalchemy, in django was very simple
to override save method to do own stuff before save the value to
database, for example:
class TestModel(models.Model):
field1=models.CharField(max_length=255)
def save():
self.field1=self.field1.upper()
super(TestModel,self).save()
so when I add or modify an object it is ever converted to uppercase.
In sa this simple model become:
class TestModel(Base)
__tablename__='testtable'
field1=Column(Unicode(255))
def __init__(field1):
self.field1=field1.upper()
this way if I call the init method field1 is converted to upper but if
i modify the field I have to manually convert to upper. There is some
way to override save method as in django orm?
regards
drakkan
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] On delete cascade, set null
Hi all,
I have the following mapping:
class SharesUsers(DeclarativeBase):
__tablename__='m2m_shares_users'
id=Column(Integer,autoincrement=True,primary_key=True)
user_id=Column(Integer,ForeignKey
('samba_users.id',onupdate=CASCADE,ondelete=CASCADE),nullable=False)
share_id=Column(Integer,ForeignKey
('samba_shares.id',onupdate=CASCADE,ondelete=CASCADE),nullable=False)
access_type=Column(SmallInteger,default=0,nullable=False)
user = relation(SambaUsers, backref=backref
('share_users'),lazy=False,passive_deletes=True)
share = relation(SambaShares, backref=backref
('users_shares'),lazy=False,passive_deletes=True)
when I delete a share object in the table m2m_share_users the
corresponding key is set to null and not deleted, I have on delete
cascade in my database definition,
how can I do a delete cascade without load related objects in memory
and without perform an update?
regards
drakkan
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
[sqlalchemy] many to many extra fields and association object
Hi all,
I'm new to sqlalchemy, I'm using 0.5rc4 and I want to map a many to
many relation with extra fields on the association table using
declarative way, I read
http://www.sqlalchemy.org/docs/05/mappers.html#association-object
but there there is the non declarative way, here is what I have in my
mappig:
access_type=0 - read only
access_type=1 - read write
m2m_users_shares=Table('m2m_users_share',metadata,
Column('user_id',Integer,ForeignKey('samba_users.id',
onupdate=CASCADE, ondelete=CASCADE,nullable=False)),
Column('share_id',Integer,ForeignKey('samba_shares.id',
onupdate=CASCADE, ondelete=CASCADE,nullable=False)),
Column('access_type',SmallInteger,default=0,nullable=False)
)
class SambaUsers(DeclarativeBase):
__tablename__='samba_users'
id=Column(Integer,primary_key=True)
username=Column(Unicode(80),nullable=False,unique=True,index=True)
password=Column(Unicode(255),nullable=False)
attivo=Column(Boolean(),default=True,nullable=False)
creato = Column(DateTime)
modificato = Column(DateTime, default=datetime.now)
def __init__(self,username,password):
self.creato=datetime.now()
self.username=username
self.password=password
def __repr__(self):
return 'SambaUser: username=%s' % (self.username)
class SambaShares(DeclarativeBase):
__tablename__='samba_shares'
id=Column(Integer,primary_key=True)
nome=Column(Unicode(255),nullable=False,unique=True,index=True)
path=Column(Unicode(255),nullable=False)
creato = Column(DateTime)
modificato = Column(DateTime, default=datetime.now)
#users = relation('SambaUsers', secondary=m2m_users_shares,
backref='shareusers')
users = relation('SambaUsers', backref='shareusers')
def __init__(self,nome,path):
self.creato=datetime.now()
self.nome=nome
self.path=path
def __repr__(self):
return 'SambaShare: nome=%s, path=%s' % (
self.nome,self.path)
in samba share I removed secondary but now how sqlalchemy know about
m2m relation?
thanks
drakkan
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To post to this group, send email to sqlalchemy@googlegroups.com
To unsubscribe from this group, send email to
sqlalchemy+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sqlalchemy?hl=en
-~--~~~~--~~--~--~---
