Thanks a lot for the suggestions, I will give it a much closer look now... Lynton
On Dec 18, 7:19 pm, "Michael Bayer" <mike...@zzzcomputing.com> wrote: > Lynton Grice wrote: > > Hi there, > > > I am writing a little persistant Queue class/s using sqlalchemy and > > sqlite. All is going fine with the development but now I have reached > > the stage where I would like to have each request spawned as a new > > process (from Python's muliprocessing module). > > The result of "sessionmaker()" is a class. Its defined at the module > level and should be globally present in memory, and automatically > available to any child processes that are spawned thereafter. It's not > something that's really appropriate to be passing around even within a > single process in most cases - its part of your application's globally > available structure. > > Additionally, when you use multiprocessing, any arguments passed across > the boundary of a process are pickled, sent over a pipe, and then > unpickled on the other side. For this reason, it is critical that the > arguments passed between parent/child are minimized in terms of their > size. This is in the multiprocessing > docs:http://docs.python.org/library/multiprocessing.html#programming-guide... > . So I would also argue that you really shouldn't be passing *any* > significant state across process boundaries. I would pass only small > easily-packageable bits of state that are just enough for the child > process to reconstruct what it needs from the application's global state > as well as the database. Things like classes, sessions, engines, > connections and transactions are not candidates for sending over a pipe. > > > > > > > The error I am getting is: "Can't pickle <class > > 'sqlalchemy.orm.session.Session'>" it's not the same object as > > sqlalchemy.orm.session.Session > > > pickle.PicklingError: > > > In my one class that is the closest to SQLite I have the following in > > the INIT method: > > > def __init__(self, queueConnection): > > self._queue = queueConnection.queue > > self._queueName = queueConnection.queue.name > > self._engine = queueConnection.engine > > self._sessionmaker = sessionmaker(bind=self._engine) > > self._session = self._sessionmaker() > > mapper(_Message, self._queue) > > > And if the calling client called the "SEND" method to send a message > > to the queue...the code is as follows: > > > def send(self, message, timeout=10, callback=None): > > start_pipe, end_pipe = Pipe() > > start_pipe.send(message) > > p = Process(target=self._queue_proxy.send, args= > > (end_pipe,)).start() > > result = p.join(timeout = timeout) > > if result == False: > > return "Timeout" > > else: > > return start_pipe.recv() > > > The exception is raised when the following line of code runs above: > > > p = Process(target=self._queue_proxy.send, args=(end_pipe,)).start > > () > > > NOTE: The "self._queue_proxy" object above is essentailly the object > > that holds the reference to the "SessionMaker" above. > > > Does anyone have any idea how I can get around this Pickling error? > > > Any help will be greatly appreciated ;-) > > > Lynton > > > -- > > > You received this message because you are subscribed to the Google Groups > > "sqlalchemy" group. > > To post to this group, send email to sqlalch...@googlegroups.com. > > To unsubscribe from this group, send email to > > sqlalchemy+unsubscr...@googlegroups.com. > > For more options, visit this group at > >http://groups.google.com/group/sqlalchemy?hl=en.- Hide quoted text - > > - Show quoted text - -- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To post to this group, send email to sqlalch...@googlegroups.com. To unsubscribe from this group, send email to sqlalchemy+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en.
