Re: Sharing objects between processes
On Mon, 2009-03-09 at 14:57 -0700, Aaron Brady wrote:\
>
> import threading
> import time
>
>
> class Globals:
> cont= True
> content= { }
> lock_content= threading.Lock( )
>
> def read( ):
> out= open( 'temp.txt', 'w' )
> while Globals.cont:
> with Globals.lock_content:
> rep= repr( Globals.content )
> out.write( rep )
> out.write( '\n' )
> time.sleep( 1 )
>
> def write_random( ):
> import random
> while Globals.cont:
> num= random.randint( 1, 100 )
> letter= random.choice( 'abcedfghij' )
> with Globals.lock_content:
> Globals.content[ num ]= letter
> time.sleep( 1 )
>
> def write_user_inp( ):
> next_key= 101
> while Globals.cont:
> us_in= input( '%i--> '% next_key )
> if not us_in:
> Globals.cont= False
> return
> us_in= us_in[ :10 ]
> print( 'You entered: %s'% us_in )
> with Globals.lock_content:
> Globals.content[ next_key ]= us_in
> next_key+= 1
>
> read_thr= threading.Thread( target= read )
> read_thr.start( )
> wri_rand_thr= threading.Thread( target= write_random )
> wri_rand_thr.start( )
> wri_user_thr= threading.Thread( target= write_user_inp )
> wri_user_thr.start( )
>
> read_thr.join( )
> wri_rand_thr.join( )
> wri_user_thr.join( )
>
> Which is about the complexity of what you asked for. It wasn't tested
> with multiprocessing. Ver 3.0.1.
> --
> http://mail.python.org/mailman/listinfo/python-list
Wow, thanks for taking the time to put that together! Unfortunately,
I've attempted to modify it to use multiprocessing without success.
This works when the threading import is used, falls through with
multiprocessing, and runs with processing, though modifications to the
Globals class do not stick:
from __future__ import with_statement
#from threading import Thread as ControlType, Lock
#from multiprocessing import Process as ControlType, Lock
from processing import Process as ControlType, Lock
import time
class Globals:
cont= True
content= { }
lock_content= Lock( )
inputs = ['itext', 'text input', 'testing text', 'test']
def read( ):
#out= open( 'temp.txt', 'w' )
while Globals.cont:
with Globals.lock_content:
rep= repr( Globals.content )
print rep
#out.write( rep )
#out.write( '\n' )
time.sleep( 1 )
def write_random( ):
import random
while Globals.cont:
num= random.randint( 1, 100 )
letter= random.choice( 'abcedfghij' )
with Globals.lock_content:
Globals.content[ num ]= letter
time.sleep( 1 )
def write_user_inp( ):
import random
next_key= 101
while Globals.cont:
if len(Globals.inputs):
us_in = Globals.inputs.pop()
else:
Globals.cont= False
return
us_in= us_in[ :10 ]
print( 'You entered: %s'% us_in )
with Globals.lock_content:
Globals.content[ next_key ]= us_in
next_key+= 1
time.sleep( 1 )
read_thr= ControlType( target= read )
read_thr.start( )
wri_rand_thr= ControlType( target= write_random )
wri_rand_thr.start( )
wri_user_thr= ControlType( target= write_user_inp )
wri_user_thr.start( )
read_thr.join( )
wri_rand_thr.join( )
wri_user_thr.join( )
However, I think I (freaking finally) managed to get this working after
reading the syncmanager documentation for the umpteenth time; it
involves extending the SyncManager class to allow an arbitrary class as
a member (this might be possible just by calling register(...) on
SyncManager itself, but this is how the doc said to handle it). It also
provides its own Lock implementations, which I used within the new
managed class to apply with locking:
from __future__ import with_statement
from multiprocessing import Process, Lock, current_process
from multiprocessing.managers import SyncManager
from random import choice, randint
from time import sleep
class TestClass(object):
name = 'testclassname'
lock = None
def setval(self, value):
with self.lock:
self.name = value
def getval(self):
with self.lock:
return self.name
def __init__(self, lock):
self.lock = lock
class TestManager(SyncManager):
pass
TestManager.register('TC', TestClass)
manager = TestManager()
manager.start()
class TestProcess(Process):
obj = None
new_name = None
def __init__(self, obj):
self.obj = obj
Process._
Re: Sharing objects between processes
On Mar 9, 4:21 pm, ET wrote:
> On Mon, 2009-03-09 at 13:58 -0700, Aaron Brady wrote:
> > On Mar 9, 2:17 pm, ET wrote:
> > > On Mon, 2009-03-09 at 11:04 -0700, Aaron Brady wrote:
snip
> > Here's what we have to work with from you:
>
> > > > > > > The problem, as briefly as possible:
> > > > > > > I have three processes which need to safely read and update two
> > > > > > > objects.
>
> > Can they be subprocesses of eachother? That is, can one master spawn
> > the others as you desire? Can you have one process running, and
> > connect to it with sockets, pipes, (mailslots,) etc., and just give
> > and get the information to it? Then, synch. is a lot easier.
>
> > Do you need MROW multiple-reader one-writer synchro., or can they all
> > go one at a time? Is deadlock a concern? Can you use OBL(OE) one big
> > lock over everything, or do you need individual locks on elements of
> > the data structure?
>
> > Can you use fire-and-forget access, or do you need return values from
> > your calls? Do you need to wait for completion of anything?
>
> > 'xmlrpc': remote procedure calls might pertain.
> > --
> >http://mail.python.org/mailman/listinfo/python-list
>
> My intention is to have one object which is updated by two threads, and
> read by a third... one updating thread takes user input, and the other
> talks to a remote server. Which are children of the others really
> doesn't matter to me; if there's a way to handle that to make this work
> more efficiently, excellent.
>
> Yes, I can use a single lock over the entire object; finer control isn't
> necessary. I also don't need to avoid blocking on reads; reading during
> a write is something I also want to avoid.
>
> I was hoping to end up with something simple and readable (hence trying
> 'with' and then regular locks first), but if pipes are the best way to
> do this, I'll have to start figuring those out. If I have to resort to
> XMLRPC I'll probably ditch the process idea and leave it at threads,
> though I'd rather avoid this.
>
> Is there, perhaps, a sensible way to apply queues and/or managers to
> this? Namespaces also seemed promising, but having no experience with
> these things, the doc did not get me far.
Here is some code and the output. There's a risk that the 'read'
thread drops the last input from the writers. It's not tested well
(only once).
One thread writes a key in range 1-100, and value in range 'a-j' to a
dictionary, every second. Another thread waits for user input, and
writes it as a value with successively higher keys, starting at 101,
also to the dictionary. The reader thread obtains a representation of
the dictionary, and appends it to a file, without sorting, also every
second.
The output is this:
{}
{87: 'd'}
{76: 'd', 87: 'd'}
{80: 'e', 76: 'd', 87: 'd'}
{80: 'e', 12: 'e', 76: 'd', 87: 'd'}
{80: 'e', 12: 'e', 71: 'h', 76: 'd', 87: 'd'}
{99: 'f', 101: 'abcd', 71: 'h', 76: 'd', 12: 'e', 80: 'e', 87: 'd'}
{99: 'f', 101: 'abcd', 102: 'efgh', 71: 'h', 76: 'd', 10: 'd', 12:
'e', 80: 'e', 87: 'd'}
{99: 'f', 101: 'abcd', 102: 'efgh', 71: 'h', 76: 'd', 10: 'd', 11:
'g', 12: 'e', 80: 'e', 87: 'd', 103: 'ijklm'}
Which is highly uninteresting. You can see my inputs at 101, 102, and
103. The code is this.
import threading
import time
class Globals:
cont= True
content= { }
lock_content= threading.Lock( )
def read( ):
out= open( 'temp.txt', 'w' )
while Globals.cont:
with Globals.lock_content:
rep= repr( Globals.content )
out.write( rep )
out.write( '\n' )
time.sleep( 1 )
def write_random( ):
import random
while Globals.cont:
num= random.randint( 1, 100 )
letter= random.choice( 'abcedfghij' )
with Globals.lock_content:
Globals.content[ num ]= letter
time.sleep( 1 )
def write_user_inp( ):
next_key= 101
while Globals.cont:
us_in= input( '%i--> '% next_key )
if not us_in:
Globals.cont= False
return
us_in= us_in[ :10 ]
print( 'You entered: %s'% us_in )
with Globals.lock_content:
Globals.content[ next_key ]= us_in
next_key+= 1
read_thr= threading.Thread( target= read )
read_thr.start( )
wri_rand_thr= threading.Thread( target= write_random )
wri_rand_thr.start( )
wri_user_thr= threading.Thread( target= write_user_inp )
wri_user_thr.start( )
read_thr.join( )
wri_rand_thr.join( )
wri_user_thr.join( )
Which is about the complexity of what you asked for. It wasn't tested
with multiprocessing. Ver 3.0.1.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Sharing objects between processes
On Mar 9, 4:21 pm, ET wrote: > On Mon, 2009-03-09 at 13:58 -0700, Aaron Brady wrote: > > On Mar 9, 2:17 pm, ET wrote: > > > On Mon, 2009-03-09 at 11:04 -0700, Aaron Brady wrote: snip > > Here's what we have to work with from you: > > > > > > > > The problem, as briefly as possible: > > > > > > > I have three processes which need to safely read and update two > > > > > > > objects. > > > Can they be subprocesses of eachother? That is, can one master spawn > > the others as you desire? Can you have one process running, and > > connect to it with sockets, pipes, (mailslots,) etc., and just give > > and get the information to it? Then, synch. is a lot easier. > > > Do you need MROW multiple-reader one-writer synchro., or can they all > > go one at a time? Is deadlock a concern? Can you use OBL(OE) one big > > lock over everything, or do you need individual locks on elements of > > the data structure? > > > Can you use fire-and-forget access, or do you need return values from > > your calls? Do you need to wait for completion of anything? > > > 'xmlrpc': remote procedure calls might pertain. > > -- > >http://mail.python.org/mailman/listinfo/python-list > > My intention is to have one object which is updated by two threads, and > read by a third... one updating thread takes user input, and the other > talks to a remote server. Which are children of the others really > doesn't matter to me; if there's a way to handle that to make this work > more efficiently, excellent. > > Yes, I can use a single lock over the entire object; finer control isn't > necessary. I also don't need to avoid blocking on reads; reading during > a write is something I also want to avoid. > > I was hoping to end up with something simple and readable (hence trying > 'with' and then regular locks first), but if pipes are the best way to > do this, I'll have to start figuring those out. If I have to resort to > XMLRPC I'll probably ditch the process idea and leave it at threads, > though I'd rather avoid this. > > Is there, perhaps, a sensible way to apply queues and/or managers to > this? Namespaces also seemed promising, but having no experience with > these things, the doc did not get me far. I will hack at your requirements for a while. -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing objects between processes
On Mon, 2009-03-09 at 13:58 -0700, Aaron Brady wrote: > On Mar 9, 2:17 pm, ET wrote: > > On Mon, 2009-03-09 at 11:04 -0700, Aaron Brady wrote: > > > On Mar 9, 12:47 pm, ET wrote: > > > > > Message: 2 > > > > > Date: Sun, 8 Mar 2009 12:00:40 -0700 (PDT) > > > > > From: Aaron Brady > > > > > Subject: Re: Sharing objects between processes > > > > > To: python-l...@python.org > > > > > Message-ID: > > > > ><5514c3df-d74e-47d8-93fc-34dd5119e...@c11g2000yqj.googlegroups.com> > > > > > Content-Type: text/plain; charset=ISO-8859-1 > > > > > > > On Mar 8, 1:36?pm, ET wrote: > > > > > > I have been using the 'threading' library and decided to try > > > > > > swapping it > > > > > > out for 'processing'... while it's awesome that processing so > > > > > > closely > > > > > > mirrors the threading interface, I've been having trouble getting my > > > > > > processes to share an object in a similar way. > > > > > > > > Using the 'with' keyword didn't work, and using normal locks doesn't > > > > > > result in the expected behavior (I can get an object to be > > > > > > accessible in > > > > > > more than one process, and Python indicates that the instances are > > > > > > living at the same address in memory, but changes in one process > > > > > > are not > > > > > > reflected in the other[s]). ?I'm sure this is because my > > > > > > expectations > > > > > > are incorrect. :) > > > > > > > > The problem, as briefly as possible: > > > > > > I have three processes which need to safely read and update two > > > > > > objects. > > > > > > > > I've been working with processing, multiprocessing, and parallel > > > > > > python, > > > > > > trying to get this working... I suspect it can be accomplished with > > > > > > managers and/or queues, but if there's an elegant way to handle it, > > > > > > I > > > > > > have thus far failed to understand it. > > > > > > > > I don't particularly care which library I use; if someone has done > > > > > > this > > > > > > or can recommend a good method they're aware of, it'd be incredibly > > > > > > helpful. > > > > > > > > Thank you! > > > > > > > There is POSH: Python Object Sharing, which I learned about a while > > > > > ago, but never used much. > > > > > > >http://poshmodule.sourceforge.net/ > > > > > > > It's UNIX only. > > > > > > Thanks, I'll definitely keep that link handy... unfortunately, this > > > > particular project needs to run on Windows as well as Linux-based > > > > systems. > > > > > I don't recall whether there was anything in the source that's a deal- > > > breaker on Windows. The source is open, as you could see. > > > > > Other possibilities are 'shelve' and any database... fixed-length > > > pickles, a directory of pickles, etc. Maybe 'multiprocessing' would > > > work for your synchronization, while you use a more custom technique > > > for data exchange. > > > > > The only other thing I can do is bring to your attention an idea of > > > mine for sharing primitives. It's in the drawing board stage if you > > > want to help. > > > > > Of course, it's only after you turned down 'multiprocessing' and > > > 'POSH'. It does things they don't and vice versa. > > > -- > > >http://mail.python.org/mailman/listinfo/python-list > > > > I assumed it wouldn't work in Windows as you mentioned it was UNIX-only; > > the readme also states that it's for POSIX systems only. > > > > I'd be more than happy to use multiprocessing; I've attempted to do so. > > My question is largely how to implement it, as I have not managed to get > > it working despite several attempts from different angles. > > > > Unfortunately, I do need to handle more than primitives, otherwise I'd > > have attempted to use the shared ctypes present in at least one of > > processing/multi
Re: Sharing objects between processes
On Mar 9, 2:17 pm, ET wrote: > On Mon, 2009-03-09 at 11:04 -0700, Aaron Brady wrote: > > On Mar 9, 12:47 pm, ET wrote: > > > > Message: 2 > > > > Date: Sun, 8 Mar 2009 12:00:40 -0700 (PDT) > > > > From: Aaron Brady > > > > Subject: Re: Sharing objects between processes > > > > To: python-l...@python.org > > > > Message-ID: > > > > <5514c3df-d74e-47d8-93fc-34dd5119e...@c11g2000yqj.googlegroups.com> > > > > Content-Type: text/plain; charset=ISO-8859-1 > > > > > On Mar 8, 1:36?pm, ET wrote: > > > > > I have been using the 'threading' library and decided to try swapping > > > > > it > > > > > out for 'processing'... while it's awesome that processing so closely > > > > > mirrors the threading interface, I've been having trouble getting my > > > > > processes to share an object in a similar way. > > > > > > Using the 'with' keyword didn't work, and using normal locks doesn't > > > > > result in the expected behavior (I can get an object to be accessible > > > > > in > > > > > more than one process, and Python indicates that the instances are > > > > > living at the same address in memory, but changes in one process are > > > > > not > > > > > reflected in the other[s]). ?I'm sure this is because my expectations > > > > > are incorrect. :) > > > > > > The problem, as briefly as possible: > > > > > I have three processes which need to safely read and update two > > > > > objects. > > > > > > I've been working with processing, multiprocessing, and parallel > > > > > python, > > > > > trying to get this working... I suspect it can be accomplished with > > > > > managers and/or queues, but if there's an elegant way to handle it, I > > > > > have thus far failed to understand it. > > > > > > I don't particularly care which library I use; if someone has done > > > > > this > > > > > or can recommend a good method they're aware of, it'd be incredibly > > > > > helpful. > > > > > > Thank you! > > > > > There is POSH: Python Object Sharing, which I learned about a while > > > > ago, but never used much. > > > > >http://poshmodule.sourceforge.net/ > > > > > It's UNIX only. > > > > Thanks, I'll definitely keep that link handy... unfortunately, this > > > particular project needs to run on Windows as well as Linux-based > > > systems. > > > I don't recall whether there was anything in the source that's a deal- > > breaker on Windows. The source is open, as you could see. > > > Other possibilities are 'shelve' and any database... fixed-length > > pickles, a directory of pickles, etc. Maybe 'multiprocessing' would > > work for your synchronization, while you use a more custom technique > > for data exchange. > > > The only other thing I can do is bring to your attention an idea of > > mine for sharing primitives. It's in the drawing board stage if you > > want to help. > > > Of course, it's only after you turned down 'multiprocessing' and > > 'POSH'. It does things they don't and vice versa. > > -- > >http://mail.python.org/mailman/listinfo/python-list > > I assumed it wouldn't work in Windows as you mentioned it was UNIX-only; > the readme also states that it's for POSIX systems only. > > I'd be more than happy to use multiprocessing; I've attempted to do so. > My question is largely how to implement it, as I have not managed to get > it working despite several attempts from different angles. > > Unfortunately, I do need to handle more than primitives, otherwise I'd > have attempted to use the shared ctypes present in at least one of > processing/multiprocessing/parallel python. Here's what we have to work with from you: > > > > > The problem, as briefly as possible: > > > > > I have three processes which need to safely read and update two > > > > > objects. Can they be subprocesses of eachother? That is, can one master spawn the others as you desire? Can you have one process running, and connect to it with sockets, pipes, (mailslots,) etc., and just give and get the information to it? Then, synch. is a lot easier. Do you need MROW multiple-reader one-writer synchro., or can they all go one at a time? Is deadlock a concern? Can you use OBL(OE) one big lock over everything, or do you need individual locks on elements of the data structure? Can you use fire-and-forget access, or do you need return values from your calls? Do you need to wait for completion of anything? 'xmlrpc': remote procedure calls might pertain. -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing objects between processes
On Mon, 2009-03-09 at 11:04 -0700, Aaron Brady wrote: > On Mar 9, 12:47 pm, ET wrote: > > > Message: 2 > > > Date: Sun, 8 Mar 2009 12:00:40 -0700 (PDT) > > > From: Aaron Brady > > > Subject: Re: Sharing objects between processes > > > To: python-l...@python.org > > > Message-ID: > > ><5514c3df-d74e-47d8-93fc-34dd5119e...@c11g2000yqj.googlegroups.com> > > > Content-Type: text/plain; charset=ISO-8859-1 > > > > > On Mar 8, 1:36?pm, ET wrote: > > > > I have been using the 'threading' library and decided to try swapping it > > > > out for 'processing'... while it's awesome that processing so closely > > > > mirrors the threading interface, I've been having trouble getting my > > > > processes to share an object in a similar way. > > > > > > Using the 'with' keyword didn't work, and using normal locks doesn't > > > > result in the expected behavior (I can get an object to be accessible in > > > > more than one process, and Python indicates that the instances are > > > > living at the same address in memory, but changes in one process are not > > > > reflected in the other[s]). ?I'm sure this is because my expectations > > > > are incorrect. :) > > > > > > The problem, as briefly as possible: > > > > I have three processes which need to safely read and update two objects. > > > > > > I've been working with processing, multiprocessing, and parallel python, > > > > trying to get this working... I suspect it can be accomplished with > > > > managers and/or queues, but if there's an elegant way to handle it, I > > > > have thus far failed to understand it. > > > > > > I don't particularly care which library I use; if someone has done this > > > > or can recommend a good method they're aware of, it'd be incredibly > > > > helpful. > > > > > > Thank you! > > > > > There is POSH: Python Object Sharing, which I learned about a while > > > ago, but never used much. > > > > >http://poshmodule.sourceforge.net/ > > > > > It's UNIX only. > > > > Thanks, I'll definitely keep that link handy... unfortunately, this > > particular project needs to run on Windows as well as Linux-based > > systems. > > I don't recall whether there was anything in the source that's a deal- > breaker on Windows. The source is open, as you could see. > > Other possibilities are 'shelve' and any database... fixed-length > pickles, a directory of pickles, etc. Maybe 'multiprocessing' would > work for your synchronization, while you use a more custom technique > for data exchange. > > The only other thing I can do is bring to your attention an idea of > mine for sharing primitives. It's in the drawing board stage if you > want to help. > > Of course, it's only after you turned down 'multiprocessing' and > 'POSH'. It does things they don't and vice versa. > -- > http://mail.python.org/mailman/listinfo/python-list I assumed it wouldn't work in Windows as you mentioned it was UNIX-only; the readme also states that it's for POSIX systems only. I'd be more than happy to use multiprocessing; I've attempted to do so. My question is largely how to implement it, as I have not managed to get it working despite several attempts from different angles. Unfortunately, I do need to handle more than primitives, otherwise I'd have attempted to use the shared ctypes present in at least one of processing/multiprocessing/parallel python. -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing objects between processes
On Mar 9, 12:47 pm, ET wrote: > > Message: 2 > > Date: Sun, 8 Mar 2009 12:00:40 -0700 (PDT) > > From: Aaron Brady > > Subject: Re: Sharing objects between processes > > To: python-l...@python.org > > Message-ID: > > <5514c3df-d74e-47d8-93fc-34dd5119e...@c11g2000yqj.googlegroups.com> > > Content-Type: text/plain; charset=ISO-8859-1 > > > On Mar 8, 1:36?pm, ET wrote: > > > I have been using the 'threading' library and decided to try swapping it > > > out for 'processing'... while it's awesome that processing so closely > > > mirrors the threading interface, I've been having trouble getting my > > > processes to share an object in a similar way. > > > > Using the 'with' keyword didn't work, and using normal locks doesn't > > > result in the expected behavior (I can get an object to be accessible in > > > more than one process, and Python indicates that the instances are > > > living at the same address in memory, but changes in one process are not > > > reflected in the other[s]). ?I'm sure this is because my expectations > > > are incorrect. :) > > > > The problem, as briefly as possible: > > > I have three processes which need to safely read and update two objects. > > > > I've been working with processing, multiprocessing, and parallel python, > > > trying to get this working... I suspect it can be accomplished with > > > managers and/or queues, but if there's an elegant way to handle it, I > > > have thus far failed to understand it. > > > > I don't particularly care which library I use; if someone has done this > > > or can recommend a good method they're aware of, it'd be incredibly > > > helpful. > > > > Thank you! > > > There is POSH: Python Object Sharing, which I learned about a while > > ago, but never used much. > > >http://poshmodule.sourceforge.net/ > > > It's UNIX only. > > Thanks, I'll definitely keep that link handy... unfortunately, this > particular project needs to run on Windows as well as Linux-based > systems. I don't recall whether there was anything in the source that's a deal- breaker on Windows. The source is open, as you could see. Other possibilities are 'shelve' and any database... fixed-length pickles, a directory of pickles, etc. Maybe 'multiprocessing' would work for your synchronization, while you use a more custom technique for data exchange. The only other thing I can do is bring to your attention an idea of mine for sharing primitives. It's in the drawing board stage if you want to help. Of course, it's only after you turned down 'multiprocessing' and 'POSH'. It does things they don't and vice versa. -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing objects between processes
> Message: 2 > Date: Sun, 8 Mar 2009 12:00:40 -0700 (PDT) > From: Aaron Brady > Subject: Re: Sharing objects between processes > To: python-list@python.org > Message-ID: > <5514c3df-d74e-47d8-93fc-34dd5119e...@c11g2000yqj.googlegroups.com> > Content-Type: text/plain; charset=ISO-8859-1 > > On Mar 8, 1:36?pm, ET wrote: > > I have been using the 'threading' library and decided to try swapping it > > out for 'processing'... while it's awesome that processing so closely > > mirrors the threading interface, I've been having trouble getting my > > processes to share an object in a similar way. > > > > Using the 'with' keyword didn't work, and using normal locks doesn't > > result in the expected behavior (I can get an object to be accessible in > > more than one process, and Python indicates that the instances are > > living at the same address in memory, but changes in one process are not > > reflected in the other[s]). ?I'm sure this is because my expectations > > are incorrect. :) > > > > The problem, as briefly as possible: > > I have three processes which need to safely read and update two objects. > > > > I've been working with processing, multiprocessing, and parallel python, > > trying to get this working... I suspect it can be accomplished with > > managers and/or queues, but if there's an elegant way to handle it, I > > have thus far failed to understand it. > > > > I don't particularly care which library I use; if someone has done this > > or can recommend a good method they're aware of, it'd be incredibly > > helpful. > > > > Thank you! > > There is POSH: Python Object Sharing, which I learned about a while > ago, but never used much. > > http://poshmodule.sourceforge.net/ > > It's UNIX only. Thanks, I'll definitely keep that link handy... unfortunately, this particular project needs to run on Windows as well as Linux-based systems. -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing objects between processes
> Message: 1 > Date: Sun, 08 Mar 2009 18:47:09 + > From: Tim Golden > Subject: Re: Sharing objects between processes > Cc: python-list@python.org > Message-ID: <49b412ad.1030...@timgolden.me.uk> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > ET wrote: > > Using the 'with' keyword didn't work... > > Just an aside here for any multiprocessing maintainers > watching ;) . I expect that the "didn't work" here > refers to this bug: > > http://bugs.python.org/issue5261 > > Altho' if the OP cares to clarify, it might be something > else. > > TJG Sorry; unfortunately, it's been a while since I ran into this and don't recall what the particular problem was (there's been too much rewriting with different libraries since then to quickly reproduce this issue). -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing objects between processes
On Mar 8, 1:36 pm, ET wrote: > I have been using the 'threading' library and decided to try swapping it > out for 'processing'... while it's awesome that processing so closely > mirrors the threading interface, I've been having trouble getting my > processes to share an object in a similar way. > > Using the 'with' keyword didn't work, and using normal locks doesn't > result in the expected behavior (I can get an object to be accessible in > more than one process, and Python indicates that the instances are > living at the same address in memory, but changes in one process are not > reflected in the other[s]). I'm sure this is because my expectations > are incorrect. :) > > The problem, as briefly as possible: > I have three processes which need to safely read and update two objects. > > I've been working with processing, multiprocessing, and parallel python, > trying to get this working... I suspect it can be accomplished with > managers and/or queues, but if there's an elegant way to handle it, I > have thus far failed to understand it. > > I don't particularly care which library I use; if someone has done this > or can recommend a good method they're aware of, it'd be incredibly > helpful. > > Thank you! There is POSH: Python Object Sharing, which I learned about a while ago, but never used much. http://poshmodule.sourceforge.net/ It's UNIX only. -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing objects between processes
ET wrote: Using the 'with' keyword didn't work... Just an aside here for any multiprocessing maintainers watching ;) . I expect that the "didn't work" here refers to this bug: http://bugs.python.org/issue5261 Altho' if the OP cares to clarify, it might be something else. TJG -- http://mail.python.org/mailman/listinfo/python-list
