Re: multiprocessing and dictionaries

2009-07-13 Thread Piet van Oostrum 
 Bjorn Meyer bjorn.m.me...@gmail.com (BM) wrote:

BM I am trying to convert a piece of code that I am using the thread module 
with 
BM to the multiprocessing module.

BM The way that I have it set up is a chunk of code reads a text file and 
assigns 
BM a dictionary key multiple values from the text file. I am using locks to 
write 
BM the values to the dictionary.
BM The way that the values are written is as follows:
BMmydict.setdefault(key, []).append(value)

BM The problem that I have run into is that using multiprocessing, the key 
gets 
BM set, but the values don't get appended.
BM I've even tried the Manager().dict() option, but it doesn't seem to work.

BM Is this not supported at this time or am I missing something?

I think you should give more information. Try to make a *minimal* program
that shows the problem and include it in your posting or supply a
download link.
-- 
Piet van Oostrum p...@cs.uu.nl
URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
Private email: p...@vanoostrum.org
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing and dictionaries

2009-07-13 Thread Bjorn Meyer 

On Monday 13 July 2009 01:56:08 Piet van Oostrum wrote:

  Bjorn Meyer bjorn.m.me...@gmail.com (BM) wrote:
 
 BM I am trying to convert a piece of code that I am using the thread
  module with BM to the multiprocessing module.
 
 BM The way that I have it set up is a chunk of code reads a text file and
  assigns BM a dictionary key multiple values from the text file. I am
  using locks to write BM the values to the dictionary.
 BM The way that the values are written is as follows:
 BM  mydict.setdefault(key, []).append(value)
 
 BM The problem that I have run into is that using multiprocessing, the
  key gets BM set, but the values don't get appended.
 BM I've even tried the Manager().dict() option, but it doesn't seem to
  work.
 
 BM Is this not supported at this time or am I missing something?

 I think you should give more information. Try to make a *minimal* program
 that shows the problem and include it in your posting or supply a
 download link.
 --
 Piet van Oostrum p...@cs.uu.nl
 URL: http://pietvanoostrum.com [PGP 8DAE142BE17999C4]
 Private email: p...@vanoostrum.org

Here is what I have been using as a test.
This pretty much mimics what I am trying to do.
I put both threading and multiprocessing in the example which shows the output 
that I am looking for.

#!/usr/bin/env python

import threading
from multiprocessing import Manager, Process

name = ('test1','test2','test3')
data1 = ('dat1','dat2','dat3')
data2 = ('datA','datB','datC')

def thread_test(name,data1,data2, d):
  for nam in name:
for num in range(0,3):
  d.setdefault(nam, []).append(data1[num])
  d.setdefault(nam, []).append(data2[num])
  print 'Thread test dict:',d

def multiprocess_test(name,data1,data2, mydict):
  for nam in name:
for num in range(0,3):
  mydict.setdefault(nam, []).append(data1[num])
  mydict.setdefault(nam, []).append(data2[num])
  print 'Multiprocess test dic:',mydict

if __name__ == '__main__':
  mgr = Manager()
  md = mgr.dict()
  d = {}

  m = Process(target=multiprocess_test, args=(name,data1,data2,md))
  m.start()
  t = threading.Thread(target=thread_test, args=(name,data1,data2,d))
  t.start()
  
  m.join()
  t.join()
  
  print 'Thread test:',d
  print 'Multiprocess test:',md


Thanks
Bjorn

-- 
http://mail.python.org/mailman/listinfo/python-list

multiprocessing and dictionaries

2009-07-12 Thread Bjorn Meyer 
I am trying to convert a piece of code that I am using the thread module with 
to the multiprocessing module.

The way that I have it set up is a chunk of code reads a text file and assigns 
a dictionary key multiple values from the text file. I am using locks to write 
the values to the dictionary.
The way that the values are written is as follows:
mydict.setdefault(key, []).append(value)

The problem that I have run into is that using multiprocessing, the key gets 
set, but the values don't get appended.
I've even tried the Manager().dict() option, but it doesn't seem to work.

Is this not supported at this time or am I missing something?

Thanks in advance.

Bjorn
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: multiprocessing and dictionaries

2009-07-12 Thread Chris Rebert 
On Sun, Jul 12, 2009 at 10:16 AM, Bjorn Meyerbjorn.m.me...@gmail.com wrote:
 I am trying to convert a piece of code that I am using the thread module with
 to the multiprocessing module.

 The way that I have it set up is a chunk of code reads a text file and assigns
 a dictionary key multiple values from the text file. I am using locks to write
 the values to the dictionary.
 The way that the values are written is as follows:
        mydict.setdefault(key, []).append(value)

 The problem that I have run into is that using multiprocessing, the key gets
 set, but the values don't get appended.

Don't have much concurrency experience, but have you tried using a
defaultdict instead?
It's possible its implementation might solve the problem.
http://docs.python.org/dev/library/collections.html#collections.defaultdict

Cheers,
Chris
-- 
http://blog.rebertia.com
-- 
http://mail.python.org/mailman/listinfo/python-list