On 11 apr, 21:11, sturlamolden wrote:
> import numpy as np
> import sharedmem as sm
> private_array = np.zeros((10,10))
> shared_array = sm.zeros((10,10))
I am also using this to implement synchronization primitives (barrier,
lock, semaphore, event) that can be sent over an instance of
multipro
On 11 apr, 09:21, John Nagle wrote:
> Because nobody can fix the Global Interpreter Lock problem in CPython.
>
> The multiprocessing module is a hack to get around the fact
> that Python threads don't run concurrently, and thus, threaded
> programs don't effectively use multi-core CPUs.
On 4/10/2011 3:29 PM, sturlamolden wrote:
On 10 apr, 18:27, John Nagle wrote:
Unless you have a performance problem, don't bother with shared
memory.
If you have a performance problem, Python is probably the wrong
tool for the job anyway.
Then why does Python have a multiprocessin
On 8 apr, 03:10, sturlamolden wrote:
> That was easy, 64-bit support for Windows is done :-)
>
> Now I'll just have to fix the Linux code, and figure out what to do
> with os._exit preventing clean-up on exit... :-(
Now it feel dumb, it's not worse than monkey patching os._exit, which
I should h
On 10 apr, 18:27, John Nagle wrote:
> Unless you have a performance problem, don't bother with shared
> memory.
>
> If you have a performance problem, Python is probably the wrong
> tool for the job anyway.
Then why does Python have a multiprocessing module?
In my opinion, if Python has
On 4/10/2011 9:11 AM, Miki Tebeka wrote:
Now, I don't know that I actually HAVE to pass my neural network and
input data as copies -- they're both READ-ONLY objects for the
duration of an evaluate function (which can go on for quite a while).
One option in that case is to use "fork" (if you're o
> Now, I don't know that I actually HAVE to pass my neural network and
> input data as copies -- they're both READ-ONLY objects for the
> duration of an evaluate function (which can go on for quite a while).
One option in that case is to use "fork" (if you're on a *nix machine).
See http://pythonwi
On 9 apr, 22:18, John Ladasky wrote:
> So, there are limited advantages to trying to parallelize the
> evaluation of ONE cascade network's weights against ONE input vector.
> However, evaluating several copies of one cascade network's output,
> against several different test inputs simultaneously
On Apr 9, 10:15 am, sturlamolden wrote:
> On 9 apr, 09:36, John Ladasky wrote:
>
> > Thanks for finding my discussion! Yes, it's about passing numpy
> > arrays to multiple processors. I'll accomplish that any way that I
> > can.
>
> My preferred ways of doing this are:
>
> 1. Most cases for par
On 9 apr, 09:36, John Ladasky wrote:
> Thanks for finding my discussion! Yes, it's about passing numpy
> arrays to multiple processors. I'll accomplish that any way that I
> can.
My preferred ways of doing this are:
1. Most cases for parallel processing are covered by libraries, even
for neur
On Apr 7, 6:10 pm, sturlamolden wrote:
> On 8 apr, 02:38, sturlamolden wrote:
>
> > I should probably fix it for 64-bit now. Just recompiliong with 64-bit
> > integers will not work, because I intentionally hardcoded the higher
> > 32 bits to 0.
>
> That was easy, 64-bit support for Windows is do
On 8 apr, 02:38, sturlamolden wrote:
> I should probably fix it for 64-bit now. Just recompiliong with 64-bit
> integers will not work, because I intentionally hardcoded the higher
> 32 bits to 0.
That was easy, 64-bit support for Windows is done :-)
Now I'll just have to fix the Linux code, an
On 8 apr, 02:03, sturlamolden wrote:
> http://folk.uio.no/sturlamo/python/sharedmem-feb13-2009.zip
> Known issues/bugs: 64-bit support is lacking, and os._exit in
> multiprocessing causes a memory leak on Linux.
I should probably fix it for 64-bit now. Just recompiliong with 64-bit
integers will
On 4 apr, 22:20, John Ladasky wrote:
> https://bitbucket.org/cleemesser/numpy-sharedmem/src/3fa526d11578/shm...
>
> I've added a few lines to this code which allows subclassing the
> shared memory array, which I need (because my neural net objects are
> more than just the array, they also contain
On 5 apr, 02:05, Robert Kern wrote:
> PicklingError: Can't pickle 'multiprocessing.sharedctypes.c_double_Array_10'>: attribute lookup
> multiprocessing.sharedctypes.c_double_Array_10 failed
Hehe :D
That is why programmers should not mess with code they don't
understand!
Gaël and I wrote shmem
On 4/7/11 1:39 PM, John Ladasky wrote:
On Apr 7, 10:44 am, Robert Kern wrote:
On 4/7/11 1:40 AM, John Ladasky wrote:
On Apr 5, 10:43 am, Philip Semanchukwrote:
And as Robert Kern pointed out, numpy arrays are also pickle-able.
OK, but SUBCLASSES of numpy.ndarray are not, in my hands,
On Apr 7, 10:44 am, Robert Kern wrote:
> On 4/7/11 1:40 AM, John Ladasky wrote:
>
> > On Apr 5, 10:43 am, Philip Semanchuk wrote:
> >> And as Robert Kern pointed out, numpy arrays are also pickle-able.
>
> > OK, but SUBCLASSES of numpy.ndarray are not, in my hands, pickling as
> > I would expect.
On 4/7/11 1:40 AM, John Ladasky wrote:
On Apr 5, 10:43 am, Philip Semanchuk wrote:
And as Robert Kern pointed out, numpy arrays are also pickle-able.
OK, but SUBCLASSES of numpy.ndarray are not, in my hands, pickling as
I would expect. I already have lots of code that is based on such
sub
On Apr 7, 2011, at 3:41 AM, John Ladasky wrote:
> Following up to my own post...
>
> On Apr 6, 11:40 pm, John Ladasky wrote:
>
>> What's up with that?
>
> Apparently, "what's up" is that I will need to implement a third
> method in my ndarray subclass -- namely, __reduce__.
>
> http://www.ma
Following up to my own post...
On Apr 6, 11:40 pm, John Ladasky wrote:
> What's up with that?
Apparently, "what's up" is that I will need to implement a third
method in my ndarray subclass -- namely, __reduce__.
http://www.mail-archive.com/numpy-discussion@scipy.org/msg02446.html
I'm burned o
Hello again, Philip,
I really appreciate you sticking with me. Hopefully this will help
someone else, too. I've done some more reading, and will offer some
minimal code below.
I've known about this page for a while, and it describes some of the
unconventional things one needs to consider when s
On Apr 5, 2011, at 12:58 PM, John Ladasky wrote:
> Hi Philip,
>
> Thanks for the reply.
>
> On Apr 4, 4:34 pm, Philip Semanchuk wrote:
>> So if you're going to use multiprocessing, you're going to use pickle, and
>> you
>> need pickleable objects.
>
> OK, that's good to know.
But as Dan Str
Hi Philip,
Thanks for the reply.
On Apr 4, 4:34 pm, Philip Semanchuk wrote:
> So if you're going to use multiprocessing, you're going to use pickle, and you
> need pickleable objects.
OK, that's good to know.
> > Pickling is still a pretty
> > vague progress to me, but I can see that you have
On 4/4/11 7:05 PM, Robert Kern wrote:
On 4/4/11 3:20 PM, John Ladasky wrote:
However, at least in Python 2.7, multiprocessing seems to have a C extension
module defining the Connection objects. Unfortunately, it looks like this C
extension just imports the regular pickler that is not aware of
On Apr 4, 2011, at 9:03 PM, Dan Stromberg wrote:
> On Mon, Apr 4, 2011 at 4:34 PM, Philip Semanchuk wrote:
>
>> So if you're going to use multiprocessing, you're going to use pickle, and
>> you need pickleable objects.
>>
>
> http://docs.python.org/library/multiprocessing.html#sharing-state-be
On Mon, Apr 4, 2011 at 4:34 PM, Philip Semanchuk wrote:
> So if you're going to use multiprocessing, you're going to use pickle, and
> you need pickleable objects.
>
http://docs.python.org/library/multiprocessing.html#sharing-state-between-processes
--
http://mail.python.org/mailman/listinfo/pyt
On 4/4/11 3:20 PM, John Ladasky wrote:
Hi folks,
I'm developing some custom neural network code. I'm using Python 2.6,
Numpy 1.5, and Ubuntu Linux 10.10. I have an AMD 1090T six-core CPU,
and I want to take full advantage of it. I love to hear my CPU fan
running, and watch my results come bac
On Apr 4, 2011, at 4:20 PM, John Ladasky wrote:
> I have been playing with multiprocessing for a while now, and I have
> some familiarity with Pool. Apparently, arguments passed to a Pool
> subprocess must be able to be pickled.
Hi John,
multiprocessing's use of pickle is not limited to Pool.
On Mon, Apr 4, 2011 at 1:20 PM, John Ladasky wrote:
> When should one pickle and copy? When to implement an object in
> shared memory? Why is pickling apparently such a non-trivial process
> anyway? And, given that multi-core CPU's are apparently here to stay,
> should it be so difficult to ma
Hi folks,
I'm developing some custom neural network code. I'm using Python 2.6,
Numpy 1.5, and Ubuntu Linux 10.10. I have an AMD 1090T six-core CPU,
and I want to take full advantage of it. I love to hear my CPU fan
running, and watch my results come back faster.
When I'm training a neural net
30 matches
Mail list logo
