A Sunday 07 March 2010 20:03:21 Gael Varoquaux escrigué:
On Sun, Mar 07, 2010 at 07:00:03PM +, René Dudfield wrote:
1. Mmap'd files are useful since you can reuse disk cache as program
memory. So large files don't waste ram on the disk cache.
I second that. mmaping has worked very
On Thu, Mar 11, 2010 at 10:04:36AM +0100, Francesc Alted wrote:
As far as I know, memmap files (or better, the underlying OS) *use* all
available RAM for loading data until RAM is exhausted and then start to use
SWAP, so the memory pressure is still there. But I may be wrong...
I believe
-Original Message-
From: numpy-discussion-boun...@scipy.org on behalf of Gael Varoquaux
Sent: Thu 11-Mar-10 11:36
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] multiprocessing shared arrays and numpy
On Thu, Mar 11, 2010 at 10:04:36AM +0100, Francesc Alted wrote
A Thursday 11 March 2010 10:36:42 Gael Varoquaux escrigué:
On Thu, Mar 11, 2010 at 10:04:36AM +0100, Francesc Alted wrote:
As far as I know, memmap files (or better, the underlying OS) *use* all
available RAM for loading data until RAM is exhausted and then start to
use SWAP, so the memory
On Thu, Mar 11, 2010 at 02:26:49PM +0100, Francesc Alted wrote:
I believe that your above assertion is 'half' right. First I think that
it is not SWAP that the memapped file uses, but the original disk space,
thus you avoid running out of SWAP. Second, if you open several times the
same
A Thursday 11 March 2010 14:35:49 Gael Varoquaux escrigué:
So, in my experience, numpy.memmap is really using that large chunk of
memory (unless my testbed is badly programmed, in which case I'd be
grateful if you can point out what's wrong).
OK, so what you are saying is that my
On Fri, Mar 5, 2010 at 7:29 PM, Brian Granger ellisonbg@gmail.com wrote:
Francesc,
Yeah, 10% of improvement by using multi-cores is an expected figure for
memory
bound problems. This is something people must know: if their computations
are
memory bound (and this is much more common
On Sun, Mar 07, 2010 at 07:00:03PM +, René Dudfield wrote:
1. Mmap'd files are useful since you can reuse disk cache as program
memory. So large files don't waste ram on the disk cache.
I second that. mmaping has worked very well for me for large datasets,
especialy in the context of
to the memory mapped
file from the child process.
Nadav
-Original Message-
From: numpy-discussion-boun...@scipy.org on behalf of Brian Granger
Sent: Fri 05-Mar-10 21:29
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] multiprocessing shared arrays and numpy
Francesc,
Yeah
-
From: numpy-discussion-boun...@scipy.org on behalf of Francesc Alted
Sent: Thu 04-Mar-10 15:12
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] multiprocessing shared arrays and numpy
What kind of calculations are you doing with this module? Can you please
send some examples
On Fri, Mar 05, 2010 at 09:53:02AM +0100, Francesc Alted wrote:
Yeah, 10% of improvement by using multi-cores is an expected figure for
memory bound problems. This is something people must know: if their
computations are memory bound (and this is much more common that one
may initially
Gael,
On Fri, Mar 05, 2010 at 10:51:12AM +0100, Gael Varoquaux wrote:
On Fri, Mar 05, 2010 at 09:53:02AM +0100, Francesc Alted wrote:
Yeah, 10% of improvement by using multi-cores is an expected figure for
memory bound problems. This is something people must know: if their
computations
On Fri, Mar 05, 2010 at 08:14:51AM -0500, Francesc Alted wrote:
FWIW, I observe very good speedups on my problems (pretty much linear in
the number of CPUs), and I have data parallel problems on fairly large
data (~100Mo a piece, doesn't fit in cache), with no synchronisation at
all
A Friday 05 March 2010 14:46:00 Gael Varoquaux escrigué:
On Fri, Mar 05, 2010 at 08:14:51AM -0500, Francesc Alted wrote:
FWIW, I observe very good speedups on my problems (pretty much linear
in the number of CPUs), and I have data parallel problems on fairly
large data (~100Mo a piece,
on behalf of Nadav Horesh
Sent: Thu 04-Mar-10 11:55
To: Discussion of Numerical Python
Subject: RE: [Numpy-discussion] multiprocessing shared arrays and numpy
Maybe the attached file can help. Adpted and tested on amd64 linux
Nadav
-Original Message-
From: numpy
Maybe the attached file can help. Adpted and tested on amd64 linux
Nadav
-Original Message-
From: numpy-discussion-boun...@scipy.org on behalf of Nadav Horesh
Sent: Thu 04-Mar-10 10:54
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] multiprocessing shared arrays
...@scipy.org on behalf of Nadav Horesh
Sent: Thu 04-Mar-10 11:55
To: Discussion of Numerical Python
Subject: RE: [Numpy-discussion] multiprocessing shared arrays and numpy
Maybe the attached file can help. Adpted and tested on amd64 linux
Nadav
-Original Message-
From: numpy-discussion-boun
of Numerical Python
Subject: RE: [Numpy-discussion] multiprocessing shared arrays and numpy
Maybe the attached file can help. Adpted and tested on amd64 linux
Nadav
-Original Message-
From: numpy-discussion-boun...@scipy.org on behalf of Nadav Horesh
Sent: Thu 04-Mar-10 10:54
: [Numpy-discussion] multiprocessing shared arrays and numpy
What kind of calculations are you doing with this module? Can you please send
some examples and the speed-ups you are getting?
Thanks,
Francesc
A Thursday 04 March 2010 14:06:34 Nadav Horesh escrigué:
Extended module that I used
Hi people,
I was wondering about the status of using the standard library
multiprocessing module with numpy. I found a cookbook example last
updated one year ago which states that:
This page was obsolete as multiprocessing's internals have changed.
More information will come shortly; a link to
A Wednesday 03 March 2010 15:31:29 Jesper Larsen escrigué:
Hi people,
I was wondering about the status of using the standard library
multiprocessing module with numpy. I found a cookbook example last
updated one year ago which states that:
This page was obsolete as multiprocessing's
There is a work by Sturla Molden: look for multiprocessing-tutorial.pdf
and sharedmem-feb13-2009.zip. The tutorial includes what is dropped in
the cookbook page. I am into the same issue and going to test it today.
Nadav
On Wed, 2010-03-03 at 15:31 +0100, Jesper Larsen wrote:
Hi people,
I
22 matches
Mail list logo
