Thanks, np.lib.format.open_memmap() works great! With prediction procs
using minimal sys memory, I can get twice as many on GPU, with fewer
optimization warnings.
Why even have the number of records in the header? Shouldn't record size
plus system-reported/growable file size be enough?
I'd love to have a shared-mem analog for smaller-scale data; now I load
data and fork to emulate that effect.
My file sizes will exceed memory, so I'm hoping to get the most out of
memmap. Will this in-loop assignment to predsum work to avoid loading
all to memory?
predsum = np.lib.format.open_memmap(outfile, mode='w+',
shape=(ids_sq,), dtype=np.float32)
for i in range(len(IN_FILES)):
pred = numpy.lib.format.open_memmap(IN_FILES[i])
predsum = np.add(predsum, pred) ################# <-
del pred
del predsum
--
Phobrain.com
On 2022-08-23 18:02, Robert Kern wrote:
> On Tue, Aug 23, 2022 at 8:47 PM <bross_phobr...@sonic.net> wrote:
>
>> I want to calc multiple ndarrays at once and lack memory, so want to write
>> in chunks (here sized to GPU batch capacity). It seems there should be an
>> interface to write the header, then write a number of elements cyclically,
>> then add any closing rubric and close the file.
>>
>> Is it as simple as lib.format.write_array_header_2_0(fp, d)
>> then writing multiple shape(N,) arrays of float by fp.write(item.tobytes())?
>
> `item.tofile(fp)` is more efficient, but yes, that's the basic scheme. There
> is no footer after the data.
>
> The alternative is to use `np.lib.format.open_memmap(filename, mode='w+',
> dtype=dtype, shape=shape)`, then assign slices sequentially to the returned
> memory-mapped array. A memory-mapped array is usually going to be friendlier
> to whatever memory limits you are running into than a nominally "in-memory"
> array.
> --
> Robert Kern
> _______________________________________________
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: bross_phobr...@sonic.net_______________________________________________
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
