@Robert, good point, always good to try out code before speculating on a
thread. ;)
Here’s working code to do the averaging, though it’s not block-wise, you’ll
have to add that on top with dask/util.apply_parallel. Note also that because
of the C-order of numpy arrays, it’s much more efficient
On Sat, Sep 16, 2017 at 7:16 AM, Chris Barker - NOAA Federal <
chris.bar...@noaa.gov> wrote:
>
> No thoughts on optimizing memory, but that indexing error probably comes
from np.mean producing float results. An astype call shoulder that work.
Why? It's not being used as an index. It's being assign
+1 on the astype(int) call.
+1 also on using dask. scikit-image has a couple of functions that might be
useful:
- skimage.util.apply_parallel: applies a function to an input array in chunks,
with user-selectable chunk size and margins. This is powered by dask.
- skimage.util.view_as_windows: us
No thoughts on optimizing memory, but that indexing error probably comes
from np.mean producing float results. An astype call shoulder that work.
-CHB
Sent from my iPhone
On Sep 15, 2017, at 5:51 PM, Robert McLeod wrote:
On Fri, Sep 15, 2017 at 2:37 PM, Elliot Hallmark
wrote:
> Nope. Numpy
On Fri, Sep 15, 2017 at 2:37 PM, Elliot Hallmark
wrote:
> Nope. Numpy only works on in memory arrays. You can determine your own
> chunking strategy using hdf5, or something like dask can figure that
> strategy out for you. With numpy you might worry about not accidentally
> making duplicates or
Nope. Numpy only works on in memory arrays. You can determine your own
chunking strategy using hdf5, or something like dask can figure that
strategy out for you. With numpy you might worry about not accidentally
making duplicates or intermediate arrays, but that's the extent of memory
optimization
I was hoping that numpy doing this in a vectorised way would only load the
surrounding traces into memory for each X and Y as it needs to rather than
the whole cube. I'm using hdf5 for the storage. My example was just a short
example without using hdf5.
On 15 Sep 2017 1:16 am, "Elliot Hallmark" w
Won't any solution not using hdf5 or some other chunked on disk storage
method load the whole cube into memory?
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
Hi,
I am trying to do a slding window on a cube (3D array) to get the average
over a block of vertical 1D arrays. I have achieved this using the
stride_tricks.asstrided but this will load the whole cube into memory at
once and is not suitable for large cubes. I have also achieved it using an
ndite
