Re: [Numpy-discussion] Only integer scalar arrays can be converted to a scalar index

[Juan Nunez-Iglesias]

@Robert, good point, always good to try out code before speculating on a 
thread. ;)

Here’s working code to do the averaging, though it’s not block-wise, you’ll 
have to add that on top with dask/util.apply_parallel. Note also that because 
of the C-order of numpy arrays, it’s much more efficient to think of axis 0 as 
the “vertical” axis, rather than axis 2. See 
http://scikit-image.org/docs/dev/user_guide/numpy_images.html#notes-on-array-order
 for more info.

import numpy as np
from skimage import util

vol = np.linspace(1, 125, 125, dtype=np.int32).reshape(5, 5, 5)
window_shape = (1, 3, 3)
windows = util.view_as_windows(vol, window_shape)
print(windows.shape)  # (5, 3, 3, 1, 3, 3)
averaged = np.mean(windows, axis=(3, 4, 5))

HTH!

Juan.


On 16 Sep 2017, 12:34 PM +1000, Robert Kern , wrote:
> On Sat, Sep 16, 2017 at 7:16 AM, Chris Barker - NOAA Federal 
>  wrote:
> >
> > No thoughts on optimizing memory, but that indexing error probably comes 
> > from np.mean producing float results. An astype call shoulder that work.
>
> Why? It's not being used as an index. It's being assigned into a float array.
>
> Rather, it's the slicing inside of `trace_block()` when it's being given 
> arrays as inputs for `x` and `y`. numpy simply doesn't support that because 
> in general the result wouldn't have a uniform shape.
>
> --
> Robert Kern
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Only integer scalar arrays can be converted to a scalar index

[Juan Nunez-Iglesias]

+1 on the astype(int) call.

+1 also on using dask. scikit-image has a couple of functions that might be 
useful:

- skimage.util.apply_parallel: applies a function to an input array in chunks, 
with user-selectable chunk size and margins. This is powered by dask.
- skimage.util.view_as_windows: uses stride tricks to produce a sliding window 
view over an n-dimensional array.

On 16 Sep 2017, 8:16 AM +1000, Chris Barker - NOAA Federal 
, wrote:
> No thoughts on optimizing memory, but that indexing error probably comes from 
> np.mean producing float results. An astype call shoulder that work.
>
> -CHB
>
> Sent from my iPhone
>
> On Sep 15, 2017, at 5:51 PM, Robert McLeod  wrote:
>
> >
> > > On Fri, Sep 15, 2017 at 2:37 PM, Elliot Hallmark  
> > > wrote:
> > > > Nope. Numpy only works on in memory arrays. You can determine your own 
> > > > chunking strategy using hdf5, or something like dask can figure that 
> > > > strategy out for you. With numpy you might worry about not accidentally 
> > > > making duplicates or intermediate arrays, but that's the extent of 
> > > > memory optimization you can do in numpy itself.
> >
> > NumPy does have it's own memory map variant on ndarray:
> >
> > https://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html
> >
> >
> >
> > --
> > Robert McLeod, Ph.D.
> > robbmcl...@gmail.com
> > robbmcl...@protonmail.com
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@python.org
> > https://mail.python.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Only integer scalar arrays can be converted to a scalar index

[Chris Barker - NOAA Federal]

No thoughts on optimizing memory, but that indexing error probably comes
from np.mean producing float results. An astype call shoulder that work.

-CHB

Sent from my iPhone

On Sep 15, 2017, at 5:51 PM, Robert McLeod  wrote:


On Fri, Sep 15, 2017 at 2:37 PM, Elliot Hallmark 
wrote:

> Nope. Numpy only works on in memory arrays. You can determine your own
> chunking strategy using hdf5, or something like dask can figure that
> strategy out for you. With numpy you might worry about not accidentally
> making duplicates or intermediate arrays, but that's the extent of memory
> optimization you can do in numpy itself.
>

NumPy does have it's own memory map variant on ndarray:

https://docs.scipy.org/doc/numpy/reference/generated/numpy.memmap.html



-- 
Robert McLeod, Ph.D.
robbmcl...@gmail.com
robbmcl...@protonmail.com

___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Only integer scalar arrays can be converted to a scalar index

[Elliot Hallmark]

Nope. Numpy only works on in memory arrays. You can determine your own
chunking strategy using hdf5, or something like dask can figure that
strategy out for you. With numpy you might worry about not accidentally
making duplicates or intermediate arrays, but that's the extent of memory
optimization you can do in numpy itself.

Elliot
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Only integer scalar arrays can be converted to a scalar index

[Michael Bostock]

Hi,

I am trying to do a slding window on a cube (3D array) to get the average
over a block of vertical 1D arrays. I have achieved this using the
stride_tricks.asstrided but this will load the whole cube into memory at
once and is not suitable for large cubes. I have also achieved it using an
nditer but this is a lot slower:

def trace_block(vol, x, y, half_window):
return vol[x - half_window:x + half_window + 1, y - half_window:y
+ half_window + 1]

vol = np.linspace(1, 125, 125, dtype=np.int32).reshape(5, 5, 5)
window_size = 3

x, y, z = vol.shape
half_window = (window_size - 1) // 2
xs = np.arange(half_window, x - half_window, dtype=np.int16)
ys = np.arange(half_window, y - half_window, dtype=np.int16)
averaged = np.zeros((5, 5, 5))
for x, y in np.nditer(np.ix_(xs, ys)):
averaged[x, y] = np.mean(trace_block(vol, x, y, half_window), (0, 1))


My attempt at using numpy vectorisation to avoid the for loop throws the
error in the subject:

vol = np.linspace(1, 125, 125, dtype=np.int32).reshape(5, 5, 5)
window_size = 3

x, y, z = vol.shape
half_window = (window_size - 1) // 2
xs = np.arange(half_window, x - half_window, dtype=np.int16)
ys = np.arange(half_window, y - half_window, dtype=np.int16)
averaged = np.zeros((5, 5, 5))
xi, yi = np.ix_(xs, ys)
averaged[xi, yi] = np.mean(trace_block(vol, xi, yi, half_window), (0, 1))

Is there any way to do slicing as shown in the trace_block function to
support the xi and yi grid arrays?

Any help you can provide will be greatly appreciated.
___
NumPy-Discussion mailing list
NumPy-Discussion@python.org
https://mail.python.org/mailman/listinfo/numpy-discussion