[Numpy-discussion] ndarray shape permutation
Hellos, I would like to propose `numpy.ndarray.permute_shape()` method to predictably permute the shape of an ndarray. In my opinion, the current alternatives (`swapaxes`, `transform`, `moveaxes` and friends) are counterintuitive and rely on referring to the axis indices. It would be abundantly helpful to have something like reshape but which moves the data around (unlike reshape, which doesn't). Scenario: structural biology uses MRC files, which define a number of fields that describe a 3D volume. There is a field which describes the dimensions of the 3D image and another which associates each image axis with a physical axis. There are six such assignments between the image shape and the axes assignments (if we keep the shape fixed we can assign XYZ, XZY, YXZ, YZX, ZXY, ZYX to the columns, rows and stacks) and working out how to correctly transform the data is generally non-trivial. I've written a package (https://github.com/emdb-empiar/maptools) which does this but I use swapaxes to reorient the 3D image. It would be so much easier to use `numpy.ndarray.permute_shape()` instead. Any thoughts? Also, any helpful hints on how to get started with such a contribution would be helpful. Paul ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] ENH: Support Random Unit Vector
### Proposed new feature or change: Description Often you want to randomize a "direction" that doesn't have "size". This can be useful when: * You want to randomize the direction of an NPC walk developing in a 2D or 3D games, where its speed is a predefined constant. * You're developing particles simulation with all particles moving in random directions * etc. This random direction should be an `n`th dimensional unit vector which is randomize **uniformly** from the unit sphere. That means that sections of the unit sphere with equal areas should have the same chance of getting a vector from. Implementation === If one wants to randomize `n`th dimensional direction uniformly, one must do the following: 1. Generate `n` components of the vector using the standard normal distribution 2. Find the norm the generated vector 3. Divide the vector by its norm In simple Python code: ```python import numpy as np def sphere_uniform(n: int): vec = np.random.normal(size=n) norm = np.sqrt(np.sum(vec ** 2)) return vec / norm ``` This implementation suggestion is based on [this blog post](https://towardsdatascience.com/the-best-way-to-pick-a-unit-vector-7bd0cc54f9b). I'm not sure if there's a faster way of implementing it using C++ and Cython, but I'm confident that someone here will know. Why Implement It in Numpy? === I believe that random unit vectors are common enough to be a part of Numpy. Scientists, gamers and engineers use random directions all the time, and I believe there is no reason why Numpy won't provide this ability. ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: ndarray shape permutation
On Mon, May 16, 2022, at 17:54, Paul Korir wrote: > Hellos, > I would like to propose `numpy.ndarray.permute_shape()` > method to predictably permute the shape of an ndarray. In my opinion, > the current alternatives (`swapaxes`, `transform`, `moveaxes` and > friends) are counterintuitive and rely on referring to the axis > indices. It would be abundantly helpful to have something like reshape > but which moves the data around (unlike reshape, which doesn't). Hi Paul, Could you please show a small example of how `permute_shape()` is meant to work? It's not clear to me what it's meant to do, especially in contrast to `transpose` and `swapaxes` etc. "Like reshape" and "permute" together specifically sounds confusing to me. From your description of the problem I'd have thought that `transpose` (with a potential `copy`) would do what you need, so I'm probably misunderstanding your use case. What would it do when applied to a 3d array of, say, shape (2, 3, 5)? András > Scenario: structural biology uses MRC files, which define a number of > fields that describe a 3D volume. There is a field which describes the > dimensions of the 3D image and another which associates each image axis > with a physical axis. There are six such assignments between the image > shape and the axes assignments (if we keep the shape fixed we can > assign XYZ, XZY, YXZ, YZX, ZXY, ZYX to the columns, rows and stacks) > and working out how to correctly transform the data is generally > non-trivial. I've written a package > (https://github.com/emdb-empiar/maptools) which does this but I use > swapaxes to reorient the 3D image. It would be so much easier to use > `numpy.ndarray.permute_shape()` instead. > Any thoughts? Also, any helpful hints on how to get started with such a > contribution would be helpful. > Paul > ___ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: deak.and...@gmail.com ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: ndarray shape permutation
On Tue, 2022-05-17 at 12:16 +0200, Andras Deak wrote: > On Mon, May 16, 2022, at 17:54, Paul Korir wrote: > > Hellos, > > I would like to propose > > `numpy.ndarray.permute_shape()` > > method to predictably permute the shape of an ndarray. In my > > opinion, > > the current alternatives (`swapaxes`, `transform`, `moveaxes` and > > friends) are counterintuitive and rely on referring to the axis > > indices. It would be abundantly helpful to have something like > > reshape > > but which moves the data around (unlike reshape, which doesn't). > > Hi Paul, > > Could you please show a small example of how `permute_shape()` is > meant to work? It's not clear to me what it's meant to do, especially An example would indeed be great, since it would make it much clearer what operation you are looking for exactly (that is different from transpose). I suspect that another approach would be adding a syntax similar (or identical) to einops to NumPy: https://github.com/arogozhnikov/einops/ which generalizes the einsum notation to allow e.g. comprehensive reshaping. Cheers, Sebastian > in contrast to `transpose` and `swapaxes` etc. "Like reshape" and > "permute" together specifically sounds confusing to me. From your > description of the problem I'd have thought that `transpose` (with a > potential `copy`) would do what you need, so I'm probably > misunderstanding your use case. What would it do when applied to a 3d > array of, say, shape (2, 3, 5)? > > András > > > Scenario: structural biology uses MRC files, which define a number > > of > > fields that describe a 3D volume. There is a field which describes > > the > > dimensions of the 3D image and another which associates each image > > axis > > with a physical axis. There are six such assignments between the > > image > > shape and the axes assignments (if we keep the shape fixed we can > > assign XYZ, XZY, YXZ, YZX, ZXY, ZYX to the columns, rows and > > stacks) > > and working out how to correctly transform the data is generally > > non-trivial. I've written a package > > (https://github.com/emdb-empiar/maptools) which does this but I use > > swapaxes to reorient the 3D image. It would be so much easier to > > use > > `numpy.ndarray.permute_shape()` instead. > > Any thoughts? Also, any helpful hints on how to get started with > > such a > > contribution would be helpful. > > Paul > > ___ > > NumPy-Discussion mailing list -- numpy-discussion@python.org > > To unsubscribe send an email to numpy-discussion-le...@python.org > > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > > Member address: deak.and...@gmail.com > ___ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: sebast...@sipsolutions.net ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: ENH: Support Random Unit Vector
On Tue, May 17, 2022 at 5:20 AM wrote: > ### Proposed new feature or change: > > Description > > > Often you want to randomize a "direction" that doesn't have "size". This > can be useful when: > * You want to randomize the direction of an NPC walk developing in a 2D or > 3D games, where its speed is a predefined constant. > * You're developing particles simulation with all particles moving in > random directions > * etc. > > This random direction should be an `n`th dimensional unit vector which is > randomize **uniformly** from the unit sphere. That means that sections of > the unit sphere with equal areas should have the same chance of getting a > vector from. > > Implementation > === > > If one wants to randomize `n`th dimensional direction uniformly, one must > do the following: > 1. Generate `n` components of the vector using the standard normal > distribution > 2. Find the norm the generated vector > 3. Divide the vector by its norm > > In simple Python code: > ```python > import numpy as np > > > def sphere_uniform(n: int): > vec = np.random.normal(size=n) > norm = np.sqrt(np.sum(vec ** 2)) > return vec / norm > ``` > > This implementation suggestion is based on [this blog post]( > https://towardsdatascience.com/the-best-way-to-pick-a-unit-vector-7bd0cc54f9b). > I'm not sure if there's a faster way of implementing it using C++ and > Cython, but I'm confident that someone here will know. > I would probably use np.linalg.norm() to calculate the norm, but mostly for readability. > Why Implement It in Numpy? > === > > I believe that random unit vectors are common enough to be a part of > Numpy. Scientists, gamers and engineers use random directions all the time, > and I believe there is no reason why Numpy won't provide this ability. > As a general policy, we do not plan on adding more methods to Generator (c.f. discussion of a previous suggested addition[1]). This would be a good multivariate distribution to add to scipy.stats, though. [1] https://www.mail-archive.com/numpy-discussion@python.org/msg04724.html -- Robert Kern ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: ndarray shape permutation
Thanks for your replies.
In retrospect, I realise that using the shape will not be helpful for a cubic
array i.e. the permutations of (10, 10, 10) are all (10, 10, 10)! However, the
problem remains. Let me try to explain.
Short version
The problem boils down to the meaning of axis indices as arguments to swapaxes
and transpose (and any related functions). Swapping axes or transposing an
array gives new meanings to the indices. For example, suppose I have a volume
of shape C, R, S. Then 0 will refer to C, 1 will refer to R and 2 will refer to
S. After I transpose it, say using (1, 2, 0) so that the shape becomes R, S, C
then now 0 will refer to R, 1 will refer to S and 2 will refer to C. I can no
longer reverse the transposition or transpose it predictably to achieve a
certain shape, which is an important operation in some applications where the
meaning of the axes is significant.
Long version
Suppose I have a volume of shape (C, R, S) and I have a corresponding
assignment of physical axes so that C=X, R=Y and S=Z. This is equivalent to
placing the volume with C along the X axis, R along Y axis and S along the Z
axis. Now, suppose I would like to permute the axes by only making reference to
the axis names: what is the shape corresponding to the orientation (Z, Y, X)?
This is a simple example because we only swap two axes and the resulting shape
is the same as performing the same swap in the shape: (S, R, C). If we knew the
indices of the axis names then we can infer these and pass them to swapaxes or
transpose:
vol = numpy.random.rand(C, R, S) # of shape (C, R, S) -> (X, Y, Z)
# now (Z, Y, X)
new_vol = numpy.swapaxes(vol, 0, 2)
new_vol.shape # (S, R, C)
The same applies to a double swap e.g. (Y, Z, X), though it is less
straightforward using swapaxes. swapaxes only takes two indices (obviously) so
we would need to call it twice reflecting the two swaps required. So we have to
somehow figure which axes to swap successively: ((0, 2) then (0, 1)). We can do
this in one step with numpy.transpose simply using indices (1, 2, 0).
However, (and this is the big 'however'), how would we reverse this? The array
has no memory of the original axes and 0, 1, and 2 now refer to the current
axes. This is where the axes names (e.g. X, Y and Z) would come in handy.
Axis names will allow permutations to happen predictably since the array will
'remember' what the original references were.
Here is what I propose: an API to numpy.ndarray with some identity e.g.
vol = numpy.random.rand(C, R, S)
vol.shape # C, R, S
vol.axes = ('X', 'Y', 'Z') # C=X, R=Y, S=Z
new_vol = vol.permute_axes(('Z', 'Y', 'X'))
# either
new_vol.axes # ('X', 'Y', 'Z') # preserve orientation but change shape
new_vol.shape # S, R, C
# or
new_vol.axes # ('Z', 'Y', 'X') # preserve shape but change orientation
new_vol.shape # C, R, S
# we can now reverse the permutation
old_vol = new_vol.permute_axes(('X', 'Y', 'Z'))
numpy.array_equal(vol, old_vol) # True
I've checked the numpy API documentation and there is no attribute .axes
present so this is the best candidate. Additionally, this will require the
.permute_axes() method/function.
Thanks for your consideration.
Paul
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
[Numpy-discussion] next NumPy Newcomers' Hour
The next Newcomers' Hour will be held this Thursday, May 19th, at 4 pm UTC. Stop by to ask questions or just to say hi as we don't have an agenda for this meeting. Join us via Zoom: https://us02web.zoom.us/j/87192457898 Cheers, Inessa Inessa Pawson NumPy Contributor Experience Lead ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: ndarray shape permutation
You could easily write an extension to ndarray that maps axis names to
indices and vice versa.
Joe
On Tue, May 17, 2022, 21:32 Paul Korir wrote:
> Thanks for your replies.
>
> In retrospect, I realise that using the shape will not be helpful for a
> cubic array i.e. the permutations of (10, 10, 10) are all (10, 10, 10)!
> However, the problem remains. Let me try to explain.
>
> Short version
> The problem boils down to the meaning of axis indices as arguments to
> swapaxes and transpose (and any related functions). Swapping axes or
> transposing an array gives new meanings to the indices. For example,
> suppose I have a volume of shape C, R, S. Then 0 will refer to C, 1 will
> refer to R and 2 will refer to S. After I transpose it, say using (1, 2, 0)
> so that the shape becomes R, S, C then now 0 will refer to R, 1 will refer
> to S and 2 will refer to C. I can no longer reverse the transposition or
> transpose it predictably to achieve a certain shape, which is an important
> operation in some applications where the meaning of the axes is significant.
>
> Long version
> Suppose I have a volume of shape (C, R, S) and I have a corresponding
> assignment of physical axes so that C=X, R=Y and S=Z. This is equivalent to
> placing the volume with C along the X axis, R along Y axis and S along the
> Z axis. Now, suppose I would like to permute the axes by only making
> reference to the axis names: what is the shape corresponding to the
> orientation (Z, Y, X)?
>
> This is a simple example because we only swap two axes and the resulting
> shape is the same as performing the same swap in the shape: (S, R, C). If
> we knew the indices of the axis names then we can infer these and pass them
> to swapaxes or transpose:
>
> vol = numpy.random.rand(C, R, S) # of shape (C, R, S) -> (X, Y, Z)
> # now (Z, Y, X)
> new_vol = numpy.swapaxes(vol, 0, 2)
> new_vol.shape # (S, R, C)
>
> The same applies to a double swap e.g. (Y, Z, X), though it is less
> straightforward using swapaxes. swapaxes only takes two indices (obviously)
> so we would need to call it twice reflecting the two swaps required. So we
> have to somehow figure which axes to swap successively: ((0, 2) then (0,
> 1)). We can do this in one step with numpy.transpose simply using indices
> (1, 2, 0).
>
> However, (and this is the big 'however'), how would we reverse this? The
> array has no memory of the original axes and 0, 1, and 2 now refer to the
> current axes. This is where the axes names (e.g. X, Y and Z) would come in
> handy.
>
> Axis names will allow permutations to happen predictably since the array
> will 'remember' what the original references were.
>
> Here is what I propose: an API to numpy.ndarray with some identity e.g.
>
> vol = numpy.random.rand(C, R, S)
> vol.shape # C, R, S
> vol.axes = ('X', 'Y', 'Z') # C=X, R=Y, S=Z
> new_vol = vol.permute_axes(('Z', 'Y', 'X'))
> # either
> new_vol.axes # ('X', 'Y', 'Z') # preserve orientation but change shape
> new_vol.shape # S, R, C
> # or
> new_vol.axes # ('Z', 'Y', 'X') # preserve shape but change orientation
> new_vol.shape # C, R, S
> # we can now reverse the permutation
> old_vol = new_vol.permute_axes(('X', 'Y', 'Z'))
> numpy.array_equal(vol, old_vol) # True
>
> I've checked the numpy API documentation and there is no attribute .axes
> present so this is the best candidate. Additionally, this will require the
> .permute_axes() method/function.
>
> Thanks for your consideration.
>
> Paul
> ___
> NumPy-Discussion mailing list -- numpy-discussion@python.org
> To unsubscribe send an email to numpy-discussion-le...@python.org
> https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
> Member address: jfoxrabinov...@gmail.com
>
___
NumPy-Discussion mailing list -- numpy-discussion@python.org
To unsubscribe send an email to numpy-discussion-le...@python.org
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
Member address: arch...@mail-archive.com
[Numpy-discussion] Re: ndarray shape permutation
On Tue, May 17, 2022 at 10:36 PM Joseph Fox-Rabinovitz < jfoxrabinov...@gmail.com> wrote: > You could easily write an extension to ndarray that maps axis names to > indices and vice versa. > Indeed. There have been several over the years. The current most-maintained one is xarray, near as I can tell. https://xarray.pydata.org/en/stable/index.html -- Robert Kern ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
