Hi Isaac,
just out of curiosity, which operation do you consider where something
like Z,X,Y order is so much more efficient to justify such a data
transformation at I/O?
Werner
On 26.04.2017 13:58, Isaac Gerg wrote:
I think of hyperslab as a sort of fancy click+drag selection tool of a
tensor (fancy in the sense that you can do strided blocks of
selection). The original problem stemmed from a desire to do matlab
shiftdim-like or numpy rollaxis-like operation. i.e. I have axes in
memory of Z,X,Y because of processing efficiency, but wish to store
the data X,Y,Z for visualization efficiency. Its really not a big
deal to simply do a memcopy but why waste the memory if I don't need
to; this gives us a chance to really do a deep dive on hyperslab to
see if it can do the job.
In trying to get this to work with hyperslabbing, this led me to the
source for np.rollaxis which led me to np.tranpose (how numpy
implements rollaxis under the good). The solution is rather elegant
for them. Every array in np stores a stride tuple which are the
strides over dimension. So to do a transpose they simply permute this
tuple. Perhaps having a similar interface for HDF is something to be
considered? On the other hand, I could see providing another
interface to mangle data as "giving yourself more rope to trip over"
so to speak.
In any case, I figured I'd share where the problem came from and in
digging around how others (i.e numpy) handle it given their use cases.
Isaac
On Wed, Apr 26, 2017 at 3:34 AM, Werner Benger <[email protected]
<mailto:[email protected]>> wrote:
Probably correct, Quincey might know better how far the hyperslab
implementation could be modified to allow a negative stride. Maybe
it wouldn't be much of a change, just there was no need so far to
do it. If that is the case, then at least it would mean that the
"hyperslab grammar" would indeed allow operations like transpose
(and rotation) of a dataset, just the implementation does not
support it. That aside, it's probably not efficient as it would
require multiple hyperslab operations, so a point selection would
be faster, but for big data and out-of-core data management,
sometimes seemingly inefficient operations turn out to be better.
On 26.04.2017 00:13, Isaac Gerg wrote:
I am not sure if you are allowed a negative stride but even if
you are, I believe the transpose still won't work because the
stride doesnt stride dimension like numpy but strides across the
memory space and wrap is not allowed. Does this line of thinking
make sense?
On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger
<[email protected] <mailto:[email protected]>> wrote:
Hi Isaac,
it's just thought that it could be done like this:
Copy operation: start = 0, stride = 1
target_index = 0 + source_index * 1
Reflection: start = max_index, stride = -1
target_index = max_index + source_index * -1
So a negative stride in one dimensions would go "backwards",
thus do a reflection in this direction. In practice the
stride parameter is probably an unsigned integer and thus
running out of bounds instead of going backwards, as you say.
Werner
On 25.04.2017 23:48, Isaac Gerg wrote:
Hi Werner, thanks for the reply. Let's wrestle with this a
bit. So numpy does their indexing by strides, the stride
for each dimension. hdf5 does their "striding" across the
data in the sense of more of a selection than a stride. I
would be inclined to say that even a negative stride wouldnt
allow this. The problem is the stride is a simply stride
and not one based on dimension index so the stride can't
wrap so you end up with out of bounds issues with the
datasets. Does this seem right?
On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger
<[email protected] <mailto:[email protected]>> wrote:
Hi Isaac,
without proof, but I could imagine it's possible to
implement a shear operation via hyperslabs. If so, then
hyperslabs could be used to implement rotations:
e.g.
https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/rotation_by_shearing.html
<https://www.ocf.berkeley.edu/%7Efricke/projects/israel/paeth/rotation_by_shearing.html>
To do a transpose, one would need a reflection in
addition to rotation (e.g.
http://techieme.in/matrix-rotation/
<http://techieme.in/matrix-rotation/> ) , and I guess
doing this operation of a reflection is not possible via
hyperslabs, so doing a transpose is out of reach.
Probably it would require to allow something like a
negative stride value...
Werner
On 25.04.2017 22:24, Isaac Gerg wrote:
Hi Quincy, I just was reading about point selections
and was wondering if that's the way we would go. Thanks
for confirming!
On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol
<[email protected] <mailto:[email protected]>> wrote:
> On Apr 25, 2017, at 1:02 PM, Isaac Gerg
<[email protected]
<mailto:[email protected]>> wrote:
>
> A coworker and I are mulling over if its possible
to take a simple 3x3 matrix (stored row major) in a
dataspace and write it out transposed to disk using
a hyperslab. It seems that the grammer provided
by start, stride, blocks, and count won't allow for
this. We hope we are wrong! It is possible to do this?
Hyperslabs won’t do this, but you could use a point
selection instead.
Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
<mailto:[email protected]>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
<mailto:[email protected]>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter:https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
<mailto:[email protected]>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
<mailto:[email protected]>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter:https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>
_______________________________________________ Hdf-forum is
for HDF software users discussion.
[email protected]
<mailto:[email protected]>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected] <mailto:[email protected]>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter:https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.:+1 225 578 4809 <tel:%28225%29%20578-4809> Fax.:+1 225 578-5362 <tel:%28225%29%20578-5362>
_______________________________________________ Hdf-forum is for
HDF software users discussion. [email protected]
<mailto:[email protected]>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
<http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
