Hi Werner, I am doing remote sensing into the seafloor with refraction being modeled. Our beamformer creates imagery via layers and its a useful way to deal with the refraction. But when you go to display the data, its useful to look at the side projection (where the vertical axis is depth and the horizontal is along track or range).
Isaac On Wed, Apr 26, 2017 at 3:14 PM, Werner Benger <[email protected]> wrote: > Hi Isaac, > > just out of curiosity, which operation do you consider where something > like Z,X,Y order is so much more efficient to justify such a data > transformation at I/O? > > Werner > > On 26.04.2017 13:58, Isaac Gerg wrote: > > I think of hyperslab as a sort of fancy click+drag selection tool of a > tensor (fancy in the sense that you can do strided blocks of selection). > The original problem stemmed from a desire to do matlab shiftdim-like or > numpy rollaxis-like operation. i.e. I have axes in memory of Z,X,Y because > of processing efficiency, but wish to store the data X,Y,Z for > visualization efficiency. Its really not a big deal to simply do a memcopy > but why waste the memory if I don't need to; this gives us a chance to > really do a deep dive on hyperslab to see if it can do the job. > > In trying to get this to work with hyperslabbing, this led me to the > source for np.rollaxis which led me to np.tranpose (how numpy implements > rollaxis under the good). The solution is rather elegant for them. Every > array in np stores a stride tuple which are the strides over dimension. So > to do a transpose they simply permute this tuple. Perhaps having a similar > interface for HDF is something to be considered? On the other hand, I > could see providing another interface to mangle data as "giving yourself > more rope to trip over" so to speak. > > In any case, I figured I'd share where the problem came from and in > digging around how others (i.e numpy) handle it given their use cases. > > Isaac > > On Wed, Apr 26, 2017 at 3:34 AM, Werner Benger <[email protected]> wrote: > >> Probably correct, Quincey might know better how far the hyperslab >> implementation could be modified to allow a negative stride. Maybe it >> wouldn't be much of a change, just there was no need so far to do it. If >> that is the case, then at least it would mean that the "hyperslab grammar" >> would indeed allow operations like transpose (and rotation) of a dataset, >> just the implementation does not support it. That aside, it's probably not >> efficient as it would require multiple hyperslab operations, so a point >> selection would be faster, but for big data and out-of-core data >> management, sometimes seemingly inefficient operations turn out to be >> better. >> >> On 26.04.2017 00:13, Isaac Gerg wrote: >> >> I am not sure if you are allowed a negative stride but even if you are, I >> believe the transpose still won't work because the stride doesnt stride >> dimension like numpy but strides across the memory space and wrap is not >> allowed. Does this line of thinking make sense? >> >> On Tue, Apr 25, 2017 at 6:06 PM, Werner Benger <[email protected]> >> wrote: >> >>> Hi Isaac, >>> >>> it's just thought that it could be done like this: >>> >>> Copy operation: start = 0, stride = 1 >>> >>> target_index = 0 + source_index * 1 >>> >>> Reflection: start = max_index, stride = -1 >>> >>> target_index = max_index + source_index * -1 >>> So a negative stride in one dimensions would go "backwards", thus do a >>> reflection in this direction. In practice the stride parameter is probably >>> an unsigned integer and thus running out of bounds instead of going >>> backwards, as you say. >>> >>> Werner >>> >>> >>> >>> On 25.04.2017 23:48, Isaac Gerg wrote: >>> >>> Hi Werner, thanks for the reply. Let's wrestle with this a bit. So >>> numpy does their indexing by strides, the stride for each dimension. hdf5 >>> does their "striding" across the data in the sense of more of a selection >>> than a stride. I would be inclined to say that even a negative stride >>> wouldnt allow this. The problem is the stride is a simply stride and not >>> one based on dimension index so the stride can't wrap so you end up with >>> out of bounds issues with the datasets. Does this seem right? >>> >>> On Tue, Apr 25, 2017 at 5:29 PM, Werner Benger <[email protected]> >>> wrote: >>> >>>> Hi Isaac, >>>> >>>> without proof, but I could imagine it's possible to implement a shear >>>> operation via hyperslabs. If so, then hyperslabs could be used to implement >>>> rotations: >>>> >>>> e.g. https://www.ocf.berkeley.edu/~fricke/projects/israel/paeth/r >>>> otation_by_shearing.html >>>> To do a transpose, one would need a reflection in addition to rotation >>>> (e.g. http://techieme.in/matrix-rotation/ ) , and I guess doing this >>>> operation of a reflection is not possible via hyperslabs, so doing a >>>> transpose is out of reach. Probably it would require to allow something >>>> like a negative stride value... >>>> >>>> Werner >>>> >>>> >>>> >>>> On 25.04.2017 22:24, Isaac Gerg wrote: >>>> >>>> Hi Quincy, I just was reading about point selections and was wondering >>>> if that's the way we would go. Thanks for confirming! >>>> >>>> On Tue, Apr 25, 2017 at 4:21 PM, Quincey Koziol <[email protected]> wrote: >>>> >>>>> >>>>> > On Apr 25, 2017, at 1:02 PM, Isaac Gerg <[email protected]> >>>>> wrote: >>>>> > >>>>> > A coworker and I are mulling over if its possible to take a simple >>>>> 3x3 matrix (stored row major) in a dataspace and write it out transposed >>>>> to >>>>> disk using a hyperslab. It seems that the grammer provided by start, >>>>> stride, blocks, and count won't allow for this. We hope we are wrong! It >>>>> is possible to do this? >>>>> >>>>> Hyperslabs won’t do this, but you could use a point selection >>>>> instead. >>>>> >>>>> Quincey >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Hdf-forum is for HDF software users discussion. >>>>> [email protected] >>>>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.h >>>>> dfgroup.org >>>>> Twitter: https://twitter.com/hdf5 >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Hdf-forum is for HDF software users >>>> [email protected]http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>>> Twitter: https://twitter.com/hdf5 >>>> >>>> -- >>>> ___________________________________________________________________________ >>>> Dr. Werner Benger Visualization Research >>>> Center for Computation & Technology at Louisiana State University (CCT/LSU) >>>> 2019 Digital Media Center, Baton Rouge, Louisiana 70803 >>>> Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: >>>> +1 225 578-5362 <%28225%29%20578-5362> >>>> >>>> _______________________________________________ Hdf-forum is for HDF >>>> software users discussion. [email protected] >>>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>>> Twitter: https://twitter.com/hdf5 >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users >>> [email protected]http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> Twitter: https://twitter.com/hdf5 >>> >>> -- >>> ___________________________________________________________________________ >>> Dr. Werner Benger Visualization Research >>> Center for Computation & Technology at Louisiana State University (CCT/LSU) >>> 2019 Digital Media Center, Baton Rouge, Louisiana 70803 >>> Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: >>> +1 225 578-5362 <%28225%29%20578-5362> >>> >>> _______________________________________________ Hdf-forum is for HDF >>> software users discussion. [email protected] >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> Twitter: https://twitter.com/hdf5 >> >> _______________________________________________ >> Hdf-forum is for HDF software users >> [email protected]http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> Twitter: https://twitter.com/hdf5 >> >> -- >> ___________________________________________________________________________ >> Dr. Werner Benger Visualization Research >> Center for Computation & Technology at Louisiana State University (CCT/LSU) >> 2019 Digital Media Center, Baton Rouge, Louisiana 70803 >> Tel.: +1 225 578 4809 <%28225%29%20578-4809> Fax.: +1 >> 225 578-5362 <%28225%29%20578-5362> >> >> _______________________________________________ Hdf-forum is for HDF >> software users discussion. [email protected] >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> Twitter: https://twitter.com/hdf5 > > _______________________________________________ > Hdf-forum is for HDF software users > [email protected]http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 > > -- > ___________________________________________________________________________ > Dr. Werner Benger Visualization Research > Center for Computation & Technology at Louisiana State University (CCT/LSU) > 2019 Digital Media Center, Baton Rouge, Louisiana 70803 > Tel.: +1 225 578 4809 <(225)%20578-4809> Fax.: +1 225 > 578-5362 <(225)%20578-5362> > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 >
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
