Right so, from your desire to avoid a copy, I must inform you that you are making a copy - maybe I didn't understand your case though - seemed you didn't want to have any overhead of memory and maybe cpu time.
What happens is the type conversion system is being ran inside the dataset write/read which unless provided with a background buffer via transfer properties, allocates a buffer, then proceeds to run a batch conversion from the source memory with source type to destination memory with destination type. It's not very heavy at all if you do it in batches of the right size - and you must weight this against the fairly high overhead of the HDF api relative to C++ with smart memory management techniques and user conversion code. But its usually not necessary to do that and the conversion api is there for free + robust. -Jason On Tue, Oct 13, 2015 at 10:07 PM, Jiaxin Han <[email protected]> wrote: > Thank you Jason! > > I actually just found a solution after digging really hard into the > documentation. I've posted my answer here > > http://stackoverflow.com/questions/33085684/hide-certain-fields-of-a-compound-datatype-from-being-written-to-or-read-back-f > > The key is to specify a properly packed datatype in file that is different > from the datatype in memory. > > Bests, > > Jiaxin > > > 2015-10-14 2:42 GMT+01:00 Jason Newton <[email protected]>: > >> You can't hide it. >> >> Your options are copy it to a new packed struct layout - maybe performing >> the conversions in chunks of a suitable number. This would probably be the >> fastest and probably pretty friendly on memory & high performance if you >> choose the right constants. You can use the type conversion routines to go >> from the originanl memory layout of the structs to the packed ones, or >> simply reduce it yourself - which will of course be higher performance / >> lower overhead if you preallocate (rather than append) and avoid all the >> complicated conversion code. >> >> Your second option is structure of arrays which decouples where each >> field goes, which of course allows you to specify exactly what you want. >> You could then either post-process or preprocess them back in as structures >> wherever is most appropriate. >> >> HTH, >> -Jason >> >> On Mon, Oct 12, 2015 at 11:55 AM, Jiaxin Han <[email protected]> wrote: >> >>> Hello, >>> >>> I was expecting that by not inserting a field into memtype, I could >>> avoid outputing that field to disk. However, the attached test file shows >>> that this is not the case. Even if I do not insert the field, everything is >>> written and everything is read back as well. >>> >>> I guess it has to do with the size of memtype. I tried pack() to reduce >>> the size but the data interpretation then went wrong. >>> >>> Defining a new struct containing only these wanted fields is not >>> optimal, since it would require copying the data to the new struct or back, >>> while my application involves huge amounts of data. What I try to hide is >>> actually a vector field, which I write out separately as an array of >>> variable-length arrays. Currently although I have omitted the vector field >>> in memtype, it is still written and then read back as well, which corrupts >>> the memory (the reading automatically fills the size and memory pointer of >>> the vectors with their written values, which are not valid pointers >>> anymore). >>> >>> So is there a way that I can truly hide a particular field from being >>> written as well as from being read back, without having to define a new >>> temporary class? >>> >>> Thank you! >>> >>> Jiaxin >>> >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> Twitter: https://twitter.com/hdf5 >>> >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> Twitter: https://twitter.com/hdf5 >> > > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5 >
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
