Thanks for explaining! This was very helpful :) Best, Samer
On 02.01.2017 4:02 AM, Martijn Jasperse wrote: > Hi Samer, > Sorry I had a copy-paste malfunction there. I actually meant to > use H5Fget_file_image > <https://support.hdfgroup.org/HDF5/doc/RM/RM_H5F.html#File-GetFileImage> twice > instead. First using a NULL pointer to retrieve the "file size", then > allocating some memory and calling it again to get the actual data. > > I think the purpose of setting your own callbacks is that you can > handle the /realloc /call so you know when the pointer gets changed. > Because even if you use the "in-place" flag, adding attributes will > likely cause the image to be expanded and hence require a new buffer. > But having your own realloc lets you catch that. According to the > manual, callback functions are allowed to be NULL to indicate the > defaults should be used. > > I did some work with file images a while back, but admittedly we > didn't bother with in-place for our application as the copy overhead > was relatively small. But maybe I can modify some old code to make an > example. > > Cheers, > Martijn > > > On 2 January 2017 at 13:28, Samer Afach <[email protected] > <mailto:[email protected]>> wrote: > > Hi Martijn: > > Thank you very much for your response. Since you split your answer > into two parts, I also have to do the same and ask two questions > about these parts, if you don't mind. > > 1. You said "you can get the file image back from two calls to > H5LTopen_file_image" But I don't get what two calls you meant. Are > you talking about the calls of "H5Pset_file_image_callbacks"? > > 2. I would love to do this as "in place", but there's a memory > leak I reported in the HDF5 library that happens when opening > images as in-place (ref. HDFFV-10019), and I'm waiting for > feedback about that. But ignoring that memory leak, are you saying > this is different from the first case (when it's not in-place)? > And are you saying that to get the size of the image, the only way > is to re-implement *all* the callback functions of > H5Pset_file_image_callbacks? > > Honestly that's a little scary that I'd love to see an example > about this. > > Thank you for all this information! You helped a lot :) > > Best, > Samer > > > On 01.01.2017 11:16 PM, Martijn Jasperse wrote: >> Hi Samer, >> Since you opened the file without H5LT_FILE_IMAGE_DONT_COPY, the >> library makes a copy of the original buffer and operates on that. >> So once you've created the attribute you wanted and done a flush, >> you can get the file image back from two calls >> to H5LTopen_file_image as usual (first to get the size and then >> to get the actual data). >> >> If you wanted to do it "in place" without making a copies you'd >> likely need to look into |H5Pset_file_image_callbacks|* *to >> handle when the file image is resized and hence when the buffer >> changes. >> >> Cheers, >> Martijn >> >> On 27 December 2016 at 10:47, Samer Afach <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Landon: >> >> Thanks for the response. Actually I'm not talking about file >> objects, but /file images/, which is basically the whole file >> stored in some buffer in memory. Imagine you use >> std::fstream::read() or the classic C FILE's fread() to read >> the whole file into memory, and then wanting to open it with >> the HDF5 library. This is why I'm using >> "H5LTopen_file_image()", which doesn't take a file name or >> file path, but takes a void* buffer, which is a pointer to >> the array where the file is stored /in memory/. >> >> Now after opening that using "H5LTopen_file_image()", we'll >> get an HDF5 file object (or file handle), the one you're >> talking about. My question is: Now after we do the changes to >> the file under that object, and after we H5Fflush(), that >> buffer may be changed. My question is: How do we keep track >> of that buffer (and its size) to write it back to a file >> eventually (using a std::fstream::write() or the classic C >> FILE's fwrite())? >> >> I would appreciate the help of anyone who knows about this. >> >> Best, >> Samer Afach >> >> >> On 27.12.2016 12:19 AM, Landon Clipp wrote: >>> Hello Samer, >>> >>> I think I get the gist of what you're asking. If you are >>> writing to the object and then want to read from the updated >>> file, you will need to call H5Fflush() in order to write >>> your data buffers to the file on your disk. Modifying >>> objects on your file does not actually immediately change >>> the file itself. The changes are only made when the object >>> and/or file is closed or flushed. I believe that should >>> solve your problem. >>> >>> Regards, >>> Landon Clipp >>> >>> On Thu, Dec 22, 2016 at 2:39 AM, Samer Afach <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hello guys: >>> >>> In a program, I use some relatively small HDF5 files >>> multiple times and transfer them then through TCP/IP, so >>> I read them into memory as `std::string`, and then read >>> them as HDF5 using: >>> >>> >>> fileHandle=H5LTopen_file_image((void*)&FileImage.front(),FileImage.size(),H5LT_FILE_IMAGE_OPEN_RW); >>> >>> >>> *My question is:* Since I'm opening this as RW, how can >>> I write an attribute to it and retrieve the image back? >>> >>> The motivation behind my question is simple. I have a >>> simple function that can write an attribute to an object >>> by its ID. Here it's: >>> >>> >>> voidWriteStringAttribute(hid_tdataset_id,conststd::string&name,conststd::string&value) >>> { >>> hid_tAttributeType=H5Tcopy(H5T_C_S1); >>> H5Tset_size(AttributeType,value.length()); >>> hid_tdataspaceHandle=H5Screate(H5S_SCALAR); >>> >>> hid_tAttrHandle=H5Acreate(dataset_id,name.c_str(),AttributeType,dataspaceHandle,H5P_DEFAULT,H5P_DEFAULT); >>> H5Awrite(AttrHandle,AttributeType,&value.front()); >>> >>> >>> H5Aclose(AttrHandle); >>> H5Sclose(dataspaceHandle); >>> } >>> >>> What I don't understand is that after writing the >>> attribute using this function (Is it right to do this? >>> >>> WriteStringAttribute(fileHandle,"MyAttr","MyAttrValue"); >>> >>> ), the image size should've been changed. How can I get >>> the new correct image size and write it back to a file >>> (or do anything else with it, for that matter)? >>> >>> Thank you. >>> >>> Best, Samer Afach >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> <mailto:[email protected]> >>> >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> >>> <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org> >>> Twitter: https://twitter.com/hdf5 >>> >>> _______________________________________________ >>> Hdf-forum is for HDF software users discussion. >>> [email protected] >>> <mailto:[email protected]> >>> >>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >>> >>> <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org> >>> Twitter: https://twitter.com/hdf5 >> _______________________________________________ Hdf-forum is >> for HDF software users discussion. >> [email protected] >> <mailto:[email protected]> >> >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> >> <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org> >> Twitter: https://twitter.com/hdf5 >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] <mailto:[email protected]> >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org> >> Twitter: https://twitter.com/hdf5 > _______________________________________________ Hdf-forum is for > HDF software users discussion. [email protected] > <mailto:[email protected]> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > <http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org> > Twitter: https://twitter.com/hdf5 > > _______________________________________________ > Hdf-forum is for HDF software users discussion. > [email protected] > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org > Twitter: https://twitter.com/hdf5
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
