Hi Daniel,
It looks like you are writing chunks of size 100, where each struct is
maybe 40 bytes? I'm not sure what all the types are in the struct - but
if that is the case each chunk is about 4k. It is my understanding that
each chunk equates to one system write to disk, and these are expensive.
A good rule of thumb is to target 1MB chunks.
best,
David
software engineer at SLAC
On 07/19/15 06:26, Daniel Rimmelspacher wrote:
Dear hdf-forum,
I am trying to write compound data to an extendible hdf-dataset. For
the code snippet below, I am writing ~30000 compound items one-by-one,
resulting in an approximately 15MB h5-file.
For dumping this amount of data the hdf library requires roughly 15
seconds. This seems a little bit long to me. My guess is that
requesting the proper hyperslab for each new item wastes most of the
time.
Here, however, I am struggling a little bit, since I don't manage to
find out more about this.
I'd appreciate if someone would have a quick look at the code below in
order to give me a hint.
Thanks and regards,
Daniel
////////////////////////////////////////////////////////////////////
// Header: definition of struct type characteristic_t
//////////////////////////////////////////////// ////
...
/////////////////////////////////////////////////////////////////////
// This section initializes the dataset (once) for incremental read
/////////////////////////////////////////////////////////////////////
// initialize variable length string type
constStrType vlst(PredType::C_S1, H5T_VARIABLE);
// Create memory space for compound datatype
memspace = CompType(sizeof(characteristic_t));
H5Tinsert(memspace.getId(), "Name", HOFFSET(characteristic_t ,
name), vlst.getId());
H5Tinsert(memspace.getId(), "LongIdentifier",
HOFFSET(characteristic_t , longId), vlst.getId());
H5Tinsert(memspace.getId(), "Type", HOFFSET(characteristic_t
, type), vlst.getId());
H5Tinsert(memspace.getId(), "Address", HOFFSET(characteristic_t ,
address), vlst.getId());
H5Tinsert(memspace.getId(), "Deposit", HOFFSET(characteristic_t ,
deposit), vlst.getId());
H5Tinsert(memspace.getId(), "MaxDiff", HOFFSET(characteristic_t ,
maxDiff), vlst.getId());
H5Tinsert(memspace.getId(), "Conversion",
HOFFSET(characteristic_t , conversion), vlst.getId());
H5Tinsert(memspace.getId(), "LowerLimit",
HOFFSET(characteristic_t , lowerLimit), vlst.getId());
H5Tinsert(memspace.getId(), "UpperLimit",
HOFFSET(characteristic_t , upperLimit), vlst.getId());
// Prepare data set
dims[0] = 0; // Initial size
hsize_t rank = 1;
// data will be
alligned in array style
hsize_t maxDims[1] = {H5S_UNLIMITED};
// dataset will be
extendible
hsize_t chunkDims[1] = {100}; // some random chunksize
DataSpace *dataspace = newDataSpace (rank, dims,
maxDims); // set dataspace for dataset
// Modify dataset creation property to enable chunking
DSetCreatPropList prop;
prop.setChunk(rank, chunkDims);
// Create the chunked dataset. Note the use of pointer.
charData = file.createDataSet( "Characteristic", memspace, *dataspace,
prop);
// Init helper
hsize_t chunk[1] = {1};
chunkSpace = DataSpace(1, chunk, NULL);
filespace = DataSpace(charData.getSpace());
//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// This section will be called repeatadly in order to write the
compound items iteratively
/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
// Create the new item.
characteristic_t s1[1];
s1[0].name = name;
s1[0].longId = Id;
s1[0].type = type;
s1[0].address = address;
s1[0].deposit = deposit;
s1[0].maxDiff = maxDiff;
s1[0].conversion = conversion;
s1[0].lowerLimit = lowerLimit;
s1[0].upperLimit = upperLimit;
// Extend dataset
dims[0]++;
charData.extend(dims);
// Compute new dims
hsize_t chunk[1] = {1};
hsize_t start[1] = {0};
start[0] = dims[0]-1;
// Select a hyperslab in extended portion of the dataset.
filespace = charData.getSpace();
filespace.selectHyperslab(H5S_SELECT_SET, chunk, start);
// Write data to the extended portion of the dataset.
charData.write(s1, memspace, chunkSpace, filespace);
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
