Hello,
Thank you everyone for your help. I figured out the problem, I was just
misunderstanding how the functions worked. I was able to successfully read
the dataset into a buffer. I did not realize that a 1D array was
sufficient, I was for some reason thinking that it had to be a contiguous
multidimensional array but it turns out that the functions know how to read
the arrays if you give it the rank and the size of each dimension.
Turns out I have another problem however. I am trying to now write this
buffer into a new file. The error happens when I try to create a new
dataset. When I ran my code, I got errors such as: "H5D.c line 194 in
H5Dcreate2(): unable to create dataset." I looked online and it turns out
that there is a size limit to the buffer and mine most certainly exceeds
that. So the solution is to create a dataset creation property list and set
it to chunk. Even after I have set a reasonable chunk size, I still get the
same errors. I will attach my code and the errors I am receiving. Relevant
code starts at line 122. Thank you SO MUCH for your help, I'm still trying
to learn all of this.
Landon
On Sat, Sep 3, 2016 at 11:45 AM, Michael Jackson <
[email protected]> wrote:
> You can take a look at the following source files. They are mean for C++
> and templates but assuming you know a bit of C++ you can convert them back
> to pure "C" without any issues. The template parameter is on the POD type.
>
> https://github.com/BlueQuartzSoftware/SIMPL/tree/develop/Source/H5Support
>
> Take a look at H5Lite.h and H5Lite.cpp. There are functions in there to
> "readPointerDataset()", writePointerDataSet() and getDatasetInfo().
>
> The basic flow would be the following (using some pure "C").
>
> // Open the file and get the "Location ID"
> hid_t fileId = ...
>
> char* datasetName = ....
> //Since you know it is a 5D array:
> hsize_t_t dims[5];
> H5T_class_t classType;
> size_t type_size;
> H5Lite::getDatasetInfo(fileId, datasetName, dims, classType, typesize);
>
> // Now loop over all the dim[] values to compute the total number
> // of elements that need to allocate, lets assume they are 32 bit
> // signed ints
> size_t totalElements = dims[0] * dims[1] * dims[2] * dims[3] * dims[4];
> // Allocate the data
> signed int* dataPtr = malloc(totalElements * sizeof(signed int));
>
> herr_t err = H5Lite::readPointerDataset(fileId, datasetName, dataPtr);
> // Check error
> if (err < 0) { ..... }
>
> // Open New file for writing
> hid_t outFileId = ...
> signed int rank = 5;
> err = H5Lite::writePointerDataset(outFileid, datasetName, rank, dims,
> dataPtr);
> // Check error
> if (err < 0) { ..... }
>
> This assumes that you take the code from GitHub and convert the necessary
> functions into pure "C" which should be straight forward to do.
>
> The code referenced above is BSD licensed.
>
> --
> Michael A. Jackson
> BlueQuartz Software, LLC
> [e]: [email protected]
>
>
> Nelson, Jarom wrote:
>
>> You might look at h5copy as a reference, or just use that tool to do the
>> work for you.
>>
>> Jarom
>>
>> *From:*Hdf-forum [mailto:[email protected]] *On
>> Behalf Of *Landon Clipp
>> *Sent:* Friday, September 02, 2016 11:56 AM
>> *To:* [email protected]
>> *Subject:* [Hdf-forum] Best way to repackage a dataset? (C program)
>>
>> Hello everyone,
>>
>> I am working with an HDF5 file that has a 5D dataset. What I'm wanting
>> to do is to create a C program that reads this dataset into memory and
>> then outputs it into a newly created file with only that dataset in it
>> (perhaps at the root directory of the file tree). What I don't
>> understand is how to read this entire 5D array using H5Dread into a 5D
>> buffer that has been previously allocated on the heap (note I cannot use
>> an array allocated on the stack, it would be too large and would create
>> seg faults).
>>
>> What is the general process I need to employ to do such a thing, and is
>> there maybe a more elegant solution to this than reading the entire
>> dataset into memory? This process seems easy to me for a 1 or 2D array
>> but I am lost with larger dimension arrays. Thanks.
>>
>> Regards,
>>
>> Landon
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> [email protected]
>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>> Twitter: https://twitter.com/hdf5
>>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> [email protected]
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5
>
HDF5-DIAG: Error detected in HDF5 (1.8.17) thread 0:
#000: H5D.c line 194 in H5Dcreate2(): unable to create dataset
major: Dataset
minor: Unable to initialize object
#001: H5Dint.c line 455 in H5D__create_named(): unable to create and link to
dataset
major: Dataset
minor: Unable to initialize object
#002: H5L.c line 1638 in H5L_link_object(): unable to create new link to
object
major: Links
minor: Unable to initialize object
#003: H5L.c line 1882 in H5L_create_real(): can't insert link
major: Symbol table
minor: Unable to insert object
#004: H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed
major: Symbol table
minor: Object not found
#005: H5Gtraverse.c line 755 in H5G_traverse_real(): component not found
major: Symbol table
minor: Object not found
HDF5-DIAG: Error detected in HDF5 (1.8.17) thread 0:
#000: H5D.c line 415 in H5Dclose(): not a dataset
major: Invalid arguments to routine
minor: Inappropriate type
#include <stdio.h>
#include <hdf5.h> // default HDF5 library
#include <stdlib.h>
#define DATASETNAME "HDFEOS/SWATHS/MOP01/Data Fields/MOPITTRadiances"
#define MOPITT_RANK 5
/*
NOTE: argv[1] = INPUT_FILE
argv[2] = OUTPUT_FILE
*/
/*********************
*FUNCTION PROTOTYPES*
*********************/
float getElement5D( float *array, hsize_t dimSize[5], int *position ); // note, this function is used for testing. It's temporary
int main( int argc, char* argv[] )
{
if ( argc != 3 )
{
fprintf( stderr, "\nError! Program expects format: %s [input HDF5 file] [output HDF5 file]\n\n", argv[0] );
return EXIT_FAILURE;
}
hid_t file; // file ID
hid_t dataset; // dataset ID
hid_t filespace; // filespace ID
hid_t memspace; // memoryspace ID
hid_t dcpl_id;
hsize_t MOPITTdims[MOPITT_RANK]; // size of each MOPITT dimension
hsize_t MOPITTmaxsize[MOPITT_RANK]; // maximum allowed size of each dimension in MOPITTdims
hsize_t MOPITTChunkSize[MOPITT_RANK];
herr_t status_n, status;
/*
* Open the file and do error checking
*/
file = H5Fopen( argv[1], H5F_ACC_RDONLY, H5P_DEFAULT );
if ( file < 0 )
{
fprintf( stderr, "\nError! Could not open HDF5 file. Exiting program.\n\n" );
return EXIT_FAILURE;
}
/*
* open radiance dataset and do error checking
*/
dataset = H5Dopen( file, DATASETNAME, H5F_ACC_RDONLY );
if ( dataset < 0 )
{
fprintf( stderr, "\nError! Could not open dataset. Exiting program.\n\n" );
return EXIT_FAILURE;
}
/*
* Get dataset dimension
*/
filespace = H5Dget_space(dataset);
status_n = H5Sget_simple_extent_dims(filespace, MOPITTdims, MOPITTmaxsize );
/*
* Create a data_out buffer using the dimensions of the dataset specified by MOPITTdims
*/
//float *****data_out = allocateMOPITTBuffer( MOPITTdims );
float *data_out = malloc( sizeof(float ) * MOPITTdims[0] * MOPITTdims[1] * MOPITTdims[2] * MOPITTdims[3] * MOPITTdims[4]);
/*
* get memory space
*/
memspace = H5Screate_simple( MOPITT_RANK, MOPITTdims, NULL );
/*
* Now read dataset into data_out buffer
* H5T_NATIVE_FLOAT is specifying that we are reading floating points
* H5S_ALL specifies that we are reading the entire dataset instead of just a portion
*/
status = H5Dread( dataset, H5T_NATIVE_FLOAT, memspace, filespace, H5P_DEFAULT, data_out );
/* Free all memory associated with the input data */
H5Sclose(memspace);
H5Sclose(filespace);
H5Dclose(dataset);
H5Fclose(file);
/*******************************************************************************************
*Reading the dataset is now complete. We have freed all related memory for the input data *
*(except of course the data_out array) and can now work on writing this data to a new file*
*******************************************************************************************/
/* Create a file. H5F_ACC_TRUNC says to overwrite if file already exists, H5P_DEFAULT
* says to create with default properties.
*/
file = H5Fcreate( argv[2], H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT );
/* Because dataset is large, we need to use chunking. Create dataset creation property list,
* set to use chunking.
*/
MOPITTChunkSize[0] = 100;
MOPITTChunkSize[1] = MOPITTdims[1];
MOPITTChunkSize[2] = MOPITTdims[2];
MOPITTChunkSize[3] = MOPITTdims[3];
MOPITTChunkSize[4] = MOPITTdims[4];
/* set dataspace creation property list to chunk */
dcpl_id = H5Pcreate(H5P_DATASET_CREATE);
H5Pset_chunk(dcpl_id, MOPITT_RANK, MOPITTChunkSize);
memspace = H5Screate_simple( MOPITT_RANK, MOPITTdims, NULL );
dataset = H5Dcreate( file, "/MOPITT/Data Fields/MOPITT Radiances", H5T_NATIVE_FLOAT, memspace,
H5P_DEFAULT, dcpl_id, H5P_DEFAULT );
/* Free all remaining memory */
H5Dclose(dataset);
H5Sclose(memspace);
H5Pclose(dcpl_id);
H5Fclose(file);
free(data_out);
return 0;
}
/*
DESCRIPTION:
This function returns the value specified by the dimensions given to it in the array
ARGUMENTS:
1. float array
2. Size of each dimension (given by a 5 element 1D array. Elements 0-4 specify size of dimensions 0-4).
3. Requested position in the array (given by a 5 element 1D array. Elements 0-4 specify which position in each dimension)
EFFECTS:
None. Does not write anything.
RETURN:
Value stored at specified dimension
array[dim0][dim1][dim2][dim3][dim4]
*/
float getElement5D( float *array, hsize_t dimSize[5], int position[5] )
{
if ( position[0] < 0 || position[0] >= dimSize[0] ||
position[1] < 0 || position[1] >= dimSize[1] ||
position[2] < 0 || position[2] >= dimSize[2] ||
position[3] < 0 || position[3] >= dimSize[3] ||
position[4] < 0 || position[4] >= dimSize[3] )
{
fprintf( stderr, "\ngetElement5D: Error! Invalid position. Check that dimSize and position arguments are correct.\n");
exit (-1);
}
return array[ (position[0] * dimSize[1] * dimSize[2] * dimSize[3] * dimSize[4]) +
(position[1] * dimSize[2] * dimSize[3] * dimSize[4]) +
(position[2] * dimSize[3] * dimSize[4]) +
(position[3] * dimSize[4]) +
position[4] ];
}
_______________________________________________
Hdf-forum is for HDF software users discussion.
[email protected]
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
