Hi, On 21/04/16 17:33, Pedro Vicente wrote:
[...]
>>>Hmmm. Is there any big reason NOT to try to read a netCDF produced HDF5 file with the native HDF5 library if someone so chooses? It's possible to read a netCDF file using HDF5, yes. There are 2 things that you will miss doing this: 1) the ability to inquire about shared netCDF dimensions. 2) the ability to read remotely with openDAP. Reading with HDF5 also exposes metadata that is supposed to be private to netCDF. See below
I know that there are several groups (KNMI on L1B data for Sentinel 5 precursor, Jaxa for the EarthCare mission) that use the plain HDF-5 library to *write* netCDF4 files, including shared dimensions. This is done for performance reasons, including memory management issues within the netCDF-4 library. The issues were reported, but a solution was not forthcoming within the time frame required for the S5P schedule.
The L1B processor runs into these issues because of the large number of groups they needed to support the on-ground calibration of the instrument (think several thousand groups in a file, there is no interface for closing a group - or the memory associated with a group was never actually released, either of these two). For level 2 processing we use the netCDF-4 library, as the number of groups is much smaller, and the overall data volume is a lot smaller as well.
These L1B files can be read using the netCDF4 library, but not modified by this library, opening in 'a' mode fails. The same happens if you use h5repack to compress a netCDF-4 file or add a checksum filter, reading is fine, but modifying fails. For the S5P project read-only compatibility is sufficient, but having a compression utility for netCDF4 files that keeps them available for modification sure would be nice.
It would be a complete nuisance if the netCDF-4 library would refuse to open hdf-5 files that it considers incompatible based on this flag. Yes, I realise that duplicating the effort of unicar for netCDF-4 and creating your own netCDF-4 write tool is not efficient, but neither are memory leaks deep in netCDF.
Best, Maarten Sneep -- KNMI T: 030 2206747 E: maarten.sn...@knmi.nl R: A2.14 _______________________________________________ CF-metadata mailing list CF-metadata@cgd.ucar.edu http://mailman.cgd.ucar.edu/mailman/listinfo/cf-metadata
