Hi, Even! I'm making some progress in utilizing OpenOptions to modify HDF drivers.
For automated testing of the new HDF drivers, should I use gdal.OpenEx() like below in gor2vrt.py? src_ds = gdal.OpenEx( infile, gdal.OF_VECTOR, open_options = openoptions ) Or does gdal.Open() also support open_options argument? -- HDF:Antifragile Solution for Elastic Bigdata Analytics On Fri, Aug 5, 2016 at 2:59 AM, Even Rouault <even.roua...@spatialys.com> wrote: > Frank, > > From what I understand from Joe's needs, it looks like simple transposing of a > 2D raster wouldn't be enough. Here we would need to "transpose" pixels > scattered through different subdatasets due to the Nd > 2 dimensionality of > the > original dataset, which would be impractical to express at the VRT level, and > likely inefficient. > > Something I've thought about would be to make Nd > 2 rasters native objects at > the GDAL level, but this would have likely deep implications on the code base > and should be considered carefully, and would be of interest for a limited set > of drivers (netCDF, HDF4, HDF5, Rasdaman). > > Even > >> Brian / Even, >> >> Certainly it is desirable for the HDF (and perhaps other super >> flexible formats like netcdf) to support an open option to select >> alternative axes. But the ability to transpose a dataset could also >> be quite valuable in the VRT driver to "fix" any input transposed >> dataset. >> >> I'm also not entirely certain why one couldn't supply an appopriately >> transposed geotransform to accomplish something similar. This could >> be done without any code changes in the existing VRT format. >> >> Best regards, >> Frank >> >> On Thu, Aug 4, 2016 at 3:32 PM, Even Rouault <even.roua...@spatialys.com> > wrote: >> > On Thursday 04 August 2016 16:31:25 H. Joe Lee wrote: >> >> Hi, >> >> >> >> My name is Joe Lee and I'm very interested in improving GDAL's >> >> >> >> capability to access NASA HDF4/HDF5 data so that users can work with >> >> HDF easily through GDAL. For example, my goal is to allow users to >> >> translate any HDF data into GeoTIFF via gdal_translate. >> >> >> >> I've worked with diverse NASA HDF products and provided solution for >> >> >> >> visualizing data correctly through Python/MATLAB/IDL/NCL [1] and I >> >> know that many products, other than HDF-EOS, may not work well with >> >> GDAL because HDF is flexible and NASA data producers do not >> >> necessarily follow the conventions that GDAL uses. >> >> >> >> By default, GDAL HDF4/HDF5 driver uses the following convention for >> >> >> >> unknown products. >> >> >> >> For HDF4 (frmts/hdf4/hdf4imagedataset.cpp): >> >> >> >> // Search for the starting "X" and "Y" in the names or take >> >> // the last two dimensions as X and Y sizes >> >> iXDim = nDimCount - 1; >> >> iYDim = nDimCount - 2; >> >> >> >> For HDF5 (frmts/hdf5/hdf5imagedataset.cpp): >> >> int GetYIndex() const { return IsComplexCSKL1A() ? 0 : ndims - 2; >> >> } >> >> int GetXIndex() const { return IsComplexCSKL1A() ? 1 : ndims - 1; >> >> } >> >> >> >> The above code works well as long as Unknown HDF product follows the >> >> >> >> above convention. However, in reality, HDF data can have an arbitrary >> >> >> >> order in terms of Band, X and Y dimension like this: >> >> dset4D[XDim=360][YDim=180][Band1=2][Band2=3] >> >> dimindex: 0 1 2 3 >> >> >> >> Since ndims=4, ndims-2 becomes 2 and ndims-1=3. In such case, GDAL >> >> >> >> generates 360x180 bands of 2x3 images, instead of the desired 2x3 >> >> bands of 360x180 images. >> >> >> >> Thus, I'm wondering if GDAL can expand VRT schema so that VRT allows >> >> >> >> users to specify the correct dimension index because specifying >> >> dimension order for each different NASA product in [1] is >> >> impractical. For example, I'd like suggest a new tag like >> >> >> >> "SourceDimension" like below: >> >> <VRTRasterBand dataType="UInt16" band="1"> >> >> <SimpleSource> >> >> >> >> <SourceFilename >> >> >> >> relativeToVRT="0">HDF4_SDS:UNKNOWN:"DATA_WITH_4D_DATASET.hdf":7</SourceFi >> >> len ame> <SourceDimension RasterXDim="0" RasterYDim="1" /> >> >> >> >> <SourceBand>1</SourceBand> >> >> <SourceProperties RasterXSize="360" RasterYSize="180" >> >> >> >> DataType="UInt16" BlockXSize="360" BlockYSize="180" /> >> >> >> >> ... >> >> >> >> </SimpleSource> >> >> >> >> </VRTRasterBand> >> >> >> >> Once user specifies correct dimensions by editing VRT, it can be >> >> >> >> used by GDAL HDF4/HDF5 drivers and the HDF drivers read the data >> >> correctly for GDAL's image buffer. >> >> >> >> Do you think it's right and feasible approach to solve wrong X/Y >> >> >> >> dimension order problem? Or do you have any other existing solution >> >> that can mitigate this problem in GDAL? The GEE project team has >> >> experimented the idea by creating another separate XML file [2] but I >> >> think it's time to sync with GDAL development team and come up with >> >> the most elegant solution. In my opinion, VRT looks best and I wish >> >> GDAL development team can give me some feedback on this idea. >> > >> > Joe, >> > >> > I would rather suggest to add open options to the drivers and pass them >> > with the exiting VRT OpenOptions element, rather than adding a new >> > element in the VRT that would be specific of a few drivers >> > >> > <SimpleSource> >> > >> > <SourceFilename> >> > >> > relativeToVRT="0">HDF4_SDS:UNKNOWN:"DATA_WITH_4D_DATASET.hdf":7</SourceFil >> > ename>> >> > <OpenOptions> >> > >> > <OOI key="RASTERXDIM">0</OOI> >> > <OOI key="RASTERYDIM">1</OOI> >> > >> > </OpenOptions> >> > >> > <SourceBand>1</SourceBand> >> > <SourceProperties RasterXSize="360" RasterYSize="180" >> > DataType="UInt16" >> > >> > BlockXSize="360" BlockYSize="180" /> >> > >> > ... >> > >> > </SimpleSource> >> > >> > Which is equivalent to: >> > >> > gdalinfo HDF4_SDS:UNKNOWN:"DATA_WITH_4D_DATASET.hdf":7 -oo RASTERXDIM=0 >> > -oo >> > RASTERYDIM=0 >> > >> > >> > Even >> > >> > -- >> > Spatialys - Geospatial professional services >> > http://www.spatialys.com >> > _______________________________________________ >> > gdal-dev mailing list >> > gdal-dev@lists.osgeo.org >> > http://lists.osgeo.org/mailman/listinfo/gdal-dev > > -- > Spatialys - Geospatial professional services > http://www.spatialys.com > _______________________________________________ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev