Re: [gdal-dev] NetCDF and datum handling suggestion
On Wed, Sep 7, 2011 at 2:33 AM, Even Rouault even.roua...@mines-paris.org wrote: In case you feel like testing it, the functions needed are OGRSpatialReference::importFromProj4() and exportToProj4(), the later which is not documented. Perhaps others can comment on the suitability of exporttoProj4() ??? It is documented but doxygen didn't like a macro that was inserted between the comment and the function. Should be fixed by r23070 thanks! I'd note that proj4 strings have not all the expressivity of WKT. In particular they cannot capture citation strings, or authority name and code. So the suitability of using them depends on what you want to do with them. How important is this information? Assuming we drop all the citation strings, or authority names and codes, but retain the important parameters of each projection (in CF and proj format), are we missing anything important numerically? In other words, are there any show-stopper issues that would keep us from using proj4 instead of wkt? ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] NetCDF and datum handling suggestion
How important is this information? Assuming we drop all the citation strings, or authority names and codes, but retain the important parameters of each projection (in CF and proj format), are we missing anything important numerically? In other words, are there any show-stopper issues that would keep us from using proj4 instead of wkt? Generally not. You indeed mentionned a significant word, numerically, which indicates that you are interested more in the information for reprojection purposes, than strict (not to say pedantic ;-)) identification of the SRS. If the software that process the dataset all use proj4 and the proj4 string is an adequate representation of the SRS, then it is a perfect fit. There are situations where 2 distincts SRS from the EPSG database will translate into identical proj4 string. For example, just looking at the very beginning of the epsg file provided with proj4, I can see : # TWD97 3824 +proj=longlat +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +no_defs # IGRS 3889 +proj=longlat +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +no_defs The corresponding WKT (given by using gdal utility 'testepsg EPSG:') are : GEOGCS[TWD97, DATUM[Taiwan_Datum_1997, SPHEROID[GRS 1980,6378137,298.257222101, AUTHORITY[EPSG,7019]], TOWGS84[0,0,0,0,0,0,0], AUTHORITY[EPSG,1026]], PRIMEM[Greenwich,0, AUTHORITY[EPSG,8901]], UNIT[degree,0.0174532925199433, AUTHORITY[EPSG,9122]], AUTHORITY[EPSG,3824]] GEOGCS[IGRS, DATUM[Iraqi_Geospatial_Reference_System, SPHEROID[GRS 1980,6378137,298.257222101, AUTHORITY[EPSG,7019]], TOWGS84[0,0,0,0,0,0,0], AUTHORITY[EPSG,1029]], PRIMEM[Greenwich,0, AUTHORITY[EPSG,8901]], UNIT[degree,0.0174532925199433, AUTHORITY[EPSG,9122]], AUTHORITY[EPSG,3889]] Are the Taiwan_Datum_1997 and Iraqi_Geospatial_Reference_System really the same, or is it because we haven't captured the difference, or it is too subtle to be modelled with proj4, or it would require some grids that are not available, etc... I have no idea. Also you should be aware that some SRS have no translation into proj4 strings due to unsupported projection methods for example. Using proj4 strings is not a total heresy however. Postgis is a famous and widespread example of the use of proj4 strings. Although in the Postgis case, the spatial_ref_sys table includes authority names and code columns, and the SRID is used as a key in the geometry_columns table. So you can fix proj4 strings in a centralized way if needed. Another point to have in mind is that the proj4 strings (at least the ones provided in the epsg file that comes with proj4) are generally derived from the EPSG database, and there can be imperfections in the way they are translated, or later improvements. A recent proj4 change has added for example TOWGS84 statements in a lot of cases where they were missing before. You can look at the following page which gives details on how GDAL, proj4, libtiff and postgis SRS databases are built from the EPSG database : http://svn.osgeo.org/metacrs/geotiff/trunk/libgeotiff/csv/README Some could also say that proj4 strings are not a very formal definition, and that WKT has a better potential to act as a standard. However, reading http://home.gdal.org/projects/opengis/wktproblems.html might be enlightning. (Note : apparently the page has been written quite some time ago, and some statements do no longer apply, for example the one about PostGIS. PostGIS has used proj4 strings and not WKT for a long time. But most of the rest still apply however.) Not sure to really have helped you. There are issues with either using WKT or proj4 strings. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] NetCDF and datum handling suggestion
Thanks for the info Even I can see that there is not yet One standard to rule them all... Correct me if I am wrong: is WKT the preferred way GDAL stores the information? There was a proposal some time ago in the CF trac to add descriptive variables such as crs_id (EPSG code), crs_name and crs_wkt. https://cf-pcmdi.llnl.gov/trac/ticket/18 (pointed out by Patrick) https://cf-pcmdi.llnl.gov/trac/ticket/9 https://cf-pcmdi.llnl.gov/trac/ticket/27 However, these were dropped because of disagreement between the participants and the simpler and current approach was adopted (specify the numeric values of the projection and datum). Unfortunately, the issue has not been discussed since. I think our solution should address the following 1) Storing projection and datum in the netcdf file with 100% CF compliance. 2) Storing complete information so that it can be retrieved later by GDAL, This is already done by writing and reading the WKT. 3) Making sure that the CF and WKT are not contradictory, in the case that the projection was modified by another program, but left the WKT intact. 4) Reading CF-compliant projection information and translating that into WKT (this could eliminate the need for 2 and 3) Perhaps 4 can be achieved with another tool external to GDAL (as suggested in the CF discussions). Is there anyone who would be inclined to engage in a discussion within the CF community so that any or all of proj4, epsg and wkt be included as a CF standard? Regards, Etienne On Wed, Sep 7, 2011 at 4:54 PM, Even Rouault even.roua...@mines-paris.org wrote: How important is this information? Assuming we drop all the citation strings, or authority names and codes, but retain the important parameters of each projection (in CF and proj format), are we missing anything important numerically? In other words, are there any show-stopper issues that would keep us from using proj4 instead of wkt? Generally not. You indeed mentionned a significant word, numerically, which indicates that you are interested more in the information for reprojection purposes, than strict (not to say pedantic ;-)) identification of the SRS. If the software that process the dataset all use proj4 and the proj4 string is an adequate representation of the SRS, then it is a perfect fit. There are situations where 2 distincts SRS from the EPSG database will translate into identical proj4 string. For example, just looking at the very beginning of the epsg file provided with proj4, I can see : # TWD97 3824 +proj=longlat +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +no_defs # IGRS 3889 +proj=longlat +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +no_defs The corresponding WKT (given by using gdal utility 'testepsg EPSG:') are : GEOGCS[TWD97, DATUM[Taiwan_Datum_1997, SPHEROID[GRS 1980,6378137,298.257222101, AUTHORITY[EPSG,7019]], TOWGS84[0,0,0,0,0,0,0], AUTHORITY[EPSG,1026]], PRIMEM[Greenwich,0, AUTHORITY[EPSG,8901]], UNIT[degree,0.0174532925199433, AUTHORITY[EPSG,9122]], AUTHORITY[EPSG,3824]] GEOGCS[IGRS, DATUM[Iraqi_Geospatial_Reference_System, SPHEROID[GRS 1980,6378137,298.257222101, AUTHORITY[EPSG,7019]], TOWGS84[0,0,0,0,0,0,0], AUTHORITY[EPSG,1029]], PRIMEM[Greenwich,0, AUTHORITY[EPSG,8901]], UNIT[degree,0.0174532925199433, AUTHORITY[EPSG,9122]], AUTHORITY[EPSG,3889]] Are the Taiwan_Datum_1997 and Iraqi_Geospatial_Reference_System really the same, or is it because we haven't captured the difference, or it is too subtle to be modelled with proj4, or it would require some grids that are not available, etc... I have no idea. Also you should be aware that some SRS have no translation into proj4 strings due to unsupported projection methods for example. Using proj4 strings is not a total heresy however. Postgis is a famous and widespread example of the use of proj4 strings. Although in the Postgis case, the spatial_ref_sys table includes authority names and code columns, and the SRID is used as a key in the geometry_columns table. So you can fix proj4 strings in a centralized way if needed. Another point to have in mind is that the proj4 strings (at least the ones provided in the epsg file that comes with proj4) are generally derived from the EPSG database, and there can be imperfections in the way they are translated, or later improvements. A recent proj4 change has added for example TOWGS84 statements in a lot of cases where they were missing before. You can look at the following page which gives details on how GDAL, proj4, libtiff and postgis SRS databases are built from the EPSG database : http://svn.osgeo.org/metacrs/geotiff/trunk/libgeotiff/csv/README Some could also say that proj4 strings are not a very formal definition, and that WKT has a better potential to act as a standard. However, reading http://home.gdal.org/projects/opengis/wktproblems.html might be
Re: [gdal-dev] NetCDF and datum handling suggestion
Le mercredi 07 septembre 2011 22:49:58, Etienne Tourigny a écrit : Thanks for the info Even I can see that there is not yet One standard to rule them all... Correct me if I am wrong: is WKT the preferred way GDAL stores the information? I'd say that WKT serves as a pivot representation of SRS within GDAL/OGR. An OGRSpatialReference object can be bijectively represented as a WKT string (because the OGRSpatialReference object is really a tree representation build from the parsing of the WKT string). But behind that apparent simplicity, there are often a lot of transformations implied because few formats natively speak WKT. You have the Geotiff way, the MapInfo way, the PCIDSK way, the USGS way (used HDF4, NDF and FAST drivers for example), the ERMapper way, etc etc. You have also the ESRI WKT that must be morphed from/to GDAL WKT... And the netCDF CF way ;-) ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] NetCDF and datum handling suggestion
On Mon, Sep 5, 2011 at 11:23 PM, Etienne Tourigny etiennesky@gmail.com wrote: Patrick, A) your suggestion does make sense, it is imperial that basic datum information be saved as to adhere to CF-x standard. Are those CF variables (radius, flattening etc) really sufficient for software such as THREDDS? Yes, tests I did showed it was able to project correctly after adding these (whereas beforehand the WMS layers were off - presumably TDS assumes a spherical earth by default). The datum attributes are highlighted in color at CF conventions website, in the table at the end of the page: http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/apf.html B) The other aspect (to save projection_ref so that GDAL can import it) also makes sense, however I have the following concern: If another software changes the datum and or projection information of a netcdf file created by GDAL, but keeps the GDAL 'spatial_ref' tag intact, then GDAL would import the wrong projection. I don't think that this should happen, as a program would be expected to write a new grid_mapping variable, without the GDAL tags. Can you think of possible software which does this and make a test? I can only think of CDO, and I do know that it discards anything from a previous projection definition. I can't think of any software that does this off the top of my head either ... though we've been looking into a NetCDF driver for the Geotoolkit project, which may be able to reproject existing NetCDF datasets. In any case GDAL should probably print a warning if the 2 get 'out of sync' when it imports? C) With that concern in mind,I was wondering if we can do one of the following instead: 1) import the standard CF variables and guess the WKT from that 2) just be content with the basic DATUM values from the CF variables for example, here is what a wkt looks like when importing a CF netcdf file wich a WGS84 datum: GEOGCS[unknown, DATUM[unknown, SPHEROID[Spheroid,6378137,298.257223563]], PRIMEM[Greenwich,0], UNIT[degree,0.0174532925199433]] and here is the full WGS84 WKT generated by GDAL GEOGCS[WGS 84, DATUM[WGS_1984, SPHEROID[WGS 84,6378137,298.257223563, AUTHORITY[EPSG,7030]], AUTHORITY[EPSG,6326]], PRIMEM[Greenwich,0], UNIT[degree,0.0174532925199433], AUTHORITY[EPSG,4326]] As you can see, the important values are there, but there are no descriptive names and EPSG authority. This is not strictly identified as WGS84 EPSG datum, but the important stuff is there. I thought about this too ... it does seem a bit concerning though to try and recognise the named datums from the values - possibly you'd have issues of the accuracy they are stored at determining whether GDAL could recognise them correctly or not? I've heard there may be a 'dynamic datum' in development that has its values adjusted more frequently - in this case referencing a standard name would likely be necessary. Having checked the GDAL NetCDF format page though, it does check the CF encoded projection before it's own WKT, so it'd be good if we could consistently use CF as first reference for the datum too. D) I doubt that the CF conventions have plans to support PROJ.4/WKT in the near future. Have a look at the draft for CF-1.6, nothing new about PROJ.4 or WKT : http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#grid-mappings-and-projections However, I would suggest to discuss this on the CF mailing list. They might be receptive, as they already make references to PROJ.4 in their projection definitions. BUT it would be better in this case to use PROJ.4 strings instead of WKT, as (I think) that PROJ.4 strings are better supported than WKT (including Proj.4 itself). For example, the folks that work on CDO have plans to support PROJ.4 strings, but now WKT. Thanks, I'll need to read up a bit more on Proj.4 strings but sounds like it could be a good way to go. I just had a bit more of a look around the CF webpages and they did consider putting more CRS information in the specs, e.g. a crs_id attribute that could contain an EPSG code, but it didn't make it through as yet: https://cf-pcmdi.llnl.gov/trac/wiki/GridMappingAndCrs https://cf-pcmdi.llnl.gov/trac/ticket/18 So yes discussing this on their mailing list again sounds like a good idea. Am I correct in thinking that GDAL could output a PROJ.4 string and import it in the same manner that it currently does with WKT? Not sure - clearly you can specify Proj4 from the command line to commands, but not sure you can ask it to store this, will do some checking. BTW, there is a CF-NetCDF 1.0 Standards Working Group part of OGC, perhaps we can discuss with someone from that group? http://www.opengeospatial.org/projects/groups/cf-netcdf1.0swg Yes, sounds like a good idea. We actually read through the OGC standards on NetCDF recently, the basic standard already accepted really just
Re: [gdal-dev] NetCDF and datum handling suggestion
On Tue, Sep 6, 2011 at 8:31 PM, Patrick Sunter patdeve...@gmail.com wrote: On Mon, Sep 5, 2011 at 11:23 PM, Etienne Tourigny etiennesky@gmail.com wrote: Patrick, A) your suggestion does make sense, it is imperial that basic datum information be saved as to adhere to CF-x standard. Are those CF variables (radius, flattening etc) really sufficient for software such as THREDDS? Yes, tests I did showed it was able to project correctly after adding these (whereas beforehand the WMS layers were off - presumably TDS assumes a spherical earth by default). good The datum attributes are highlighted in color at CF conventions website, in the table at the end of the page: http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.5/apf.html B) The other aspect (to save projection_ref so that GDAL can import it) also makes sense, however I have the following concern: If another software changes the datum and or projection information of a netcdf file created by GDAL, but keeps the GDAL 'spatial_ref' tag intact, then GDAL would import the wrong projection. I don't think that this should happen, as a program would be expected to write a new grid_mapping variable, without the GDAL tags. Can you think of possible software which does this and make a test? I can only think of CDO, and I do know that it discards anything from a previous projection definition. I can't think of any software that does this off the top of my head either ... though we've been looking into a NetCDF driver for the Geotoolkit project, which may be able to reproject existing NetCDF datasets. In any case GDAL should probably print a warning if the 2 get 'out of sync' when it imports? How could we know if it is 'out of sync'? My feeling is that it's ok to assume that any software that messes with the projection info will delete any custom tags, but still safer to use as much standard information as possible, and stay away from GDAL tags when possible. C) With that concern in mind,I was wondering if we can do one of the following instead: 1) import the standard CF variables and guess the WKT from that 2) just be content with the basic DATUM values from the CF variables for example, here is what a wkt looks like when importing a CF netcdf file wich a WGS84 datum: GEOGCS[unknown, DATUM[unknown, SPHEROID[Spheroid,6378137,298.257223563]], PRIMEM[Greenwich,0], UNIT[degree,0.0174532925199433]] and here is the full WGS84 WKT generated by GDAL GEOGCS[WGS 84, DATUM[WGS_1984, SPHEROID[WGS 84,6378137,298.257223563, AUTHORITY[EPSG,7030]], AUTHORITY[EPSG,6326]], PRIMEM[Greenwich,0], UNIT[degree,0.0174532925199433], AUTHORITY[EPSG,4326]] As you can see, the important values are there, but there are no descriptive names and EPSG authority. This is not strictly identified as WGS84 EPSG datum, but the important stuff is there. I thought about this too ... it does seem a bit concerning though to try and recognise the named datums from the values - possibly you'd have issues of the accuracy they are stored at determining whether GDAL could recognise them correctly or not? I've heard there may be a 'dynamic datum' in development that has its values adjusted more frequently - in this case referencing a standard name would likely be necessary. Having checked the GDAL NetCDF format page though, it does check the CF encoded projection before it's own WKT, so it'd be good if we could consistently use CF as first reference for the datum too. agreed, I propose we stay away from WKT/PROJ.4 when a suitable CF representation exists, and if not fall back to WKT / PROJ.4 - unless it becomes a standard in later CF conventions, which could take time IMHO. D) I doubt that the CF conventions have plans to support PROJ.4/WKT in the near future. Have a look at the draft for CF-1.6, nothing new about PROJ.4 or WKT : http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#grid-mappings-and-projections However, I would suggest to discuss this on the CF mailing list. They might be receptive, as they already make references to PROJ.4 in their projection definitions. BUT it would be better in this case to use PROJ.4 strings instead of WKT, as (I think) that PROJ.4 strings are better supported than WKT (including Proj.4 itself). For example, the folks that work on CDO have plans to support PROJ.4 strings, but now WKT. Thanks, I'll need to read up a bit more on Proj.4 strings but sounds like it could be a good way to go. I just had a bit more of a look around the CF webpages and they did consider putting more CRS information in the specs, e.g. a crs_id attribute that could contain an EPSG code, but it didn't make it through as yet: https://cf-pcmdi.llnl.gov/trac/wiki/GridMappingAndCrs https://cf-pcmdi.llnl.gov/trac/ticket/18 So yes
Re: [gdal-dev] NetCDF and datum handling suggestion
In case you feel like testing it, the functions needed are OGRSpatialReference::importFromProj4() and exportToProj4(), the later which is not documented. Perhaps others can comment on the suitability of exporttoProj4() ??? It is documented but doxygen didn't like a macro that was inserted between the comment and the function. Should be fixed by r23070 I'd note that proj4 strings have not all the expressivity of WKT. In particular they cannot capture citation strings, or authority name and code. So the suitability of using them depends on what you want to do with them. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] NetCDF and datum handling suggestion
Patrick, A) your suggestion does make sense, it is imperial that basic datum information be saved as to adhere to CF-x standard. Are those CF variables (radius, flattening etc) really sufficient for software such as THREDDS? B) The other aspect (to save projection_ref so that GDAL can import it) also makes sense, however I have the following concern: If another software changes the datum and or projection information of a netcdf file created by GDAL, but keeps the GDAL 'spatial_ref' tag intact, then GDAL would import the wrong projection. I don't think that this should happen, as a program would be expected to write a new grid_mapping variable, without the GDAL tags. Can you think of possible software which does this and make a test? I can only think of CDO, and I do know that it discards anything from a previous projection definition. C) With that concern in mind,I was wondering if we can do one of the following instead: 1) import the standard CF variables and guess the WKT from that 2) just be content with the basic DATUM values from the CF variables for example, here is what a wkt looks like when importing a CF netcdf file wich a WGS84 datum: GEOGCS[unknown, DATUM[unknown, SPHEROID[Spheroid,6378137,298.257223563]], PRIMEM[Greenwich,0], UNIT[degree,0.0174532925199433]] and here is the full WGS84 WKT generated by GDAL GEOGCS[WGS 84, DATUM[WGS_1984, SPHEROID[WGS 84,6378137,298.257223563, AUTHORITY[EPSG,7030]], AUTHORITY[EPSG,6326]], PRIMEM[Greenwich,0], UNIT[degree,0.0174532925199433], AUTHORITY[EPSG,4326]] As you can see, the important values are there, but there are no descriptive names and EPSG authority. This is not strictly identified as WGS84 EPSG datum, but the important stuff is there. D) I doubt that the CF conventions have plans to support PROJ.4/WKT in the near future. Have a look at the draft for CF-1.6, nothing new about PROJ.4 or WKT : http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#grid-mappings-and-projections However, I would suggest to discuss this on the CF mailing list. They might be receptive, as they already make references to PROJ.4 in their projection definitions. BUT it would be better in this case to use PROJ.4 strings instead of WKT, as (I think) that PROJ.4 strings are better supported than WKT (including Proj.4 itself). For example, the folks that work on CDO have plans to support PROJ.4 strings, but now WKT. Am I correct in thinking that GDAL could output a PROJ.4 string and import it in the same manner that it currently does with WKT? BTW, there is a CF-NetCDF 1.0 Standards Working Group part of OGC, perhaps we can discuss with someone from that group? http://www.opengeospatial.org/projects/groups/cf-netcdf1.0swg Regards, Etienne On Mon, Sep 5, 2011 at 6:40 AM, Patrick Sunter patdeve...@gmail.com wrote: Hi Etienne and others interested in NetCDF driver, I did some work on WMS display of NetCDF files generated by GDAL today, and it gave me a chance to play around with think about how to handle the Datum issue as described at http://trac.osgeo.org/gdal/wiki/NetCDF_Improvements#Datumissues . Here's a first cut suggestion of how GDAL should behave when writing a new NetCDF file using gdal_translate: * Keep saving the current datum info, with names e.g. WGS84, as part of the WKT that GDAL saves in it's special spatial_ref attribute. * But _also_ save the actual spheroid parameters for the datum in use in attributes CF-1 can read, e.g. semi_major_axis, semi_minor_axis, and inverse_flattening. Then on import, GDAL would look for its spatial_ref by default as authoritative for datum info, but could use the CF-1 as a backup. This would parallel the way projection parameters are handled in the NetCDF driver, i.e. writing WKT in the 'spatial_ref' attribute, but also writes duplicate CF-1 convention ones if possible to help with display of the file in NetCDF tools and things like THREDDS. Perhaps this could be a passable option to the driver whether you write the duplicate CF-1 datum info or not, with default to on. (That is, unless the CF conventions have plan to support named datums coming up imminently?) cheers, Patrick. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
