Re: [gdal-dev] RFC 61: Call for vote on adoption
I've fixed this so that currently all tests pass (at least locally). Now Shapefiles, which are created XY come back as XY, XYM as XYM, XYZ as XYZ, and XYZM as XYZM. Also one should be able to set the required coordinate combination using parameters - that's not comprehensively tested however. There are still a couple of fixme's in the code (shape2ogr.cpp), writing a shape should take into account the requested coordinate combination and whether to skip Ms or write them as "no data" (value < 1038 according to the doc). I changed the code for OGRFeature::SetGeomFieldDirectly to set or unset the geometry M or Z coordinates depending on the geometry field type of the feature. That's somewhat bold but the way I did it at least did not break anything in the tests. Ari 10.02.2016, 13:26, Ari Jolma kirjoitti: One more issue came up. Shapefiles, which have Z will also have M, at least a placeholder for it. Having shapefiles as XYZM always in that case may be problematic. It is possible to use the open options SHPT=type to override the default. We could have Z => XYZ as default for backwards compatibility and require open option for Z => XYZM. Ari 09.02.2016, 17:07, Ari Jolma kirjoitti: I declare this motion passed with +1 from Even, Tamas, Jukka and Daniel and no -1. I'll commit soon the changes to the core and then later the drivers: memory, shape and pg. The goal is to keep the travis test build passing. Ari 05.02.2016, 10:04, Ari Jolma kirjoitti: I'd like to ask the PSC and others to vote on adopting RFC 61: Support for measured geometries. The draft RFC is at https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries and a draft implementation is at https://github.com/ajolma/GDAL-XYZM which is tested at https://travis-ci.org/ajolma/GDAL-XYZM (the test #34, which passed, was the so far last one after core changes with unchanged autotests and a rather comprehensive set of WKT tests in the Perl tests directory) This is seemingly a small change but it is at the heart of OGR so it has some important implications. The only backwards incompatibilities that have appeared so far are with some drivers, for example shapefile, which can be lessened with, e.g., open options. Best, Ari ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
> ok, I should perhaps set up a better local test environment. > > I now limit the dimension harmonization to the shape driver @33440. Travis seems to be stalled right now, but locally I get a crash with latest rev : TEST: ogr_shape_15 ... ==4768== Invalid read of size 8 ==4768==at 0x8928771: SHPReadOGRFeature(SHPInfo*, DBFInfo*, OGRFeatureDefn*, int, tagSHPObject*, char const*) (shape2ogr.cpp:1164) ==4768==by 0x8810A30: OGRShapeLayer::FetchShape(int) (ogrshapelayer.cpp:730) ==4768==by 0x8810C14: OGRShapeLayer::GetNextFeature() (ogrshapelayer.cpp:794) ==4768==by 0x86B1359: OGR_L_GetNextFeature (ogrlayer.cpp:550) ==4768==by 0x2047E979: _wrap_Layer_GetNextFeature (ogr_wrap.cpp:3845) ==4768==by 0x4A7766: PyEval_EvalFrameEx (in /usr/bin/python2.6) ==4768==by 0x4A93D0: PyEval_EvalCodeEx (in /usr/bin/python2.6) ==4768==by 0x4A7568: PyEval_EvalFrameEx (in /usr/bin/python2.6) ==4768==by 0x4A82AF: PyEval_EvalFrameEx (in /usr/bin/python2.6) ==4768==by 0x4A82AF: PyEval_EvalFrameEx (in /usr/bin/python2.6) ==4768==by 0x4A82AF: PyEval_EvalFrameEx (in /usr/bin/python2.6) ==4768==by 0x4A93D0: PyEval_EvalCodeEx (in /usr/bin/python2.6) ==4768== Address 0x0 is not stack'd, malloc'd or (recently) free'd > > Ari -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
Le vendredi 12 février 2016 11:26:46, Ari Jolma a écrit : > I've fixed this so that currently all tests pass (at least locally). > > Now Shapefiles, which are created XY come back as XY, XYM as XYM, XYZ as > XYZ, and XYZM as XYZM. Also one should be able to set the required > coordinate combination using parameters - that's not comprehensively > tested however. > Great > There are still a couple of fixme's in the code (shape2ogr.cpp), writing > a shape should take into account the requested coordinate combination > and whether to skip Ms or write them as "no data" (value < 1038 > according to the doc). -10^38 > > I changed the code for OGRFeature::SetGeomFieldDirectly to set or unset > the geometry M or Z coordinates depending on the geometry field type of > the feature. That's somewhat bold but the way I did it at least did not > break anything in the tests. Agreed it is a non trivial change. But it does break things. Crashes in the CSW driver for example : https://travis-ci.org/rouault/gdal_coverage/builds/108752731 . And other non crashing failures. Perhaps do this dimension harmonization not in SetGeomFieldDirectly() but in a helper function and call it in the shapefile driver if you need it ? We could also try to fix drivers/tests, but there may be cases where drivers cannot declare a consistant coordinate dimensionnality because they have a mix of {Geom} and {Geom}Z features > > Ari > > 10.02.2016, 13:26, Ari Jolma kirjoitti: > > One more issue came up. > > > > Shapefiles, which have Z will also have M, at least a placeholder for it. > > > > Having shapefiles as XYZM always in that case may be problematic. > > > > It is possible to use the open options SHPT=type to override the > > default. We could have Z => XYZ as default for backwards compatibility > > and require open option for Z => XYZM. > > > > Ari > > > > 09.02.2016, 17:07, Ari Jolma kirjoitti: > >> I declare this motion passed with +1 from Even, Tamas, Jukka and > >> Daniel and no -1. > >> > >> I'll commit soon the changes to the core and then later the drivers: > >> memory, shape and pg. The goal is to keep the travis test build passing. > >> > >> Ari > >> > >> 05.02.2016, 10:04, Ari Jolma kirjoitti: > >>> I'd like to ask the PSC and others to vote on adopting RFC 61: > >>> Support for measured geometries. > >>> > >>> The draft RFC is at > >>> > >>> https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries > >>> > >>> and a draft implementation is at > >>> > >>> https://github.com/ajolma/GDAL-XYZM > >>> > >>> which is tested at > >>> > >>> https://travis-ci.org/ajolma/GDAL-XYZM > >>> > >>> (the test #34, which passed, was the so far last one after core > >>> changes with unchanged autotests and a rather comprehensive set of > >>> WKT tests in the Perl tests directory) > >>> > >>> This is seemingly a small change but it is at the heart of OGR so it > >>> has some important implications. The only backwards > >>> incompatibilities that have appeared so far are with some drivers, > >>> for example shapefile, which can be lessened with, e.g., open options. > >>> > >>> Best, > >>> > >>> Ari -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
12.02.2016, 12:43, Even Rouault kirjoitti: There are still a couple of fixme's in the code (shape2ogr.cpp), writing a shape should take into account the requested coordinate combination and whether to skip Ms or write them as "no data" (value < 1038 according to the doc). -10^38 copy paste error from pdf :) I changed the code for OGRFeature::SetGeomFieldDirectly to set or unset the geometry M or Z coordinates depending on the geometry field type of the feature. That's somewhat bold but the way I did it at least did not break anything in the tests. Agreed it is a non trivial change. But it does break things. Crashes in the CSW driver for example : https://travis-ci.org/rouault/gdal_coverage/builds/108752731 . And other non crashing failures. Perhaps do this dimension harmonization not in SetGeomFieldDirectly() but in a helper function and call it in the shapefile driver if you need it ? We could also try to fix drivers/tests, but there may be cases where drivers cannot declare a consistant coordinate dimensionnality because they have a mix of {Geom} and {Geom}Z features ok, I should perhaps set up a better local test environment. I now limit the dimension harmonization to the shape driver @33440. Ari ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
12.02.2016, 14:54, Even Rouault kirjoitti: ok, I should perhaps set up a better local test environment. I now limit the dimension harmonization to the shape driver @33440. Travis seems to be stalled right now, but locally I get a crash with latest rev : Thanks. Fixed. It is sometime difficult to spot errors in the log output of the autotests. Ari ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
10.02.2016, 15:21, Kurt Schwehr kirjoitti: One request: #define OGR_G_NOT_EMPTY_POINT 0x1 #define OGR_G_3D 0x2 #define OGR_G_MEASURED 0x4 #define OGR_G_IGNORE_MEASURED 0x8 Can you make these a static const int or static const unsigned int member of the OGRGeometry class? #defines make life more difficult. These constants are used also outside the classes. How does one access those for example in OGRWktReadPointsM, which receives the flags as a parameter? Ari ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
I've started a wiki page for the status of the drivers regarding measured geometries: https://trac.osgeo.org/gdal/wiki/MeasuredGeometriesInDrivers Ari 10.02.2016, 15:05, Even Rouault kirjoitti: Le mercredi 10 février 2016 13:53:28, Peter Halls a écrit : Types on page 2). PointZ A PointZ consists of a triplet of double-precision coordinates in the order X, Y, Z plus a measure" I had misinterpreted their meaning of "optional" here and submitted a documentation query to ESRI on being told I was wrong. The response was that M being optional means that it can be valueless, but is always present. Woo, really That's a genuine scoop. I reported it to Frank at the time ... but several email systems later, I no longer have my copies of the correspondence with ESRI nor the OGR bug report, in which these were copied. Somewhere I may still have a backup of my work area, to see what I did in my code. However, as you say, this is 20+ years ago It was something which, at the time, was causing us some problems I think you to be correct in your Shapelib summary below: Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles without M values for more than 20 years. I'm surprised we wouldn't have heard about that if such shapefiles couldn't be read. Or perhaps ESRI software is robust to missing M too Would someone looking at this discussion and having access to ESRI software be willing to generate tiny (meaning just one single shape, with the smallest number of vertices) shapefiles of type PointZ, ArcZ, PolygonZ and MultiPointZ, and *without* M values ? So as to enhance the test suite and see in particular which value is set exactly as the nodata value in the M component I was wondering if we should change the behaviour of the shape driver to add that nodata M component for XYZ shapes. This wouldn't cause compat problems with previous versions of GDAL, but it would cause .shp sizes to grow by ~ 1/3. So perhaps go on with current behaviour of stripping M component if most implementations seem to be OK with that ? Best wishes, Peter On 10 February 2016 at 12:31, Even Rouaultwrote: Le mercredi 10 février 2016 13:05:20, Peter Halls a écrit : Ari, et al, ESRI handle this in a non-intuitive way: XYM is supported, but Z always has a Measure, so is XYZM! The formal definition is here: https://www.*esri*.com/library/whitepapers/pdfs/*shapefile*.pdf (1998) Shape Types having Z are defined on pp19ff where it states: "Shape Types inX,Y,Z Space Shapes of this type have an optional coordinate M. Note that "no data" value can be specified as a value for M (see Numeric Types on page 2). PointZ A PointZ consists of a triplet of double-precision coordinates in the order X, Y, Z plus a measure" I had misinterpreted their meaning of "optional" here and submitted a documentation query to ESRI on being told I was wrong. The response was that M being optional means that it can be valueless, but is always present. Woo, really That's a genuine scoop. Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles without M values for more than 20 years. I'm surprised we wouldn't have heard about that if such shapefiles couldn't be read. Or perhaps ESRI software is robust to missing M too Best wishes, Peter -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
OGRGeometry::OGR_G_NOT_EMPTY_POINT On Wed, Feb 10, 2016 at 7:34 AM, Ari Jolmawrote: > > 10.02.2016, 15:21, Kurt Schwehr kirjoitti: > >> >> One request: >> >> #define OGR_G_NOT_EMPTY_POINT 0x1 >> #define OGR_G_3D 0x2 >> #define OGR_G_MEASURED 0x4 >> #define OGR_G_IGNORE_MEASURED 0x8 >> Can you make these a static const int or static const unsigned int member >> of the OGRGeometry class? #defines make life more difficult. >> > > These constants are used also outside the classes. How does one access > those for example in OGRWktReadPointsM, which receives the flags as a > parameter? > > Ari > > -- -- http://schwehr.org ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
On 11 February 2016 at 01:31, Even Rouaultwrote: > Le mercredi 10 février 2016 13:05:20, Peter Halls a écrit : >> ESRI handle this in a non-intuitive way: XYM is supported, but Z >> always has a Measure, so is XYZM! The formal definition is here: >> https://www.*esri*.com/library/whitepapers/pdfs/*shapefile*.pdf (1998) >> >> Shape Types having Z are defined on pp19ff where it states: >> >> "Shape Types inX,Y,Z Space >> Shapes of this type have an optional coordinate > > Woo, really That's a genuine scoop. > > Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles > without M values for more than 20 years. I'm surprised we wouldn't have heard > about that if such shapefiles couldn't be read. Or perhaps ESRI software is > robust to missing M too I've looked into this a few years ago, because I found differences in XYZ files written by GDAL and Esri software. There are two styles of (e.g.) PointZ files, where the coordinates are written with either three or four doubles (the last with "no data" if M is not used). The style of PointZ is determined by the content length provided in the record header as 16-bit words (either 14 or 18). Therefore, the specs are correct when they describe M as an "optional coordinate" (p.15). For XYZ shapefiles, GDAL always write three doubles, whereas Esri write four doubles with "no data" as the last coordinate. As a result, Esri's XYZ shp files are slightly larger than GDAL's. Occasionally I use software that requires PointZ shapefiles to have four doubles, so my existing workaround is to use Esri's CopyFeatures_management tool to rewrite the shapefile. Could "ogr2ogr -dim 4" be added to this RFC to write XYZ[M] Shapefiles that always use four floats for coordinates, as Esri software does? On 11 February 2016 at 02:05, Even Rouault wrote: > Would someone looking at this discussion and having access to ESRI software be > willing to generate tiny (meaning just one single shape, with the smallest > number of vertices) shapefiles of type PointZ, ArcZ, PolygonZ and MultiPointZ, > and *without* M values ? So as to enhance the test suite and see in particular > which value is set exactly as the nodata value in the M component I can generate these today, and will attach a zip file to https://trac.osgeo.org/gdal/wiki/MeasuredGeometriesInDrivers ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
One more issue came up. Shapefiles, which have Z will also have M, at least a placeholder for it. Having shapefiles as XYZM always in that case may be problematic. It is possible to use the open options SHPT=type to override the default. We could have Z => XYZ as default for backwards compatibility and require open option for Z => XYZM. Ari 09.02.2016, 17:07, Ari Jolma kirjoitti: I declare this motion passed with +1 from Even, Tamas, Jukka and Daniel and no -1. I'll commit soon the changes to the core and then later the drivers: memory, shape and pg. The goal is to keep the travis test build passing. Ari 05.02.2016, 10:04, Ari Jolma kirjoitti: I'd like to ask the PSC and others to vote on adopting RFC 61: Support for measured geometries. The draft RFC is at https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries and a draft implementation is at https://github.com/ajolma/GDAL-XYZM which is tested at https://travis-ci.org/ajolma/GDAL-XYZM (the test #34, which passed, was the so far last one after core changes with unchanged autotests and a rather comprehensive set of WKT tests in the Perl tests directory) This is seemingly a small change but it is at the heart of OGR so it has some important implications. The only backwards incompatibilities that have appeared so far are with some drivers, for example shapefile, which can be lessened with, e.g., open options. Best, Ari ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
Le mercredi 10 février 2016 12:26:59, Ari Jolma a écrit : > One more issue came up. > > Shapefiles, which have Z will also have M, at least a placeholder for it. > > Having shapefiles as XYZM always in that case may be problematic. Small precision for those not necessarily familiar with the internals of the shapefile format. The .shp and .shx header have a declared shape type (used to set the OGR layer geometry type), that can be (restricting to the case of polygon for the sake of clarity, but true for other geometry types) : SHPT_POLYGON : the shapefile contains only Polygon 2D shapes SHPT_POLYGONZ: the shapefile may contain a mix of PolygonZ or PolygonZM shapes SHPT_POLYGONM: the shapefile may contain a mix of Polygon or PolygonM shapes Declaring a SHPT_POLYGONM as a PolygonM layer is OK (if the shapefile doesn't contain PolygonM shapes, it should declare only SHPT_POLYGON) But declaring SHPT_POLYGONZ as a PolygonZM layer, when there are in fact no PolygonZM shapes in it but just PolygonZ, is more problematic for backward compatibility issues. Ideally when dealing with shapefiles without M values we shouldn't advertize M neither in the layer geometry type, nor in the feature geometry. > > It is possible to use the open options SHPT=type to override the > default. We could have Z => XYZ as default for backwards compatibility > and require open option for Z => XYZM. That's one possibility. As open options should be a last resort, if SHPT isn't specified perhaps we could do a probing of the first shape. If it has a M component then advertize ZM geometry type, otherwise just xZ. That's of course not completely bullet proof as there can be in theory a mix of Z and ZM shapes, but my feeling is that this should work in > 90% of the cases. That's an improvement I could have a look at during the Code Sprint if you don't feel confident enough in the shape driver. ~ Somehow related issue I've noticed but that should be more easily fixable. Now creating a Polygon25D shapefile creates PolygonZM and not PolygonZ shapes (and perhaps true for other shape types). trunk: $ ogr2ogr poly25d.shp poly.shp -nlt multipolygon25d -overwrite $ ll poly25d.* -rw-r--r-- 1 even even 529 2016-02-10 12:36 poly25d.dbf -rw-r--r-- 1 even even 557 2016-02-10 12:36 poly25d.prj -rw-r--r-- 1 even even 8820 2016-02-10 12:36 poly25d.shp -rw-r--r-- 1 even even 180 2016-02-10 12:36 poly25d.shx 2.0: $ ogr2ogr poly25d.shp poly.shp -nlt multipolygon25d -overwrite $ ll poly25d.* -rw-r--r-- 1 even even 529 2016-02-10 12:36 poly25d.dbf -rw-r--r-- 1 even even 557 2016-02-10 12:36 poly25d.prj -rw-r--r-- 1 even even 6700 2016-02-10 12:36 poly25d.shp -rw-r--r-- 1 even even 180 2016-02-10 12:36 poly25d.shx On the read size, I've commited https://trac.osgeo.org/gdal/changeset/33408 "Shape: fix crash when reading a MultiPointZ without M values, and when reading a PointZ without M do not create a PointZM geometry (trunk only)" Quickly checking at the code, I think other geometry types are OK, but perhaps we should have more extensive cases : - adding Z shapefiles of other geometry types generated with GDAL 2.0 and checking that the ExportToIsoWKT() read with trunk is the one expected - and/or generating Z shapes with trunk and checking they are read as Z only with ExportToIsoWKT() > > Ari > > 09.02.2016, 17:07, Ari Jolma kirjoitti: > > I declare this motion passed with +1 from Even, Tamas, Jukka and > > Daniel and no -1. > > > > I'll commit soon the changes to the core and then later the drivers: > > memory, shape and pg. The goal is to keep the travis test build passing. > > > > Ari > > > > 05.02.2016, 10:04, Ari Jolma kirjoitti: > >> I'd like to ask the PSC and others to vote on adopting RFC 61: > >> Support for measured geometries. > >> > >> The draft RFC is at > >> > >> https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries > >> > >> and a draft implementation is at > >> > >> https://github.com/ajolma/GDAL-XYZM > >> > >> which is tested at > >> > >> https://travis-ci.org/ajolma/GDAL-XYZM > >> > >> (the test #34, which passed, was the so far last one after core > >> changes with unchanged autotests and a rather comprehensive set of > >> WKT tests in the Perl tests directory) > >> > >> This is seemingly a small change but it is at the heart of OGR so it > >> has some important implications. The only backwards incompatibilities > >> that have appeared so far are with some drivers, for example > >> shapefile, which can be lessened with, e.g., open options. > >> > >> Best, > >> > >> Ari -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
Le mercredi 10 février 2016 13:05:20, Peter Halls a écrit : > Ari, et al, > > ESRI handle this in a non-intuitive way: XYM is supported, but Z > always has a Measure, so is XYZM! The formal definition is here: > https://www.*esri*.com/library/whitepapers/pdfs/*shapefile*.pdf (1998) > > Shape Types having Z are defined on pp19ff where it states: > > "Shape Types inX,Y,Z Space > Shapes of this type have an optional coordinate > M. Note that "no data" value can be specified as a value for M (see Numeric > Types on page 2). > PointZ > A PointZ consists of a triplet of double-precision coordinates in the order > X, Y, Z plus a measure" > > I had misinterpreted their meaning of "optional" here and submitted a > documentation query to ESRI on being told I was wrong. The response was > that M being optional means that it can be valueless, but is always > present. Woo, really That's a genuine scoop. Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles without M values for more than 20 years. I'm surprised we wouldn't have heard about that if such shapefiles couldn't be read. Or perhaps ESRI software is robust to missing M too > > Best wishes, > > Peter -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
Le mercredi 10 février 2016 13:22:49, Ari Jolma a écrit : > 10.02.2016, 13:55, Even Rouault kirjoitti: > > Le mercredi 10 février 2016 12:26:59, Ari Jolma a écrit : > >> One more issue came up. > >> > >> Shapefiles, which have Z will also have M, at least a placeholder for > >> it. > >> > >> Having shapefiles as XYZM always in that case may be problematic. > > > > Small precision for those not necessarily familiar with the internals of > > the shapefile format. The .shp and .shx header have a declared shape > > type (used to set the OGR layer geometry type), that can be (restricting > > to the case of polygon for the sake of clarity, but true for other > > geometry types) : SHPT_POLYGON : the shapefile contains only Polygon 2D > > shapes > > SHPT_POLYGONZ: the shapefile may contain a mix of PolygonZ or PolygonZM > > shapes > > Would you differentiate between *Z or *ZM and * and *M shapes using the > ESRI "no data": > > "Nevertheless, shapefiles support the concept of "no data" values, but > they are currently used only for measures. Any floating point number > smaller than –1038 is considered by a shapefile reader to > represent a "no data" value." Given latest Peter's message, yes likely. And also completely missing M as produced by shapelib up to now. > > > SHPT_POLYGONM: the shapefile may contain a mix of Polygon or PolygonM > > shapes > > > > Declaring a SHPT_POLYGONM as a PolygonM layer is OK (if the shapefile > > doesn't contain PolygonM shapes, it should declare only SHPT_POLYGON) > > > > But declaring SHPT_POLYGONZ as a PolygonZM layer, when there are in fact > > no PolygonZM shapes in it but just PolygonZ, is more problematic for > > backward compatibility issues. Ideally when dealing with shapefiles > > without M values we shouldn't advertize M neither in the layer geometry > > type, nor in the feature geometry. > > > >> It is possible to use the open options SHPT=type to override the > >> default. We could have Z => XYZ as default for backwards compatibility > >> and require open option for Z => XYZM. > > > > That's one possibility. As open options should be a last resort, if SHPT > > isn't specified perhaps we could do a probing of the first shape. If it > > has a M component then advertize ZM geometry type, otherwise just > > xZ. That's of course not completely bullet proof as there can be in > > theory a mix of Z and ZM shapes, but my feeling is that this should work > > in > 90% of the cases. > > > > That's an improvement I could have a look at during the Code Sprint if > > you don't feel confident enough in the shape driver. > > Feel free to have a look. I'm now stuck with the pg driver :) Hopefully it is less tricky than shapefiles. I see there are other test failures related to shapefiles as well > > Maybe, if what I wrote above is correct, read the whole shapefile in and > then, if all M values are "no data" downgrade all geometries to XY or XYZ? For performance reason, I'd perhaps not read all M values of all shapes, but only the ones of the first shape. An extra enhancement would be to have an option to do a full scan of the shapefile. We could also use that to avoid advertizing Polygon layer geometry type when there's a mix of Polygon/MultiPolygon. > > Ari > > > ~ > > > > Somehow related issue I've noticed but that should be more easily > > fixable. Now creating a Polygon25D shapefile creates PolygonZM and not > > PolygonZ shapes (and perhaps true for other shape types). > > > > trunk: > > $ ogr2ogr poly25d.shp poly.shp -nlt multipolygon25d -overwrite > > $ ll poly25d.* > > -rw-r--r-- 1 even even 529 2016-02-10 12:36 poly25d.dbf > > -rw-r--r-- 1 even even 557 2016-02-10 12:36 poly25d.prj > > -rw-r--r-- 1 even even 8820 2016-02-10 12:36 poly25d.shp > > -rw-r--r-- 1 even even 180 2016-02-10 12:36 poly25d.shx > > > > 2.0: > > $ ogr2ogr poly25d.shp poly.shp -nlt multipolygon25d -overwrite > > $ ll poly25d.* > > -rw-r--r-- 1 even even 529 2016-02-10 12:36 poly25d.dbf > > -rw-r--r-- 1 even even 557 2016-02-10 12:36 poly25d.prj > > -rw-r--r-- 1 even even 6700 2016-02-10 12:36 poly25d.shp > > -rw-r--r-- 1 even even 180 2016-02-10 12:36 poly25d.shx > > > > On the read size, I've commited > > https://trac.osgeo.org/gdal/changeset/33408 "Shape: fix crash when > > reading a MultiPointZ without M values, and when reading a PointZ > > without M do not create a PointZM geometry (trunk only)" > > > > Quickly checking at the code, I think other geometry types are OK, but > > perhaps we should have more extensive cases : > > - adding Z shapefiles of other geometry types generated with GDAL 2.0 and > > checking that the ExportToIsoWKT() read with trunk is the one expected > > - and/or generating Z shapes with trunk and checking they are read as Z > > only with ExportToIsoWKT() > > > >> Ari > >> > >> 09.02.2016, 17:07, Ari Jolma kirjoitti: > >>> I declare this motion passed with +1 from Even, Tamas, Jukka and > >>> Daniel and no -1. > >>> > >>>
Re: [gdal-dev] RFC 61: Call for vote on adoption
Ari Jolma gmail.com> writes: > > > One more issue came up. > Shapefiles, which have Z will also have M, at least a placeholder > for it. > Having shapefiles as XYZM always in that case may be problematic. > It is possible to use the open options SHPT=type to override the > default. We could have Z => XYZ as default for backwards > compatibility and require open option for Z => XYZM. > Ari09.02.2016, 17:07, Ari Jolma kirjoitti: I noticed your changeset and I was just reading the shapefile specification. I never knew that there are no separate XYZ and XYZM shapetypes. I also believe I have never met a shapefile that really contains XYZM data. I agree that the default should be to convert such shapefiles to XYZ instead of XYZM. That would affect shapetype values 11 PointZ 13 PolyLineZ 15 PolygonZ 18 MultiPointZ -Jukka Rahkonen- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
Ari, et al, ESRI handle this in a non-intuitive way: XYM is supported, but Z always has a Measure, so is XYZM! The formal definition is here: https://www.*esri*.com/library/whitepapers/pdfs/*shapefile*.pdf (1998) Shape Types having Z are defined on pp19ff where it states: "Shape Types inX,Y,Z Space Shapes of this type have an optional coordinate M. Note that "no data" value can be specified as a value for M (see Numeric Types on page 2). PointZ A PointZ consists of a triplet of double-precision coordinates in the order X, Y, Z plus a measure" I had misinterpreted their meaning of "optional" here and submitted a documentation query to ESRI on being told I was wrong. The response was that M being optional means that it can be valueless, but is always present. Best wishes, Peter ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
10.02.2016, 13:55, Even Rouault kirjoitti: Le mercredi 10 février 2016 12:26:59, Ari Jolma a écrit : One more issue came up. Shapefiles, which have Z will also have M, at least a placeholder for it. Having shapefiles as XYZM always in that case may be problematic. Small precision for those not necessarily familiar with the internals of the shapefile format. The .shp and .shx header have a declared shape type (used to set the OGR layer geometry type), that can be (restricting to the case of polygon for the sake of clarity, but true for other geometry types) : SHPT_POLYGON : the shapefile contains only Polygon 2D shapes SHPT_POLYGONZ: the shapefile may contain a mix of PolygonZ or PolygonZM shapes Would you differentiate between *Z or *ZM and * and *M shapes using the ESRI "no data": "Nevertheless, shapefiles support the concept of "no data" values, but they are currently used only for measures. Any floating point number smaller than –1038 is considered by a shapefile reader to represent a "no data" value." SHPT_POLYGONM: the shapefile may contain a mix of Polygon or PolygonM shapes Declaring a SHPT_POLYGONM as a PolygonM layer is OK (if the shapefile doesn't contain PolygonM shapes, it should declare only SHPT_POLYGON) But declaring SHPT_POLYGONZ as a PolygonZM layer, when there are in fact no PolygonZM shapes in it but just PolygonZ, is more problematic for backward compatibility issues. Ideally when dealing with shapefiles without M values we shouldn't advertize M neither in the layer geometry type, nor in the feature geometry. It is possible to use the open options SHPT=type to override the default. We could have Z => XYZ as default for backwards compatibility and require open option for Z => XYZM. That's one possibility. As open options should be a last resort, if SHPT isn't specified perhaps we could do a probing of the first shape. If it has a M component then advertize ZM geometry type, otherwise just xZ. That's of course not completely bullet proof as there can be in theory a mix of Z and ZM shapes, but my feeling is that this should work in > 90% of the cases. That's an improvement I could have a look at during the Code Sprint if you don't feel confident enough in the shape driver. Feel free to have a look. I'm now stuck with the pg driver :) Maybe, if what I wrote above is correct, read the whole shapefile in and then, if all M values are "no data" downgrade all geometries to XY or XYZ? Ari ~ Somehow related issue I've noticed but that should be more easily fixable. Now creating a Polygon25D shapefile creates PolygonZM and not PolygonZ shapes (and perhaps true for other shape types). trunk: $ ogr2ogr poly25d.shp poly.shp -nlt multipolygon25d -overwrite $ ll poly25d.* -rw-r--r-- 1 even even 529 2016-02-10 12:36 poly25d.dbf -rw-r--r-- 1 even even 557 2016-02-10 12:36 poly25d.prj -rw-r--r-- 1 even even 8820 2016-02-10 12:36 poly25d.shp -rw-r--r-- 1 even even 180 2016-02-10 12:36 poly25d.shx 2.0: $ ogr2ogr poly25d.shp poly.shp -nlt multipolygon25d -overwrite $ ll poly25d.* -rw-r--r-- 1 even even 529 2016-02-10 12:36 poly25d.dbf -rw-r--r-- 1 even even 557 2016-02-10 12:36 poly25d.prj -rw-r--r-- 1 even even 6700 2016-02-10 12:36 poly25d.shp -rw-r--r-- 1 even even 180 2016-02-10 12:36 poly25d.shx On the read size, I've commited https://trac.osgeo.org/gdal/changeset/33408 "Shape: fix crash when reading a MultiPointZ without M values, and when reading a PointZ without M do not create a PointZM geometry (trunk only)" Quickly checking at the code, I think other geometry types are OK, but perhaps we should have more extensive cases : - adding Z shapefiles of other geometry types generated with GDAL 2.0 and checking that the ExportToIsoWKT() read with trunk is the one expected - and/or generating Z shapes with trunk and checking they are read as Z only with ExportToIsoWKT() Ari 09.02.2016, 17:07, Ari Jolma kirjoitti: I declare this motion passed with +1 from Even, Tamas, Jukka and Daniel and no -1. I'll commit soon the changes to the core and then later the drivers: memory, shape and pg. The goal is to keep the travis test build passing. Ari 05.02.2016, 10:04, Ari Jolma kirjoitti: I'd like to ask the PSC and others to vote on adopting RFC 61: Support for measured geometries. The draft RFC is at https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries and a draft implementation is at https://github.com/ajolma/GDAL-XYZM which is tested at https://travis-ci.org/ajolma/GDAL-XYZM (the test #34, which passed, was the so far last one after core changes with unchanged autotests and a rather comprehensive set of WKT tests in the Perl tests directory) This is seemingly a small change but it is at the heart of OGR so it has some important implications. The only backwards incompatibilities that have appeared so far are with some drivers, for example shapefile, which can be lessened with, e.g., open options. Best, Ari
Re: [gdal-dev] RFC 61: Call for vote on adoption
> Types on page 2). > PointZ > A PointZ consists of a triplet of double-precision coordinates in the order > X, Y, Z plus a measure" > > I had misinterpreted their meaning of "optional" here and submitted a > documentation query to ESRI on being told I was wrong. The response was > that M being optional means that it can be valueless, but is always > present. Woo, really That's a genuine scoop. I reported it to Frank at the time ... but several email systems later, I no longer have my copies of the correspondence with ESRI nor the OGR bug report, in which these were copied. Somewhere I may still have a backup of my work area, to see what I did in my code. However, as you say, this is 20+ years ago It was something which, at the time, was causing us some problems I think you to be correct in your Shapelib summary below: Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles without M values for more than 20 years. I'm surprised we wouldn't have heard about that if such shapefiles couldn't be read. Or perhaps ESRI software is robust to missing M too Best wishes, Peter On 10 February 2016 at 12:31, Even Rouaultwrote: > Le mercredi 10 février 2016 13:05:20, Peter Halls a écrit : > > Ari, et al, > > > > ESRI handle this in a non-intuitive way: XYM is supported, but Z > > always has a Measure, so is XYZM! The formal definition is here: > > https://www.*esri*.com/library/whitepapers/pdfs/*shapefile*.pdf (1998) > > > > Shape Types having Z are defined on pp19ff where it states: > > > > "Shape Types inX,Y,Z Space > > Shapes of this type have an optional coordinate > > M. Note that "no data" value can be specified as a value for M (see > Numeric > > Types on page 2). > > PointZ > > A PointZ consists of a triplet of double-precision coordinates in the > order > > X, Y, Z plus a measure" > > > > I had misinterpreted their meaning of "optional" here and submitted a > > documentation query to ESRI on being told I was wrong. The response was > > that M being optional means that it can be valueless, but is always > > present. > > Woo, really That's a genuine scoop. > > Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles > without M values for more than 20 years. I'm surprised we wouldn't have > heard > about that if such shapefiles couldn't be read. Or perhaps ESRI software is > robust to missing M too > > > > > Best wishes, > > > > Peter > > -- > Spatialys - Geospatial professional services > http://www.spatialys.com > -- Peter J Halls, PhD Student, Post-war Reconstruction and Development Unit (PRDU), University of York Snail mail: PRDU, Derwent College, University of York, Heslington, York YO10 5DD This message has the status of a private and personal communication ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
Le mercredi 10 février 2016 13:53:28, Peter Halls a écrit : > > Types on page 2). > > PointZ > > A PointZ consists of a triplet of double-precision coordinates in the > > order > > > X, Y, Z plus a measure" > > > > I had misinterpreted their meaning of "optional" here and submitted a > > documentation query to ESRI on being told I was wrong. The response was > > that M being optional means that it can be valueless, but is always > > present. > > Woo, really That's a genuine scoop. > > I reported it to Frank at the time ... but several email systems later, I > no longer have my copies of the correspondence with ESRI nor the OGR bug > report, in which these were copied. Somewhere I may still have a backup of > my work area, to see what I did in my code. However, as you say, this is > 20+ years ago It was something which, at the time, was causing us > some problems > > I think you to be correct in your Shapelib summary below: > > Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles > without M values for more than 20 years. I'm surprised we wouldn't have > heard about that if such shapefiles couldn't be read. Or perhaps ESRI > software is robust to missing M too Would someone looking at this discussion and having access to ESRI software be willing to generate tiny (meaning just one single shape, with the smallest number of vertices) shapefiles of type PointZ, ArcZ, PolygonZ and MultiPointZ, and *without* M values ? So as to enhance the test suite and see in particular which value is set exactly as the nodata value in the M component I was wondering if we should change the behaviour of the shape driver to add that nodata M component for XYZ shapes. This wouldn't cause compat problems with previous versions of GDAL, but it would cause .shp sizes to grow by ~ 1/3. So perhaps go on with current behaviour of stripping M component if most implementations seem to be OK with that ? > > Best wishes, > > Peter > > > > > On 10 February 2016 at 12:31, Even Rouault> > wrote: > > Le mercredi 10 février 2016 13:05:20, Peter Halls a écrit : > > > Ari, et al, > > > > > > ESRI handle this in a non-intuitive way: XYM is supported, but Z > > > > > > always has a Measure, so is XYZM! The formal definition is here: > > > https://www.*esri*.com/library/whitepapers/pdfs/*shapefile*.pdf > > > (1998) > > > > > > Shape Types having Z are defined on pp19ff where it states: > > > "Shape Types inX,Y,Z Space > > > Shapes of this type have an optional coordinate > > > M. Note that "no data" value can be specified as a value for M (see > > > > Numeric > > > > > Types on page 2). > > > PointZ > > > A PointZ consists of a triplet of double-precision coordinates in the > > > > order > > > > > X, Y, Z plus a measure" > > > > > > I had misinterpreted their meaning of "optional" here and submitted a > > > documentation query to ESRI on being told I was wrong. The response > > > was that M being optional means that it can be valueless, but is > > > always present. > > > > Woo, really That's a genuine scoop. > > > > Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles > > without M values for more than 20 years. I'm surprised we wouldn't have > > heard > > about that if such shapefiles couldn't be read. Or perhaps ESRI software > > is robust to missing M too > > > > > Best wishes, > > > > > > Peter > > > > -- > > Spatialys - Geospatial professional services > > http://www.spatialys.com -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
I'm +0. Neither for nor against. One request: #define OGR_G_NOT_EMPTY_POINT 0x1 #define OGR_G_3D 0x2 #define OGR_G_MEASURED 0x4 #define OGR_G_IGNORE_MEASURED 0x8 Can you make these a static const int or static const unsigned int member of the OGRGeometry class? #defines make life more difficult. find . -name \*.h -o -name \*.cpp | xargs grep '#define' | egrep '[0-9"]' | egrep -v '[(]|\[' | wc -l 4517 A random example of doing this: https://trac.osgeo.org/gdal/browser/trunk/gdal/alg/gdal_simplesurf.h?rev=33127#L93 // Descriptor length static const int DESC_SIZE = 64; -kurt On Wed, Feb 10, 2016 at 4:53 AM, Peter Hallswrote: > > Types on page 2). > > PointZ > > A PointZ consists of a triplet of double-precision coordinates in the > order > > X, Y, Z plus a measure" > > > > I had misinterpreted their meaning of "optional" here and submitted a > > documentation query to ESRI on being told I was wrong. The response was > > that M being optional means that it can be valueless, but is always > > present. > > Woo, really That's a genuine scoop. > > I reported it to Frank at the time ... but several email systems later, I > no longer have my copies of the correspondence with ESRI nor the OGR bug > report, in which these were copied. Somewhere I may still have a backup of > my work area, to see what I did in my code. However, as you say, this is > 20+ years ago It was something which, at the time, was causing us > some problems > > I think you to be correct in your Shapelib summary below: > > Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles > without M values for more than 20 years. I'm surprised we wouldn't have > heard about that if such shapefiles couldn't be read. Or perhaps ESRI > software is robust to missing M too > > Best wishes, > > Peter > > > > > On 10 February 2016 at 12:31, Even Rouault > wrote: > >> Le mercredi 10 février 2016 13:05:20, Peter Halls a écrit : >> > Ari, et al, >> > >> > ESRI handle this in a non-intuitive way: XYM is supported, but Z >> > always has a Measure, so is XYZM! The formal definition is here: >> > https://www.*esri*.com/library/whitepapers/pdfs/*shapefile*.pdf >> (1998) >> > >> > Shape Types having Z are defined on pp19ff where it states: >> > >> > "Shape Types inX,Y,Z Space >> > Shapes of this type have an optional coordinate >> > M. Note that "no data" value can be specified as a value for M (see >> Numeric >> > Types on page 2). >> > PointZ >> > A PointZ consists of a triplet of double-precision coordinates in the >> order >> > X, Y, Z plus a measure" >> > >> > I had misinterpreted their meaning of "optional" here and submitted a >> > documentation query to ESRI on being told I was wrong. The response was >> > that M being optional means that it can be valueless, but is always >> > present. >> >> Woo, really That's a genuine scoop. >> >> Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles >> without M values for more than 20 years. I'm surprised we wouldn't have >> heard >> about that if such shapefiles couldn't be read. Or perhaps ESRI software >> is >> robust to missing M too >> >> > >> > Best wishes, >> > >> > Peter >> >> -- >> Spatialys - Geospatial professional services >> http://www.spatialys.com >> > > > > -- > > > Peter J Halls, PhD Student, Post-war Reconstruction and Development Unit > (PRDU), > University of York > > Snail mail: PRDU, Derwent College, University of York, > Heslington, York YO10 5DD > This message has the status of a private and personal communication > > > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev > -- -- http://schwehr.org ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
On 11 February 2016 at 02:05, Even Rouaultwrote: > Would someone looking at this discussion and having access to ESRI software be > willing to generate tiny (meaning just one single shape, with the smallest > number of vertices) shapefiles of type PointZ, ArcZ, PolygonZ and MultiPointZ, > and *without* M values ? So as to enhance the test suite and see in particular > which value is set exactly as the nodata value in the M component Done, see https://trac.osgeo.org/gdal/attachment/wiki/MeasuredGeometriesInDrivers/ I created two geometry examples per shapefile, and added an "Expected" attribute to describe the WKT of the expected geometry. While the M values were editable (despite not requesting this creation option in ArcCatalog), I left these as "NaN". ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
09.02.2016, 11:17, Even Rouault kirjoitti: Tests are not independant from each other, so I suspect the failure in ogr_pg_21 might let a (implicit) transaction non committed. Thus ogr_pg_24 which opens a new connection doesn't see the table created before. Whereas if you quit, the transaction will be committed. That's it. I think it is a bug. Ari ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
Thanks, We now have +1 from Even, Tamas, Jukka and Daniel. Is that formally enough? I have a small change to the draft. I now think that the coordinate flags should be managed/queried solely through Is3D(), IsMeasured() (which are standard) and set3D(boolean) and setMeasured(boolean), (which are extensions to the SF specification). I.e., the getFlags() and setFlags(int) should be removed. The reason is that there is one flag, which is used only for points, and thus the API can be easily misused. Similarly the bit #defines for the coordinate flags sh/could be private. In practice that means to move them to ogr_p.h. Even, there are strange errors in the ogr_pg.py tests. https://s3.amazonaws.com/archive.travis-ci.org/jobs/107782519/log.txt Test 24 fails because the layer is NoneType, i.e., it does not open the datatypetest table. However, if I run the tests up to 24 but quit before it, the table exists and seems to be ok. I had similar problems with the shape driver - temporary datasets disappearing midtests. Any idea what's the reason? Ari 08.02.2016, 19:06, Daniel Morissette kirjoitti: On 2016-02-05 3:04 AM, Ari Jolma wrote: I'd like to ask the PSC and others to vote on adopting RFC 61: Support for measured geometries. The draft RFC is at https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries +1 Daniel ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
09.02.2016, 10:18, Ari Jolma kirjoitti: Even, there are strange errors in the ogr_pg.py tests. https://s3.amazonaws.com/archive.travis-ci.org/jobs/107782519/log.txt Test 24 fails because the layer is NoneType, i.e., it does not open the datatypetest table. However, if I run the tests up to 24 but quit before it, the table exists and seems to be ok. I had similar problems with the shape driver - temporary datasets disappearing midtests. Any idea what's the reason? A bit more on this: If I run the tests until 24, i.e., including it, and then quit. The table datatypetest is in the database. If I run the test under gdb just until where test 24 attempts to open the table, the table is not in the database. I can see this also from pgAdmin. The table seems to disappear just for the test. Which is very confusing. Ari ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
Le mardi 09 février 2016 09:18:00, Ari Jolma a écrit : > Thanks, > > We now have +1 from Even, Tamas, Jukka and Daniel. Is that formally enough? Yes. at least 2 business days, 2 +1 votes and no -1. > > I have a small change to the draft. I now think that the coordinate > flags should be managed/queried solely through Is3D(), IsMeasured() > (which are standard) and set3D(boolean) and setMeasured(boolean), (which > are extensions to the SF specification). I.e., the getFlags() and > setFlags(int) should be removed. The reason is that there is one flag, > which is used only for points, and thus the API can be easily misused. Sounds good. I had a bit that feeling too when looking at the code but didn't dive enough into the implementation to have a strong opinion on it. > > Similarly the bit #defines for the coordinate flags sh/could be private. > In practice that means to move them to ogr_p.h. > > Even, there are strange errors in the ogr_pg.py tests. > > https://s3.amazonaws.com/archive.travis-ci.org/jobs/107782519/log.txt > > Test 24 fails because the layer is NoneType, i.e., it does not open the > datatypetest table. However, if I run the tests up to 24 but quit before > it, the table exists and seems to be ok. Tests are not independant from each other, so I suspect the failure in ogr_pg_21 might let a (implicit) transaction non committed. Thus ogr_pg_24 which opens a new connection doesn't see the table created before. Whereas if you quit, the transaction will be committed. > I had similar problems with the > shape driver - temporary datasets disappearing midtests. Any idea what's > the reason? This cannot be due to transactions here, but here too steps may have dependencies. > > Ari > > 08.02.2016, 19:06, Daniel Morissette kirjoitti: > > On 2016-02-05 3:04 AM, Ari Jolma wrote: > >> I'd like to ask the PSC and others to vote on adopting RFC 61: Support > >> for measured geometries. > >> > >> The draft RFC is at > >> > >> https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries > > > > +1 > > > > Daniel > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
I declare this motion passed with +1 from Even, Tamas, Jukka and Daniel and no -1. I'll commit soon the changes to the core and then later the drivers: memory, shape and pg. The goal is to keep the travis test build passing. Ari 05.02.2016, 10:04, Ari Jolma kirjoitti: I'd like to ask the PSC and others to vote on adopting RFC 61: Support for measured geometries. The draft RFC is at https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries and a draft implementation is at https://github.com/ajolma/GDAL-XYZM which is tested at https://travis-ci.org/ajolma/GDAL-XYZM (the test #34, which passed, was the so far last one after core changes with unchanged autotests and a rather comprehensive set of WKT tests in the Perl tests directory) This is seemingly a small change but it is at the heart of OGR so it has some important implications. The only backwards incompatibilities that have appeared so far are with some drivers, for example shapefile, which can be lessened with, e.g., open options. Best, Ari ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
Le lundi 08 février 2016 08:17:28, vous avez écrit : > 06.02.2016, 18:00, Sean Gillies kirjoitti: > > Can you explain more in detail about the backwards incompatibilities > > and how to soften them? What's going to break and what are the > > specific options? > > Hi Sean, > > In the C++ and C API there are no backwards incompatibilities but the > get/setCoordinateDimension() methods become deprecated since XYM and XYZ > have the same dimension. Is3D(), IsMeasured(), set3D(), and > setMeasured() should be used instead. > > I don't know about the other drivers but Shapefile driver currently > reads XYM geometries as XYZ and that will be changed so that XYM are > read as XYM. I think that is the only incompatibility. A new open option > MEASURE_AS_Z=YES/NO (or something) can be defined for backwards > compatibility. I don't think there's a strong need for such an option without confirmation from users. The XYM support as XYZ was really a hack (people had no way to know if the XYZ was XYM or XYZ, unless they read the .shp/.shx themselves). And it is not clear how that would work in update scenarios (currently the M/Z is discarded on update : https://trac.osgeo.org/gdal/ticket/6331 ) Pirmin: as I see from https://trac.osgeo.org/gdal/ticket/2374 you were the one to contribute the patch, what is your opinion on this ? > Similar option can be defined for other drivers if they > have the same feature. I believe all other drivers currently discard the M values -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
Hi, Am Montag, 8. Februar 2016, 10.04:55 schrieb Even Rouault: > Le lundi 08 février 2016 08:17:28, vous avez écrit : > > 06.02.2016, 18:00, Sean Gillies kirjoitti: > > > Can you explain more in detail about the backwards incompatibilities > > > and how to soften them? What's going to break and what are the > > > specific options? > > > > Hi Sean, > > > > In the C++ and C API there are no backwards incompatibilities but the > > get/setCoordinateDimension() methods become deprecated since XYM and XYZ > > have the same dimension. Is3D(), IsMeasured(), set3D(), and > > setMeasured() should be used instead. > > > > I don't know about the other drivers but Shapefile driver currently > > reads XYM geometries as XYZ and that will be changed so that XYM are > > read as XYM. I think that is the only incompatibility. A new open option > > MEASURE_AS_Z=YES/NO (or something) can be defined for backwards > > compatibility. > > I don't think there's a strong need for such an option without confirmation > from users. The XYM support as XYZ was really a hack (people had no way to > know if the XYZ was XYM or XYZ, unless they read the .shp/.shx themselves). > And it is not clear how that would work in update scenarios (currently the > M/Z is discarded on update : https://trac.osgeo.org/gdal/ticket/6331 ) > > Pirmin: as I see from https://trac.osgeo.org/gdal/ticket/2374 you were the > one to contribute the patch, what is your opinion on this ? Completeley forgot that... We had a customer request for accessing M values in Shapefiles from QGIS and the result was this hacky solution. In my opinion we can live without a compatibilty switch to this intermediate solution when a proper API for XYZM is provided. It will be easy to adapt that in a coming version of QGIS (update would be needed also for setting the compatibility switch) and in the meantime they have to stay on their GDAL version (what they usually do anyway). So an incompatibility warning in the release notes would be enough for me. Regards Pirmin -- Pirmin Kalberer Sourcepole - Linux & Open Source Solutions http://www.sourcepole.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
On 2016-02-05 3:04 AM, Ari Jolma wrote: I'd like to ask the PSC and others to vote on adopting RFC 61: Support for measured geometries. The draft RFC is at https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries +1 Daniel -- Daniel Morissette http://www.mapgears.com/ T: +1 418-696-5056 #201 http://evouala.com/ - Location Intelligence Made Easy ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
06.02.2016, 18:00, Sean Gillies kirjoitti: Can you explain more in detail about the backwards incompatibilities and how to soften them? What's going to break and what are the specific options? Hi Sean, In the C++ and C API there are no backwards incompatibilities but the get/setCoordinateDimension() methods become deprecated since XYM and XYZ have the same dimension. Is3D(), IsMeasured(), set3D(), and setMeasured() should be used instead. I don't know about the other drivers but Shapefile driver currently reads XYM geometries as XYZ and that will be changed so that XYM are read as XYM. I think that is the only incompatibility. A new open option MEASURE_AS_Z=YES/NO (or something) can be defined for backwards compatibility. Similar option can be defined for other drivers if they have the same feature. Ari ps: Sorry for the late answer, I was offline the weekend. -- Sean Gillies ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
Hi Ari, On Fri, Feb 5, 2016 at 1:04 AM, Ari Jolmawrote: > I'd like to ask the PSC and others to vote on adopting RFC 61: Support > for measured geometries. > > The draft RFC is at > > https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries > > and a draft implementation is at > > https://github.com/ajolma/GDAL-XYZM > > which is tested at > > https://travis-ci.org/ajolma/GDAL-XYZM > > (the test #34, which passed, was the so far last one after core changes > with unchanged autotests and a rather comprehensive set of WKT tests in the > Perl tests directory) > > This is seemingly a small change but it is at the heart of OGR so it has > some important implications. The only backwards incompatibilities that have > appeared so far are with some drivers, for example shapefile, which can be > lessened with, e.g., open options. > Can you explain more in detail about the backwards incompatibilities and how to soften them? What's going to break and what are the specific options? -- Sean Gillies ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] RFC 61: Call for vote on adoption
I'd like to ask the PSC and others to vote on adopting RFC 61: Support for measured geometries. The draft RFC is at https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries and a draft implementation is at https://github.com/ajolma/GDAL-XYZM which is tested at https://travis-ci.org/ajolma/GDAL-XYZM (the test #34, which passed, was the so far last one after core changes with unchanged autotests and a rather comprehensive set of WKT tests in the Perl tests directory) This is seemingly a small change but it is at the heart of OGR so it has some important implications. The only backwards incompatibilities that have appeared so far are with some drivers, for example shapefile, which can be lessened with, e.g., open options. Best, Ari ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
Le vendredi 05 février 2016 09:04:00, Ari Jolma a écrit : > I'd like to ask the PSC and others to vote on adopting RFC 61: Support > for measured geometries. +1 > > The draft RFC is at > > https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries > > and a draft implementation is at > > https://github.com/ajolma/GDAL-XYZM > > which is tested at > > https://travis-ci.org/ajolma/GDAL-XYZM > > (the test #34, What is test #34 ? > which passed, was the so far last one after core changes > with unchanged autotests and a rather comprehensive set of WKT tests in > the Perl tests directory) > I see some failures in latest build, apparently related to -nlt POLYGON25D / - lco SHPT=POLYGONZ in the shapefile driver https://s3.amazonaws.com/archive.travis-ci.org/jobs/107177078/log.txt Failures Script: utilities/test_ogr2ogr_lib.py TEST: test_ogr2ogr_lib_7 ... fail Script: utilities/test_gdal_grid_lib.py TEST: test_gdal_grid_lib_1 ... fail Script: utilities/test_ogr2ogr.py TEST: test_ogr2ogr_11 ... fail TEST: test_ogr2ogr_12 ... fail TEST: test_ogr2ogr_43 ... fail Script: pyscripts/test_ogr2ogr_py.py TEST: test_ogr2ogr_py_11 ... fail TEST: test_ogr2ogr_py_12 ... fail TEST: test_ogr2ogr_py_43 ... fail -- > This is seemingly a small change but it is at the heart of OGR so it has > some important implications. The only backwards incompatibilities that > have appeared so far are with some drivers, for example shapefile, which > can be lessened with, e.g., open options. > > Best, > > Ari -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
+1 I'll update the mssql spatial driver accordingly. Best regards, Tamas 2016-02-05 9:04 GMT+01:00 Ari Jolma: > I'd like to ask the PSC and others to vote on adopting RFC 61: Support > for measured geometries. > > The draft RFC is at > > https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries > > and a draft implementation is at > > https://github.com/ajolma/GDAL-XYZM > > which is tested at > > https://travis-ci.org/ajolma/GDAL-XYZM > > (the test #34, which passed, was the so far last one after core changes > with unchanged autotests and a rather comprehensive set of WKT tests in the > Perl tests directory) > > This is seemingly a small change but it is at the heart of OGR so it has > some important implications. The only backwards incompatibilities that have > appeared so far are with some drivers, for example shapefile, which can be > lessened with, e.g., open options. > > Best, > > Ari > > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > http://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
05.02.2016, 10:49, Even Rouault kirjoitti: Le vendredi 05 février 2016 09:04:00, Ari Jolma a écrit : I'd like to ask the PSC and others to vote on adopting RFC 61: Support for measured geometries. +1 The draft RFC is at https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries and a draft implementation is at https://github.com/ajolma/GDAL-XYZM which is tested at https://travis-ci.org/ajolma/GDAL-XYZM (the test #34, What is test #34 ? https://travis-ci.org/ajolma/GDAL-XYZM/builds/106980691 ...it has #34 which passed, was the so far last one after core changes with unchanged autotests and a rather comprehensive set of WKT tests in the Perl tests directory) I see some failures in latest build, apparently related to -nlt POLYGON25D / - lco SHPT=POLYGONZ in the shapefile driver Yes, it fails now since the shapefile driver is partially implemented - only points and linestrings so far. I made it skip one test in ogr_shape.py, which relied on the M to Z hacks. https://s3.amazonaws.com/archive.travis-ci.org/jobs/107177078/log.txt Failures Script: utilities/test_ogr2ogr_lib.py TEST: test_ogr2ogr_lib_7 ... fail Script: utilities/test_gdal_grid_lib.py TEST: test_gdal_grid_lib_1 ... fail Script: utilities/test_ogr2ogr.py TEST: test_ogr2ogr_11 ... fail TEST: test_ogr2ogr_12 ... fail TEST: test_ogr2ogr_43 ... fail Script: pyscripts/test_ogr2ogr_py.py TEST: test_ogr2ogr_py_11 ... fail TEST: test_ogr2ogr_py_12 ... fail TEST: test_ogr2ogr_py_43 ... fail -- This is seemingly a small change but it is at the heart of OGR so it has some important implications. The only backwards incompatibilities that have appeared so far are with some drivers, for example shapefile, which can be lessened with, e.g., open options. Best, Ari ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] RFC 61: Call for vote on adoption
Ari Jolma gmail.com> writes: > > > I'd like to ask the PSC and others to vote on adopting RFC 61: > > Support for measured geometries. +1 -Jukka Rahkonen- ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev