Re: [gdal-dev] RFC 61: Call for vote on adoption

[Ari Jolma]

I've fixed this so that currently all tests pass (at least locally).

Now Shapefiles, which are created XY come back as XY, XYM as XYM, XYZ as 
XYZ, and XYZM as XYZM. Also one should be able to set the required 
coordinate combination using parameters - that's not comprehensively 
tested however.


There are still a couple of fixme's in the code (shape2ogr.cpp), writing 
a shape should take into account the requested coordinate combination 
and whether to skip Ms or write them as "no data" (value < 1038 
according to the doc).


I changed the code for OGRFeature::SetGeomFieldDirectly to set or unset 
the geometry M or Z coordinates depending on the geometry field type of 
the feature. That's somewhat bold but the way I did it at least did not 
break anything in the tests.


Ari

10.02.2016, 13:26, Ari Jolma kirjoitti:

One more issue came up.

Shapefiles, which have Z will also have M, at least a placeholder for it.

Having shapefiles as XYZM always in that case may be problematic.

It is possible to use the open options SHPT=type to override the 
default. We could have Z => XYZ as default for backwards compatibility 
and require open option for Z  => XYZM.


Ari


09.02.2016, 17:07, Ari Jolma kirjoitti:
I declare this motion passed with +1 from Even, Tamas, Jukka and 
Daniel and no -1.


I'll commit soon the changes to the core and then later the drivers: 
memory, shape and pg. The goal is to keep the travis test build passing.


Ari

05.02.2016, 10:04, Ari Jolma kirjoitti:
I'd like to ask the PSC and others to vote on adopting RFC 61: 
Support for measured geometries.


The draft RFC is at

https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries

and a draft implementation is at

https://github.com/ajolma/GDAL-XYZM

which is tested at

https://travis-ci.org/ajolma/GDAL-XYZM

(the test #34, which passed, was the so far last one after core 
changes with unchanged autotests and a rather comprehensive set of 
WKT tests in the Perl tests directory)


This is seemingly a small change but it is at the heart of OGR so it 
has some important implications. The only backwards 
incompatibilities that have appeared so far are with some drivers, 
for example shapefile, which can be lessened with, e.g., open options.


Best,

Ari







___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Even Rouault]

> ok, I should perhaps set up a better local test environment.
> 
> I now limit the dimension harmonization to the shape driver @33440.

Travis seems to be stalled right now, but locally I get a crash with latest 
rev :

  TEST: ogr_shape_15 ... ==4768== Invalid read of size 8
==4768==at 0x8928771: SHPReadOGRFeature(SHPInfo*, DBFInfo*, 
OGRFeatureDefn*, int, tagSHPObject*, char const*) (shape2ogr.cpp:1164)
==4768==by 0x8810A30: OGRShapeLayer::FetchShape(int) 
(ogrshapelayer.cpp:730)
==4768==by 0x8810C14: OGRShapeLayer::GetNextFeature() 
(ogrshapelayer.cpp:794)
==4768==by 0x86B1359: OGR_L_GetNextFeature (ogrlayer.cpp:550)
==4768==by 0x2047E979: _wrap_Layer_GetNextFeature (ogr_wrap.cpp:3845)
==4768==by 0x4A7766: PyEval_EvalFrameEx (in /usr/bin/python2.6)
==4768==by 0x4A93D0: PyEval_EvalCodeEx (in /usr/bin/python2.6)
==4768==by 0x4A7568: PyEval_EvalFrameEx (in /usr/bin/python2.6)
==4768==by 0x4A82AF: PyEval_EvalFrameEx (in /usr/bin/python2.6)
==4768==by 0x4A82AF: PyEval_EvalFrameEx (in /usr/bin/python2.6)
==4768==by 0x4A82AF: PyEval_EvalFrameEx (in /usr/bin/python2.6)
==4768==by 0x4A93D0: PyEval_EvalCodeEx (in /usr/bin/python2.6)
==4768==  Address 0x0 is not stack'd, malloc'd or (recently) free'd


> 
> Ari

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Even Rouault]

Le vendredi 12 février 2016 11:26:46, Ari Jolma a écrit :
> I've fixed this so that currently all tests pass (at least locally).
> 
> Now Shapefiles, which are created XY come back as XY, XYM as XYM, XYZ as
> XYZ, and XYZM as XYZM. Also one should be able to set the required
> coordinate combination using parameters - that's not comprehensively
> tested however.
> 

Great

> There are still a couple of fixme's in the code (shape2ogr.cpp), writing
> a shape should take into account the requested coordinate combination
> and whether to skip Ms or write them as "no data" (value < 1038
> according to the doc).

-10^38

> 
> I changed the code for OGRFeature::SetGeomFieldDirectly to set or unset
> the geometry M or Z coordinates depending on the geometry field type of
> the feature. That's somewhat bold but the way I did it at least did not
> break anything in the tests.

Agreed it is a non trivial change. But it does break things. Crashes in the 
CSW driver for example :
https://travis-ci.org/rouault/gdal_coverage/builds/108752731 .
And other non crashing failures.
Perhaps do this dimension harmonization not in SetGeomFieldDirectly() but in a 
helper function and call it in the shapefile driver if you need it ?
We could also try to fix drivers/tests, but there may be cases where drivers 
cannot declare a consistant coordinate dimensionnality because they have a mix 
of {Geom} and {Geom}Z features

> 
> Ari
> 
> 10.02.2016, 13:26, Ari Jolma kirjoitti:
> > One more issue came up.
> > 
> > Shapefiles, which have Z will also have M, at least a placeholder for it.
> > 
> > Having shapefiles as XYZM always in that case may be problematic.
> > 
> > It is possible to use the open options SHPT=type to override the
> > default. We could have Z => XYZ as default for backwards compatibility
> > and require open option for Z  => XYZM.
> > 
> > Ari
> > 
> > 09.02.2016, 17:07, Ari Jolma kirjoitti:
> >> I declare this motion passed with +1 from Even, Tamas, Jukka and
> >> Daniel and no -1.
> >> 
> >> I'll commit soon the changes to the core and then later the drivers:
> >> memory, shape and pg. The goal is to keep the travis test build passing.
> >> 
> >> Ari
> >> 
> >> 05.02.2016, 10:04, Ari Jolma kirjoitti:
> >>> I'd like to ask the PSC and others to vote on adopting RFC 61:
> >>> Support for measured geometries.
> >>> 
> >>> The draft RFC is at
> >>> 
> >>> https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries
> >>> 
> >>> and a draft implementation is at
> >>> 
> >>> https://github.com/ajolma/GDAL-XYZM
> >>> 
> >>> which is tested at
> >>> 
> >>> https://travis-ci.org/ajolma/GDAL-XYZM
> >>> 
> >>> (the test #34, which passed, was the so far last one after core
> >>> changes with unchanged autotests and a rather comprehensive set of
> >>> WKT tests in the Perl tests directory)
> >>> 
> >>> This is seemingly a small change but it is at the heart of OGR so it
> >>> has some important implications. The only backwards
> >>> incompatibilities that have appeared so far are with some drivers,
> >>> for example shapefile, which can be lessened with, e.g., open options.
> >>> 
> >>> Best,
> >>> 
> >>> Ari

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Ari Jolma]

12.02.2016, 12:43, Even Rouault kirjoitti:

There are still a couple of fixme's in the code (shape2ogr.cpp), writing
a shape should take into account the requested coordinate combination
and whether to skip Ms or write them as "no data" (value < 1038
according to the doc).

-10^38


copy paste error from pdf :)




I changed the code for OGRFeature::SetGeomFieldDirectly to set or unset
the geometry M or Z coordinates depending on the geometry field type of
the feature. That's somewhat bold but the way I did it at least did not
break anything in the tests.

Agreed it is a non trivial change. But it does break things. Crashes in the
CSW driver for example :
https://travis-ci.org/rouault/gdal_coverage/builds/108752731 .
And other non crashing failures.
Perhaps do this dimension harmonization not in SetGeomFieldDirectly() but in a
helper function and call it in the shapefile driver if you need it ?
We could also try to fix drivers/tests, but there may be cases where drivers
cannot declare a consistant coordinate dimensionnality because they have a mix
of {Geom} and {Geom}Z features


ok, I should perhaps set up a better local test environment.

I now limit the dimension harmonization to the shape driver @33440.

Ari

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Ari Jolma]

12.02.2016, 14:54, Even Rouault kirjoitti:

ok, I should perhaps set up a better local test environment.

I now limit the dimension harmonization to the shape driver @33440.

Travis seems to be stalled right now, but locally I get a crash with latest
rev :



Thanks. Fixed. It is sometime difficult to spot errors in the log output 
of the autotests.


Ari

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Ari Jolma]

10.02.2016, 15:21, Kurt Schwehr kirjoitti:


One request:

#define OGR_G_NOT_EMPTY_POINT 0x1
#define OGR_G_3D 0x2
#define OGR_G_MEASURED 0x4
#define OGR_G_IGNORE_MEASURED 0x8
Can you make these a static const int or static const unsigned int 
member of the OGRGeometry class?  #defines make life more difficult.


These constants are used also outside the classes. How does one access 
those for example in OGRWktReadPointsM, which receives the flags as a 
parameter?


Ari

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Ari Jolma]

I've started a wiki page for the status of the drivers regarding 
measured geometries:


https://trac.osgeo.org/gdal/wiki/MeasuredGeometriesInDrivers

Ari

10.02.2016, 15:05, Even Rouault kirjoitti:

Le mercredi 10 février 2016 13:53:28, Peter Halls a écrit :

Types on page 2).
PointZ
A PointZ consists of a triplet of double-precision coordinates in the

order


X, Y, Z plus a measure"

I had misinterpreted their meaning of "optional" here and submitted a
documentation query to ESRI on being told I was wrong.  The response was
that M being optional means that it can be valueless, but is always
present.

Woo, really  That's a genuine scoop.

I reported it to  Frank at the time ... but several email systems later, I
no longer have my copies of the correspondence with ESRI nor the OGR bug
report, in which these were copied.  Somewhere I may still have a backup of
my work area, to see what I did in my code.  However, as you say, this is
20+ years ago   It was something which, at the time, was causing us
some problems 

I think you to be correct in your Shapelib summary below:

Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles
without M values for more than 20 years. I'm surprised we wouldn't have
heard about that if such shapefiles couldn't be read. Or perhaps ESRI
software is robust to missing M too

Would someone looking at this discussion and having access to ESRI software be
willing to generate tiny (meaning just one single shape, with the smallest
number of vertices) shapefiles of type PointZ, ArcZ, PolygonZ and MultiPointZ,
and *without* M values ? So as to enhance the test suite and see in particular
which value is set exactly as the nodata value in the M component

I was wondering if we should change the behaviour of the shape driver to add
that nodata M component for XYZ shapes. This wouldn't cause compat problems
with previous versions of GDAL, but it would cause .shp sizes to grow by ~
1/3. So perhaps go on with current behaviour of stripping M component if most
implementations seem to be OK with that ?


Best wishes,

Peter




On 10 February 2016 at 12:31, Even Rouault 

wrote:

Le mercredi 10 février 2016 13:05:20, Peter Halls a écrit :

Ari, et al,

   ESRI handle this in a non-intuitive way:  XYM is supported, but Z

always has a Measure, so is XYZM!   The formal definition is here:
https://www.*esri*.com/library/whitepapers/pdfs/*shapefile*.pdf
(1998)

   Shape Types having Z are defined on pp19ff where it states:
"Shape Types inX,Y,Z Space
Shapes of this type have an optional coordinate
M. Note that "no data" value can be specified as a value for M (see

Numeric


Types on page 2).
PointZ
A PointZ consists of a triplet of double-precision coordinates in the

order


X, Y, Z plus a measure"

I had misinterpreted their meaning of "optional" here and submitted a
documentation query to ESRI on being told I was wrong.  The response
was that M being optional means that it can be valueless, but is
always present.

Woo, really  That's a genuine scoop.

Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles
without M values for more than 20 years. I'm surprised we wouldn't have
heard
about that if such shapefiles couldn't be read. Or perhaps ESRI software
is robust to missing M too


Best wishes,

Peter

--
Spatialys - Geospatial professional services
http://www.spatialys.com


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Kurt Schwehr]

OGRGeometry::OGR_G_NOT_EMPTY_POINT

On Wed, Feb 10, 2016 at 7:34 AM, Ari Jolma  wrote:

>
> 10.02.2016, 15:21, Kurt Schwehr kirjoitti:
>
>>
>> One request:
>>
>> #define OGR_G_NOT_EMPTY_POINT 0x1
>> #define OGR_G_3D 0x2
>> #define OGR_G_MEASURED 0x4
>> #define OGR_G_IGNORE_MEASURED 0x8
>> Can you make these a static const int or static const unsigned int member
>> of the OGRGeometry class?  #defines make life more difficult.
>>
>
> These constants are used also outside the classes. How does one access
> those for example in OGRWktReadPointsM, which receives the flags as a
> parameter?
>
> Ari
>
>


-- 
--
http://schwehr.org
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Mike Toews]

On 11 February 2016 at 01:31, Even Rouault  wrote:
> Le mercredi 10 février 2016 13:05:20, Peter Halls a écrit :
>>   ESRI handle this in a non-intuitive way:  XYM is supported, but Z
>> always has a Measure, so is XYZM!   The formal definition is here:
>> https://www.*esri*.com/library/whitepapers/pdfs/*shapefile*.pdf   (1998)
>>
>>   Shape Types having Z are defined on pp19ff where it states:
>>
>> "Shape Types inX,Y,Z Space
>> Shapes of this type have an optional coordinate
>
> Woo, really  That's a genuine scoop.
>
> Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles
> without M values for more than 20 years. I'm surprised we wouldn't have heard
> about that if such shapefiles couldn't be read. Or perhaps ESRI software is
> robust to missing M too

I've looked into this a few years ago, because I found differences in
XYZ files written by GDAL and Esri software. There are two styles of
(e.g.) PointZ files, where the coordinates are written with either
three or four doubles (the last with "no data" if M is not used). The
style of PointZ is determined by the content length provided in the
record header as 16-bit words (either 14 or 18). Therefore, the specs
are correct when they describe M as an "optional coordinate" (p.15).

For XYZ shapefiles, GDAL always write three doubles, whereas Esri
write four doubles with "no data" as the last coordinate. As a result,
Esri's XYZ shp files are slightly larger than GDAL's.

Occasionally I use software that requires PointZ shapefiles to have
four doubles, so my existing workaround is to use Esri's
CopyFeatures_management tool to rewrite the shapefile.

Could "ogr2ogr -dim 4" be added to this RFC to write XYZ[M] Shapefiles
that always use four floats for coordinates, as Esri software does?


On 11 February 2016 at 02:05, Even Rouault  wrote:
> Would someone looking at this discussion and having access to ESRI software be
> willing to generate tiny (meaning just one single shape, with the smallest
> number of vertices) shapefiles of type PointZ, ArcZ, PolygonZ and MultiPointZ,
> and *without* M values ? So as to enhance the test suite and see in particular
> which value is set exactly as the nodata value in the M component

I can generate these today, and will attach a zip file to
https://trac.osgeo.org/gdal/wiki/MeasuredGeometriesInDrivers
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Ari Jolma]

One more issue came up.

Shapefiles, which have Z will also have M, at least a placeholder for it.

Having shapefiles as XYZM always in that case may be problematic.

It is possible to use the open options SHPT=type to override the 
default. We could have Z => XYZ as default for backwards compatibility 
and require open option for Z  => XYZM.


Ari


09.02.2016, 17:07, Ari Jolma kirjoitti:
I declare this motion passed with +1 from Even, Tamas, Jukka and 
Daniel and no -1.


I'll commit soon the changes to the core and then later the drivers: 
memory, shape and pg. The goal is to keep the travis test build passing.


Ari

05.02.2016, 10:04, Ari Jolma kirjoitti:
I'd like to ask the PSC and others to vote on adopting RFC 61: 
Support for measured geometries.


The draft RFC is at

https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries

and a draft implementation is at

https://github.com/ajolma/GDAL-XYZM

which is tested at

https://travis-ci.org/ajolma/GDAL-XYZM

(the test #34, which passed, was the so far last one after core 
changes with unchanged autotests and a rather comprehensive set of 
WKT tests in the Perl tests directory)


This is seemingly a small change but it is at the heart of OGR so it 
has some important implications. The only backwards incompatibilities 
that have appeared so far are with some drivers, for example 
shapefile, which can be lessened with, e.g., open options.


Best,

Ari





___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Even Rouault]

Le mercredi 10 février 2016 12:26:59, Ari Jolma a écrit :
> One more issue came up.
> 
> Shapefiles, which have Z will also have M, at least a placeholder for it.
> 
> Having shapefiles as XYZM always in that case may be problematic.

Small precision for those not necessarily familiar with the internals of the 
shapefile format. The .shp and .shx header have a declared shape type (used to 
set the OGR layer geometry type), that can be (restricting to the case of 
polygon for the sake of clarity, but true for other geometry types) :
SHPT_POLYGON : the shapefile contains only Polygon 2D shapes
SHPT_POLYGONZ:  the shapefile may contain a mix of PolygonZ or PolygonZM shapes
SHPT_POLYGONM: the shapefile may contain a mix of Polygon or PolygonM shapes

Declaring a SHPT_POLYGONM as a PolygonM layer is OK (if the shapefile doesn't 
contain PolygonM shapes, it should declare only SHPT_POLYGON)

But declaring SHPT_POLYGONZ as a PolygonZM layer, when there are in fact no 
PolygonZM shapes in it but just PolygonZ, is more problematic for backward 
compatibility issues. Ideally when dealing with shapefiles without M values we 
shouldn't advertize M neither in the layer geometry type, nor in the feature 
geometry.

> 
> It is possible to use the open options SHPT=type to override the
> default. We could have Z => XYZ as default for backwards compatibility
> and require open option for Z  => XYZM.

That's one possibility. As open options should be a last resort, if SHPT isn't 
specified perhaps we could do a probing of the first shape. If it has a M 
component then advertize ZM geometry type, otherwise just xZ. That's 
of course not completely bullet proof as there can be in theory a mix of Z and 
ZM shapes, but my feeling is that this should work in > 90% of the cases.

That's an improvement I could have a look at during the Code Sprint if you 
don't feel confident enough in the shape driver.


~

Somehow related issue I've noticed but that should be more easily fixable. Now 
creating a Polygon25D shapefile creates PolygonZM and not PolygonZ shapes (and 
perhaps true for other shape types).

trunk:
$ ogr2ogr poly25d.shp poly.shp -nlt multipolygon25d -overwrite
$ ll poly25d.*
-rw-r--r-- 1 even even  529 2016-02-10 12:36 poly25d.dbf
-rw-r--r-- 1 even even  557 2016-02-10 12:36 poly25d.prj
-rw-r--r-- 1 even even 8820 2016-02-10 12:36 poly25d.shp
-rw-r--r-- 1 even even  180 2016-02-10 12:36 poly25d.shx

2.0:
$ ogr2ogr poly25d.shp poly.shp -nlt multipolygon25d -overwrite
$ ll poly25d.*
-rw-r--r-- 1 even even  529 2016-02-10 12:36 poly25d.dbf
-rw-r--r-- 1 even even  557 2016-02-10 12:36 poly25d.prj
-rw-r--r-- 1 even even 6700 2016-02-10 12:36 poly25d.shp
-rw-r--r-- 1 even even  180 2016-02-10 12:36 poly25d.shx

On the read size, I've commited https://trac.osgeo.org/gdal/changeset/33408
"Shape: fix crash when reading a MultiPointZ without M values, and when reading 
a PointZ without M do not create a PointZM geometry (trunk only)"

Quickly checking at the code, I think other geometry types are OK, but perhaps 
we should have more extensive cases :
- adding Z shapefiles of other geometry types generated with GDAL 2.0 and 
checking that the ExportToIsoWKT() read with trunk is the one expected
- and/or generating Z shapes with trunk and checking they are read as Z only 
with ExportToIsoWKT()

> 
> Ari
> 
> 09.02.2016, 17:07, Ari Jolma kirjoitti:
> > I declare this motion passed with +1 from Even, Tamas, Jukka and
> > Daniel and no -1.
> > 
> > I'll commit soon the changes to the core and then later the drivers:
> > memory, shape and pg. The goal is to keep the travis test build passing.
> > 
> > Ari
> > 
> > 05.02.2016, 10:04, Ari Jolma kirjoitti:
> >> I'd like to ask the PSC and others to vote on adopting RFC 61:
> >> Support for measured geometries.
> >> 
> >> The draft RFC is at
> >> 
> >> https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries
> >> 
> >> and a draft implementation is at
> >> 
> >> https://github.com/ajolma/GDAL-XYZM
> >> 
> >> which is tested at
> >> 
> >> https://travis-ci.org/ajolma/GDAL-XYZM
> >> 
> >> (the test #34, which passed, was the so far last one after core
> >> changes with unchanged autotests and a rather comprehensive set of
> >> WKT tests in the Perl tests directory)
> >> 
> >> This is seemingly a small change but it is at the heart of OGR so it
> >> has some important implications. The only backwards incompatibilities
> >> that have appeared so far are with some drivers, for example
> >> shapefile, which can be lessened with, e.g., open options.
> >> 
> >> Best,
> >> 
> >> Ari

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Even Rouault]

Le mercredi 10 février 2016 13:05:20, Peter Halls a écrit :
> Ari, et al,
> 
>   ESRI handle this in a non-intuitive way:  XYM is supported, but Z
> always has a Measure, so is XYZM!   The formal definition is here:
> https://www.*esri*.com/library/whitepapers/pdfs/*shapefile*.pdf   (1998)
> 
>   Shape Types having Z are defined on pp19ff where it states:
> 
> "Shape Types inX,Y,Z Space
> Shapes of this type have an optional coordinate
> M. Note that "no data" value can be specified as a value for M (see Numeric
> Types on page 2).
> PointZ
> A PointZ consists of a triplet of double-precision coordinates in the order
> X, Y, Z plus a measure"
> 
> I had misinterpreted their meaning of "optional" here and submitted a
> documentation query to ESRI on being told I was wrong.  The response was
> that M being optional means that it can be valueless, but is always
> present.

Woo, really  That's a genuine scoop.

Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles 
without M values for more than 20 years. I'm surprised we wouldn't have heard 
about that if such shapefiles couldn't be read. Or perhaps ESRI software is 
robust to missing M too

> 
> Best wishes,
> 
> Peter

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Even Rouault]

Le mercredi 10 février 2016 13:22:49, Ari Jolma a écrit :
> 10.02.2016, 13:55, Even Rouault kirjoitti:
> > Le mercredi 10 février 2016 12:26:59, Ari Jolma a écrit :
> >> One more issue came up.
> >> 
> >> Shapefiles, which have Z will also have M, at least a placeholder for
> >> it.
> >> 
> >> Having shapefiles as XYZM always in that case may be problematic.
> > 
> > Small precision for those not necessarily familiar with the internals of
> > the shapefile format. The .shp and .shx header have a declared shape
> > type (used to set the OGR layer geometry type), that can be (restricting
> > to the case of polygon for the sake of clarity, but true for other
> > geometry types) : SHPT_POLYGON : the shapefile contains only Polygon 2D
> > shapes
> > SHPT_POLYGONZ:  the shapefile may contain a mix of PolygonZ or PolygonZM
> > shapes
> 
> Would you differentiate between *Z or *ZM and * and *M shapes using the
> ESRI "no data":
> 
> "Nevertheless, shapefiles support the concept of "no data" values, but
> they are currently used only for measures. Any floating point number
> smaller than –1038 is considered by a shapefile reader to
> represent a "no data" value."

Given latest Peter's message, yes likely. And also completely missing M as 
produced by shapelib up to now.

> 
> > SHPT_POLYGONM: the shapefile may contain a mix of Polygon or PolygonM
> > shapes
> > 
> > Declaring a SHPT_POLYGONM as a PolygonM layer is OK (if the shapefile
> > doesn't contain PolygonM shapes, it should declare only SHPT_POLYGON)
> > 
> > But declaring SHPT_POLYGONZ as a PolygonZM layer, when there are in fact
> > no PolygonZM shapes in it but just PolygonZ, is more problematic for
> > backward compatibility issues. Ideally when dealing with shapefiles
> > without M values we shouldn't advertize M neither in the layer geometry
> > type, nor in the feature geometry.
> > 
> >> It is possible to use the open options SHPT=type to override the
> >> default. We could have Z => XYZ as default for backwards compatibility
> >> and require open option for Z  => XYZM.
> > 
> > That's one possibility. As open options should be a last resort, if SHPT
> > isn't specified perhaps we could do a probing of the first shape. If it
> > has a M component then advertize ZM geometry type, otherwise just
> > xZ. That's of course not completely bullet proof as there can be in
> > theory a mix of Z and ZM shapes, but my feeling is that this should work
> > in > 90% of the cases.
> > 
> > That's an improvement I could have a look at during the Code Sprint if
> > you don't feel confident enough in the shape driver.
> 
> Feel free to have a look. I'm now stuck with the pg driver :)

Hopefully it is less tricky than shapefiles. I see there are other test 
failures related to shapefiles as well

> 
> Maybe, if what I wrote above is correct, read the whole shapefile in and
> then, if all M values are "no data" downgrade all geometries to XY or XYZ?

For performance reason, I'd perhaps not read all M values of all shapes, but 
only the ones of the first shape.

An extra enhancement would be to have an option to do a full scan of the 
shapefile. We could also use that to avoid advertizing Polygon layer geometry 
type when there's a mix of Polygon/MultiPolygon. 

> 
> Ari
> 
> > ~
> > 
> > Somehow related issue I've noticed but that should be more easily
> > fixable. Now creating a Polygon25D shapefile creates PolygonZM and not
> > PolygonZ shapes (and perhaps true for other shape types).
> > 
> > trunk:
> > $ ogr2ogr poly25d.shp poly.shp -nlt multipolygon25d -overwrite
> > $ ll poly25d.*
> > -rw-r--r-- 1 even even  529 2016-02-10 12:36 poly25d.dbf
> > -rw-r--r-- 1 even even  557 2016-02-10 12:36 poly25d.prj
> > -rw-r--r-- 1 even even 8820 2016-02-10 12:36 poly25d.shp
> > -rw-r--r-- 1 even even  180 2016-02-10 12:36 poly25d.shx
> > 
> > 2.0:
> > $ ogr2ogr poly25d.shp poly.shp -nlt multipolygon25d -overwrite
> > $ ll poly25d.*
> > -rw-r--r-- 1 even even  529 2016-02-10 12:36 poly25d.dbf
> > -rw-r--r-- 1 even even  557 2016-02-10 12:36 poly25d.prj
> > -rw-r--r-- 1 even even 6700 2016-02-10 12:36 poly25d.shp
> > -rw-r--r-- 1 even even  180 2016-02-10 12:36 poly25d.shx
> > 
> > On the read size, I've commited
> > https://trac.osgeo.org/gdal/changeset/33408 "Shape: fix crash when
> > reading a MultiPointZ without M values, and when reading a PointZ
> > without M do not create a PointZM geometry (trunk only)"
> > 
> > Quickly checking at the code, I think other geometry types are OK, but
> > perhaps we should have more extensive cases :
> > - adding Z shapefiles of other geometry types generated with GDAL 2.0 and
> > checking that the ExportToIsoWKT() read with trunk is the one expected
> > - and/or generating Z shapes with trunk and checking they are read as Z
> > only with ExportToIsoWKT()
> > 
> >> Ari
> >> 
> >> 09.02.2016, 17:07, Ari Jolma kirjoitti:
> >>> I declare this motion passed with +1 from Even, Tamas, Jukka and
> >>> Daniel and no -1.
> >>> 
> >>> 

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Jukka Rahkonen]

Ari Jolma  gmail.com> writes:

> 
> 
> One more issue came up.
> Shapefiles, which have Z will also have M, at least a placeholder
> for it.
> Having shapefiles as XYZM always in that case may be problematic.
> It is possible to use the open options SHPT=type to override the
> default. We could have Z => XYZ as default for backwards
> compatibility and require open option for Z  => XYZM.
> Ari09.02.2016, 17:07, Ari Jolma kirjoitti:

I noticed your changeset and I was just reading the shapefile specification.
I never knew that there are no separate XYZ and XYZM shapetypes. I also
believe I have never met a shapefile that really contains XYZM data.

I agree that the default should be to convert such shapefiles to XYZ instead
of XYZM. That would affect shapetype values

11 PointZ
13 PolyLineZ
15 PolygonZ
18 MultiPointZ

-Jukka Rahkonen-


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Peter Halls]

Ari, et al,

  ESRI handle this in a non-intuitive way:  XYM is supported, but Z
always has a Measure, so is XYZM!   The formal definition is here:
https://www.*esri*.com/library/whitepapers/pdfs/*shapefile*.pdf   (1998)

  Shape Types having Z are defined on pp19ff where it states:

"Shape Types inX,Y,Z Space
Shapes of this type have an optional coordinate
M. Note that "no data" value can be specified as a value for M (see Numeric
Types on page 2).
PointZ
A PointZ consists of a triplet of double-precision coordinates in the order
X, Y, Z plus a measure"

I had misinterpreted their meaning of "optional" here and submitted a
documentation query to ESRI on being told I was wrong.  The response was
that M being optional means that it can be valueless, but is always present.

Best wishes,

Peter
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Ari Jolma]

10.02.2016, 13:55, Even Rouault kirjoitti:

Le mercredi 10 février 2016 12:26:59, Ari Jolma a écrit :

One more issue came up.

Shapefiles, which have Z will also have M, at least a placeholder for it.

Having shapefiles as XYZM always in that case may be problematic.

Small precision for those not necessarily familiar with the internals of the
shapefile format. The .shp and .shx header have a declared shape type (used to
set the OGR layer geometry type), that can be (restricting to the case of
polygon for the sake of clarity, but true for other geometry types) :
SHPT_POLYGON : the shapefile contains only Polygon 2D shapes
SHPT_POLYGONZ:  the shapefile may contain a mix of PolygonZ or PolygonZM shapes


Would you differentiate between *Z or *ZM and * and *M shapes using the 
ESRI "no data":


"Nevertheless, shapefiles support the concept of "no data" values, but 
they are currently used only for measures. Any floating point number 
smaller than –1038 is considered by a shapefile reader to

represent a "no data" value."




SHPT_POLYGONM: the shapefile may contain a mix of Polygon or PolygonM shapes

Declaring a SHPT_POLYGONM as a PolygonM layer is OK (if the shapefile doesn't
contain PolygonM shapes, it should declare only SHPT_POLYGON)

But declaring SHPT_POLYGONZ as a PolygonZM layer, when there are in fact no
PolygonZM shapes in it but just PolygonZ, is more problematic for backward
compatibility issues. Ideally when dealing with shapefiles without M values we
shouldn't advertize M neither in the layer geometry type, nor in the feature
geometry.


It is possible to use the open options SHPT=type to override the
default. We could have Z => XYZ as default for backwards compatibility
and require open option for Z  => XYZM.

That's one possibility. As open options should be a last resort, if SHPT isn't
specified perhaps we could do a probing of the first shape. If it has a M
component then advertize ZM geometry type, otherwise just xZ. That's
of course not completely bullet proof as there can be in theory a mix of Z and
ZM shapes, but my feeling is that this should work in > 90% of the cases.

That's an improvement I could have a look at during the Code Sprint if you
don't feel confident enough in the shape driver.


Feel free to have a look. I'm now stuck with the pg driver :)

Maybe, if what I wrote above is correct, read the whole shapefile in and 
then, if all M values are "no data" downgrade all geometries to XY or XYZ?


Ari





~

Somehow related issue I've noticed but that should be more easily fixable. Now
creating a Polygon25D shapefile creates PolygonZM and not PolygonZ shapes (and
perhaps true for other shape types).

trunk:
$ ogr2ogr poly25d.shp poly.shp -nlt multipolygon25d -overwrite
$ ll poly25d.*
-rw-r--r-- 1 even even  529 2016-02-10 12:36 poly25d.dbf
-rw-r--r-- 1 even even  557 2016-02-10 12:36 poly25d.prj
-rw-r--r-- 1 even even 8820 2016-02-10 12:36 poly25d.shp
-rw-r--r-- 1 even even  180 2016-02-10 12:36 poly25d.shx

2.0:
$ ogr2ogr poly25d.shp poly.shp -nlt multipolygon25d -overwrite
$ ll poly25d.*
-rw-r--r-- 1 even even  529 2016-02-10 12:36 poly25d.dbf
-rw-r--r-- 1 even even  557 2016-02-10 12:36 poly25d.prj
-rw-r--r-- 1 even even 6700 2016-02-10 12:36 poly25d.shp
-rw-r--r-- 1 even even  180 2016-02-10 12:36 poly25d.shx

On the read size, I've commited https://trac.osgeo.org/gdal/changeset/33408
"Shape: fix crash when reading a MultiPointZ without M values, and when reading
a PointZ without M do not create a PointZM geometry (trunk only)"

Quickly checking at the code, I think other geometry types are OK, but perhaps
we should have more extensive cases :
- adding Z shapefiles of other geometry types generated with GDAL 2.0 and
checking that the ExportToIsoWKT() read with trunk is the one expected
- and/or generating Z shapes with trunk and checking they are read as Z only
with ExportToIsoWKT()


Ari

09.02.2016, 17:07, Ari Jolma kirjoitti:

I declare this motion passed with +1 from Even, Tamas, Jukka and
Daniel and no -1.

I'll commit soon the changes to the core and then later the drivers:
memory, shape and pg. The goal is to keep the travis test build passing.

Ari

05.02.2016, 10:04, Ari Jolma kirjoitti:

I'd like to ask the PSC and others to vote on adopting RFC 61:
Support for measured geometries.

The draft RFC is at

https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries

and a draft implementation is at

https://github.com/ajolma/GDAL-XYZM

which is tested at

https://travis-ci.org/ajolma/GDAL-XYZM

(the test #34, which passed, was the so far last one after core
changes with unchanged autotests and a rather comprehensive set of
WKT tests in the Perl tests directory)

This is seemingly a small change but it is at the heart of OGR so it
has some important implications. The only backwards incompatibilities
that have appeared so far are with some drivers, for example
shapefile, which can be lessened with, e.g., open options.

Best,

Ari

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Peter Halls]

> Types on page 2).
> PointZ
> A PointZ consists of a triplet of double-precision coordinates in the
order
> X, Y, Z plus a measure"
>
> I had misinterpreted their meaning of "optional" here and submitted a
> documentation query to ESRI on being told I was wrong.  The response was
> that M being optional means that it can be valueless, but is always
> present.

Woo, really  That's a genuine scoop.

I reported it to  Frank at the time ... but several email systems later, I
no longer have my copies of the correspondence with ESRI nor the OGR bug
report, in which these were copied.  Somewhere I may still have a backup of
my work area, to see what I did in my code.  However, as you say, this is
20+ years ago   It was something which, at the time, was causing us
some problems 

I think you to be correct in your Shapelib summary below:

Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles
without M values for more than 20 years. I'm surprised we wouldn't have
heard about that if such shapefiles couldn't be read. Or perhaps ESRI
software is robust to missing M too

Best wishes,

Peter




On 10 February 2016 at 12:31, Even Rouault 
wrote:

> Le mercredi 10 février 2016 13:05:20, Peter Halls a écrit :
> > Ari, et al,
> >
> >   ESRI handle this in a non-intuitive way:  XYM is supported, but Z
> > always has a Measure, so is XYZM!   The formal definition is here:
> > https://www.*esri*.com/library/whitepapers/pdfs/*shapefile*.pdf   (1998)
> >
> >   Shape Types having Z are defined on pp19ff where it states:
> >
> > "Shape Types inX,Y,Z Space
> > Shapes of this type have an optional coordinate
> > M. Note that "no data" value can be specified as a value for M (see
> Numeric
> > Types on page 2).
> > PointZ
> > A PointZ consists of a triplet of double-precision coordinates in the
> order
> > X, Y, Z plus a measure"
> >
> > I had misinterpreted their meaning of "optional" here and submitted a
> > documentation query to ESRI on being told I was wrong.  The response was
> > that M being optional means that it can be valueless, but is always
> > present.
>
> Woo, really  That's a genuine scoop.
>
> Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles
> without M values for more than 20 years. I'm surprised we wouldn't have
> heard
> about that if such shapefiles couldn't be read. Or perhaps ESRI software is
> robust to missing M too
>
> >
> > Best wishes,
> >
> > Peter
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>



-- 

Peter J Halls, PhD Student, Post-war Reconstruction and Development Unit
(PRDU),
  University of York

Snail mail: PRDU, Derwent College, University of York,
Heslington, York YO10 5DD
This message has the status of a private and personal communication

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Even Rouault]

Le mercredi 10 février 2016 13:53:28, Peter Halls a écrit :
> > Types on page 2).
> > PointZ
> > A PointZ consists of a triplet of double-precision coordinates in the
> 
> order
> 
> > X, Y, Z plus a measure"
> > 
> > I had misinterpreted their meaning of "optional" here and submitted a
> > documentation query to ESRI on being told I was wrong.  The response was
> > that M being optional means that it can be valueless, but is always
> > present.
> 
> Woo, really  That's a genuine scoop.
> 
> I reported it to  Frank at the time ... but several email systems later, I
> no longer have my copies of the correspondence with ESRI nor the OGR bug
> report, in which these were copied.  Somewhere I may still have a backup of
> my work area, to see what I did in my code.  However, as you say, this is
> 20+ years ago   It was something which, at the time, was causing us
> some problems 
> 
> I think you to be correct in your Shapelib summary below:
> 
> Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles
> without M values for more than 20 years. I'm surprised we wouldn't have
> heard about that if such shapefiles couldn't be read. Or perhaps ESRI
> software is robust to missing M too

Would someone looking at this discussion and having access to ESRI software be 
willing to generate tiny (meaning just one single shape, with the smallest 
number of vertices) shapefiles of type PointZ, ArcZ, PolygonZ and MultiPointZ, 
and *without* M values ? So as to enhance the test suite and see in particular 
which value is set exactly as the nodata value in the M component

I was wondering if we should change the behaviour of the shape driver to add 
that nodata M component for XYZ shapes. This wouldn't cause compat problems 
with previous versions of GDAL, but it would cause .shp sizes to grow by ~ 
1/3. So perhaps go on with current behaviour of stripping M component if most 
implementations seem to be OK with that ?

> 
> Best wishes,
> 
> Peter
> 
> 
> 
> 
> On 10 February 2016 at 12:31, Even Rouault 
> 
> wrote:
> > Le mercredi 10 février 2016 13:05:20, Peter Halls a écrit :
> > > Ari, et al,
> > > 
> > >   ESRI handle this in a non-intuitive way:  XYM is supported, but Z
> > > 
> > > always has a Measure, so is XYZM!   The formal definition is here:
> > > https://www.*esri*.com/library/whitepapers/pdfs/*shapefile*.pdf  
> > > (1998)
> > > 
> > >   Shape Types having Z are defined on pp19ff where it states:
> > > "Shape Types inX,Y,Z Space
> > > Shapes of this type have an optional coordinate
> > > M. Note that "no data" value can be specified as a value for M (see
> > 
> > Numeric
> > 
> > > Types on page 2).
> > > PointZ
> > > A PointZ consists of a triplet of double-precision coordinates in the
> > 
> > order
> > 
> > > X, Y, Z plus a measure"
> > > 
> > > I had misinterpreted their meaning of "optional" here and submitted a
> > > documentation query to ESRI on being told I was wrong.  The response
> > > was that M being optional means that it can be valueless, but is
> > > always present.
> > 
> > Woo, really  That's a genuine scoop.
> > 
> > Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles
> > without M values for more than 20 years. I'm surprised we wouldn't have
> > heard
> > about that if such shapefiles couldn't be read. Or perhaps ESRI software
> > is robust to missing M too
> > 
> > > Best wishes,
> > > 
> > > Peter
> > 
> > --
> > Spatialys - Geospatial professional services
> > http://www.spatialys.com

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Kurt Schwehr]

I'm +0.  Neither for nor against.

One request:

#define OGR_G_NOT_EMPTY_POINT 0x1
#define OGR_G_3D 0x2
#define OGR_G_MEASURED 0x4
#define OGR_G_IGNORE_MEASURED 0x8

Can you make these a static const int or static const unsigned int member
of the OGRGeometry class?  #defines make life more difficult.

find . -name \*.h -o -name \*.cpp | xargs grep '#define' | egrep '[0-9"]' |
egrep -v '[(]|\[' | wc -l
4517

A random example of doing this:

https://trac.osgeo.org/gdal/browser/trunk/gdal/alg/gdal_simplesurf.h?rev=33127#L93

// Descriptor length
static const int DESC_SIZE = 64;

-kurt


On Wed, Feb 10, 2016 at 4:53 AM, Peter Halls  wrote:

> > Types on page 2).
> > PointZ
> > A PointZ consists of a triplet of double-precision coordinates in the
> order
> > X, Y, Z plus a measure"
> >
> > I had misinterpreted their meaning of "optional" here and submitted a
> > documentation query to ESRI on being told I was wrong.  The response was
> > that M being optional means that it can be valueless, but is always
> > present.
>
> Woo, really  That's a genuine scoop.
>
> I reported it to  Frank at the time ... but several email systems later, I
> no longer have my copies of the correspondence with ESRI nor the OGR bug
> report, in which these were copied.  Somewhere I may still have a backup of
> my work area, to see what I did in my code.  However, as you say, this is
> 20+ years ago   It was something which, at the time, was causing us
> some problems 
>
> I think you to be correct in your Shapelib summary below:
>
> Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles
> without M values for more than 20 years. I'm surprised we wouldn't have
> heard about that if such shapefiles couldn't be read. Or perhaps ESRI
> software is robust to missing M too
>
> Best wishes,
>
> Peter
>
>
>
>
> On 10 February 2016 at 12:31, Even Rouault 
> wrote:
>
>> Le mercredi 10 février 2016 13:05:20, Peter Halls a écrit :
>> > Ari, et al,
>> >
>> >   ESRI handle this in a non-intuitive way:  XYM is supported, but Z
>> > always has a Measure, so is XYZM!   The formal definition is here:
>> > https://www.*esri*.com/library/whitepapers/pdfs/*shapefile*.pdf
>>  (1998)
>> >
>> >   Shape Types having Z are defined on pp19ff where it states:
>> >
>> > "Shape Types inX,Y,Z Space
>> > Shapes of this type have an optional coordinate
>> > M. Note that "no data" value can be specified as a value for M (see
>> Numeric
>> > Types on page 2).
>> > PointZ
>> > A PointZ consists of a triplet of double-precision coordinates in the
>> order
>> > X, Y, Z plus a measure"
>> >
>> > I had misinterpreted their meaning of "optional" here and submitted a
>> > documentation query to ESRI on being told I was wrong.  The response was
>> > that M being optional means that it can be valueless, but is always
>> > present.
>>
>> Woo, really  That's a genuine scoop.
>>
>> Unless I'm seriously mistaken, shapelib / OGR has produced XYZ shapefiles
>> without M values for more than 20 years. I'm surprised we wouldn't have
>> heard
>> about that if such shapefiles couldn't be read. Or perhaps ESRI software
>> is
>> robust to missing M too
>>
>> >
>> > Best wishes,
>> >
>> > Peter
>>
>> --
>> Spatialys - Geospatial professional services
>> http://www.spatialys.com
>>
>
>
>
> --
>
> 
> Peter J Halls, PhD Student, Post-war Reconstruction and Development Unit
> (PRDU),
>   University of York
>
> Snail mail: PRDU, Derwent College, University of York,
> Heslington, York YO10 5DD
> This message has the status of a private and personal communication
>
> 
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>



-- 
--
http://schwehr.org
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Mike Toews]

On 11 February 2016 at 02:05, Even Rouault  wrote:
> Would someone looking at this discussion and having access to ESRI software be
> willing to generate tiny (meaning just one single shape, with the smallest
> number of vertices) shapefiles of type PointZ, ArcZ, PolygonZ and MultiPointZ,
> and *without* M values ? So as to enhance the test suite and see in particular
> which value is set exactly as the nodata value in the M component

Done, see 
https://trac.osgeo.org/gdal/attachment/wiki/MeasuredGeometriesInDrivers/

I created two geometry examples per shapefile, and added an "Expected"
attribute to describe the WKT of the expected geometry. While the M
values were editable (despite not requesting this creation option in
ArcCatalog), I left these as "NaN".
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Ari Jolma]

09.02.2016, 11:17, Even Rouault kirjoitti:

Tests are not independant from each other, so I suspect the failure in
ogr_pg_21 might let a (implicit) transaction non committed. Thus ogr_pg_24
which opens a new connection doesn't see the table created before. Whereas if
you quit, the transaction will be committed.


That's it. I think it is a bug.

Ari

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Ari Jolma]

Thanks,

We now have +1 from Even, Tamas, Jukka and Daniel. Is that formally enough?

I have a small change to the draft. I now think that the coordinate 
flags should be managed/queried solely through Is3D(), IsMeasured() 
(which are standard) and set3D(boolean) and setMeasured(boolean), (which 
are extensions to the SF specification). I.e., the getFlags() and 
setFlags(int) should be removed. The reason is that there is one flag, 
which is used only for points, and thus the API can be easily misused.


Similarly the bit #defines for the coordinate flags sh/could be private. 
In practice that means to move them to ogr_p.h.


Even, there are strange errors in the ogr_pg.py tests.

https://s3.amazonaws.com/archive.travis-ci.org/jobs/107782519/log.txt

Test 24 fails because the layer is NoneType, i.e., it does not open the 
datatypetest table. However, if I run the tests up to 24 but quit before 
it, the table exists and seems to be ok. I had similar problems with the 
shape driver - temporary datasets disappearing midtests. Any idea what's 
the reason?


Ari

08.02.2016, 19:06, Daniel Morissette kirjoitti:

On 2016-02-05 3:04 AM, Ari Jolma wrote:

I'd like to ask the PSC and others to vote on adopting RFC 61: Support
for measured geometries.

The draft RFC is at

https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries




+1

Daniel



___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Ari Jolma]

09.02.2016, 10:18, Ari Jolma kirjoitti:



Even, there are strange errors in the ogr_pg.py tests.

https://s3.amazonaws.com/archive.travis-ci.org/jobs/107782519/log.txt

Test 24 fails because the layer is NoneType, i.e., it does not open 
the datatypetest table. However, if I run the tests up to 24 but quit 
before it, the table exists and seems to be ok. I had similar problems 
with the shape driver - temporary datasets disappearing midtests. Any 
idea what's the reason?


A bit more on this:

If I run the tests until 24, i.e., including it, and then quit. The 
table datatypetest is in the database.


If I run the test under gdb just until where test 24 attempts to open 
the table, the table is not in the database. I can see this also from 
pgAdmin.


The table seems to disappear just for the test. Which is very confusing.

Ari

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Even Rouault]

Le mardi 09 février 2016 09:18:00, Ari Jolma a écrit :
> Thanks,
> 
> We now have +1 from Even, Tamas, Jukka and Daniel. Is that formally enough?

Yes. at least 2 business days, 2 +1 votes and no -1.

> 
> I have a small change to the draft. I now think that the coordinate
> flags should be managed/queried solely through Is3D(), IsMeasured()
> (which are standard) and set3D(boolean) and setMeasured(boolean), (which
> are extensions to the SF specification). I.e., the getFlags() and
> setFlags(int) should be removed. The reason is that there is one flag,
> which is used only for points, and thus the API can be easily misused.

Sounds good. I had a bit that feeling too when looking at the code but didn't 
dive enough into the implementation to have a strong opinion on it.

> 
> Similarly the bit #defines for the coordinate flags sh/could be private.
> In practice that means to move them to ogr_p.h.
> 
> Even, there are strange errors in the ogr_pg.py tests.
> 
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/107782519/log.txt
> 
> Test 24 fails because the layer is NoneType, i.e., it does not open the
> datatypetest table. However, if I run the tests up to 24 but quit before
> it, the table exists and seems to be ok.

Tests are not independant from each other, so I suspect the failure in 
ogr_pg_21 might let a (implicit) transaction non committed. Thus ogr_pg_24 
which opens a new connection doesn't see the table created before. Whereas if 
you quit, the transaction will be committed.

> I had similar problems with the
> shape driver - temporary datasets disappearing midtests. Any idea what's
> the reason?

This cannot be due to transactions here, but here too steps may have 
dependencies.

> 
> Ari
> 
> 08.02.2016, 19:06, Daniel Morissette kirjoitti:
> > On 2016-02-05 3:04 AM, Ari Jolma wrote:
> >> I'd like to ask the PSC and others to vote on adopting RFC 61: Support
> >> for measured geometries.
> >> 
> >> The draft RFC is at
> >> 
> >> https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries
> > 
> > +1
> > 
> > Daniel
> 
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Ari Jolma]

I declare this motion passed with +1 from Even, Tamas, Jukka and Daniel 
and no -1.


I'll commit soon the changes to the core and then later the drivers: 
memory, shape and pg. The goal is to keep the travis test build passing.


Ari

05.02.2016, 10:04, Ari Jolma kirjoitti:
I'd like to ask the PSC and others to vote on adopting RFC 61: Support 
for measured geometries.


The draft RFC is at

https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries

and a draft implementation is at

https://github.com/ajolma/GDAL-XYZM

which is tested at

https://travis-ci.org/ajolma/GDAL-XYZM

(the test #34, which passed, was the so far last one after core 
changes with unchanged autotests and a rather comprehensive set of WKT 
tests in the Perl tests directory)


This is seemingly a small change but it is at the heart of OGR so it 
has some important implications. The only backwards incompatibilities 
that have appeared so far are with some drivers, for example 
shapefile, which can be lessened with, e.g., open options.


Best,

Ari



___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Even Rouault]

Le lundi 08 février 2016 08:17:28, vous avez écrit :
> 06.02.2016, 18:00, Sean Gillies kirjoitti:
> > Can you explain more in detail about the backwards incompatibilities
> > and how to soften them? What's going to break and what are the
> > specific options?
> 
> Hi Sean,
> 
> In the C++ and C API there are no backwards incompatibilities but the
> get/setCoordinateDimension() methods become deprecated since XYM and XYZ
> have the same dimension.  Is3D(), IsMeasured(), set3D(), and
> setMeasured() should be used instead.
> 
> I don't know about the other drivers but Shapefile driver currently
> reads XYM geometries as XYZ and that will be changed so that XYM are
> read as XYM. I think that is the only incompatibility. A new open option
> MEASURE_AS_Z=YES/NO (or something) can be defined for backwards
> compatibility. 

I don't think there's a strong need for such an option without confirmation 
from users. The XYM support as XYZ was really a hack (people had no way to 
know if the XYZ was XYM or XYZ, unless they read the .shp/.shx themselves). 
And it is not clear how that would work in update scenarios (currently the M/Z 
is discarded on update :  https://trac.osgeo.org/gdal/ticket/6331 )

Pirmin: as I see from https://trac.osgeo.org/gdal/ticket/2374 you were the one 
to contribute the patch, what is your opinion on this ?

> Similar option can be defined for other drivers if they
> have the same feature.

I believe all other drivers currently discard the M values

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Pirmin Kalberer]

Hi,

Am Montag, 8. Februar 2016, 10.04:55 schrieb Even Rouault:
> Le lundi 08 février 2016 08:17:28, vous avez écrit :
> > 06.02.2016, 18:00, Sean Gillies kirjoitti:
> > > Can you explain more in detail about the backwards incompatibilities
> > > and how to soften them? What's going to break and what are the
> > > specific options?
> > 
> > Hi Sean,
> > 
> > In the C++ and C API there are no backwards incompatibilities but the
> > get/setCoordinateDimension() methods become deprecated since XYM and XYZ
> > have the same dimension.  Is3D(), IsMeasured(), set3D(), and
> > setMeasured() should be used instead.
> > 
> > I don't know about the other drivers but Shapefile driver currently
> > reads XYM geometries as XYZ and that will be changed so that XYM are
> > read as XYM. I think that is the only incompatibility. A new open option
> > MEASURE_AS_Z=YES/NO (or something) can be defined for backwards
> > compatibility.
> 
> I don't think there's a strong need for such an option without confirmation
> from users. The XYM support as XYZ was really a hack (people had no way to
> know if the XYZ was XYM or XYZ, unless they read the .shp/.shx themselves).
> And it is not clear how that would work in update scenarios (currently the
> M/Z is discarded on update :  https://trac.osgeo.org/gdal/ticket/6331 )
> 
> Pirmin: as I see from https://trac.osgeo.org/gdal/ticket/2374 you were the
> one to contribute the patch, what is your opinion on this ?

Completeley forgot that... We had a customer request for accessing M values in 
Shapefiles from QGIS and the result was this hacky solution.
In my opinion we can live without a compatibilty switch to this intermediate 
solution when a proper API for XYZM is provided. It will be easy to adapt that 
in a coming version of QGIS (update would be needed also for setting the 
compatibility switch) and in the meantime they have to stay on their GDAL 
version (what they usually do anyway). So an incompatibility warning in the 
release notes would be enough for me.

Regards
Pirmin

-- 
Pirmin Kalberer
Sourcepole  -  Linux & Open Source Solutions
http://www.sourcepole.com

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Daniel Morissette]

On 2016-02-05 3:04 AM, Ari Jolma wrote:

I'd like to ask the PSC and others to vote on adopting RFC 61: Support
for measured geometries.

The draft RFC is at

https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries




+1

Daniel

--
Daniel Morissette
http://www.mapgears.com/
T: +1 418-696-5056 #201

http://evouala.com/ - Location Intelligence Made Easy
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Ari Jolma]

06.02.2016, 18:00, Sean Gillies kirjoitti:



Can you explain more in detail about the backwards incompatibilities 
and how to soften them? What's going to break and what are the 
specific options?


Hi Sean,

In the C++ and C API there are no backwards incompatibilities but the 
get/setCoordinateDimension() methods become deprecated since XYM and XYZ 
have the same dimension.  Is3D(), IsMeasured(), set3D(), and 
setMeasured() should be used instead.


I don't know about the other drivers but Shapefile driver currently 
reads XYM geometries as XYZ and that will be changed so that XYM are 
read as XYM. I think that is the only incompatibility. A new open option 
MEASURE_AS_Z=YES/NO (or something) can be defined for backwards 
compatibility. Similar option can be defined for other drivers if they 
have the same feature.


Ari

ps: Sorry for the late answer, I was offline the weekend.



--
Sean Gillies


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Sean Gillies]

Hi Ari,

On Fri, Feb 5, 2016 at 1:04 AM, Ari Jolma  wrote:

> I'd like to ask the PSC and others to vote on adopting RFC 61:  Support
> for measured geometries.
>
> The draft RFC is at
>
> https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries
>
> and a draft implementation is at
>
> https://github.com/ajolma/GDAL-XYZM
>
> which is tested at
>
> https://travis-ci.org/ajolma/GDAL-XYZM
>
> (the test #34, which passed, was the so far last one after core changes
> with unchanged autotests and a rather comprehensive set of WKT tests in the
> Perl tests directory)
>
> This is seemingly a small change but it is at the heart of OGR so it has
> some important implications. The only backwards incompatibilities that have
> appeared so far are with some drivers, for example shapefile, which can be
> lessened with, e.g., open options.
>

Can you explain more in detail about the backwards incompatibilities and
how to soften them? What's going to break and what are the specific options?

-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] RFC 61: Call for vote on adoption

[Ari Jolma]

I'd like to ask the PSC and others to vote on adopting RFC 61: Support 
for measured geometries.


The draft RFC is at

https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries

and a draft implementation is at

https://github.com/ajolma/GDAL-XYZM

which is tested at

https://travis-ci.org/ajolma/GDAL-XYZM

(the test #34, which passed, was the so far last one after core changes 
with unchanged autotests and a rather comprehensive set of WKT tests in 
the Perl tests directory)


This is seemingly a small change but it is at the heart of OGR so it has 
some important implications. The only backwards incompatibilities that 
have appeared so far are with some drivers, for example shapefile, which 
can be lessened with, e.g., open options.


Best,

Ari

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Even Rouault]

Le vendredi 05 février 2016 09:04:00, Ari Jolma a écrit :
> I'd like to ask the PSC and others to vote on adopting RFC 61: Support
> for measured geometries.

+1

> 
> The draft RFC is at
> 
> https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries
> 
> and a draft implementation is at
> 
> https://github.com/ajolma/GDAL-XYZM
> 
> which is tested at
> 
> https://travis-ci.org/ajolma/GDAL-XYZM
> 
> (the test #34, 

What is test #34 ?

> which passed, was the so far last one after core changes
> with unchanged autotests and a rather comprehensive set of WKT tests in
> the Perl tests directory)
> 

I see some failures in latest build, apparently related to -nlt POLYGON25D / -
lco SHPT=POLYGONZ in the shapefile driver

https://s3.amazonaws.com/archive.travis-ci.org/jobs/107177078/log.txt

  Failures 
Script: utilities/test_ogr2ogr_lib.py
  TEST: test_ogr2ogr_lib_7 ... fail
Script: utilities/test_gdal_grid_lib.py
  TEST: test_gdal_grid_lib_1 ... fail
Script: utilities/test_ogr2ogr.py
  TEST: test_ogr2ogr_11 ... fail
  TEST: test_ogr2ogr_12 ... fail
  TEST: test_ogr2ogr_43 ... fail
Script: pyscripts/test_ogr2ogr_py.py
  TEST: test_ogr2ogr_py_11 ... fail
  TEST: test_ogr2ogr_py_12 ... fail
  TEST: test_ogr2ogr_py_43 ... fail
 --

> This is seemingly a small change but it is at the heart of OGR so it has
> some important implications. The only backwards incompatibilities that
> have appeared so far are with some drivers, for example shapefile, which
> can be lessened with, e.g., open options.
> 
> Best,
> 
> Ari

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Tamas Szekeres]

+1

I'll update the mssql spatial driver accordingly.

Best regards,

Tamas


2016-02-05 9:04 GMT+01:00 Ari Jolma :

> I'd like to ask the PSC and others to vote on adopting RFC 61:  Support
> for measured geometries.
>
> The draft RFC is at
>
> https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries
>
> and a draft implementation is at
>
> https://github.com/ajolma/GDAL-XYZM
>
> which is tested at
>
> https://travis-ci.org/ajolma/GDAL-XYZM
>
> (the test #34, which passed, was the so far last one after core changes
> with unchanged autotests and a rather comprehensive set of WKT tests in the
> Perl tests directory)
>
> This is seemingly a small change but it is at the heart of OGR so it has
> some important implications. The only backwards incompatibilities that have
> appeared so far are with some drivers, for example shapefile, which can be
> lessened with, e.g., open options.
>
> Best,
>
> Ari
>
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Ari Jolma]

05.02.2016, 10:49, Even Rouault kirjoitti:

Le vendredi 05 février 2016 09:04:00, Ari Jolma a écrit :

I'd like to ask the PSC and others to vote on adopting RFC 61: Support
for measured geometries.

+1


The draft RFC is at

https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries

and a draft implementation is at

https://github.com/ajolma/GDAL-XYZM

which is tested at

https://travis-ci.org/ajolma/GDAL-XYZM

(the test #34,

What is test #34 ?


https://travis-ci.org/ajolma/GDAL-XYZM/builds/106980691

...it has #34





which passed, was the so far last one after core changes
with unchanged autotests and a rather comprehensive set of WKT tests in
the Perl tests directory)


I see some failures in latest build, apparently related to -nlt POLYGON25D / -
lco SHPT=POLYGONZ in the shapefile driver


Yes, it fails now since the shapefile driver is partially implemented - 
only points and linestrings so far.


I made it skip one test in ogr_shape.py, which relied on the M to Z hacks.




https://s3.amazonaws.com/archive.travis-ci.org/jobs/107177078/log.txt

   Failures 
Script: utilities/test_ogr2ogr_lib.py
   TEST: test_ogr2ogr_lib_7 ... fail
Script: utilities/test_gdal_grid_lib.py
   TEST: test_gdal_grid_lib_1 ... fail
Script: utilities/test_ogr2ogr.py
   TEST: test_ogr2ogr_11 ... fail
   TEST: test_ogr2ogr_12 ... fail
   TEST: test_ogr2ogr_43 ... fail
Script: pyscripts/test_ogr2ogr_py.py
   TEST: test_ogr2ogr_py_11 ... fail
   TEST: test_ogr2ogr_py_12 ... fail
   TEST: test_ogr2ogr_py_43 ... fail
  --


This is seemingly a small change but it is at the heart of OGR so it has
some important implications. The only backwards incompatibilities that
have appeared so far are with some drivers, for example shapefile, which
can be lessened with, e.g., open options.

Best,

Ari


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RFC 61: Call for vote on adoption

[Jukka Rahkonen]

Ari Jolma  gmail.com> writes:

> 
> 
> I'd like to ask the PSC and others to vote on adopting RFC 61: 
> 
> Support for measured geometries.

+1

-Jukka Rahkonen-
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev