[gdal-dev] Request for advice in unexpected slowdown working with LVBAG

[Jan Heckman via gdal-dev]

My c++ utility-app collects Kadaster BAG data in postgresql tables for use
in "populatie service".
It has functioned for several years, with a typical runtime of 40 minutes.
In the process of rounding this off, I updated the gdal binaries from 3.03
to 3.53, as well as added some threaded workloads for analyzing the tables.
Confusingly, the runtime went up to 4 hours and more. This is after the
download, so internet speed has no role here. It's all local, no network
and my PC has not degraded. I have tried to reconstruct an older setup, but
failed to get it back up to speed.
Profiling and random breaks during waiting point conclusively to the
function getting features from a layer.
LVBAG processes lots of xml of a specific (but not highly particular)
structure.
My app is compiled, if so indicated, against expat. Xerces is also a
required dll.
So my first idea was that there might be a serious performance difference
between xml libs, but without direct evidence.
I am using gdal windows binaries from Gisinternals.
All suggestions are welcome!
Thanks in advance,
Jan
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Proposed RFC 8 amendment regarding (prohibited use of) generative AI tools

[Jan Heckman via gdal-dev]

Some reading

.

On Wed, Oct 9, 2024 at 9:11 AM ElPaso via gdal-dev 
wrote:

> Il 09/10/24 00:55, Greg Troxel via gdal-dev ha scritto:
> > ElPaso via gdal-dev  writes:
> >
> >> I have read the discussion on lwn and I must say that I am more in
> >> line with the debian position.
> > My view is that code that comes out of generative AI should be viewed as
> > an improper derived work, and lacking adequate provenance/permission to
> > be added to an open source project, period.  As Even says, we could
> > relax this in the future.  It's very difficult to go back and remove
> > things.
>
>
> This may be true in some circumstances but that's not what I have seen
> so far using copilot wih GDAL: most of the times what it does for me
> much faster than me is cut-and-paste-replace or autocomplete taking code
> from other parts of GDAL or most frequently from other parts of the same
> file that I am editing, for example, when I was changing  ogrlineref to
> use gdalargumentparser, after manually changing the first part of the file
>
>
> https://github.com/OSGeo/gdal/pull/10147/files#diff-b906434b9e6a52aef54e0894ba43a7202290b1af4964cc5d9f1ec8ae7a1c4e15R1271
>
> the AI was very useful to autocomplete the other command line switches
> (one by one, not all of them), I had to change/edit almost everything
> but the scaffolding was there, the source of the autocomplete was
> obviously GDAL itself.
>
>
> Where is the copyright issue in this use case?
>
>
> Another situation where I find it useful is when writing tests: most of
> the times tests are boring boilerplate code to construct the test data,
> for example: the next line here was generate from the comment
>
>
> https://github.com/qgis/QGIS/pull/58986/files#diff-8eebd3707bdc54c42ecd1a2abfca8f623a4c6a8c44c89f344a6762fe95a95059R4682
>
>
> also the checks were automatically generated after I entered the first
> couple (they are essentially a copy-paste), the source is QGIS itself.
>
>
> That said I agree that when an AI will be smart enough to be able to be
> "creative" (I know, hard to define what it exactly is, but a one-line
> cut-paste-replace from the same code base certainly isn't) we will have
> a problem to define who/what is the author of the code.
>
>
> Perhaps we could find a way to allow the limited use of AI tools as
> autocompleters as long as the source of the generated material is
> obviously the code base itself (for instance when using the GDAL API).
>
>
> --
> Alessandro Pasotti
> w3: www.itopen.it
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Proposed RFC 8 amendment regarding (prohibited use of) generative AI tools

[Jan Heckman via gdal-dev]

Sorry to butt in.
I have no say in this whatsoever.
A few mayhaps/perby useful points:
- not using GAI is not a lasting option, so rather focus on precautions and
be aware of dangers.
Prime danger, to me, seams copyright trolling ala Oracle. They have
bought/adopted stuff in order to sue people.
Of course, GDAL is not rich enough to profit, but reputation damage is also
to be avoided.
Precaution would be rewrite with gdal naming convention, see whether the
usually general code can be more specific to the context.
Another weird idea: perhaps you can ask your copilot or whatever GAI
whether copyrighted material is involved. Experimental, wishful thinking,
Don't know yet.
Suppose you get a positive answer, then do a strong rewrite or do not use
at all.
Sometimes framing your questions in a slightly more general way may help
you without too much copy paste.
- SO. identifying each successful SO inquiry can be quite obnoxious.
Can you persuade SO into a simpler agreement?
I am thinking of decorating all modules with an SO mention and
adding an option to each (ogrinfo, ogr2ogr etc) executable: --credits.
Hardly have to explain more, I guess. If you do, mention --credits in
--help.
I have no suggestion for RFC8, except the general tone: don't use unless,
which is already there.
And the final advice to anyone would be: do not just copy-paste.
I'll show myself out 😉

On Tue, Oct 8, 2024 at 6:21 PM Howard Butler via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

>
>
> > On Oct 7, 2024, at 11:38 AM, Even Rouault via gdal-dev <
> gdal-dev@lists.osgeo.org> wrote:
> >
> > Hi,
> >
> > I propose we update RFC 8 with the following changes:
> https://github.com/OSGeo/gdal/pull/10958
> >
> > Can we enforce that? No (similarly we can't prevent someone from
> stealing copyrighted material from source code of proprietary software they
> have access to, or "re-license" or "forget" copyright attribution of open
> source code without authorization), but at least we set the policy.
> >
> > This is admittedly a highly controversial topic. Debian for example has
> had related discussions but didn't come with a position for now, whereas
> Gentoo banned use of generative AI: https://lwn.net/Articles/972331/
> >
> > It seems more prudent to me to ban for now, and potentially relax in the
> future, when/if the legal status of this becomes clearer. (not even
> mentioning the ethical or environmental impact sides of such tools)
> >
> > Even
>
> I wonder if the policy should be to require any contributions that use
> generative AI *be labeled as such*. This would serve two purposes – it
> would allow us to react to contributions that would be problematic to
> include and it would allow us to preserve those labels in the codebase for
> any of the contributions for future users of the software.
>
> It is always the responsibility of the users of the software to be aware
> of the copyright implications of their use of the software. The project
> should do its best to give users the tools to know the status of the
> software in regard to where code that is included originated.
>
> As to how to functionally implement it, maybe a simple tag on PRs that
> users must check ("did you use copilot or similar when making this PR?" and
> then some kind of automation that push that tag into the changeset history
> when it is merged.
>
> Howard
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Inquire mapped column names for conversion to shapefile

[Jan Heckman via gdal-dev]

That's pretty cool, you just saved me an awful lot of time!
Thanks,
Jan

On Mon, Apr 29, 2024 at 4:31 PM Even Rouault 
wrote:

> Hi,
>
> Something like:
>
>
> from osgeo import ogr
> original_field_names = [ "toolongforshapefile1", "toolongforshapefile2"]
> map_to_shp = {}
> tmpfilename = "/vsimem/temp.shp"
> ds = ogr.GetDriverByName("ESRI Shapefile").CreateDataSource(tmpfilename)
> lyr = ds.CreateLayer("temp")
> lyr_defn = lyr.GetLayerDefn()
> for name in original_field_names:
>  lyr.CreateField(ogr.FieldDefn(name))
>  shp_field_name = lyr_defn.GetFieldDefn(lyr_defn.GetFieldCount() -
> 1).GetNameRef()
>  map_to_shp[name] = shp_field_name
> ds = None
> ogr.GetDriverByName("ESRI Shapefile").DeleteDataSource(tmpfilename)
> print(map_to_shp)
>
> Even
>
> Le 29/04/2024 à 16:23, Jan Heckman via gdal-dev a écrit :
> > Hi everyone,
> > I want to make a map of original (let's say postgresql/postgis) column
> > names to those used in a shapefile after conversion; assuming that at
> > least some column names in postgis will be over 10 characters in
> > length, and shortening may produce conflicts. Basically I know how
> > this is done, appending _, but I hope that there is preferably a
> > python method available in Qgis python to get the list of column names
> > used in the shapefile.
> > Any help or pointers are welcome!
> > Thanks,
> > Jan
> >
> > ___
> > gdal-dev mailing list
> > gdal-dev@lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Inquire mapped column names for conversion to shapefile

[Jan Heckman via gdal-dev]

Hi everyone,
I want to make a map of original (let's say postgresql/postgis) column
names to those used in a shapefile after conversion; assuming that at least
some column names in postgis will be over 10 characters in length, and
shortening may produce conflicts. Basically I know how this is done,
appending _, but I hope that there is preferably a python method
available in Qgis python to get the list of column names used in the
shapefile.
Any help or pointers are welcome!
Thanks,
Jan
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Possible error in orientation of converted arcs from dxf

[Jan Heckman via gdal-dev]

Hi,

I set up a case for an issue and had to conclude that this issue does not
help in solving my problem (finding the proper vertex sequence/need to
reverse in a series of lines and arcs).
Surprisingly at first, in the case I made, there were arcs with proper as
well as improper sequence which all had gone through the same procedure
during conversion with ogr2ogr.

When specifying an arc by startangle, endangle, orientation
(counterclockwise around the positive z axis, using radius and midpoint),
the order of the vertices cannot also be specified (differently) by the
same ingredients.

Not being very good at dxf, I looked at the optional specification of an
extrusion vector (in an arc) in the hope that this relates to the sweep of
the arc,
rather than the extrusion of a plane arc to a 3D surface, why else define
this vector in the context of an object rather than in the context of an
extrusion command?
I'm still unsure of the meaning of this thing. and haven't found docs to
support the notion that the sweep might be intended.
And even if it is, I haven't seen the codes in the dxf files I worked with
sofar.

Anyway:
The codes are 210, 220, 230 (x,y,z) and by default 0.0,0.0,1.0. Setting
this to 0.0,0.0,-1.0 made the arc invisible in Qgis and in the ogr2ogr
produced shapefile.
When I have time I'll look more into the gdal sources, but it looks like I
need to handle this problem where it arises using my own linestring
dissolve routine.

Thanks,
Jan


On Sun, Feb 25, 2024 at 10:13 PM Even Rouault 
wrote:

> Please file an issue with a (as small as possible) .dxf file reproducing
> the issue
>
> Even
> Le 25/02/2024 à 22:05, Jan Heckman a écrit :
>
> Hi Even, all,
> There might be a slight connection, but my case imo does not involve
> either a bulgefactor or a smoothing routine;
> it is the basic conversion/render of a circular arc where apparently
> start- and endangle are interchanged sofar without apparent cause.
> Possibly the dxf has, earlier on, something which influences (rightly or
> wrongly) the clockwise setting/bool seen in this type of driver function.
> So I'll first try to find any global setting in the dxf which might
> encourage clockwise circle handling instead of default anti-clockwise.
> As an afterthought, this cannot be the case as then the arc would have
> been drawn in the other direction, giving a very large arc instead of a
> smallish one.
> As I said earlier, the correct arc is used but with the (imo) wrong
> orientation/sequence of the vertices.
> Jan
>
>
> On Sun, Feb 25, 2024 at 2:08 PM Even Rouault 
> wrote:
>
>> Hi,
>>
>> I'm wondering if there woulnd't be a connection with
>> https://github.com/OSGeo/gdal/issues/1839#issuecomment-537185826 ?
>>
>> Even
>>
>> Le 25/02/2024 à 13:32, Jan Heckman via gdal-dev a écrit :
>> > Hi all,
>> > Sorry for a somewhat long post, but here we go:
>> > I am converting a .dxf having a sequence of simple lines and arcs
>> > which form a continuous (poly)line, (correctly) ordered by
>> > EntityHandle. The arcs are converted (to shp) as well as displayed
>> > (Qgis) in the opposite direction as the simple lines. Checking the
>> > codes and docs, this appears wrong.
>> > Below the minimal relevant information, I can add the complete .dxf,
>> > but I think attached files are frowned upon?
>> >
>> > Docs cited:
>> >
>> https://help.autodesk.com/view/OARX/2024/ENU/?guid=GUID-0B14D8F1-0EBA-44BF-9108-57D8CE614BC8
>> > ARC (DXF)
>> > The following group codes apply to arc entities.
>> > Arc group codes
>> > Group code Description
>> > 100 Subclass marker (AcDbCircle)
>> > 39 Thickness (optional; default = 0)
>> > 10 Center point (in OCS) DXF: X value; APP: 3D point
>> > 20, 30 DXF: Y and Z values of center point (in OCS)
>> > 40 Radius
>> > 100 Subclass marker (AcDbArc)
>> > 50 Start angle
>> > 51 End angle
>> > 210 Extrusion direction (optional; default = 0, 0, 1) DXF: X value;
>> > APP: 3D vector
>> > 220, 230 DXF: Y and Z values of extrusion direction (optional)
>> >
>> > https://ezdxf.readthedocs.io/en/stable/tutorials/dxf_primitives.html
>> > The arc goes always in counter-clockwise orientation around the z-axis
>> > more precisely the extrusion vector of OCS, but this is beyond the
>> > scope of this tutorial.
>> >
>> > The case (extracted, whole .dxf file available):
>> > AcDbEntity
>> >   8
>> > -001_09_AND1N
>> >   6
>> > Continuous
>> >  62
>> > 18
>> > 100
>> > AcDbCircle
>> >  10

Re: [gdal-dev] Possible error in orientation of converted arcs from dxf

[Jan Heckman via gdal-dev]

Hi Even, all,
There might be a slight connection, but my case imo does not involve either
a bulgefactor or a smoothing routine;
it is the basic conversion/render of a circular arc where apparently start-
and endangle are interchanged sofar without apparent cause.
Possibly the dxf has, earlier on, something which influences (rightly or
wrongly) the clockwise setting/bool seen in this type of driver function.
So I'll first try to find any global setting in the dxf which might
encourage clockwise circle handling instead of default anti-clockwise.
As an afterthought, this cannot be the case as then the arc would have been
drawn in the other direction, giving a very large arc instead of a smallish
one.
As I said earlier, the correct arc is used but with the (imo) wrong
orientation/sequence of the vertices.
Jan


On Sun, Feb 25, 2024 at 2:08 PM Even Rouault 
wrote:

> Hi,
>
> I'm wondering if there woulnd't be a connection with
> https://github.com/OSGeo/gdal/issues/1839#issuecomment-537185826 ?
>
> Even
>
> Le 25/02/2024 à 13:32, Jan Heckman via gdal-dev a écrit :
> > Hi all,
> > Sorry for a somewhat long post, but here we go:
> > I am converting a .dxf having a sequence of simple lines and arcs
> > which form a continuous (poly)line, (correctly) ordered by
> > EntityHandle. The arcs are converted (to shp) as well as displayed
> > (Qgis) in the opposite direction as the simple lines. Checking the
> > codes and docs, this appears wrong.
> > Below the minimal relevant information, I can add the complete .dxf,
> > but I think attached files are frowned upon?
> >
> > Docs cited:
> >
> https://help.autodesk.com/view/OARX/2024/ENU/?guid=GUID-0B14D8F1-0EBA-44BF-9108-57D8CE614BC8
> > ARC (DXF)
> > The following group codes apply to arc entities.
> > Arc group codes
> > Group code Description
> > 100 Subclass marker (AcDbCircle)
> > 39 Thickness (optional; default = 0)
> > 10 Center point (in OCS) DXF: X value; APP: 3D point
> > 20, 30 DXF: Y and Z values of center point (in OCS)
> > 40 Radius
> > 100 Subclass marker (AcDbArc)
> > 50 Start angle
> > 51 End angle
> > 210 Extrusion direction (optional; default = 0, 0, 1) DXF: X value;
> > APP: 3D vector
> > 220, 230 DXF: Y and Z values of extrusion direction (optional)
> >
> > https://ezdxf.readthedocs.io/en/stable/tutorials/dxf_primitives.html
> > The arc goes always in counter-clockwise orientation around the z-axis
> > more precisely the extrusion vector of OCS, but this is beyond the
> > scope of this tutorial.
> >
> > The case (extracted, whole .dxf file available):
> > AcDbEntity
> >   8
> > -001_09_AND1N
> >   6
> > Continuous
> >  62
> > 18
> > 100
> > AcDbCircle
> >  10
> > 130974.661837
> >  20
> > 497502.478468
> >  30
> > 0.0
> >  40
> > 7576.09443001
> > 100
> > AcDbArc
> >  50
> > 305.544433001
> >  51
> > 344.795475
> >   0
> > LINE
> >   5
> > 2E
> > 330
> > 1A0
> > 100
> >
> > The arc is converted (.shp) and drawn (qgis) from end angle to start
> > angle, without a 210 code to alter default behavior, which obviously
> > is the other way around, noting counter clock around (positive) Z-axis.
> > The applicability of this convention also follows from the (correct)
> > position of the arc in the conversion and drawing.
> > Only begin- and endpoints are interchanged.
> >
> > Please advice,
> > Thanks in advance,
> > Jan
> >
> > ___
> > gdal-dev mailing list
> > gdal-dev@lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Possible error in orientation of converted arcs from dxf

[Jan Heckman via gdal-dev]

Hi all,
Sorry for a somewhat long post, but here we go:
I am converting a .dxf having a sequence of simple lines and arcs which
form a continuous (poly)line, (correctly) ordered by EntityHandle. The arcs
are converted (to shp) as well as displayed (Qgis) in the opposite
direction as the simple lines. Checking the codes and docs, this appears
wrong.
Below the minimal relevant information, I can add the complete .dxf, but I
think attached files are frowned upon?

Docs cited:
https://help.autodesk.com/view/OARX/2024/ENU/?guid=GUID-0B14D8F1-0EBA-44BF-9108-57D8CE614BC8
ARC (DXF)
The following group codes apply to arc entities.
Arc group codes
Group code Description
100 Subclass marker (AcDbCircle)
39 Thickness (optional; default = 0)
10 Center point (in OCS) DXF: X value; APP: 3D point
20, 30 DXF: Y and Z values of center point (in OCS)
40 Radius
100 Subclass marker (AcDbArc)
50 Start angle
51 End angle
210 Extrusion direction (optional; default = 0, 0, 1) DXF: X value; APP: 3D
vector
220, 230 DXF: Y and Z values of extrusion direction (optional)

https://ezdxf.readthedocs.io/en/stable/tutorials/dxf_primitives.html
The arc goes always in counter-clockwise orientation around the z-axis more
precisely the extrusion vector of OCS, but this is beyond the scope of this
tutorial.

The case (extracted, whole .dxf file available):
AcDbEntity
  8
-001_09_AND1N
  6
Continuous
 62
18
100
AcDbCircle
 10
130974.661837
 20
497502.478468
 30
0.0
 40
7576.09443001
100
AcDbArc
 50
305.544433001
 51
344.795475
  0
LINE
  5
2E
330
1A0
100

The arc is converted (.shp) and drawn (qgis) from end angle to start angle,
without a 210 code to alter default behavior, which obviously is the other
way around, noting counter clock around (positive) Z-axis.
The applicability of this convention also follows from the (correct)
position of the arc in the conversion and drawing.
Only begin- and endpoints are interchanged.

Please advice,
Thanks in advance,
Jan
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] layer creation option shapefile when converting arcs in dxf

[Jan Heckman via gdal-dev]

Even,
Thanks, that does the job.
I'll lookup configuration options first next time!
Jan

On Sat, Feb 24, 2024 at 12:20 PM Even Rouault 
wrote:

> Hi,
>
> perhaps the OGR_ARC_STEPSIZE configuration option:
> https://gdal.org/user/configoptions.html#config-OGR_ARC_STEPSIZE ?
>
> Even
>
> Le 24/02/2024 à 12:15, Jan Heckman via gdal-dev a écrit :
> > Hi All,
> > I am looking for a way to influence the (density of the) linestrings
> > which are produced by converting a circle segment in dxf (to a
> > linestring in shapefile format); the aim, ideally, is to limit the
> > angle(s) between the chords produced.
> > If not available, a setting of the number of vertices for a circle
> > would also help.
> > I searched docs, and if I overlooked, apologies in advance.
> > I am hoping for a setting for ogr2ogr, but also interested in a
> > solution using the (C++) api/library.
> > thanks, Jan
> >
> > ___
> > gdal-dev mailing list
> > gdal-dev@lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] layer creation option shapefile when converting arcs in dxf

[Jan Heckman via gdal-dev]

Hi All,
I am looking for a way to influence the (density of the) linestrings which
are produced by converting a circle segment in dxf (to a linestring in
shapefile format); the aim, ideally, is to limit the angle(s) between the
chords produced.
If not available, a setting of the number of vertices for a circle would
also help.
I searched docs, and if I overlooked, apologies in advance.
I am hoping for a setting for ogr2ogr, but also interested in a solution
using the (C++) api/library.
thanks, Jan
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] layer.GetSpatialRef() fails on linux for shapefiles

[Jan Heckman via gdal-dev]

That's clear. However, environmental means SET in one environment, and
EXPORT in the other.
Sorry for the pun, but I expect Windows and Linux settings/exports for GDAL
should be fairly similar.
So the environments might be the next step to communicate, if I'm not
mistaken at root.

On Sun, Sep 24, 2023 at 11:58 PM Jonathan Moules <
jonathan-li...@lightpear.com> wrote:

> This applies to *all* shapefiles for that user. There are about 10 in the
> sample and they're from a diverse range of UK organisations. Mostly 27700
> (British National Grid), but likely one or two WGS84. Hence thinking it may
> be environmental, but I don't have any real *nix knowledge to use to
> diagnose the issue.
>
>
> On 2023-09-24 22:52, Jan Heckman wrote:
>
> Sorry to break in, but surely, we would like to see the .prj file in
> question for a simple try at reproducing?
>
> On Sun, Sep 24, 2023 at 11:37 PM Jonathan Moules via gdal-dev <
> gdal-dev@lists.osgeo.org> wrote:
>
>> Thanks Even. I don't have access to the machine either as the colleague
>> is moving to another project. I'll have to see if it fails for another
>> *nix user.
>>
>> On 2023-09-24 22:35, Even Rouault wrote:
>> >
>> > Le 24/09/2023 à 23:22, Jonathan Moules via gdal-dev a écrit :
>> >>
>> >> > Also check if the environment isn't messed up regarding PROJ and
>> >> the PROJ_LIB/PROJ_DATA environment variable
>> >>
>> >> Thanks Even; sorry, what does this line mean? I'm guessing you're
>> >> referring to: https://proj.org/en/9.3/usage/environmentvars.html -
>> >> what would a "messed up" one look like?
>> >>
>> > Hard to say without access to the machine. Perhaps just try to
>> > recreate a new Conda env from scratch
>> >
>> >
>>
>> ___
>> gdal-dev mailing list
>> gdal-dev@lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] layer.GetSpatialRef() fails on linux for shapefiles

[Jan Heckman via gdal-dev]

Sorry to break in, but surely, we would like to see the .prj file in
question for a simple try at reproducing?

On Sun, Sep 24, 2023 at 11:37 PM Jonathan Moules via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Thanks Even. I don't have access to the machine either as the colleague
> is moving to another project. I'll have to see if it fails for another
> *nix user.
>
> On 2023-09-24 22:35, Even Rouault wrote:
> >
> > Le 24/09/2023 à 23:22, Jonathan Moules via gdal-dev a écrit :
> >>
> >> > Also check if the environment isn't messed up regarding PROJ and
> >> the PROJ_LIB/PROJ_DATA environment variable
> >>
> >> Thanks Even; sorry, what does this line mean? I'm guessing you're
> >> referring to: https://proj.org/en/9.3/usage/environmentvars.html -
> >> what would a "messed up" one look like?
> >>
> > Hard to say without access to the machine. Perhaps just try to
> > recreate a new Conda env from scratch
> >
> >
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Shapefile with corrupted index: SHAPE_RESTORE_SHX=YES doesn't correctly repairs it.

[Jan Heckman]

Ok, great.
I don't use AutoCad, seems to be a good explanation for the idx, although I
have not found out whether it is
a spatial index or an .shx replacement. See
https://forums.autodesk.com/t5/autocad-map-3d-forum/2018-shape-file-idx-issue/td-p/8300100
discussing (sharing) problems with this .idx file in autocad.
I was at first convinced that some action on the .dbf was the cause of the
problem, anyway, that does not seem
too likely anymore as the dbf rows have an id field and the rows (still)
ordered ascendingly as you would expect.
Therefore the order of the shp and the dbf is likely the same.
Only the last dbf row has damage.
I am a bit unsure whether you have a valid shapefile again/still.
Unless you have a fully recovered/backed up shapefile by now, I suggest you
take a (second) look at the link
<https://www.dropbox.com/s/wj0loe0m7apuszz/test.zip?dl=0> I gave above.
I could repair the last weirdness (lines sticking out westward) and now it
looks ok, and complete.
See https://www.dropbox.com/s/bh6pfvk2haddoqf/result.png?dl=0 for a quick
glance.
In dark the shapefile as it was first shown in QGis, in pink the
(additional) objects that have been recovered.
Best regards,
Jan

On Wed, May 17, 2023 at 4:59 PM Andrea Giudiceandrea via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi Jan,
> thank you very much for looking at this issue.
>
> I had already managed to restore the .shx index file, as reported in my
> first message [1] [2] by fixing the GDAL/OGR SHPRestoreSHX routine [3].
> Even Rouault also further improved it adding extra sanity checks, a
> better logic and better error messages [4].
> Obviously, SHPRestoreSHX routing doesn't take care of the the mismatch
> between the shape and dbf records number.
>
> An old but still working Shapefile recovery tool (which also handle the
> shp / dbf mismatch) is the "Shape Checker" [5].
>
> The .idx file may have been created by AutoCAD [6], as the user reported
> that he tried to open the corrupted Shapefile layer with AutoCAD after
> the corruption occurred.
>
> Best regards.
>
> Andrea
>
>
> [1] https://lists.osgeo.org/pipermail/gdal-dev/2023-May/057229.html
> [2] https://github.com/qgis/QGIS/issues/53058#issuecomment-1547740971
> [3] https://github.com/OSGeo/gdal/pull/7774
> [4] https://github.com/OSGeo/gdal/pull/7778
> [5]
>
> https://web.archive.org/web/20091024035556/http://geocities.com/SiliconValley/Haven/2295/howto_shapechk.html
> [6]
>
> https://www.autodesk.com/support/technical/article/caas/sfdcarticles/sfdcarticles/Required-files-that-make-up-a-shapefile.html
>
>
>
> Il 17/05/2023 12:39, Jan Heckman ha scritto:
> > Hi all,
> >
> > As noted in the QGis issues <https://github.com/qgis/QGIS/issues/53058>,
>
> > the shx has problems that ogr2ogr (or QGis Repair Shapefile) will not
> fix;
> > The error I get in cmdline with SHAPE_RESTORE_SHX=YES is "ERROR 4: Error
> > parsing .shp to restore .shx".
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Shapefile with corrupted index: SHAPE_RESTORE_SHX=YES doesn't correctly repairs it.

[Jan Heckman]

Have to get back on this, I did not look well at the shapefile I generated
(see last mail).
It is both better and worse. It has most of the missing objects, to my
surprise, but appears to have some invalid coordinates
which show up as horizontal lines in the west part. Still it may help in
reconstruction, I think.
Good luck,
Jan

On Wed, May 17, 2023 at 12:39 PM Jan Heckman  wrote:

> Hi all,
>
> As noted in the QGis issues <https://github.com/qgis/QGIS/issues/53058>,
> the shx has problems that ogr2ogr (or QGis Repair Shapefile) will not fix;
> The error I get in cmdline with SHAPE_RESTORE_SHX=YES is "ERROR 4: Error
> parsing .shp to restore .shx".
> Long time ago I wrote my own version to recreate the .shx from the .shp,
> and it does restore the .shx (leaving me to wonder whether my routine is
> really correct);
> long story short, the repaired shapefile loads fine, but the shp does not
> contain the missing geometries anyway,
> and the dbf doesn't have rows of missing objects (as shown in the QGis
> issue).
>
> The missing objects are presumed lost, see issue.
>
> Looking at the .idx file I surmise that it is a foxbase/foxpro index. Good
> (very) old dbase has .mdx and .ndx.
> DBF index files are not part of the shapefile spec.
> The layout of the file does not fully fit the few descriptions I have, but
> looks like btree record blocks with keys
> that may address column H (soil erosion).
> The presence of this 'foreign' (in the context of shapefiles) file made be
> think that perhaps the .dbf has been
> modified using foxpro/foxbase, leading to an error when saving again in
> QGis.
> Another bit of evidence for such a scenario is that the last row in the
> .dbf is garbled.
> (check with LibreOffice Calc).
>
> FWIW a link <https://www.dropbox.com/s/wj0loe0m7apuszz/test.zip?dl=0> to
> the worked shapefile.
>
> On Tue, May 16, 2023 at 2:03 PM Andrea Giudiceandrea via gdal-dev <
> gdal-dev@lists.osgeo.org> wrote:
>
>> Il 16/05/2023 13:52, Jan Heckman ha scritto:
>> > Also out of curiosity,
>> > Are you willing to share the shapefile, preferably at least shp and
>> dbf,
>> > but probably .shp alone will do and .prj helps for checking conversion
>> > results.
>>
>> Hi Jan,
>> the corrupted Shapefile layer is available in the QGIS issue report at
>> https://github.com/qgis/QGIS/issues/53058
>>
>> Best regards.
>>
>> Andrea
>> ___
>> gdal-dev mailing list
>> gdal-dev@lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Shapefile with corrupted index: SHAPE_RESTORE_SHX=YES doesn't correctly repairs it.

[Jan Heckman]

Hi all,

As noted in the QGis issues <https://github.com/qgis/QGIS/issues/53058>,
the shx has problems that ogr2ogr (or QGis Repair Shapefile) will not fix;
The error I get in cmdline with SHAPE_RESTORE_SHX=YES is "ERROR 4: Error
parsing .shp to restore .shx".
Long time ago I wrote my own version to recreate the .shx from the .shp,
and it does restore the .shx (leaving me to wonder whether my routine is
really correct);
long story short, the repaired shapefile loads fine, but the shp does not
contain the missing geometries anyway,
and the dbf doesn't have rows of missing objects (as shown in the QGis
issue).

The missing objects are presumed lost, see issue.

Looking at the .idx file I surmise that it is a foxbase/foxpro index. Good
(very) old dbase has .mdx and .ndx.
DBF index files are not part of the shapefile spec.
The layout of the file does not fully fit the few descriptions I have, but
looks like btree record blocks with keys
that may address column H (soil erosion).
The presence of this 'foreign' (in the context of shapefiles) file made be
think that perhaps the .dbf has been
modified using foxpro/foxbase, leading to an error when saving again in
QGis.
Another bit of evidence for such a scenario is that the last row in the
.dbf is garbled.
(check with LibreOffice Calc).

FWIW a link <https://www.dropbox.com/s/wj0loe0m7apuszz/test.zip?dl=0> to
the worked shapefile.

On Tue, May 16, 2023 at 2:03 PM Andrea Giudiceandrea via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Il 16/05/2023 13:52, Jan Heckman ha scritto:
> > Also out of curiosity,
> > Are you willing to share the shapefile, preferably at least shp and dbf,
> > but probably .shp alone will do and .prj helps for checking conversion
> > results.
>
> Hi Jan,
> the corrupted Shapefile layer is available in the QGIS issue report at
> https://github.com/qgis/QGIS/issues/53058
>
> Best regards.
>
> Andrea
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Shapefile with corrupted index: SHAPE_RESTORE_SHX=YES doesn't correctly repairs it.

[Jan Heckman]

Also out of curiosity,
Are you willing to share the shapefile, preferably at least shp and dbf,
but probably .shp alone will do and .prj helps for checking conversion
results.
Also, to be sure, the expected number of rows is relevant so as not to call
victory too soon.
Jan

On Tue, May 16, 2023 at 1:41 PM Andrea Giudiceandrea via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Il 15/05/2023 15:34, Andrea Giudiceandrea ha scritto:
> > it seems to me ogr fails to properly create the .idx [EDIT: .shx] file:
> > it incorrectly stores, in the index file header, the total length in
> > 16-bit words of the .shp file instead of the total length in 16-bit
> > words of the .idx [EDIT: .shx] file itself.
>
> Looking at the code, SHPRestoreSHX initially copies the SHP header
> (including the SHP content length) to the SHX file and then it doesn't
> updates the SHX content length in the SHX header when an error occurred
> during the reading of the SHP file.
>
> I propose the PR https://github.com/OSGeo/gdal/pull/7774 in order to
> update the SHX content length even if an error occurred.
>
> Maybe would it be better to let SHPRestoreSHX modify the SHX file (when
> already present) only at the end of the SHP reading procedure and only
> if no error occurred?
>
> Best regards.
>
> Andrea Giudiceandrea
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] how to export all the definitions of the relationships present in a FileGDB?

[Jan Heckman]

I seem to remember that the n:m relationships can be retrieved as a table.
Look in the TOC.
1:n relationships are not stored in GDB as a table, but as a simple foreign
key relationship as far as I remember.
If you were import an n:m relationship(table) into say postgresql,
you would then have to add the needed (primary and foreign key) constraints
manually or programmatically.
With json, I have no clue how to make the relationship table(s) operational
again. Not quite my turf.
Hope this puts you in the right direction,
Jan

On Sun, Feb 19, 2023 at 1:06 PM Totò Fiandaca 
wrote:

> Is it possible to export all the relationship definitions present in a
> FileGDB or a GPKG?
>
> with $ ogrinfo -json autotest/ogr/data/filegdb/Domains.gdb I can extract
> all the domains, but for the relationships?
>
> A thousand thanks
>
> --
> *Ing. Salvatore Fiandaca*
> *mobile*.:+39 327.493.8955
> *m*: *pigrecoinfin...@gmail.com *
> *C.F*.: FNDSVT71E29Z103G
> *P.IVA*: 06597870820
> *membro QGIS Italia - http://qgis.it/ *
> *socio GFOSS.it - *http://gfoss.it/
> *member OSGeo* - *https://www.osgeo.org/member/fiandaca/*
> 
> *blog:*
> * https://pigrecoinfinito.com/  FB: Co-admin
> - https://www.facebook.com/qgis.it/ **
>  *
> *TW:  **https://twitter.com/totofiandaca
> *
>
> 43°51'0.54"N  10°34'27.62"E - EPSG:4326
>
> “Se la conoscenza deve essere aperta a tutti,
> perchè mai limitarne l’accesso?”
> R. Stallman
>
> Questo documento, allegati inclusi, contiene informazioni di proprietà di
> FIANDACA SALVATORE e deve essere utilizzato esclusivamente dal destinatario
> in relazione alle finalità per le quali è stato ricevuto. E' vietata
> qualsiasi forma di riproduzione o divulgazione senza l'esplicito consenso
> di FIANDACA SALVATORE. Qualora fosse stato ricevuto per errore si prega
> di informare tempestivamente il mittente e distruggere la copia in proprio
> possesso.
>
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] ogrinfo UPDATE performance request

[Jan Heckman]

Hi Jukka,

Thanks for all the dialect info.

Reading your last message about testing the dialect in use, I have to admit
that I did not think about native sql
(dialect) being deduced from the source type rather than from any dbms
involved.
That is clear to me now, of course.

Would it be correct, then, that, when using SQL when converting from
shapefile to postgresql/postgis,
dialect sqlite (or ogr_sql) cannot be avoided? I can live with that, of
course. Perhaps I'll start converting my
'special' geometry functions to dll's in postgresql so that I don't need to
convert and reconvert to/from shapefile.

Specifically I am working on what in arcgis is called union, which divides
overlapping polygons in constituent
parts without overlaps. This function does not seem present in postgis and
is problematic anyway when
the dataset is large enough.
(The Dutch spatial planning dataset 'Enkelbestemming' is such a dataset
with over 2.6 million entries with
overlaps both small and large, but to me, at least in terms of efficient
data access, the very essence of the idea
is that you have at most one valid geometry with attributes for each
(point)location.)

Best regards and thanks,
Jan

On Thu, Jun 9, 2022 at 12:19 PM Rahkonen Jukka <
jukka.rahko...@maanmittauslaitos.fi> wrote:

> Hi,
>
>
>
> You are getting closer but at the bottom of the page
> https://gdal.org/user/ogr_sql_dialect.html there is an important
> paragraph:
>
> Non-OGR SQL
>
> All OGR drivers for database systems: MySQL, PostgreSQL / PostGIS, Oracle
> Spatial, SQLite / Spatialite RDBMS, ODBC RDBMS, ESRI Personal GeoDatabase,
> SAP HANA and MSSQLSpatial - Microsoft SQL Server Spatial Database, override
> the GDALDataset::ExecuteSQL() function with dedicated implementation and,
> by default, pass the SQL statements directly to the underlying RDBMS. In
> these cases the SQL syntax varies in some particulars from OGR SQL. Also,
> anything possible in SQL can then be accomplished for these particular
> databases. Only the result of SQL WHERE statements will be returned as
> layers.
>
>
>
> So if you do not select SQL dialect for PostGIS you do not use the OGR SQL
> dialect but the native PostgreSQL one. For using OGR SQL with PostgreSQL
> (maybe it could make sense in some special case but I am not sure) use
> “-dialect OGRSQL”. If you are uncertain about what dialect is in use you
> can make a tests with some native-only functions like -sql "select
> postgis_version()" or -sql “select sqlite_version”.
>
>
>
> -Jukka Rahkonen-
>
>
>
> *Lähettäjä:* Andreas Oxenstierna 
> *Lähetetty:* torstai 9. kesäkuuta 2022 12.48
> *Vastaanottaja:* gdal-dev@lists.osgeo.org; Rahkonen Jukka <
> jukka.rahko...@maanmittauslaitos.fi>
> *Aihe:* Re: ogrinfo UPDATE performance request
>
>
>
> Hi
>
> Thanks a lot and sorry for the noise.
> UPDATE do work as expected using the OGR SQL, i.e. one atomic db
> transaction executing in the database = vastly faster than using -dialect
> sqlite
> .
> The doc at https://gdal.org/user/ogr_sql_dialect.html
> 
>  needs
> some addition - I assume that INSERT, DELETE etc. also works
> "While in theory any sort of command could be handled this way, in
> practice the mechanism is used to provide a subset of SQL SELECT capability
> to applications."
>
> Maybe also clarify if the sqlite dialect is necessary to keep at all ...
>
>
>
> Best Regards
>
> Andreas Oxenstierna
> T-Kartor Geospatial AB
> Olof Mohlins väg 12 Kristianstad
> mobile: +46 733 206831
> mailto: andreas.oxenstie...@t-kartor.com
> www.t-kartor.com
> 
>
> On 9 Jun 2022, 11:15 +0200, Rahkonen Jukka <
> jukka.rahko...@maanmittauslaitos.fi>, wrote:
>
> Hi,
>
>
>
> This updated 166000 rows in 15 seconds for me on my laptop without any
> workarounds:
>
> ogrinfo PG:"host=localhost port=5432 dbname=my_pg user=user password=pw"
> -sql "update buildingtest set version=99"
>
>
>
> -Jukka Rahkonen-
>
>
>
> *Lähettäjä:* Andreas Oxenstierna 
> *Lähetetty:* torstai 9. kesäkuuta 2022 12.02
> *Vastaanottaja:* gdal-dev@lists.osgeo.org; Rahkonen Jukka <
> jukka.rahko...@maanmittauslaitos.fi>
> *Aihe:* Re: ogrinfo UPDATE performance request
>
>
>
> Hi
>
> AFAIK, UPDATE only w

Re: [gdal-dev] ogrinfo UPDATE performance request

[Jan Heckman]

Hi Jukka,
Not entirely sure what you mean by 'have you tested the sql with psql',
since this is an ogr2ogr process which works.
It is a remnant of a case where multiple iv (iv1, iv2 etc.) shapefile
columns were needed, so the sql used to be
like
coalesce(iv1,'')||coalesce(iv2,'')||coalesce(iv2,'')||coalesce(iv3,'')||coalesce(iv4,'')
instead of a single (useless) coalesce,
which I kept as it might come in handy later.
I am reconstructing my stuff (new update), as the idea of a good speed up
by leaving out -dialect sqlite is attractive.
So far I noticed that one postgis function requires sqlite dialect:
D:\ro_enkel\data>ogr2ogr -f postgresql -append -gt unlimited
PG:"user=postgres dbname=test sslmode=disable" -sql "select
*,substr(plangebied,9,4) as ovh_code from enkelbestemming where
ST_GeometryType(geometrie)='POLYGON'" -nlt multipolygon -nln
ro_enkel.enkelbestemming enkelbestemming.gml.gz -progress
ERROR 1: Undefined function 'ST_GeometryType' used.
(I know that I could have used -skipfailures to get rid of linestring
geometries here, but the specific geometry type selection seemed better)
I'll check my other sqlite dialect stuff as soon as I get to it, but first
I need to wait for a conversion to complete.
Hope this rings a bell about parsing the sql with postgis functions in
postgresql dialect?
Best regards,
Jan

On Thu, Jun 9, 2022 at 10:57 AM Rahkonen Jukka <
jukka.rahko...@maanmittauslaitos.fi> wrote:

> Hi Jan,
>
>
>
> I had a try with a table that I created from the “states” shapefile from
> the Geoserver demo data.
>
>
>
> ogrinfo PG:"host=localhost port=5432 dbname=my_pg user=user password=pw"
> -sql "select trim(coalesce(state_name,state_fips),';') as fidstring,
> state_fips, st_union(wkb_geometry) as multipolygon from states group by
> state_name,state_fips"
>
>
>
> I think that the SQL is fundamentally the same and it does work for me
> with the default dialect once I edited the geometry column name to the one
> I have in PostGIS. Are you sure that it requires SQLite dialect for you?
> Have you tested the SQL with psql?
>
>
>
> -Jukka Rahkonen-
>
>
>
> *Lähettäjä:* Jan Heckman 
> *Lähetetty:* torstai 9. kesäkuuta 2022 11.17
> *Vastaanottaja:* Rahkonen Jukka 
> *Kopio:* Andreas Oxenstierna ;
> gdal-dev@lists.osgeo.org
> *Aihe:* Re: [gdal-dev] ogrinfo UPDATE performance request
>
>
>
> Afaik as I know, slightly more involved sql on a postgresql table requires
> dialect=sqlite to work at all, e.g. (picked a random example using ogr2ogr
> instead of ogrinfo)
>
> ogr2ogr -f postgresql -dialect sqlite -append PG:"user=%user%
> dbname=%dbname%" -sql "select trim(coalesce(iv1,''),';') as fidstring,
> code, st_union(geometry) as multipolygon from alltogether_pruned_out group
> by iv1,code" -nln %schema%.%baseout%_pruned -nlt multipolygon
> alltogether_pruned_out.shp
>
> This gives me acceptable performance, btw. It's handling a really large
> dataset, so having to wait for some minutes did not bother me.
>
> This is on version 3.4.2.
>
> The transaction behavior in ogr2ogr can be influenced with the -gt option,
> with -gt unlimited to do everything in a single transaction.
>
> But it would be nice if this imo strange need to refer to dialect sqlite
> were removed. Especially strange when you use postgis functions in the sql
> statement...
>
> Best regards,
>
> Jan
>
>
>
> On Thu, Jun 9, 2022 at 9:46 AM Rahkonen Jukka <
> jukka.rahko...@maanmittauslaitos.fi> wrote:
>
> Hi,
>
>
>
> Do not use “-dialect sqlite” if you play with PostgreSQL but let GDAL to
> use the native PG SQL dialect.
>
>
>
> -Jukka Rahkonen-
>
>
>
> *Lähettäjä:* gdal-dev  *Puolesta *Andreas
> Oxenstierna
> *Lähetetty:* torstai 9. kesäkuuta 2022 9.50
> *Vastaanottaja:* gdal-dev@lists.osgeo.org
> *Aihe:* [gdal-dev] ogrinfo UPDATE performance request
>
>
>
> Dear developers
>
> Ogr SQL update capabilities are really impressing but there is one major
> performance issue with update of many features, exemplified by:
> ogrinfo -dialect sqlite -sql "UPDATE  SET x = 1" PG:”"
>
> This is painfully slow because ogr updates features one by one and
> furthermore updates all existing attributes incl. geometries.
> Eg. updating 1 features in pgAdmin/psql with UPDATE  SET x = 1
> executes in milliseconds but takes several minutes with ogr.
>
> The current ogr functionality is also not correct from a database
> transactional point of view.
>
> I found an old RFC,
> https://gdal.org/development/rfc/rfc13_createfeatures.html
&

Re: [gdal-dev] ogrinfo UPDATE performance request

[Jan Heckman]

Afaik as I know, slightly more involved sql on a postgresql table requires
dialect=sqlite to work at all, e.g. (picked a random example using ogr2ogr
instead of ogrinfo)
ogr2ogr -f postgresql -dialect sqlite -append PG:"user=%user%
dbname=%dbname%" -sql "select trim(coalesce(iv1,''),';') as fidstring,
code, st_union(geometry) as multipolygon from alltogether_pruned_out group
by iv1,code" -nln %schema%.%baseout%_pruned -nlt multipolygon
alltogether_pruned_out.shp
This gives me acceptable performance, btw. It's handling a really large
dataset, so having to wait for some minutes did not bother me.
This is on version 3.4.2.
The transaction behavior in ogr2ogr can be influenced with the -gt option,
with -gt unlimited to do everything in a single transaction.
But it would be nice if this imo strange need to refer to dialect sqlite
were removed. Especially strange when you use postgis functions in the sql
statement...
Best regards,
Jan

On Thu, Jun 9, 2022 at 9:46 AM Rahkonen Jukka <
jukka.rahko...@maanmittauslaitos.fi> wrote:

> Hi,
>
>
>
> Do not use “-dialect sqlite” if you play with PostgreSQL but let GDAL to
> use the native PG SQL dialect.
>
>
>
> -Jukka Rahkonen-
>
>
>
> *Lähettäjä:* gdal-dev  *Puolesta *Andreas
> Oxenstierna
> *Lähetetty:* torstai 9. kesäkuuta 2022 9.50
> *Vastaanottaja:* gdal-dev@lists.osgeo.org
> *Aihe:* [gdal-dev] ogrinfo UPDATE performance request
>
>
>
> Dear developers
>
> Ogr SQL update capabilities are really impressing but there is one major
> performance issue with update of many features, exemplified by:
> ogrinfo -dialect sqlite -sql "UPDATE  SET x = 1" PG:”"
>
> This is painfully slow because ogr updates features one by one and
> furthermore updates all existing attributes incl. geometries.
> Eg. updating 1 features in pgAdmin/psql with UPDATE  SET x = 1
> executes in milliseconds but takes several minutes with ogr.
>
> The current ogr functionality is also not correct from a database
> transactional point of view.
>
> I found an old RFC,
> https://gdal.org/development/rfc/rfc13_createfeatures.html, requesting
> this but it was withdrawn for reasons not anymore digitally available.
>
>
>
> Best Regards
>
> Andreas Oxenstierna
> T-Kartor Geospatial AB
> Olof Mohlins väg 12 Kristianstad
> mobile: +46 733 206831
> mailto: andreas.oxenstie...@t-kartor.com
> www.t-kartor.com
> 
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] ogr2ogr Postgres upload performance

[Jan Heckman]

Perhaps the spatial index update (in de DB) slows the insertion.
It might be more efficient to not create the SI in the first run,
then insert more rows and create the SI after all is done.
I don't know how to turn off SI creation in the command line.

On Mon, May 23, 2022 at 3:04 PM Alexandre Gacon 
wrote:

> Hello,
>
> I am using ogr2ogr to upload data from several geopackages to a postgis
> database. Some tables contain thousands of rows (buildings for example).
>
> The import of the first file is quite fast (tables are created for the
> first file so PG_USE_COPY is used) but the following file is much slower
> (using INSERT instead of COPY).
>
> How could I data insertion for the other files? Force PG_USE_COPY ?
> Increase the value of GT ? Postpone spatial index creation ?
>
> Should I concatenate all the geopackages first and then insert the data in
> Postgis?
>
> Thank you for your help
>
> --
> Alexandre Gacon
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] What's so wrong with this nc file ...

[Jan Heckman]

Hi list,
I am working with the LVBAG driver, for which I am truly grateful, by the
way, in C++.
I have a problem with OFTStringList and the corresponding
function GetAsStringList() and/or the cast operator(s) to
std::vector
In my code they produce an empty result.
switch (oField.GetType())
{
...
break;
case OFTStringList:
{
const std::string asString{ oField.GetAsString() };
cout << "Field type: " << oField.GetType() << ", string representation of
StringList: " << asString << endl;
const std::vector asList{ oField };
cout << "Vector length string list: " << asList.size() << endl;
}
break;
...
}
produces
(single item list)
Field type: 5, string representation of StringList:
(1:NL.IMBAG.Pand.051810310601)
Vector length string list: 0
(2 item list)
Field type: 5, string representation of StringList:
(2:woonfunctie,kantoorfunctie)
Vector length string list: 0
and similar when I use oField.GetAsStringList() or any other variant using
reference etc.
The library I use is gdal-303-1928-x64-core.msi obtained from gisinternals
supportsite.

I can parse the stringlist type, the strings will in this case not contain
comma's, but I appreciate a pointer
to achieving this without additional bother.
Thanks in advance,
Jan
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] modify zvalue in database when inserted using zfield option

[Jan Heckman]

Quite possibly in the Z-coordinate of your geometry column. You can check
in pgadmin or psql (or another program like omnidb) what the exact type of
the postgis geometry in your table is.
You should see something like geometry(LineStringZM,28992) what i.c. means
a dutch projection of a linestring with Z and M coordinates.
You can access these (outside gdal) only through postgis, like
select st_astext(st_pointn(polyline,1)) from ehrdtracks limit 1;
st_astext
--
 POINT ZM (71905.9 417771 1504 0)

Esp with points you can formulate an update query replacing the geometry
element with a condition,
using ST_Makepoint(), see https://postgis.net/docs/ST_MakePoint.html

Perhaps Qgis might help, too, but I can't tell for sure.

Hope this helps,
Jan

On Fri, Aug 13, 2021 at 10:34 PM Saulteau Don  wrote:

> I used ogr2ogr to append contours into a larger dataset. I opted for the
> zfield option using the ELEVATION field.
>
> I realized one of my contours has a bad value and need to modify it to
> match the other contours it is connected to.
>
> I can edit the attribute ELEVATION fine, but when I rerun a mesh tool it
> uses the zfield like its the old value instead of the modified one in the
> ELEVATION field.
>
> Where are the zvalues hidden in PostGIS to change them? Do I delete the
> old line and re-append after I've made the change? That's one option but I
> was wondering if I can just easily edit it inside postgis first.
>
>
>
> Donovan
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] unicode support in GDAL

[Jan Heckman]

Does the shapefile have a .cpg with UTF8 as spec?
Did you open the shape (or, in fact, the .dbf) in e.g.  Qgis?
You might try open it (the .dbf) in Libre office calc and specify the
encoding as UTF8.
If there's still ? or non-Russian where it should be, you are sure that the
shapefile (dbf) is not correct.
It may be quite ok and all you need is to supply the .cpg file.


On Thu, Aug 12, 2021 at 5:54 PM Claudiu Cochior via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hello everybody,
>
>
>
> I have a question related to Unicode strings in GDAL
>
>
>
> We are using GDAL 3.0.4 and at some point we would like to write to
> shapefile a string that contains Russian characters. To give you some
> context, we are in C++, we created a OGRFeature and we want to set a field
> to the string that contains the Russian characters. My machine is in
> English and the language for non-unicode programs is set to English. I
> didn’t find in the GDAL doc a definitive answer if GDAL support Unicode
> strings for filed values. As a test I converted the System::String to UTF-8
> but the shape file displays only ? for the characters. If I change the
> language for non-unicode programs to Russian then I can safely extract the
> ANSI string and give it to the SetField method and the result is OK.
>
>
>
> So, does GDAL supports unicede field string values somehow?
>
>
>
> Thanks,
>
>
>
>
>
> Claudiu
>
> --
>
>
> This email, including any attachments, may contain confidential and/or
> proprietary information intended only for the use of the recipient. If you
> are not the intended recipient, any distribution, copying, or use of this
> email or its attachments is prohibited. If you received this email in
> error, please reply to the sender immediately and delete this message and
> any copies.
>
> *Bentley Systems has taken all reasonable steps to ensure that this
> communication is free from viruses, data corruption, and unauthorized
> alteration. Bentley Systems does not accept liability for any damages that
> may be incurred as a result of this or any communication by email*
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Fwd: ogr2ogr crash attempting to drop table in filegdb

[Jan Heckman]

However, ogrinfo does no better.
It refuses to rename a table (filegdb) and crashes on drop table.
Jan

On Fri, Feb 5, 2021 at 11:43 AM jratike80 <
jukka.rahko...@maanmittauslaitos.fi> wrote:

> Hi,
>
> SQL in ogr2ogr is only for SELECT. For any other purpose use ogrinfo.
> An example:
>
> ogrinfo -sql "alter table point_p rename column test to foo" point.gpkg
> -dialect sqlite
>
> -Jukka Rahkonen-
>
>
> janhec wrote
> > Dear list,
> >
> > About the filegdb driver
> > Attempting:
> > ogr2ogr -f FILEGDB test.gdb bag2101.gdb -update -sql "drop table
> > standplaats"
> > crashed ogr2ogr (obtained from gisinternals.com,
> > gdal-302-1928-x64-core.msi
> > etc) with a heap corruption. (VS: Unhandled exception at
> > 0x7FFE1C6EF099
> > (ntdll.dll) in ogr2ogr.exe: 0xC374: A heap has been corrupted
> > (parameters: 0x7FFE1C7587F0).)
> > The table does get dropped.
> >
> > The table was inserted from a shapefile using:
> > ogr2ogr --config shape_encoding "UTF-8" --config FGDB_BULK_LOAD YES
> > -append
> > -f "filegdb" bag2101.gdb sta.shp -nln standplaats -sql "select identifica
> > as identificatie,hfdadresid as hoofdadresid,* from sta", and displays
> > properly in Qgis.
> >
> > So the filegdb driver appears to be properly installed.
> >
> > Also,  I could not get column renaming to work. Using dialects filegdb
> and
> > OGR SQL,
> > ogr2ogr -f FILEGDB test.gdb test.gdb -update -sql "alter table
> vbo_actueel
> > rename column oppervlakt to oppervlakte"  -dialect "FILEGDB"
> > gives ERROR 1: Failed at executing 'alter table vbo_actueel rename column
> > oppervlakt to oppervlakte' (An invalid SQL statement was used.);
> > using -dialect SQLITE, I get
> > ERROR 1: In ExecuteSQL(): sqlite3_prepare_v2(Alter table vbo_actueel
> > rename
> > column oppervlakt to oppervlakte): no such table: vbo_actueel
> > whereas ogrinfo clearly states the table exists.
> >
> > I am trying to clear my arcgis ties preferring Qgis and for reasons of
> > cost, but sometimes I do need to provide data in filegdb format.
> >
> > Thanks in advance for any advice,
> > Jan
> >
> > PS. I do not add the shapefile for reference, since that version of this
> > mail got "rejected by the owner" for being about 1500 bytes in size.
> >
> > ___
> > gdal-dev mailing list
>
> > gdal-dev@.osgeo
>
> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>
>
>
> --
> Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Fwd: ogr2ogr crash attempting to drop table in filegdb

[Jan Heckman]

Hi Jukka,
Shame on me. Had just found out.
thanks,
Jan

On Fri, Feb 5, 2021 at 11:43 AM jratike80 <
jukka.rahko...@maanmittauslaitos.fi> wrote:

> Hi,
>
> SQL in ogr2ogr is only for SELECT. For any other purpose use ogrinfo.
> An example:
>
> ogrinfo -sql "alter table point_p rename column test to foo" point.gpkg
> -dialect sqlite
>
> -Jukka Rahkonen-
>
>
> janhec wrote
> > Dear list,
> >
> > About the filegdb driver
> > Attempting:
> > ogr2ogr -f FILEGDB test.gdb bag2101.gdb -update -sql "drop table
> > standplaats"
> > crashed ogr2ogr (obtained from gisinternals.com,
> > gdal-302-1928-x64-core.msi
> > etc) with a heap corruption. (VS: Unhandled exception at
> > 0x7FFE1C6EF099
> > (ntdll.dll) in ogr2ogr.exe: 0xC374: A heap has been corrupted
> > (parameters: 0x7FFE1C7587F0).)
> > The table does get dropped.
> >
> > The table was inserted from a shapefile using:
> > ogr2ogr --config shape_encoding "UTF-8" --config FGDB_BULK_LOAD YES
> > -append
> > -f "filegdb" bag2101.gdb sta.shp -nln standplaats -sql "select identifica
> > as identificatie,hfdadresid as hoofdadresid,* from sta", and displays
> > properly in Qgis.
> >
> > So the filegdb driver appears to be properly installed.
> >
> > Also,  I could not get column renaming to work. Using dialects filegdb
> and
> > OGR SQL,
> > ogr2ogr -f FILEGDB test.gdb test.gdb -update -sql "alter table
> vbo_actueel
> > rename column oppervlakt to oppervlakte"  -dialect "FILEGDB"
> > gives ERROR 1: Failed at executing 'alter table vbo_actueel rename column
> > oppervlakt to oppervlakte' (An invalid SQL statement was used.);
> > using -dialect SQLITE, I get
> > ERROR 1: In ExecuteSQL(): sqlite3_prepare_v2(Alter table vbo_actueel
> > rename
> > column oppervlakt to oppervlakte): no such table: vbo_actueel
> > whereas ogrinfo clearly states the table exists.
> >
> > I am trying to clear my arcgis ties preferring Qgis and for reasons of
> > cost, but sometimes I do need to provide data in filegdb format.
> >
> > Thanks in advance for any advice,
> > Jan
> >
> > PS. I do not add the shapefile for reference, since that version of this
> > mail got "rejected by the owner" for being about 1500 bytes in size.
> >
> > ___
> > gdal-dev mailing list
>
> > gdal-dev@.osgeo
>
> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>
>
>
> --
> Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Fwd: ogr2ogr crash attempting to drop table in filegdb

[Jan Heckman]

Dear list,

About the filegdb driver
Attempting:
ogr2ogr -f FILEGDB test.gdb bag2101.gdb -update -sql "drop table
standplaats"
crashed ogr2ogr (obtained from gisinternals.com, gdal-302-1928-x64-core.msi
etc) with a heap corruption. (VS: Unhandled exception at 0x7FFE1C6EF099
(ntdll.dll) in ogr2ogr.exe: 0xC374: A heap has been corrupted
(parameters: 0x7FFE1C7587F0).)
The table does get dropped.

The table was inserted from a shapefile using:
ogr2ogr --config shape_encoding "UTF-8" --config FGDB_BULK_LOAD YES -append
-f "filegdb" bag2101.gdb sta.shp -nln standplaats -sql "select identifica
as identificatie,hfdadresid as hoofdadresid,* from sta", and displays
properly in Qgis.

So the filegdb driver appears to be properly installed.

Also,  I could not get column renaming to work. Using dialects filegdb and
OGR SQL,
ogr2ogr -f FILEGDB test.gdb test.gdb -update -sql "alter table vbo_actueel
rename column oppervlakt to oppervlakte"  -dialect "FILEGDB"
gives ERROR 1: Failed at executing 'alter table vbo_actueel rename column
oppervlakt to oppervlakte' (An invalid SQL statement was used.);
using -dialect SQLITE, I get
ERROR 1: In ExecuteSQL(): sqlite3_prepare_v2(Alter table vbo_actueel rename
column oppervlakt to oppervlakte): no such table: vbo_actueel
whereas ogrinfo clearly states the table exists.

I am trying to clear my arcgis ties preferring Qgis and for reasons of
cost, but sometimes I do need to provide data in filegdb format.

Thanks in advance for any advice,
Jan

PS. I do not add the shapefile for reference, since that version of this
mail got "rejected by the owner" for being about 1500 bytes in size.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Driver maintenance - long-term solution ?

[Jan Heckman]

I have none of the financial/organisational knowledge that some of you
have, and this is mere googling.
With that being said, is
https://bigdatastack.eu/european-open-source-initiative a potential?
It has a participation link and explicitly includes individuals and not
only academia and companies.
Jan

On Fri, Jan 15, 2021 at 8:31 PM David Strip  wrote:

> Further evidence of the challenge of getting funds to support gdal from US
> gov't agencies:
> Someone at the USGS privately emailed me to confirm that like Sandia, they
> cannot support gdal through voluntary donations or in-kind contributions
> that are outside the scope of contracted project work. Over the years the
> USGS has contracted through open solicitations for specific projects that
> are returned to the community, but these contracts are driven by
> programmatic USGS requirements which may or may not have utility to the
> broader gdal community.
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Jan Heckman]

In good company when pruning the source trees:
https://www.phoronix.com/scan.php?page=news_item&px=2021-Linux-Drop-Old-CPUs
Apparently 68000 may not be supported any longer in linux...
Just joking. Hope you don't mind...

On Tue, Jan 12, 2021 at 3:46 AM Jed O. Kaplan 
wrote:

> I for one am using the GMT vector driver for GDAL on a regular basis,
> e.g., for integration between GRASS GIS and GMT for cartography. I am
> grateful to the developers for their continued support for this driver.
>
> Thanks, Jed
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Jan Heckman]

Hey guys, I was just typing pretty much the same thing.
So these drivers could be removed gracefully with a trace allowing
gis-archeologists/archivists to use them.
I can just see the papers in my imagination.
Jan

On Mon, Jan 11, 2021 at 1:03 AM Stephen Woodbridge <
stephenwoodbridg...@gmail.com> wrote:

> Even,
>
> This makes a lot of sense to me. How would you handle this in Python?
> Would it make sense to create a GDAL-removed repository and move stuff
> into it just so it is available if someone wants it. This would not be
> supported or updated by GDAL just making it available if someone
> wants/needs to fork it, something else like this?
>
> -Steve W
>
> On 1/10/2021 6:02 PM, Even Rouault wrote:
> > Hi,
> >
> > It's not spring yet, but I'm in a mood lately of axing useless things,
> and we
> > probably have tons of candidate for that in GDAL, especially in drivers.
> > I was going to just axe the DB2 driver
> > (https://github.com/OSGeo/gdal/pull/3366) but the issue is more general.
> >
> > Any idea how we can know what is used and what isn't ? A "call-home"
> > functionality where we would track driver usage would only be acceptable
> if
> > people enable it and have network connectivity, so we won't probably get
> lots
> > of feedback. Having a spreadsheet with the driver list and asking people
> to
> > fill it would probably also receive little feedback. So the idea I had
> was to
> > do something like the following in the Open() method of a candidate for
> > removal:
> >
> > GDALDataset* FooDriver::Open(  )
> > {
> > if( !Identify(poOpenInfo) )
> >return nullptr;
> >
> > if( !CPLTestBool(CPLGetConfigOption("GDAL_ENABLE_DRIVER_FOO", "NO") )
> > {
> > CPLError(CE_Failure, CPLE_AppDefined,
> >  "Driver FOO is considered for removal in GDAL 3.5. You are invited "
> >  "to convert any dataset in that format to another more common one ."
> >  "If you need this driver in future GDAL versions, create a ticket
> at "
> >  "https://github.com/OSGeo/gdal (look first for an existing one
> first) to "
> >  "explain how critical it is for you (but the GDAL project may still
> "
> >  "remove it), and to enable it now, set the GDAL_ENABLE_DRIVER_FOO "
> >  "configuration option / environment variable to YES");
> > return nullptr;
> >  }
> >  ...
> > }
> >
> > That is, when we detect a file to be handled by the driver, emit the
> above
> > error message and do not open the dataset, unless the user defines the
> > environment variable.
> > Similarly in the Create()/CreateCopy() methods.
> > If we ship this in 3.3, with a 3.5 milestone for removal, this would
> offer a
> > feedback period of one year / 2 feature versions.
> >
> > Here's my own list of candidates for retirement (probably
> over-conservative).
> > Mostly based on gut feeling. None of them are particularly bad citizens,
> but I
> > have no indication that they are still used, which doesn't mean they
> aren't.
> >
> > * Raster side:
> > BPG
> > DB2Raster
> > DOQ1
> > DOQ2
> > E00GRID
> > Epsilon
> > FujiBAS
> > GS7BG
> > GSAG
> > IDA
> > JDEM
> > JPEG2000 (Jasper): JP2OpenJPEG is a better replacement
> > JPEGLS
> > LAN
> > MFF
> > MG4Lidar ?
> > NDF
> > NTv1
> > SDTS Raster
> > SGI
> > XPM
> > ZMap
> >
> > * Vector side:
> > AERONAVFAA
> > ESRI ArcObjects
> > ARCGEN
> > BNA
> > Cloudant
> > CouchDB
> > DB2
> > DODS
> > FMEObjects Gateway
> > Geomedia MDB
> > GMT ASCII Vectors
> > GTM
> > HTF
> > INGRES
> > MongoDB (the old one, superseded by MongoDBv3)
> > OpenAIR
> > REC
> > SDTS
> > SUA
> > SVG
> > TIGER
> > WALK
> >
> >
> > Anything you'd add / remove ?
> >
> > What is not obvious is what would be the criterion for keeping a driver:
> 1,
> > 10, 100 users asking for the driver to be kept ?
> > If a GDAL developer contributing to the overall good of the project
> needs the
> > preservation of a driver to be able to justify its continued
> involvement, I'd
> > tend to think it to be enough to keep it.
> >
> >
> > Even
> >
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Jan Heckman]

A bit of googling sometimes gives an indication. I was a little confused
about SVG, not using it myself but getting quite a few references.
E.g. this one in stackexchange, with a mention how often it was looked up:
https://gis.stackexchange.com/questions/11476/importing-svg-into-gis
Just an idea. Obviously, it is not easy to decide exactly what to do with
this sort of information.
Jan

On Mon, Jan 11, 2021 at 12:02 AM Even Rouault 
wrote:

> Hi,
>
> It's not spring yet, but I'm in a mood lately of axing useless things, and
> we
> probably have tons of candidate for that in GDAL, especially in drivers.
> I was going to just axe the DB2 driver
> (https://github.com/OSGeo/gdal/pull/3366) but the issue is more general.
>
> Any idea how we can know what is used and what isn't ? A "call-home"
> functionality where we would track driver usage would only be acceptable
> if
> people enable it and have network connectivity, so we won't probably get
> lots
> of feedback. Having a spreadsheet with the driver list and asking people
> to
> fill it would probably also receive little feedback. So the idea I had was
> to
> do something like the following in the Open() method of a candidate for
> removal:
>
> GDALDataset* FooDriver::Open(  )
> {
>if( !Identify(poOpenInfo) )
>   return nullptr;
>
>if( !CPLTestBool(CPLGetConfigOption("GDAL_ENABLE_DRIVER_FOO", "NO") )
>{
>CPLError(CE_Failure, CPLE_AppDefined,
> "Driver FOO is considered for removal in GDAL 3.5. You are invited "
> "to convert any dataset in that format to another more common one ."
> "If you need this driver in future GDAL versions, create a ticket at "
> "https://github.com/OSGeo/gdal (look first for an existing one first)
> to "
> "explain how critical it is for you (but the GDAL project may still "
> "remove it), and to enable it now, set the GDAL_ENABLE_DRIVER_FOO "
> "configuration option / environment variable to YES");
>return nullptr;
> }
> ...
> }
>
> That is, when we detect a file to be handled by the driver, emit the above
> error message and do not open the dataset, unless the user defines the
> environment variable.
> Similarly in the Create()/CreateCopy() methods.
> If we ship this in 3.3, with a 3.5 milestone for removal, this would offer
> a
> feedback period of one year / 2 feature versions.
>
> Here's my own list of candidates for retirement (probably
> over-conservative).
> Mostly based on gut feeling. None of them are particularly bad citizens,
> but I
> have no indication that they are still used, which doesn't mean they
> aren't.
>
> * Raster side:
> BPG
> DB2Raster
> DOQ1
> DOQ2
> E00GRID
> Epsilon
> FujiBAS
> GS7BG
> GSAG
> IDA
> JDEM
> JPEG2000 (Jasper): JP2OpenJPEG is a better replacement
> JPEGLS
> LAN
> MFF
> MG4Lidar ?
> NDF
> NTv1
> SDTS Raster
> SGI
> XPM
> ZMap
>
> * Vector side:
> AERONAVFAA
> ESRI ArcObjects
> ARCGEN
> BNA
> Cloudant
> CouchDB
> DB2
> DODS
> FMEObjects Gateway
> Geomedia MDB
> GMT ASCII Vectors
> GTM
> HTF
> INGRES
> MongoDB (the old one, superseded by MongoDBv3)
> OpenAIR
> REC
> SDTS
> SUA
> SVG
> TIGER
> WALK
>
>
> Anything you'd add / remove ?
>
> What is not obvious is what would be the criterion for keeping a driver:
> 1,
> 10, 100 users asking for the driver to be kept ?
> If a GDAL developer contributing to the overall good of the project needs
> the
> preservation of a driver to be able to justify its continued involvement,
> I'd
> tend to think it to be enough to keep it.
>
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] exporting all tables from PG schema

[Jan Heckman]

That's cool.
Had not yet run into this, but was probably going to pretty soon.
Thanks!

On Tue, Dec 29, 2020 at 6:28 PM jratike80 <
jukka.rahko...@maanmittauslaitos.fi> wrote:

> Hi,
>
> I would have a try with open option LIST_ALL_TABLES=YES
> https://gdal.org/drivers/vector/pg.html.
> "LIST_ALL_TABLES=YES/NO: This may be “YES” to force all tables, including
> non-spatial ones, to be listed." It is probably worth reading
> https://gdal.org/drivers/vector/gpkg.html as well, about the layer
> creation
> options and aspatial tables.
>
>
> -Jukka Rahkonen-
>
>
>
>
> ghtmtt wrote
> > Hi all,
> >
> > I'd like to export all the table of a PG schema to either GPKG or a
> > shapefiles.
> >
> > In the schema I've a lot of non-spatial tables and with something like:
> >
> > ogr2ogr -f GPKG output.gpkg "PG:dbname=mydb schemas=myschema user=myuser
> > password=mypws host=myhost" -progress
> >
> > the output is generated but it only contains geometric tables. I
> > couldn't find a parameter to force the export of also non-spatial tables.
> >
> > What I'd like to avoid is to loop on every single layer
> >
> > Cheers and thanks
> >
> > Matteo
> > ___
> > gdal-dev mailing list
>
> > gdal-dev@.osgeo
>
> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>
>
>
> --
> Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] ogr2ogr (I)LIKE not caseINsensitive

[Jan Heckman]

Hi Even,
I'll upgrade to get rid of the error.
Thanks  again.
Jan

On Thu, Nov 28, 2019 at 5:49 PM Even Rouault 
wrote:

> On jeudi 28 novembre 2019 17:40:34 CET Jan Heckman wrote:
> > That's
> > ogr2ogr plangebieden.shp -t_srs EPSG:28992 -dim 2 -spat 45000 405000
> 14
> > 485000 WFS:"http://afnemers.ruimtelijkeplannen.nl/afnemers2012/services";
> > Bestemmingsplangebied -lco ENCODING=UTF-8 -where
> > "BeleidsmatigVerantwoordelijkeOverheid LIKE '_eme%' AND planstatus NOT
> LIKE
> > 'voor%' AND planstatus NOT LIKE 'wijzig%' AND planstatus NOT LIKE
> > 'concept%' AND planstatus NOT LIKE 'geconso%' AND planstatus NOT LIKE
> > 'ontwerp%' AND datum > '2006' AND historisch=0 AND digitaalgewaarmerkt
> LIKE
> > 'ja%' AND dossierstatus <> 'vervallen' AND Naam NOT LIKE '%parke%' AND
> Naam
> > NOT LIKE '%paraplu%' AND Naam NOT LIKE '%Parke%' AND Naam NOT LIKE
> > '%Paraplu%'"
>
> I cannot reproduce a ERROR 1: Destination buffer too small. with GDAL
> master.
> I suspect this might have been fixed although I've no particular
> recollection
> of such a fix having been done.
> I did get however a
> ERROR 1: Failed to parse date '2006' evaluating OGR WHERE expression
> which can be fixed by using '2006/01/01' instead of '2006'
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] ogr2ogr (I)LIKE not caseINsensitive

[Jan Heckman]

That's
ogr2ogr plangebieden.shp -t_srs EPSG:28992 -dim 2 -spat 45000 405000 14
485000 WFS:"http://afnemers.ruimtelijkeplannen.nl/afnemers2012/services";
Bestemmingsplangebied -lco ENCODING=UTF-8 -where
"BeleidsmatigVerantwoordelijkeOverheid LIKE '_eme%' AND planstatus NOT LIKE
'voor%' AND planstatus NOT LIKE 'wijzig%' AND planstatus NOT LIKE
'concept%' AND planstatus NOT LIKE 'geconso%' AND planstatus NOT LIKE
'ontwerp%' AND datum > '2006' AND historisch=0 AND digitaalgewaarmerkt LIKE
'ja%' AND dossierstatus <> 'vervallen' AND Naam NOT LIKE '%parke%' AND Naam
NOT LIKE '%paraplu%' AND Naam NOT LIKE '%Parke%' AND Naam NOT LIKE
'%Paraplu%'"

On Thu, Nov 28, 2019 at 4:58 PM Even Rouault 
wrote:

> On jeudi 28 novembre 2019 16:42:19 CET Jan Heckman wrote:
> > Hi Even, all,
> > Thanks for the fast and helpful response!
> > So, if I need this shortly, I should compile a slightly modified version?
> > I can't see myself giving a true solution to the backward compatibility
> > issue ;)
> > My problem here is not large.
> > The main advantage of caseinsensitive is a compact and accurate query, I
> > was used to seeing if I could just omit the first (capital or not)
> > character.
>
> Try the changeset at https://github.com/OSGeo/gdal/pull/2065
>
> > But I ran into a case where this produced unwanted results and cmdline
> > gave ERROR 1: Destination buffer too small.
>
> Can you share how to reproduce that? That should be fixed
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] ogr2ogr (I)LIKE not caseINsensitive

[Jan Heckman]

Hi Even, all,
Thanks for the fast and helpful response!
So, if I need this shortly, I should compile a slightly modified version?
I can't see myself giving a true solution to the backward compatibility
issue ;)
My problem here is not large.
The main advantage of caseinsensitive is a compact and accurate query, I
was used to seeing if I could just omit the first (capital or not)
character.
But I ran into a case where this produced unwanted results and cmdline
gave ERROR 1: Destination buffer too small.
So I'll put the less compact SQL in a file and use -sql @file.sql for now.
Thanks,
Jan

On Thu, Nov 28, 2019 at 4:25 PM Even Rouault 
wrote:

> On jeudi 28 novembre 2019 16:09:28 CET Jan Heckman wrote:
> > Hi everybody,
> > According to docs <https://gdal.org/user/ogr_sql_dialect.html>, LIKE and
> > ILIKE should be caseinsensitive.
> > However, they are not.
> > ogr2ogr plangebieden.shp -t_srs EPSG:28992 -dim 2 -spat 45000 405000
> 14
> > 485000 WFS:"http://afnemers.ruimtelijkeplannen.nl/afnemers2012/services";
> > Bestemmingsplangebied -lco ENCODING=UTF-8 -where
> > "BeleidsmatigVerantwoordelijkeOverheid LIKE 'Geme%'" produces empty
> output.
> > The clue is that LIKE 'geme%'  (as in the DB 'gemeente'). does produce
> > output.
> > Replacing LIKE with ILIKE makes no difference.
>
> Jan,
>
> That's an interesting case.
>
> OGR SQL indeeds consider LIKE and ILIKE are identical and case insensitive.
>
> But... when issuing a -where to a WFS source, OGR turns the SQL into OGC
> Filter Encoding so that the filter is evaluated by the server for
> performance
> reasons. When doing so it uses the PropertyIsLike predicate, which is case
> sensitive by default. There's a matchCase attribute that could be set to
> false.
>
> So I can see 2 options:
>
> 1) change the WFS translation code to set matchCase=false. Quite easy to
> implement. Some backward compatibility consequence for users of WFS
> sources
> that would have dependended on the current behaviour (case sensitivity)
>
> 2) more work. for WFS, have LIKE and ILIKE having two different
> translations.
> But if we do so, as we would need to add a new SWQ_ILIKE operator, and we
> should also probably make LIKE and ILIKE be distinct operations on a pure
> OGR
> SQL evaluation (client side). Which has some backward compatibility
> implication.
>
> Even
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] ogr2ogr (I)LIKE not caseINsensitive

[Jan Heckman]

Hi everybody,
According to docs , LIKE and
ILIKE should be caseinsensitive.
However, they are not.
ogr2ogr plangebieden.shp -t_srs EPSG:28992 -dim 2 -spat 45000 405000 14
485000 WFS:"http://afnemers.ruimtelijkeplannen.nl/afnemers2012/services";
Bestemmingsplangebied -lco ENCODING=UTF-8 -where
"BeleidsmatigVerantwoordelijkeOverheid LIKE 'Geme%'" produces empty output.
The clue is that LIKE 'geme%'  (as in the DB 'gemeente'). does produce
output.
Replacing LIKE with ILIKE makes no difference.
ogr2ogr --version: GDAL 2.3.0dev, released 2017/99/99
Is there a way to use a LIKE like caseinsensitively in an ogr2ogr query?
Thanks,
Jan
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Numeric accuracy

[Jan Heckman]

Hi Ari and all,

why not use machine epsilon ?
somewhat over 1e-16 for double precision, or an even more tolerant
tolerance? Most cases of meter coordinates 0.001 is realistic.
Given projection errors, a bit more is also defensible.

Am I to simplistic?

On Sat, Nov 11, 2017 at 11:05 AM, Ari Jolma  wrote:

> I have been bitten twice, once with ArcGIS and now with Rasdaman WCS, with
> numeric accuracy.
>
> I'm making a data request on the corner of the bounding box, let's say it
> has minimum X of 75042.7273594. I'm setting my minX to that value and I'm
> enforcing it to that value with MAX (this was introduced because of a case
> with ArcGIS). The I print that to the request with "%.18g" (it was "%.15g"
> earlier but I changes it to that because of ArcGIS) and the result is
> 75042.727359398, which is not good for Rasdaman, since it is formally
> less than 75042.7273594. Although, in gdb
>
> (gdb) p 75042.7273594 > 75042.72735939
> $7 = false
>
> Any ideas how to detect/prevent these kinds of situations? Keep the checks
> and go back to "%.15g"? Not ok with ArcGIS.
>
> Ari
>
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Improve clipping of shapefiles

[Jan Heckman]

Can't really help you with the OGR part, however, I wrote utilities do
solve this (really) fast for calculating grids based on this type of
intersection.
Speed is often a matter of using specific tools utilizing the
characteristics of the problem, i.c. the fishnet regularities.
Can you share the shapesfiles so I can try?
Regards,
Jan

On Mon, Sep 11, 2017 at 11:09 AM, Paul Meems  wrote:

> I have a large shapefile with over 2.8 million shapes (fishnet) and I have
> a border file with only 1 multipolygon.
>
> I'm trying to clip the fishnet with the border.
> Using code is takes about 5 min. using command line it takes even longer.
>
> This is my command:
> ogr2ogr fishnetClipped.shp fishnet.shp -clipsrc border.shp
>
> And this is my C# code:
> using (var dsFishnet = Ogr.Open(fishnetFilename, 0))
> {
> // Select first layer:
> using (var layerFishnet = dsFishnet.GetLayerByIndex(0))
> {
> // Open border:
> using (var dsBorder = Ogr.Open(borderFilename, 0))
> {
> // Select first layer:
> using (var layerBorder = dsBorder.GetLayerByIndex(0))
> {
> // Get ESRI driver:
> using (var driver = Ogr.GetDriverByName("ESRI Shapefile"))
> {
> // Create new shapefile:
> using (var dsClipped = 
> driver.CreateDataSource(outputFilename,
> new string[] { }))
> {
> // Create new layer:
> var layerName = Path.GetFileNameWithoutExtension(
> outputFilename);
> using (var layerClipped = 
> dsClipped.CreateLayer(layerName,
> layerFishnet.GetSpatialRef(), layerFishnet.GetGeomType(), new string[] { }))
> {
> if (layerClipped == null)
> {
> throw new Exception("Layer creation
> failed.");
> }
>
> // Clip the layer:
> layerFishnet.Clip(layerBorder, layerClipped,
> null, null, null);
> }
> }
> }
> }
> }
> }
> }
>
> I'm using GDAL 2.1.3
>
> Can I somehow improve the speed?
> I read something about putting one of the layers in memory or using a
> spatial filter.
> But I can't find an example of how to do this.
>
> Please advise.
>
> Thanks,
> Paul
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Raster calculation optimalisation

[Jan Heckman]

Maybe it does not compile the formula but runs parser etc. as a kind of
interpreter at each function call.
Googled and saw http://beltoforion.de/article.php?a=muparsersse. Did not
try, so no recommendation, let alone guarantee ;)
You could check by defining a test function explicitly, if it is much
faster when compiled directly, that should be it, I guess.
Good luck,
Jan

On Tue, Jun 13, 2017 at 1:20 PM, Paul Meems  wrote:

> I'm using GDAL v2.1.3 with the SWIG bindings in my custom C# application.
>
> I've created a method to do some calculation of my raster file:
> public bool GdalCalculate(string input, string output, string formula,
> double? minValue = null)
> {
> if (!File.Exists(input))
> throw new FileNotFoundException("Can't find the input file", new
> Exception("Working with " + input));
>
> // 1. First copy the input file to create the output file
> try
> {
> /* 
> 
> */
> /*  Get driver
>  */
> /* 
> 
> */
> using (var drv = Gdal.GetDriverByName("GTiff"))
> {
> if (drv == null)
> {
> throw new Exception("Can't get GTiff driver");
> }
>
> /* --
> -- */
> /*  Open dataset.
>   */
> /* --
> -- */
> using (var ds = Gdal.Open(input, Access.GA_ReadOnly))
> {
> if (ds == null)
> {
> throw new Exception("Can't open GDAL dataset: " +
> input);
> }
>
> var options = new[] { "" };
> using (var newDataset = drv.CreateCopy(output, ds, 0,
> options, null, "Sample Data"))
> {
> if (newDataset == null)
> {
> throw new Exception("Can't create destination
> dataset: " + output);
> }
>
> // Close input dataset:
> ds.Dispose();
>
> // 2. Loop through all pixels and perform formula on
> it:
> using (var band = newDataset.GetRasterBand(1))
> {
> double noData;
> int hasValue;
> band.GetNoDataValue(out noData, out hasValue);
> var sizeX = band.XSize;
> var numLines = band.YSize;
> var func = new Function("f(A) = " + formula);
>
> // Loop through each line in turn.
> for (var line = 0; line < numLines; line++)
> {
> var scanline = new float[sizeX];
> // Read in data for the current line
> var cplReturn = band.ReadRaster(0, line,
> sizeX, 1, scanline, sizeX, 1, 0, 0);
> if (cplReturn != CPLErr.CE_None)
> {
> throw new Exception("band.ReadRaster
> failed: " + Gdal.GetLastErrorMsg());
> }
>
> var outputLine = new List();
> foreach (var f in scanline)
> {
> var pixelValue = f;
> if ((float)f != (float)noData)
> {
> // Calculate
> pixelValue = (float)func.calculate(f);
> if (minValue.HasValue)
> pixelValue =
> (float)Math.Max(pixelValue, minValue.GetValueOrDefault());
> }
> outputLine.Add(pixelValue);
> }
>
> // Rewrite line:
> cplReturn = band.WriteRaster(0, line, sizeX,
> 1, outputLine.ToArray(), sizeX, 1, 0, 0);
> if (cplReturn != CPLErr.CE_None)
> {
> throw new Exception("band.WriteRaster
> failed: " + Gdal.GetLastErrorMsg());
> }
> }
>
> // 3. Save changes to file:
> band.FlushCache();
> newDataset.FlushCache();
> }
> }
> }
> }
> }
> catch (Exception e)
> {
> throw new Exception("Can't open GDAL dataset: " + input, e);
> }
>
> return true;
> }
>
> This method is working fine, but is very slow.
> I'm using a 

Re: [gdal-dev] SQLite driver hit and miss with spatial index

[Jan Heckman]

Hi,
Out of interest, I tried the tack of converting to shapefile with a .dbf
only containing the primary index OGC_FID.
This might provide an alternative for indexed access to the geometry. I see
it isn't needed anymore, but anyway,
for info and comparison:
What worked:
- shapefile record count 25201411 agrees with sqlite table.
- shapefile size (.shp): 672 MB (recordcount * 28 + 100).
- shapefile index (.sbn): 194 MB. That's quite a bit more compact than the
spatialite index?
- apparently immediate response when zooming in or inquiring by clicking
(did not measure response time accurately)
What went wrong:
- could not select the primary index for conversion using ogr2ogr ...
-select OGC_FID ...
- Got incomplete conversion when using spatialite_gui with spatiaLite
version 4.3.0.a, conversion to DBF.
  recordcount of the conversion: 19927604

This using OSGEOW prompt of Qgis 2.18.0 (with GDAL 2.1.1, released
2016/07/07), on a windows 7 machine.

Perhaps worthwhile to use the (no-index.sqlite) dataset to look into the
conversion.
Error trying to convert to shapefile with OGC_FID:
F:\>ogr2ogr -f "ESRI Shapefile" -nlt POINT -select OGC_FID test.shp
j:/downloads/no-index.sqlite
ERROR 1: Field 'OGC_FID' not found in source layer.
ERROR 1: Terminating translation prematurely after failed
translation of layer osmi_addresses_nodes_with_addresses (use -skipfailures
to skip errors)

Is the OGC_FID primary index identical to a shapefile FID, that is, not in
the table, but really a counter?
So, I'm not sure I can properly link points to the sqlite data, but
otherwise it looks doable.
Jan

On Thu, Apr 6, 2017 at 2:30 PM, Frederik Ramm  wrote:

> Hi,
>
>while what I wrote in the previous post is still interesting and a
> bit strange, I have meanwhile upgraded to GDAL 2 packages from the
> ubuntugis-unstable repository for Xenial, and now the behaviour is as I
> would expect it: The ogr2ogr -spat command *with* indexes takes
> practically no time (0.05 seconds), and the -spat command *without*
> index takes about 40 seconds as before.
>
> Sorry for the noise then.
>
> Bye
> Frederik
>
> --
> Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Release of a GDAL 2.0.x and 2.1.x Visual Studio 2010 Solution (static and dynamic lib)

[Jan Heckman]

Hi Mateusz,
Thanks for your effort! It is really handy for me.
FYI, a few more things I came across:
- In the dll post build event there are 3 copy commands (dll, lib and exp).
The second and third command need a space before 'copy'.
- Enclosing source and destination file macros in quotes helps.
- in 64 bits compilation (VS2015 community update 1) the link fails with a
C1001 (internal compiler error). This is incurred by gdalexif.cpp, which is
not to blame btw.
  The 'solution' is to turn off compiler optimization for this file only
(and only for x64 compilation). Have not checked whether this happens e.g.
in VS2013.
- diminishing the number of compiler warnings:
 - /D _CRT_SECURE_NO_WARNINGS, if the secure versions are not going to be
used anyway (portability?).
 - change deprecated POSIX (acc. to VS), such as open, read, write, close,
fdopen, setmode to underscored names (_open, _read etc.)
   Not sure whether this is exclusively MS.
- warnings which are somewhat worryable: C4251, when compiling or using
DLL: X used in clas Y needs dll-interface.
  article  claims to solve
this, but not too well, imo, because the proposed solution then causes the
same warning..
  Advice (e.g. gamedev mails
)
is
not to export whole classes but only public members and ctor - dtor. More
to be found here

.
  Claims that some STL (esp vector) can be safely exported, but not all
(map has (or at least had) a static member).
I've never come to grief ignoring C2451, so far.
Jan

On Thu, Aug 4, 2016 at 4:07 PM, Mateusz Loskot  wrote:

> On 4 August 2016 at 15:41, Ivan Lucena  wrote:
> > Hi Mateusz,
> >
> > I far as I know the project generated by makegdal_gen.bat does not work
> with newer version of VS.
> >
> > It used to be the case that we can import the project to newer version
> of VS but I think that MS dropped the support for that feature.
> >
> > Have one of you manage to load the generated .vcproj into VS > 2010? I
> would like to know how to do that!
>
> Ivan,
>
> 1. makegdal_gen 10.00 32 > makegdal10.vcproj
> 2. Launch VS2015
> 3. File > Open > Project/Solution...
> 4. Select makegdal10.vcproj
> 5. Accept to upgrade
>
> I can confirm it works for me.
>
> Best regards,
> --
> Mateusz Loskot, http://mateusz.loskot.net
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Release of a GDAL 2.0.x and 2.1.x Visual Studio 2010 Solution (static and dynamic lib)

[Jan Heckman]

Hi,
Great stuff here!
A minor point: please add quotes to the copy path in post-build event.
Often people (like me) will have spaces in the project path,
e.g. C:\Users\jan\Documents\Visual Studio
2015\Projects\GDAL_2.1.x_VC-master\libgdal-2.1.1.
copy
$(Projectdir)$(PlatformShortName)\$(Configuration)\$(TargetName)$(TargetExt)
$(SolutionDir)binaries\lib\$(PlatformShortName)\$(Configuration)\$(TargetName)$(TargetExt)
becomes
copy
"$(Projectdir)$(PlatformShortName)\$(Configuration)\$(TargetName)$(TargetExt)"
"$(SolutionDir)binaries\lib\$(PlatformShortName)\$(Configuration)\$(TargetName)$(TargetExt)"
Can confirm it works fine in VS2015 and full lists of (re)source and header
files are indeed present. Neat!
Jan

On Fri, Aug 5, 2016 at 1:30 PM, Jeff McKenna 
wrote:

> On 2016-08-04 6:44 PM, Mateusz Loskot wrote:
>
>>
>>> Can we make sure to document these steps, so they are not lost in emails?
>>> Somewhere on the BuildHints wiki would be great.  Possibly here?
>>> https://trac.osgeo.org/gdal/wiki/BuildingOnWindows
>>>
>>
>> https://trac.osgeo.org/gdal/wiki/BuildingOnWindows#Generatin
>> gVisualStudioproject
>>
>>
> Thanks.
>
> -jeff
>
>
>
> --
> Jeff McKenna
> MapServer Consulting and Training Services
> http://www.gatewaygeomatics.com/
>
>
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Interoperability issues with deleted features in shapefiles

[Jan Heckman]

Hi Even, everyone,
Sorry for not including the list - my mistake.
I've experimented with larger shapefiles than 2 GB but not necessarily in
combination with editing.
I'll do a few tests when I get around to it. Doesn't the .shx file get
rewritten anyway? There could be some time-consuming actions at closing
time partially masking the additional time needed for shp.
Time needed for compacting the .shp would have a considerable potential
variation depending on the extent of editing and the displacement caused in
the shapefile.
My first idea was not to compact the shapefile (automatically), but do the
.shx only (leaving out indexes of deleted shp records or setting their
length in the shx and/or shp to zero). But there are some programs which do
not pay much attention to the .shx anyway. If we can discount such
behaviour, the shx route is ok, and a compact can be done as a separate
action, like PACK in good old Dbase.
Jan


On Wed, Jan 20, 2016 at 11:51 AM, Even Rouault 
wrote:

> Jan,
>
> Do you mind sharing your opinion with the list too ?
>
> > Hi,
> > I started a bit of a lib years and years ago when the shapelib code
> didn't
> > have delete.
> > I implemented my own delete much as you mention (both dbf and shp/shx),
> > with a repack at closing.
> > Repack at closing never bothered me in the sense of any (very) noticeable
> > delay.
> > So I think it's indeed the best solution and the price is not high.
>
> Depends on the size of shapefiles. For people with 2 GB shapefiles, that
> might
> be noticeable. But editing operations on such shapefiles aren't necessarily
> very common admitedly.
>
> > Regards,
> > Jan
> >
> > On Wed, Jan 20, 2016 at 12:42 AM, Even Rouault <
> even.roua...@spatialys.com>
> >
> > wrote:
> > > Hi,
> > >
> > > There have been some recent discussion on the qgis list about an old
> > > ticket https://hub.qgis.org/issues/11007
> > >
> > > Basically the issue seems to be that a lot / most non-shapelib /
> non-OGR
> > > based
> > > shapefile readers don't understand the way OGR delete features in
> > > shapefiles.
> > >
> > > When OGR/shapelib deletes a feature, it simply marks the corresponding
> > > record
> > > in the DBF as deleted (technically putting a '*' character in the first
> > > byte of
> > > the DBF record) and that's all. Very fast and OGR handles that
> > > consistently (with the small restriction that the feature count reports
> > > the deleted features as still existing, but iteration or getting
> > > features by id do not report them)
> > >
> > > This way of deleting a DBF record is the documented one :
> > > http://www.clicketyclick.dk/databases/xbase/format/dbf.html#DBF_STRUCT
> > > """
> > > Deleted flag:
> > > Value   Description
> > > 2Ah (*) Record is deleted
> > > 20h (blank) Record is valid
> > > """
> > >
> > > However other GIS packages, and among others, a famous proprietary one
> -
> > > let's
> > > call it "LineGIS" - when reading such shapefiles do not recognize the
> > > deleted
> > > feature as deleted and display both the geometry and attributes. More
> > > annoying, when "LineGIS" deletes another record in such a shapefile and
> > > saves
> > > the result, the shapefile can no longer be opened afterwards with an
> > > error message reporting an inconsistency in number of shapes w.r.t
> > > number of records
> > > (and on inspection, the shp/shx indeed contain N - 1 records and the
> dbf
> > > N -
> > > 2, so it looks like it would be semi-aware of deleted DBF records)
> > > When "LineGIS" starts with a "clean" shapefile and deletes a record in
> > > it, it
> > > removes the corresponding entries in the .dbf, .shp and .shx files,
> which
> > > is
> > > the result of the REPACK operation the shapefile driver can do if
> > > explicitly
> > > asked.
> > >
> > > "LineGIS" isn't the only one to have troubles with deleted DBF records.
> > > From
> > > what I can see GeoTools (just picking a random example) only fully
> handle
> > > them
> > > since 2014 :
> > > https://osgeo-org.atlassian.net/browse/GEOT-4539
> > >
> > >
> https://github.com/geotools/geotools/commit/e7333ccb284d137f3240ce5d0d09b
> > > 3d7195f1890
> > >
> > > The shapefile specification (
> > > http://www.esri.com/library/whitepapers/pdfs/shapefile.pdf ) doesn't
> > > mention
> > > about how deleted records should be handled. Particularly if the
> > > requirement
> > > "The table must contain one record per shape feature" (page 25) allows
> > > DBF records marked as deleted... Anyway the theory/spec and the
> practice
> > > are 2 different things.
> > >
> > > What surprises me is such an issue didn't raise more loud complaints
> > > before as
> > > the OGR / shapelib behaviour has been the same since forever AFAIK.
> > >
> > > I'm wondering if OGR shouldn't automatically run REPACK when closing a
> > > shapefile when deletions (as well as edit operations of existing
> features
> > > leading to holes in the .shp) have happened. The side effect of this
> > > would be

Re: [gdal-dev] Reading shapefile PRJ failing

[Jan Heckman]

afaik PRJ files should be pure (ascii)text. You could refer to getting rid
of BOM

to
clean them out.
Do not know an additional parameter for the file opener/reader.
Jan


On Sat, Jul 12, 2014 at 6:45 PM, mccorb  wrote:

> So I have tracked this down to the fact that the PRJ files that I have been
> given to use have a BOM of 0xEF 0xBB 0xBF for the first three bytes of the
> file.
>
> When I use the shapefiles that have these byte order markers the Layer
> class
> method getSpatialRef() (java JNI bindings) returns null. I am guessing that
> the PRJ file is being opened in binary mode instead of text mode possibly?
>
> Regardless, is there any setting I can make to coerce the file
> opener/reader
> into ignoring the BOM when grabbing the WKT from the PRJ file?
>
> thanks
>
>
>
> --
> View this message in context:
> http://osgeo-org.1560.x6.nabble.com/Reading-shapefile-PRJ-failing-tp5150778.html
> Sent from the GDAL - Dev mailing list archive at Nabble.com.
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] SBN shapefile indexing

[Jan Heckman]

Hi Even,

C/C++:
- lambda can probably be avoided by a (recursive) function and some
additional parameter passing.
- SBNIndex() uses a class internally. Only the function (SBNIndex()) is
exported. So I hope to retain this to avoid reprogramming.
 - exception(s) can be caught internally.
- I use std::vector which should be ok with VS7.1 (
http://msdn.microsoft.com/en-us/library/9xd04bzs(v=vs.71).aspx), gcc 4 (
https://gcc.gnu.org/onlinedocs/gcc-4.6.2/libstdc++/api/a00739.html) and
sort(). Vector can be rewritten to dynamic arrays and qsort(), but I don't
really want to.

- 'What do you mean': I meant the .sbn files created by my implementation
differ occasionally from the .sbn files Esri produces. Esri integrates bins
with few features with bins higher up in the tree, to some degree. The
differences I'm talking about are that in some cases features in my index
are 'pulled up' somewhat higher than Esri does. I cannot quite infer the
rules Esri follows in this. However, since the SBNSearch() routines follow
the tree, the features will be found; the consequences for index-efficiency
are a little hard to predict. Basically, there is a balance between bins
with a few features which may have a larger extent (for the entire bin), vs
bins with many features which should have a narrow extent. I expect the
efficiency difference to be marginal or unnoticeable.
- SBN 2D: As far as I know it's strictly 2D, but I've never found proof of
this, have to check Esri SBN index of an ZM shapefile.
- SHPReadObjectExtent() can be done by some easy copying and pasting from
SHPReadObject. We/you might consider using the new function in
SHPReadObject() for economy, but it would not be a big deal, since this
code is unlikely to change.

So, I can adapt my implementation, retaining C++ internally (class,
template, exceptions, std::vector and sort()) and exporting SBNIndex() as a
__cdecl/SHPAPI_CALL as if pure C; and implement SHPReadObjectExtent(), for
the time being within SBNIndex.cpp.

SBNIndex() would be called with the shapefilename (with or without
extension) and the shapefile would be assumed to be closed; an error when
opening the shape for (non-shared) reading would be returned to the caller.

Maybe I should add a searchroutine which just returns the (fid's of)
features in a specific bin to facilitate testing.

Jan

On Wed, Jun 18, 2014 at 7:48 PM, Even Rouault 
wrote:

> Le mercredi 18 juin 2014 18:44:17, Jan Heckman a écrit :
> > Hi all,
> >
> > Starting from https://github.com/drwelby/hasbeen I've implemented an SBN
> > shapefile indexing routine in C++. I'm happy, so far, with the results;
> > I've used it for a few weeks now.
> > With some work it could be integrated into OGR.
> > Question is, how much interest, considering pros and cons of my
> > implementation?
> >
> > On the plus side:
> > - except for occasional differences in the promotion level of lonely
> > features (see hasbeen), the results are identical to Esri. Differences do
> > not result in any features/shape being 'lost' or invisible. Achieves
> > identical results in the 'block_groups' shape example (see also
> > spatial_Idx_kit.zip
> > <
> https://code.google.com/p/pyshp/downloads/detail?name=spatial_idx_kit.zip&;
> > can=2&q=>). though.
> > - Esri AG can use the indexes, as well as SBNsearch.
> > - Fast.
> > - produces .sbx on request.
> >
> > On the minus side:
> > - uses a lambda function, so limits compilerchoice
>
> It would be best if we can avoid too fancy C++ features like that one
> (within
> GDAL code, anything more complicated than simple templates is fancy)
>
> > - compiled with VS2013, VS2012 probably is ok, too, no other compilers
> > tried, but there's no ms or intel specific or exotic stuff, really.
>
> We currently support compilers at least down to VC 7.1 (2003) and GCC 4.4.
>
> > - uses exceptions (at least from constructor).
>
> If exceptions are thrown, they should be caught somewhere within GDAL
> since it
> is callable from C. But ideally the code should go to shapelib which is C
> only, although it might be OK to keep C++ and GDAL only
>
> > - uses a simple template (internally).
> > - very frustrating I cannot achieve complete identity in spite of some
> > (serious) trying.
>
> What do you mean ?
>
> > - relies now on a tweaked version of SHPReadObject() which avoids reading
> > the vertices (just extents and ID).
>
> We want to keep the signature and behaviour of SHPReadObject() unchanged.
> So
> perhaps add a variant function for your behaviour : SHPReadObjectExtent() ?
> (possibly original SHPReadObject() and SHPReadObjectExtent() relying on an
> internal unique functio

[gdal-dev] SBN shapefile indexing

[Jan Heckman]

Hi all,

Starting from https://github.com/drwelby/hasbeen I've implemented an SBN
shapefile indexing routine in C++. I'm happy, so far, with the results;
I've used it for a few weeks now.
With some work it could be integrated into OGR.
Question is, how much interest, considering pros and cons of my
implementation?

On the plus side:
- except for occasional differences in the promotion level of lonely
features (see hasbeen), the results are identical to Esri. Differences do
not result in any features/shape being 'lost' or invisible. Achieves
identical results in the 'block_groups' shape example (see also
spatial_Idx_kit.zip
).
though.
- Esri AG can use the indexes, as well as SBNsearch.
- Fast.
- produces .sbx on request.

On the minus side:
- uses a lambda function, so limits compilerchoice
- compiled with VS2013, VS2012 probably is ok, too, no other compilers
tried, but there's no ms or intel specific or exotic stuff, really.
- uses exceptions (at least from constructor).
- uses a simple template (internally).
- very frustrating I cannot achieve complete identity in spite of some
(serious) trying.
- relies now on a tweaked version of SHPReadObject() which avoids reading
the vertices (just extents and ID).
- not a single thought about M and Z features (may or may not be relevant).
- has not been tested exhaustively.
- have not written a function to maintain an index structure during
editing. Doing so will certainly require work.

Please let me know whether being able to (SBN)index shapefiles from OGR,
even though the resulting indexfiles may differ slightly, is an attractive
prospect.

Jan
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Asking about feature field types of Shape File (SHP) ?

[Jan Heckman]

Not focusing on ogr, but on shapefile:
Shapefile uses .dbf (dBase III) to store attributes (except geometrical
attributes which are stored in the .shp file).
This format does not know about integers, only about numericals (type N or
F), with a specifiable precision (how many number-positions) and number of
decimals (behind the decimal separator), aka scale.
If this sounds textish, that is because it is: the numbers are stored in
the file as text. N as a fixed point double, F - in principle, but in fact
depending on implementation - uses scientific notation. When specifying
these values (when creating a field), you typically count the non-numerical
parts such as a - sign, the decimal separator and the exponent when using F
to specify precision. Exponent, by default, but hard to change, uses 3
digits, and E and possibly a - (negative exponent).
I hope that explains why there is not an effective type Integer for
shapefiles. There is no way for a (ogr)driver to invent one in the case of
 shapefiles. The type integer is applicable by extension because in  other
formats the distinction between numerical types is maintained more
rigorously. Number of decimals = 0 should do the trick, but ArcGis will
call it a double with some precision and scale 0. Can be used as integer
without side-effects, so far I've noticed.
Jan


On Mon, Jun 16, 2014 at 5:05 PM, Ari Jolma  wrote:

> On 06/04/2014 11:45 PM, taibc wrote:
>
>> Hi everyone,
>>
>> I created the shape file by usinge the feature filed type: OFTInteger.
>>
>> But when I open the file in Arcgis (ArcCatalog) the data type is: Double
>> (not integer).
>>
>> Do you know how to fix this (I want it is Int) ?
>>
>
> Dear Tai,
>
> This is a bit old email so I'll answer from my head. I believe you need to
> set the length of the Integer field to be short enough. If I remember
> correctly there is a ticket on the issue, but it is deemed as a feature,
> not a bug.
>
> Ari Jolma
>
>
>> Thanks and regards,
>> Tai
>>
>>
>>
>> --
>> View this message in context: http://osgeo-org.1560.x6.
>> nabble.com/Asking-about-feature-field-types-of-Shape-
>> File-SHP-tp5144127.html
>> Sent from the GDAL - Dev mailing list archive at Nabble.com.
>> ___
>> gdal-dev mailing list
>> gdal-dev@lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Difficulty finding all parts of shapefile polygon using Python and osgeo/ogr

[Jan Heckman]

Hi Bruce,
Fine, I liked looking into it as a way of refamiliarizing myself, so no
time wasted as far as I'm concerned.
Something that stuck in my mind is that the shapefile looks like a
contour-plot and that the cut-off of the little shape0 might not be
desired, at least not with the sort of minimal breach between the parts we
see in your example: the split could be regarded as an artifact.
If this is broadly correct in your case and you see the contouring as an
issue, maybe we could look into steering the contouring algorithm using
(speaking loosely) a minimal breach parameter. I've done something similar,
but it's a bit of time ago.
The problem I'm viewing occurs with narrow ridge- (or canyon-)like
structures, where, in view of the discretization, you could imagine a gap
in the ridge or a nice continuous case.
(The difference would be quite noticeable when you try to climb along the
ridge [?]). This is known in contouring as an issue of interpretation and
cannot be decided analytically without additional information, leading to a
case for supplying an additional input parameter to influence the result.
Best regards,
Jan


On Tue, May 20, 2014 at 5:54 AM, Bruce Raup  wrote:

> Jan,
>
> Thank you very much for spending time on my problem.  I ended up figuring
> out that the shapefile's structure consisted of a MULTIPOLYGON that
> contained two POLYGONs, one simple one (the small one you mentioned) and
> another with "holes".  I modified my code to account for the possibility of
> this case and it works great now.
>
> Thanks again, and best regards,
> Bruce Raup
>
>
> On Sun, May 18, 2014 at 6:19 AM, Jan Heckman wrote:
>
>> Bruce,
>> Attached is the digging - in C++, but should help for python as well.
>> I  used C++ for easier debugging, because there just might have been a
>> bug - but there wasn't.
>> Contains c++ source for reading the geometry of the testfile, the output
>> the program generates and the original zip of the nanga test shapefile.
>> Complexity of the geometric model follows from OGC specs.
>> But then, shapefiles support e.g. a collection of geometries (to be used
>> with a single set of attributes), as well.
>> (Nanga? Nanga Parbat? Couldn't make it out[?]).
>>  Jan
>>
>>
>> On Sun, May 18, 2014 at 12:53 AM, Jan Heckman wrote:
>>
>>> Hello Bruce,
>>> The point is that the geometrycount identifies two polygons, one a small
>>> little piece sort of inside a bay of your shape, the other the rest, along
>>> with the holes.
>>> See ogr_apitutorial <http://www.gdal.org/ogr/ogr_apitut.html> - all
>>> assuming you are using 1.11: *Starting with OGR 1.11, several geometry
>>> fields <http://trac.osgeo.org/gdal/wiki/rfc41_multiple_geometry_fields>** 
>>> can
>>> be associated to a feature*.
>>> The little hidden piece starts at (approx) 467767;3900845. It happens to
>>> be the first part of the shape.
>>> The example code in the link above should put you closer to all this;
>>> however, retrieving the polygon(part)s will take a bit more digging..
>>> Good luck,
>>> Jan
>>>
>>>
>>> On Thu, May 15, 2014 at 6:13 PM, Bruce Raup  wrote:
>>>
>>>> Hello,
>>>>
>>>> I have a Python script that reads a shapefile, converts holes in
>>>> polygons to top-level polygons, and writes a new polygon.  The new
>>>> top-level polygons inherit attributes from their parents, with optional
>>>> modifications of specific attributes.  For a simple test shapefile, the
>>>> output of shpdump shows what I mean:
>>>>
>>>> $ shpdump simple_shapes_with_holes.shp
>>>> Shapefile Type: Polygon   # of Shapes: 3
>>>>
>>>> File Bounds: (  -1.113,   0.146,0,0)
>>>>  to  (   0.751,   1.101,0,0)
>>>>
>>>> Shape:0 (Polygon)  nVertices=11, nParts=2
>>>>   Bounds:(  -1.113,   0.146, 0, 0)
>>>>   to (  -0.493,   0.723, 0, 0)
>>>>  (  -1.087,   0.151, 0, 0) Ring
>>>>  (  -1.113,   0.686, 0, 0)
>>>>  (  -1.109,   0.712, 0, 0)
>>>>  (  -0.506,   0.723, 0, 0)
>>>>  (  -0.493,   0.146, 0, 0)
>>>>  (  -1.087,   0.151, 0, 0)
>>>>+ (  -0.896,   0.346, 0, 0) Ring
>>>>  (  -0.682,   0.339, 0, 0)
>>>>  (  -0.684,   0.556, 0, 0)
>>>>  (  -0.909,   0.558, 0, 0)
>>>>  (  -0.896,   0.346,

Re: [gdal-dev] Difficulty finding all parts of shapefile polygon using Python and osgeo/ogr

[Jan Heckman]

Hello Bruce,
The point is that the geometrycount identifies two polygons, one a small
little piece sort of inside a bay of your shape, the other the rest, along
with the holes.
See ogr_apitutorial  - all
assuming you are using 1.11: *Starting with OGR 1.11, several geometry
fields ** can
be associated to a feature*.
The little hidden piece starts at (approx) 467767;3900845. It happens to be
the first part of the shape.
The example code in the link above should put you closer to all this;
however, retrieving the polygon(part)s will take a bit more digging..
Good luck,
Jan


On Thu, May 15, 2014 at 6:13 PM, Bruce Raup  wrote:

> Hello,
>
> I have a Python script that reads a shapefile, converts holes in polygons
> to top-level polygons, and writes a new polygon.  The new top-level
> polygons inherit attributes from their parents, with optional modifications
> of specific attributes.  For a simple test shapefile, the output of shpdump
> shows what I mean:
>
> $ shpdump simple_shapes_with_holes.shp
> Shapefile Type: Polygon   # of Shapes: 3
>
> File Bounds: (  -1.113,   0.146,0,0)
>  to  (   0.751,   1.101,0,0)
>
> Shape:0 (Polygon)  nVertices=11, nParts=2
>   Bounds:(  -1.113,   0.146, 0, 0)
>   to (  -0.493,   0.723, 0, 0)
>  (  -1.087,   0.151, 0, 0) Ring
>  (  -1.113,   0.686, 0, 0)
>  (  -1.109,   0.712, 0, 0)
>  (  -0.506,   0.723, 0, 0)
>  (  -0.493,   0.146, 0, 0)
>  (  -1.087,   0.151, 0, 0)
>+ (  -0.896,   0.346, 0, 0) Ring
>  (  -0.682,   0.339, 0, 0)
>  (  -0.684,   0.556, 0, 0)
>  (  -0.909,   0.558, 0, 0)
>  (  -0.896,   0.346, 0, 0)
>
> Shape:1 (Polygon)  nVertices=19, nParts=4
>   Bounds:(  -0.127,   0.159, 0, 0)
>   to (   0.751,   0.783, 0, 0)
>  (   0.402,   0.783, 0, 0) Ring
>  (   0.751,   0.159, 0, 0)
>  (  -0.127,   0.198, 0, 0)
>  (   0.402,   0.783, 0, 0)
>+ (   0.298,   0.539, 0, 0) Ring
>  (   0.417,   0.523, 0, 0)
>  (   0.426,   0.612, 0, 0)
>  (   0.339,   0.625, 0, 0)
>  (   0.298,   0.539, 0, 0)
>+ (   0.083,   0.263, 0, 0) Ring
>  (   0.248,   0.252, 0, 0)
>  (   0.213,   0.396, 0, 0)
>  (   0.107,   0.367, 0, 0)
>  (   0.083,   0.263, 0, 0)
>+ (   0.361,   0.281, 0, 0) Ring
>  (   0.530,   0.265, 0, 0)
>  (   0.435,   0.413, 0, 0)
>  (   0.330,   0.435, 0, 0)
>  (   0.361,   0.281, 0, 0)
>
> Shape:2 (Polygon)  nVertices=6, nParts=1
>   Bounds:(  -0.389,   0.681, 0, 0)
>   to (   0.141,   1.101, 0, 0)
>  (  -0.383,   1.005, 0, 0) Ring
>  (  -0.136,   1.101, 0, 0)
>  (   0.141,   0.964, 0, 0)
>  (  -0.154,   0.681, 0, 0)
>  (  -0.389,   0.809, 0, 0)
>  (  -0.383,   1.005, 0, 0)
>
>
>
>
> $ shpdump simple_shapes_with_holes_flattened.shp
> Shapefile Type: Polygon   # of Shapes: 7
>
> File Bounds: (  -1.113,   0.146,0,0)
>  to  (   0.751,   1.101,0,0)
>
> Shape:0 (Polygon)  nVertices=6, nParts=1
>   Bounds:(  -1.113,   0.146, 0, 0)
>   to (  -0.493,   0.723, 0, 0)
>  (  -1.087,   0.151, 0, 0) Ring
>  (  -1.113,   0.686, 0, 0)
>  (  -1.109,   0.712, 0, 0)
>  (  -0.506,   0.723, 0, 0)
>  (  -0.493,   0.146, 0, 0)
>  (  -1.087,   0.151, 0, 0)
>
> Shape:1 (Polygon)  nVertices=5, nParts=1
>   Bounds:(  -0.909,   0.339, 0, 0)
>   to (  -0.682,   0.558, 0, 0)
>  (  -0.896,   0.346, 0, 0) Ring
>  (  -0.909,   0.558, 0, 0)
>  (  -0.684,   0.556, 0, 0)
>  (  -0.682,   0.339, 0, 0)
>  (  -0.896,   0.346, 0, 0)
>
> Shape:2 (Polygon)  nVertices=4, nParts=1
>   Bounds:(  -0.127,   0.159, 0, 0)
>   to (   0.751,   0.783, 0, 0)
>  (   0.402,   0.783, 0, 0) Ring
>  (   0.751,   0.159, 0, 0)
>  (  -0.127,   0.198, 0, 0)
>  (   0.402,   0.783, 0, 0)
>
> Shape:3 (Polygon)  nVertices=5, nParts=1
>   Bounds:(   0.298,   0.523, 0, 0)
>   to (   0.426,   0.625, 0, 0)
>  (   0.298,   0.539, 0, 0) Ring
>  (   0.339,   0.625, 0, 0)
>  (   0.426,   0.612, 0, 0)
>  (   0.417,   0.523, 0, 0)
>  (   0.298,   0.539, 0, 0)
>
> Shape:4 (Polygon)  nVertices=5, nParts=1
>   Bounds:(   0.083,   0.252, 0, 0)
>   to (   0.248,   0.396, 0, 0)
>  (   0.083,   0.263, 0, 0) Ring
>  (   0.107,   0.367, 0, 0)
>  (   0.213,  

Re: [gdal-dev] shapefile enhancements

[Jan Heckman]

Thanks for the info.
I'll look into GeoPackage!
Jan


On Thu, May 1, 2014 at 8:25 PM, Even Rouault
wrote:

> Le mercredi 30 avril 2014 22:02:34, Jan Heckman a écrit :
> > Hi,
> >
> > Extending shapefile capabilities: The one reason which might persuade
> > anyone is the still abundant use of shapefiles for dataexchange.
> > Shapefile does seem to play a vital role there. I don't know of anything
> > equally 'tradeable'.
> > However, an extension would also endanger the
> > interoperability/dataexchange, so little of this would work short of
> > definining a new shapefile variant specification, working title .shp2.
> > I can see the outlines, even though I doubt it will be possible to
> > introduce such a thing. The mods would not be that hard, though.
>
> I'm not sure there's a point in defining a .shp2 spec. The GeoPackage one
> can
> be thought as the "new shapefile".
> It should have all its main advantages :
> - spatial index
> - random access
> - binary encoding of geometries
>
> Plus a few nice features :
> - attribute indexes
> - database format, with SQL capabilities
> - binary encoding of attributes
> - extensibility
>
> Its history is admitedly a bit young, but the specification is already
> standardized and has begun to be implemented in various software packages,
> including OGR.
>
> --
> Geospatial professional services
> http://even.rouault.free.fr/services.html
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] shapefile enhancements

[Jan Heckman]

Hi,

Extending shapefile capabilities: The one reason which might persuade
anyone is the still abundant use of shapefiles for dataexchange.
Shapefile does seem to play a vital role there. I don't know of anything
equally 'tradeable'.
However, an extension would also endanger the
interoperability/dataexchange, so little of this would work short of
definining a new shapefile variant specification, working title .shp2.
I can see the outlines, even though I doubt it will be possible to
introduce such a thing. The mods would not be that hard, though.

Deleting: the bit I've implemented quickly for internal use does use the
ghost-feature approach, so I agree there. At the end of the session I pack
the .dbf; now I'll consider packing the .shp as well to avoid problems with
packages that do not read the .shx. Thanks for the advice.

As done now internally, a request for a deleted record returns a null for
does not exist, but does not affect the numbering (fid) of the other
records/shapes/rows. In the .shx, only the size is set to 0 for a delete,
so an undelete is possible.
For the dbf, the only overhead is maintaining a list of deleted
record(number)s.

To properly operate a shapefile-like system in which a delete actually
reduces the number of records would require 2 things:
1) another definition of the record identifiers (fid as part of the
record). This can be done in the shapefile, of course, since the record
number is already stored (1-based); it is just never used afaik.
2) maintain a (kind of) .dbx in memory, to index the valid records much in
the same way the .shx does. One could store it or build it when reading the
.dbf file. A more efficient scheme should be possible.
Y
ou can indeed not throw such a thing at existing shapefiledrivers.

Just sharing my thoughts here.

Jan


On Wed, Apr 30, 2014 at 4:30 PM, Even Rouault
wrote:

> Le mardi 29 avril 2014 19:46:46, Jan Heckman a écrit :
> > Hi,
> > It appears I have to do some homework on ogr's shapefile functions as it
> > stands now.
> >
> > 8GB: If interoperability is more of a priority than capacity, that's a
> > valid consideration. I've not really needed anything > 4GB so far.
>
> I'm not sure there's a point in "extending" shapefile capabilities whereas
> there are other formats, more capable, that don't have a 32bit offset
> limitation.
>
> >
> > Delete:
> > By delete I mean leaving the information in the file but (shapefile)
> taking
> > it out of the index chain (.shx), and .dbf, marking the record with an
> > asterisk in its firs byte.
> > As far as arcgis, I did a delete in this way and tried to load it. When I
> > do not reduce the record count in the dbf header, arcgis will not load
> it;
> > when I do reduce the record count in the header, arcgis will load the
> > shapefile but the attributes will not match the shapes. As a cross-check,
> > you can open the .dbf in open office or excel: the delete will be
> > recognized.
> >
> > My guess is that arcgis maps the shaperecords to the physical records of
> > the dbf only.
> >
> > To allow use of the shapefile in arcgis,  I have to compact the .dbf. The
> > shape will then be handled correctly.
> >
> > A recipe to try this out:
> > create a new empty point shapefile, load it in arcgis. Using arccatalog
> to
> > create the shapefile, it will have a single ID integer attribute. That's
> > the starting point.
> > Create 3 points and give them ID's 1 - 3.
> > Now to 'delete' the second record using a diskeditor:
> > Copy the shapefile. Open the .shx. The .shx has a header and records
> > consisting of offset-length pairs. A pair takes 8 bytes. Change the 2nd
> > offset to be identical to the last (0040 -> 004E). Diminish the
> > filelength indicator in the header (offset 0x18) by 4 (003E to
> > 003A). Copy the file, except the last 8 bytes to the new .shx file.
> > DBF: open in editor, change the first byte of the second record (at
> offset
> > 0x48) to an asterisk. The recordcount in the header is at offset 4
> (little
> > endian).
> >
> > Load in arcgis, will fail.
>
> Yes I'm not surprised at all and I would say that ArcGIS behaviour is sane.
> And I guess that OGR would be defeated too if you tried to open such a
> shapefile with it.
> If you delete record ID 2, you just have to mark the .dbf record with '*'.
> For
> uniqueness purposes, feature 3 should remain feature 3. And feature 2 be a
> "ghost feature". That's whay OGR does when you use the DeleteFeature()
> API. I
> don't think it touches the .shx at all. It could possibly change the
> offset to
>

Re: [gdal-dev] shapefile enhancements

[Jan Heckman]

Hi,
It appears I have to do some homework on ogr's shapefile functions as it
stands now.

8GB: If interoperability is more of a priority than capacity, that's a
valid consideration. I've not really needed anything > 4GB so far.

Delete:
By delete I mean leaving the information in the file but (shapefile) taking
it out of the index chain (.shx), and .dbf, marking the record with an
asterisk in its firs byte.
As far as arcgis, I did a delete in this way and tried to load it. When I
do not reduce the record count in the dbf header, arcgis will not load it;
when I do reduce the record count in the header, arcgis will load the
shapefile but the attributes will not match the shapes. As a cross-check,
you can open the .dbf in open office or excel: the delete will be
recognized.

My guess is that arcgis maps the shaperecords to the physical records of
the dbf only.

To allow use of the shapefile in arcgis,  I have to compact the .dbf. The
shape will then be handled correctly.

A recipe to try this out:
create a new empty point shapefile, load it in arcgis. Using arccatalog to
create the shapefile, it will have a single ID integer attribute. That's
the starting point.
Create 3 points and give them ID's 1 - 3.
Now to 'delete' the second record using a diskeditor:
Copy the shapefile. Open the .shx. The .shx has a header and records
consisting of offset-length pairs. A pair takes 8 bytes. Change the 2nd
offset to be identical to the last (0040 -> 004E). Diminish the
filelength indicator in the header (offset 0x18) by 4 (003E to
003A). Copy the file, except the last 8 bytes to the new .shx file.
DBF: open in editor, change the first byte of the second record (at offset
0x48) to an asterisk. The recordcount in the header is at offset 4 (little
endian).

Load in arcgis, will fail.
Change  the dbf-recordcount to 2, load and open the attribute table. The
file will load. The ID's shown will be 1 and 2. The latter should have been
3.
Conclusion: shape deletes are possible, .dbf not. Compacting the .dbf will,
of course, repair things.

Best regards,
Jan



On Mon, Apr 28, 2014 at 11:48 PM, Even Rouault  wrote:

> Le lundi 28 avril 2014 23:39:34, Jan Heckman a écrit :
> > Hi,
> >
> > I've used and maintained an offshoot shapefile library (thanks, Frank),
> and
> > added bits of functionality when needed. There are two that I think might
> > be of wider interest.
> >
> > 1) possibility to have shapefiles up to 8 GB (instead of 4 GB in ogr or 2
> > GB elsewhere);
>
> I'm +/- 0 on this, since it might cause interoperabily issues. Actually
> I've
> added a creation option in GDAL 1.11 to avoid crossing the 2GB barrier.
>
> > 2) implementing deletes.
> > ad 1) this requires little more than replacing fseek() by _fseeki64() on
> a
> > windows box and using the fact that size indicators are about words
> rather
> > than bytes.
> > The use of this feature is limited to library use; afaik gis programs (at
> > least arcgis) will not be able to use such an extended shapefile.
> > Next, portability might be an issue.
> > ad 2) a quick delete without closing and rewriting the .shp and /or the
> > .dbf file can obviously be useful. Shapefile as well as dBase III files
> > offer handles to do this.
>
> Not sure to understand what you call by delete here. Is it auto-compaction
> of
> files when deleting a record ? something like DeleteFeature() + REPACK in
> the
> shapefile driver ?
>
> > Issues:
> > Arcgis wil not recognize .dbf with records marked for deletion (by
> starting
> > a record with an asterix instead of a space character);
> > the (ogr) dbase driver will equally disregard deleted records.
> > Result: at the end of the session using the shapefile library, the
> > shapefile can be handled without a major file-rewrite, but the .dbf needs
> > to have the deleted records removed 'physically'. Still, this can be done
> > relatively quickly when closing the file.
>
> Hum, shapelib already supports deleting dbf record and the ogr shapefile
> driver
> does honour that. So, you are saying that ArcGIS doesn't handle that
> properly
> ?
>
> > Portability might again be an issue.
> >
> > So,does anyone need these features, and is it useful to start a
> discussion
> > how to implement either of these enhancements?
> >
> > regards,
> > Jan
>
> --
> Geospatial professional services
> http://even.rouault.free.fr/services.html
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] shapefile enhancements

[Jan Heckman]

Hi,

I've used and maintained an offshoot shapefile library (thanks, Frank), and
added bits of functionality when needed. There are two that I think might
be of wider interest.

1) possibility to have shapefiles up to 8 GB (instead of 4 GB in ogr or 2
GB elsewhere);
2) implementing deletes.
ad 1) this requires little more than replacing fseek() by _fseeki64() on a
windows box and using the fact that size indicators are about words rather
than bytes.
The use of this feature is limited to library use; afaik gis programs (at
least arcgis) will not be able to use such an extended shapefile.
Next, portability might be an issue.
ad 2) a quick delete without closing and rewriting the .shp and /or the
.dbf file can obviously be useful. Shapefile as well as dBase III files
offer handles to do this.
Issues:
Arcgis wil not recognize .dbf with records marked for deletion (by starting
a record with an asterix instead of a space character);
the (ogr) dbase driver will equally disregard deleted records.
Result: at the end of the session using the shapefile library, the
shapefile can be handled without a major file-rewrite, but the .dbf needs
to have the deleted records removed 'physically'. Still, this can be done
relatively quickly when closing the file.
Portability might again be an issue.

So,does anyone need these features, and is it useful to start a discussion
how to implement either of these enhancements?

regards,
Jan
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Import .GDB into MSSQL Server database

[Jan Heckman]

Hey Kyle,
I had missed it can now read earlier gdb! Great!
Thanks for the head(s)-up!!
Jan


On Wed, Apr 9, 2014 at 12:52 AM, Kyle Shannon  wrote:

> On Tue, Apr 8, 2014 at 4:47 PM, Jan Heckman  wrote:
> > Depends on the geodatabase version.
> > If (AG) 10, there is a ogr driver for it, so you would not need
> shapefiles
> > and avoid losing long fieldnames etc.
> > There is no driver for earlier AG/gdb versions.
>
> OpenFileGDB should handle earlier versions, read only:
>
> http://gdal.org/ogr/drv_openfilegdb.html
>
> It's in the trunk.
>
> > Maybe it's an old thread, but it has some information.
> > Integration in ogr.
> > good luck,
> > Jan
> >
> >
> > On Tue, Apr 8, 2014 at 8:44 PM, cdp  wrote:
> >>
> >> Any ideas on how to import a .GDB into MSSQL Server database without
> >> converting to a .shp first?
> >>
> >> Thanks all!
> >> C
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://osgeo-org.1560.x6.nabble.com/Import-GDB-into-MSSQL-Server-database-tp5133917.html
> >> Sent from the GDAL - Dev mailing list archive at Nabble.com.
> >> ___
> >> gdal-dev mailing list
> >> gdal-dev@lists.osgeo.org
> >> http://lists.osgeo.org/mailman/listinfo/gdal-dev
> >
> >
> >
> > ___
> > gdal-dev mailing list
> > gdal-dev@lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>
> --
> Kyle
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Import .GDB into MSSQL Server database

[Jan Heckman]

Depends on the geodatabase version.
If (AG) 10, there is a ogr driver for it, so you would not need shapefiles
and avoid losing long fieldnames etc.
There is no driver for earlier AG/gdb versions.
Maybe it's an old
thread,
but it has some information.
Integration in ogr .
good luck,
Jan


On Tue, Apr 8, 2014 at 8:44 PM, cdp  wrote:

> Any ideas on how to import a .GDB into MSSQL Server database without
> converting to a .shp first?
>
> Thanks all!
> C
>
>
>
> --
> View this message in context:
> http://osgeo-org.1560.x6.nabble.com/Import-GDB-into-MSSQL-Server-database-tp5133917.html
> Sent from the GDAL - Dev mailing list archive at Nabble.com.
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Feature request: gauss and other interpolations in gdalwarp

[Jan Heckman]

Hello Jan,

There hasn't been a lot of response to your question, so I take the liberty
to give you my thoughts, for what they are worth.

My first guess was that you need something like a bilateral
filter.
Noise (detail) reduction is combined with edge preservation (readability).
If so, the code is quite simple and parameters can be used to optimize
results. It would have to be combined with a filter reducing the number of
pixels.
I'm not too sure about a (more or less) complete set of filters, i.e.
there's a lot of filters and I am not familiar with what might be thought
of as a complete set (gimp and image magick could be used as a source,
probably). However, see below, 'imageworsener'.

Maybe a test tile could serve to do a few tests. After that, I can take a
look at the work needed to incorporate filters in a fairly general way to
allow adding more filters.
Completely general appears unlikely and quite a strain on parameter
passing, documentation, support.

I've been reading up a bit on the subject of image downsampling /
upsampling. There is plenty afoot which might make you consider a different
approach from using a set of images, like this paper on multiscale edge
preserving  as part of a stanford
course  on filtering/image
manipulation.

There is signalprocessing analog with (down)sampling audio which would
separate low from high frequency signals and recombine them depending on
the required scale.
Don't know if that could be done continuously instead of just for integral
pixel reductions.
Would be great if this can be done in a server, perhaps using graphics
cards to speed up the image (re)construction.

There is public domain program on the subject, called
imageworsener.
There are a lot of considered options, and source code. So this might be a
point of departure, either by incorporating it into gdal(warp), or vice
versa.

Hope this helps some,
Best regards,
Jan

On Thu, Mar 28, 2013 at 12:27 PM, Jan Hartmann wrote:

>  Perhaps I should clarify a bit what I meant, I haven't had any reactions
> until now, positive or negative, and it is important for me.
>
> I use Gdalwarp and gdaladdo extensively for goereferencing and tiling
> large historical maps serieses (raster scans). To display them efficiently,
> I need to create layers at different scale levels, e.g. if the original
> maps have a pixel size of 10 meters, I need to resample them to rasters
> with pixel sizes of 20, 40, 80 , 160, 320 and 640 meters, and tile all
> those maps appropriately. So the original maps have to be resampled quite
> drastically. In Gdalwarp there is no adequeate resampling algorithm, and
> you end up with very grainy map at those lower resolutions. Gdaladdo has
> several more algorithms, with gauss in many cases the most efficient.
> However, even with the gauss filter, maps resampled at very low resolutions
> turn out too hazy. For an example see the 1930 map of the Netherlands:
>
>  http://mapserver.sara.nl/topo/triang/
>
> If you zoom in to more detailed levels, you'll see the way the image
> sharpens. For the effect on black-white image choose the TMK-map (1850)
> with the top-center button. Filters like "unsharp mask" would perform much
> better in these cases.
>
> So I would like to propose two enhancements to gdal:
>
> - add additional filters to gdalwarp, gauss and the filters mentioned
> below.
> - implement more filters for gdalwarp and gdaladdo, e.g. "unsharp mask",
> or the "mode" filter asked by Jack below. Perhaps even add the possibility
> to specify parameters, like in ImageMagick
>
> I don't know how difficult is, and whether the gdal devs would find this
> really an improvement for gdal. I could do this with some ImageMagick or
> Gimp scripts, but it would be a kludge. As we are going to georeference the
> complete cadastral and topographical map base of the Netherlands from 1832
> to 1994 the next few years (millions of map-scans), this exentsion of gdal
> would come in very handy. And funding it will really be no problem at all.
>
> I would appreciate any kind of comment on this, positive or negative.
>
> Regards,
>
> Jan
>
> Dr. J. Hartmann
> Department of Geography
> University of Amsterdam
>
>  On 03/27/2013 10:13 PM, John Twilley wrote:
>
> I'm interested in this feature request as well.  Adding the mode
> resampling algorithm to gdalwarp would be very beneficial to my projects,
> right up there with being able to access the warp API from Python.  Is this
> at all possible?  Should I submit a feature request on Trac, or what?  Just
> let me know!
>
>  Jack.
>
> --
> mathuin at gmail dot com
>
>
> On Mon, Feb 18, 2013 at 8:17 AM, Jan Hartmann wrote:
>
>>
>>  Hi devs,
>>
>> Would it be possible to add gauss and other interpolations to gdalwarp?
>> At the moment I georeference large scans to 2000*2000 tiles at the most
>> deta

Re: [gdal-dev] extract information from polygon shapefile based on raster grid?

[Jan Heckman]

Hello Chao,

I may not understand your wish quite well, but I think you need some
datastructure which conforms to the elements of the grid and receives
addtional information derived from a number of shapes with attributes.
Your output could be another shapefile,  with the summed results as
attributes, like the number of fires.
In the case that the only spatial relationship between shapes and
gridelements is whether or not a shape falls inside, you can indeed
simplify the polygon shape to pointshapes (centerpoints) which can be
converted to a raster of 1's and 0's. Then you could use a dictionary to
put the shapeattributes in the right place. I see a problem when several
shapes fall into the same grid-element, having different attribute values.
So you would be better off with a way of rasterbuilding, where each
attribute is summed per gridelement over all the shapes that fall inside.
Such routines exist at least in several programs.
Alternatively, you can make a polygon shapefile which conforms to the
rasterelements, then do a spatial join between your (centerpoint) shapefile
and and the rastershape (effectively adding a field to the pointshapes
which is the index into the grid), then add the several attribute data per
gridelement by e.g. making a pivot table.
If you have to work with shapes that are larger than the gridelements, you
would need another solution; there would be a choice between sampling the
shapes or distributing shape attributes proportionally to calculated
overlap with (polygonal) gridelements.

So, more simply still,  assuming the grid is regular and it is possible to
compute the gridindex from pointcoordinates:

Add an index field to the centerpoint shape which has the attributes;
calculate the gridindex from the coordinates and put this into the index
field;
bring the table e.g. into excel;
do a pivot table for the attributes (count or sum).
If necessary, add the centerpoint coordinates to the resulting table;
convert the table into a new point shape with the desired attributes.

Hope I got you right and this helps,
Jan

On Wed, Mar 27, 2013 at 9:52 PM, Chao YUE  wrote:

> Dear all,
>
> I am using the python binding of GDAL for reading polygon shapefiles.
>
> I have a polygon shapefile which contains the fire polygon and burned
> area, with typical fires burning on 10-100ha.
> my model simulation result is based on 0.5degree X 0.5degree. So I need to
> extract from this shapefile on each gridcell
> of the 0.5d X 0.5 grid the information about like number of fires and mean
> fire size etc.
>
> The approach I am thinking currently is to calculate the center of each of
> fire polygon, and check which 0.5d X 0.5d gridcell
> it falls in, then build a huge dictionary with key as the numpy index of
> the 0.5dX0.5d array, with the dictionary values as
> the list(or array or pandas dataframe) which include the information of
> all the polygons falling within the the same grid cell.
>
> I don't know if there is a better approach than this? I appreciate any
> comments.
>
> best regards,
>
> Chao
>
> --
>
> ***
> Chao YUE
> Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL)
> UMR 1572 CEA-CNRS-UVSQ
> Batiment 712 - Pe 119
> 91191 GIF Sur YVETTE Cedex
> Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16
>
> 
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] ReadAsArray: putting data in an already existent array

[Jan Heckman]

Hi Diego,
I haven't checked this out, but my guess is that you should make a
(python)list of the arrays if you need to keep them.  If you don't need to
keep them, you might try to del(ete) myarray before the next ReadAsArray
call.
Of course there is a lower level function which accepts an address as you
suggest, but the question is whether it should be exposed.
Because of the potential for errors, exposing it might not be the best.
You could expose it yourself, probably, but I very much suggest trying my
suggestion first.
Good luck,
Jan

On Fri, Mar 22, 2013 at 6:47 PM, Diego Gnesi Bartolani <
diego.gn...@gmail.com> wrote:

> Hi,
>
> I'm programming in Python and I need to "scan" a large image with GDAL
> reading n rows at the time. I'm using the ReadAsArray method, but every
> time I execute it I get a new array. I think this solution consumes a lot
> of resources.
> Is there a method or option to pass to ReadAsArray an array that already
> exists, in order to change it's value, like in the following pseudo-code:
>
> band1 = ds.GetRasterBand(1)
> my_arr = band1.ReadAsArrray([params])
> [Do something with my_arr]
> band1.ReadAsArray([params to get other rows], save_to=my_arr)
>
> Of course, my_arr must have the correct size for storing incoming data.
>
> Thanks,
>
> Diego
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Issues with FileGDB support

[Jan Heckman]

Hi,
I have the gdb support working on a windows box, compiled 1.9.1, some time
ago.
>From memory:
Remember filegdb support is a plugin. The dll needs to be in a specific
gdalplugins directory.
I guess that if the support is compiled correctly and the dll is not found,
this would be mentioned in an error.
I had to edit nmake.opt; have to do that anyway for lots of support.
Lastly, when I built, there were 2 windows lib/dll's from esri (for
filegdb),  1 for vs 2008, and 1 for vs 2010.
Only the first works, which compelled me to compile using VS 2008.
If you have another vs version installed, you need to run the correct
vsvars.bat before starting (commandline) compilation.

FGDB_ENABLED = YES
FGDB_PLUGIN = YES
FGDB_SDK =
"D:\win7users\jan\Documents\Visual_Studio_2012\Projects\FileGDB_API_VS2008_1_2"
FGDB_INC = $(FGDB_SDK)\include
FGDB_LIB = $(FGDB_SDK)\lib\FileGDBAPI.lib

Specific supportfiles - see if these are compiled - are
in gdal-1.9.1\ogr\ogrsf_frmts\filegdb.
You should see ogr_FileGDB.dll and ogr_FileGDB.lib.
Let me know if this helps any.
Jan

On Fri, Mar 15, 2013 at 7:31 PM, Paul Norman  wrote:

> > From: gdal-dev-boun...@lists.osgeo.org [mailto:gdal-dev-
> > To: gdal-dev@lists.osgeo.org
> > Subject: [gdal-dev] Issues with FileGDB support
> >
> > Hello
> >
> > I have built gdal from source (1.9.2, latest stable) with Esri FileGDB
> > support included. This is specified during the configuration, --with-
> > fgdb=/usr/local/FileGDB_API/.
> >
> > The configuration and build goes well and I can see that the fileGDB
> > libs get linked but when I check the format support with for example
> > 'ogrinfo --format' FileGDB doesn't show up. ldd shows that the binary is
> > linked to the fileGDB libs.
> >
> > At this point I guess that the most likely explanation for this is an
> > error in the code, but I haven't checked it.
> >
> > So, is FileGDB support working for you?
>
> It's been awhile, but I successfully built gdal with FileGDB support.
>
> My notes indicate I used ./configure --with-fgdb=/home/pnorman/FileGDB_API
> and installed files from that location to /usr/local/lib/libFileGDBAPI.so
> and /usr/local/lib/libfgdbunixrtl.so, as well as
> /usr/share/doc/filegdb-api/.
>
> I believe I installed from svn, and ogrinfo --version reports GDAL 2.0dev,
> released 2011/12/29
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] More information to understand ColorTables in GeoTiffs

[Jan Heckman]

Robert,
Afaik, you need a three tuple for the color info. See e.g.
http://osgeo-org.1560.n6.nabble.com/gdal-dev-tiff-with-colortable-loses-alpha-values-td3747611.html
.
It has a line giving an apparently working SetColorEntry() call without
also using CreateColorRamp() which just complicates things here.
For tuples in python,
http://docs.python.org/release/1.5.1p1/tut/tuples.html gives
a useful explanation.
Hope this helps a bit,
Jan

On Sat, Mar 9, 2013 at 4:52 AM, robert  wrote:

> Frank,
>
> Thanks for the response.
>
> I've tried variations of [] and (), with and without alpha (3 or 4
> values), and with various values ranging from 0 to 255. They all have
> failed so it seems i'm doing something fundamentally wrong.
>
>
> Traceback (most recent call last):
>   File "makecolortable.py", line 13, in 
> c.CreateColorRamp(1,n,65535,m)
>   File "/usr/lib/python2.7/dist-packages/osgeo/gdal.py", line 1117, in
> CreateColorRamp
> return _gdal.ColorTable_CreateColorRamp(self, *args)
> TypeError: not a sequence
>
>
> Traceback (most recent call last):
>   File "makecolortable.py", line 11, in 
> n = c.SetColorEntry(1,[0,0,0])
>   File "/usr/lib/python2.7/dist-packages/osgeo/gdal.py", line 1113, in
> SetColorEntry
> return _gdal.ColorTable_SetColorEntry(self, *args)
> TypeError: Invalid values in ColorEntry sequence
>
>
>
> import numpy as np
> import gdal
>
> driver = gdal.GetDriverByName('GTiff')
> dst_ds = driver.Create('testcolor.tif', 100, 100, 1, gdal.GDT_UInt16)
>
> band = dst_ds.GetRasterBand(1)
> band.SetRasterColorInterpretation(gdal.GCI_PaletteIndex)
>
> c = gdal.ColorTable(gdal.GCI_PaletteIndex)
> n = c.SetColorEntry(1,[0,0,0])
> m = c.SetColorEntry(65535,[0,0,0])
> c.CreateColorRamp(1,n,65535,m)
> band.SetColorTable(c)
>
>
> --nix
>
> On 2013/03/08, at 19:09, Frank Warmerdam  wrote:
>
> On Fri, Mar 8, 2013 at 2:05 PM, Robert Nix  wrote:
>
> Let me adjust my question. I changed the color-table code to:
>
> c = gdal.ColorTable(gdal.GCI_PaletteIndex)
> n = c.SetColorEntry(1,[255,128,0,0])
> m = c.SetColorEntry(65535,[0,128,255,255])
> c.CreateColorRamp(1,n,65535,m)
> band.SetColorTable(c)
>
>
> I discovered that the 64K colors are due to the UInt16 (tested GDT_Byte and
> got 255) so i realized that maybe i need to create a ramp?
>
>
> Robert,
>
> I will note that the TIFF format does not support "short colormaps".
> So if the type of the pixel is 8bit byte then the color table will be
> exactly 256 entries even if you only put in one color.  Frr UInt16 it
> will be 65536.  Most other colormap supporting formats do not impose
> this.
>
> But the above code fails with:
>
> Traceback (most recent call last):
>  File "makecolortable.py", line 14, in 
>n = c.SetColorEntry(1,[255,128,0,0])
>  File "/usr/lib/python2.7/dist-packages/osgeo/gdal.py", line 1113, in
> SetColorEntry
>return _gdal.ColorTable_SetColorEntry(self, *args)
> TypeError: Invalid values in ColorEntry sequence
>
>
> That sequence appears correct as all four values meet the short
> requirement,
> (i think this is the code that produces the error):
>
> if ( !PyArg_ParseTuple( $input,"hhh|h", &ce.c1, &ce.c2, &ce.c3, &ce.c4 ) )
> {
>
>  PyErr_SetString(PyExc_TypeError, "Invalid values in ColorEntry sequence
> ");
>
>
>
> I think the problem is that you switched from tuples to lists (ie.
> from (r,g,b,a) to [r,g,b,a]).
>
> Also, FYI, TIFF does not support alpha in colormaps.
>
> Best regards,
> Frank
>
> --nix
>
> On Mar 8, 2013, at 16:35, Robert Nix  wrote:
>
> Hi,
>
> I'm trying to add a color table to a 1-band UInt16 GeoTiff with Python as
> follows:
>
> import numpy as np
> import gdal
>
> driver = gdal.GetDriverByName('GTiff')
> dst_ds = driver.Create('testcolor.tif', 100, 100, 1, gdal.GDT_UInt16)
>
> band = dst_ds.GetRasterBand(1)
> band.SetRasterColorInterpretation(gdal.GCI_PaletteIndex)
>
> c = gdal.ColorTable(gdal.GCI_PaletteIndex)
> c.SetColorEntry(1,(255,127,0,63))
> band.SetColorTable(c)
>
> a = np.random.randint(1,10,size=(100,100))
> band.WriteArray(a)
>
> dst_ds = None
>
>
> The problem is that when i run gdalinfo on the testcolor.tif, it lists
> 65535
> color table entries:
>
> Driver: GTiff/GeoTIFF
> Files: testcolor.tif
>   testcolor.tif.aux.xml
> Size is 100, 100
> Coordinate System is `'
> Image Structure Metadata:
>  INTERLEAVE=BAND
> Corner Coordinates:
> Upper Left  (0.0,0.0)
> Lower Left  (0.0,  100.0)
> Upper Right (  100.0,0.0)
> Lower Right (  100.0,  100.0)
> Center  (   50.0,   50.0)
> Band 1 Block=100x40 Type=UInt16, ColorInterp=Palette
>  Color Table (RGB with 65536 entries)
>0: 1,0,0,255
>1: 1,0,0,255
>2: 0,0,0,255
>3: 0,0,0,255
> ...
>  65533: 0,0,0,255
>  65534: 0,0,0,255
>  65535: 0,0,0,255
>
>
> What's going on … besides the fact that maybe i'm not understanding
> color-tables? If it's clear i don't understand, is there a good resource
> for
> understanding color-tables? Or must I go learn the source code (which i
> really don't have to do)?
>
> Tha

Re: [gdal-dev] nodata in .bil format

[Jan Heckman]

Hi,
There should certainly not be a space between Nodata and Value as in your
mail below.
There are two formats relevant for (at least) arcgis:
.bil. In this case the keyword is NoData
.flt. In this case the keyword is NoData_Value.
Capitalization, afaik, is irrelevant.
So you could modify the .hdr file as a test, and if it works out as I
indicated (NoData=- in stead of NoData Value=-) you can file a
useful ticket.
.flt, by the way, is a float array which appears to be used only as an
intermediate (conversion) format, as in the conversion tools: float to
raster and vice versa.
Hope this helps a little,
Jan

On Wed, Mar 6, 2013 at 7:53 PM, Joaquim Luis  wrote:

> Hi,
>
> While playing a bit with Lastools I came across this behavior. I create
> outputs in both .bil and .asc ESRI formats
> The weird thing is that beeing the .hdr header file common (I mean exactly
> equal) to both outputs the  .bil format does not recognize the nodata value.
> Is this a knwon issue?(GDAL trunk on Windows)
>
> gdalinfo lixo.bil
> ERROR 6: lixo.bil, band 1: SetNoDataValue() not supported for this dataset.
> Driver: EHdr/ESRI .hdr Labelled
>
>
> C:\programs\lastools\data>**gdalinfo lixo.asc
> Driver: AAIGrid/Arc/Info ASCII Grid
> Files: lixo.asc
> ...
> Band 1 Block=250x1 Type=Float32, ColorInterp=Undefined
>   NoData Value=-
>
> Joaquim
> __**_
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/**mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Most optimal algorithm for contour correctness?

[Jan Heckman]

Hi,
I've tackled a similar problem.
Correctness of contour is a simple matter, except for one thing. That is
the issue of saddles.
That issue cannot be solved "correctly" since typically the data can be
interpreted in several ways.
This is also described in a wikipedia article (google for it, if you like).
Imagine a (descending) mountainridge, which may have towers and gaps in it
(sometimes referred to as sentinels when climbing).
When the (dem) grid is too coarse, these won't show.
I found that I had to write my own routine to handle this.
Roughly as follows:
When gridsquare with values equal or larger than the contourvalue are
isolated, except they touch diagonally, you have a potential saddle
situation.
My "correctness" is that I continue straight on when the average of the
four surrounding gridpoints is equal to or larger than the contour value.
If not, the contour is divided in two subcontours.
Hope this helps and tell me whether you need such code.
I have been considering offering it for the gdal and saga communities, but
have not had enough time to make the code nice enough.
Jan

On Thu, Feb 14, 2013 at 4:15 AM, nav  wrote:

> Am creating an app which correctly calculates contours from a DEM file or
> any
> height-map given to it. The correctness of the contour holding the most
> importance, since a person viewing the contours shouldn't end up taking
> wrong planning decisions and maybe losing their life by the route they
> choose after looking at the map.
>
> I have already tried using GDAL 1.7 source code, compiled it and added some
> features of my own to it for some other objective. Now I want to
> add-to/modify the source to be able to use the contour generation function
> of GDAL in C++.
>
> But before proceeding, I wanted to know:
> 1. If GDAL offers the best free contour generation algorithm available or
> is
> there any other library which I can link to my code and feed my DEM map to
> it and receive contour vector lines (or go thru research papers and
> implement the contour generation algorithm myself in C++)? If GDAL, then
> why
> is it the best?
> 2. A generated contour will basically be points connected with straight
> lines. Is it more correct for a contour to be depicted with connected
> straight lines or will it give a more correct result to use a spline to
> smoothen out the lines?
>
> I won't be interpolating any contours or the DEM because I've read on this
> website that interpolation leads to incorrect results.
>
>
>
> --
> View this message in context:
> http://osgeo-org.1560.n6.nabble.com/Most-optimal-algorithm-for-contour-correctness-tp5034148.html
> Sent from the GDAL - Dev mailing list archive at Nabble.com.
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] libFileGDBAPI.so not found with compiling gdal 1.9.2 on openSUSE 12.2 x64.

[Jan Heckman]

Hi,
On windows, using visual studio (2008 and 2012), I could compile both 32
bits and 64 bits, but only the 32 bits version actually worked.
I haven't tried to figure out why the 64 bits version didn't work, 32 bits
was good enough for me.
Hope this is an option for you as well,
Jan

On Fri, Jan 25, 2013 at 5:23 AM, Donovan Cameron wrote:

> Looks like over in the [qgis-user] list, someone is having similar
> problems:
> https://groups.google.com/d/msg/qgis_user_remote/B2J9sOsSLbo/BIpAHZ84bhgJ
>
> On Thu, Jan 24, 2013 at 9:05 PM, Donovan Cameron 
> wrote:
> > I can't seem to compile GDAL with FileGDB support on openSUSE.
> > I'm compiling locally using the OBS tools (osc) and have checked-out
> > the libgdal package from the Application:Geo project.
> >
> > Steps I've taken:
> >
> > 1) Download FileGDB_API_1_2-64.tar.gz
> >
> > 2) tar xzvf FileGDB_API_1_2-64.tar.gz -C /usr/lib64/
> >
> > 3) export LD_LIBRARY_PATH=/usr/lib64/FileGDB_API/lib
> > cd /usr/lib64/FileGDB_API/samples
> > make
> > cd bin
> > ./Querying #Works, so the API is fine.
> >
> > 4) su -
> > $ vi /etc/ld.so.conf.d/fgdb.conf #Added a single entry:
> > /usr/lib64/FileGDB_API/lib
> > $ ldconfig #A grep of this with the -v flag shows it registered fine
> >
> > 5) #Some openSUSE specific stuff, b/c I'm building locally via the OBS
> > $ zypper in osc
> > cd /home/saultdon/Development/obs
> > osc co Application:Geo libgdal
> > cd Application:Geo/libgdal
> > vi libgdal.spec #added "--with-fgdb=/usr/lib64/FileGDB_API" in
> > %configure section
> > #I also did a test build without that configure param. first and it
> > compiled fine with working RPMs
> > osc build openSUSE_12.2 x86_64 libgdal.spec --clean
> >
> > Resulting Error:
> > [  209s] checking for libFileGDBAPI.so in in
> > /usr/lib64/FileGDB_API/lib... configure: error: not found.
> > [  209s] error: Bad exit status from /var/tmp/rpm-tmp.7SOa8g (%build)
> > [  209s] RPM build errors:
> > [  209s] Bad exit status from /var/tmp/rpm-tmp.7SOa8g (%build)
> > The buildroot was: /var/tmp/build-root
> >
> > libFileGDBAPI.so file exists, ldconfig registered it, and the API test
> > worked fine, the env. var. LD_LIBRARY_PATH is also set, but I don't
> > even think that's necessary. I only had to set that to get the API
> > test to work.
> >
> > I'm not sure where this is going wrong. If anyone has any input or
> > needs more information, let me know.
> >
> >
> > Thanks!
> >
> >
> > Donovan
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev