[gdal-dev] ogrinfo behaviour with -al and -json options

[Andrea Giudiceandrea via gdal-dev]

Hi all,
IIUC, the documentation states that with the -al option, ogrinfo should 
output the info of all layers in a container (e.g. a GPKG file). Anyway, 
when a layer name is provided in the command line string alongside the 
-al option, than it looks like the -al option is overridden and not 
taken in consideration.


Is this the expected behaviour?

The same behaviour occurs with the -json option, where the documentation 
states than in such case the -al option is also implicit.


If those are the correct and expected behaviour, maybe it would be 
useful to clarify it in the documentation.


Regards.

Andrea
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] gdal_rasterize parameters not working

[Andrea Giudiceandrea via gdal-dev]

Il 03/07/2023 18:08, Scott ha scritto:

You're using the -tr and -ts. You can only use one or the other, not both.


I think it would be useful if gdal_rasterize emitted a warning when the 
-tr and -ts options are used at the same time like other GDAL tools do.


Best regards.

Andrea
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Shapefile with corrupted index: SHAPE_RESTORE_SHX=YES doesn't correctly repairs it.

[Andrea Giudiceandrea via gdal-dev]

Hi Jan,
thank you very much for looking at this issue.

I had already managed to restore the .shx index file, as reported in my 
first message [1] [2] by fixing the GDAL/OGR SHPRestoreSHX routine [3]. 
Even Rouault also further improved it adding extra sanity checks, a 
better logic and better error messages [4].
Obviously, SHPRestoreSHX routing doesn't take care of the the mismatch 
between the shape and dbf records number.


An old but still working Shapefile recovery tool (which also handle the 
shp / dbf mismatch) is the "Shape Checker" [5].


The .idx file may have been created by AutoCAD [6], as the user reported 
that he tried to open the corrupted Shapefile layer with AutoCAD after 
the corruption occurred.


Best regards.

Andrea


[1] https://lists.osgeo.org/pipermail/gdal-dev/2023-May/057229.html
[2] https://github.com/qgis/QGIS/issues/53058#issuecomment-1547740971
[3] https://github.com/OSGeo/gdal/pull/7774
[4] https://github.com/OSGeo/gdal/pull/7778
[5] 
https://web.archive.org/web/20091024035556/http://geocities.com/SiliconValley/Haven/2295/howto_shapechk.html
[6] 
https://www.autodesk.com/support/technical/article/caas/sfdcarticles/sfdcarticles/Required-files-that-make-up-a-shapefile.html




Il 17/05/2023 12:39, Jan Heckman ha scritto:

Hi all,

As noted in the QGis issues , 
the shx has problems that ogr2ogr (or QGis Repair Shapefile) will not fix;
The error I get in cmdline with SHAPE_RESTORE_SHX=YES is "ERROR 4: Error 
parsing .shp to restore .shx".

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Shapefile with corrupted index: SHAPE_RESTORE_SHX=YES doesn't correctly repairs it.

[Andrea Giudiceandrea via gdal-dev]

Il 16/05/2023 13:52, Jan Heckman ha scritto:

Also out of curiosity,
Are you willing to share the shapefile, preferably at least shp and dbf, 
but probably .shp alone will do and .prj helps for checking conversion 
results.


Hi Jan,
the corrupted Shapefile layer is available in the QGIS issue report at 
https://github.com/qgis/QGIS/issues/53058


Best regards.

Andrea
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Shapefile with corrupted index: SHAPE_RESTORE_SHX=YES doesn't correctly repairs it.

[Andrea Giudiceandrea via gdal-dev]

Il 15/05/2023 15:34, Andrea Giudiceandrea ha scritto:
it seems to me ogr fails to properly create the .idx [EDIT: .shx] file: 
it incorrectly stores, in the index file header, the total length in 
16-bit words of the .shp file instead of the total length in 16-bit 
words of the .idx [EDIT: .shx] file itself.


Looking at the code, SHPRestoreSHX initially copies the SHP header 
(including the SHP content length) to the SHX file and then it doesn't 
updates the SHX content length in the SHX header when an error occurred 
during the reading of the SHP file.


I propose the PR https://github.com/OSGeo/gdal/pull/7774 in order to 
update the SHX content length even if an error occurred.


Maybe would it be better to let SHPRestoreSHX modify the SHX file (when 
already present) only at the end of the SHP reading procedure and only 
if no error occurred?


Best regards.

Andrea Giudiceandrea
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Shapefile with corrupted index: SHAPE_RESTORE_SHX=YES doesn't correctly repairs it.

[Andrea Giudiceandrea via gdal-dev]

Il 15/05/2023 15:52, Rahkonen Jukka ha scritto:

Am I right that .idx is an attribute index? By the documentation it feels 
somehow odd


Hi Jukka Rahkonen,
I apologise there was a typo in my message... where I wrote ".idx" I 
should have written ".shx".



Il 15/05/2023 17:43, Robert Hewlett ha scritto:
Out of curiosity, if you isolate the shp, dbf and shx (make a copy) in a 
separate folder is the data still corrupt?


Yes, it is still corrupted.

Regards.

Andrea
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Shapefile with corrupted index: SHAPE_RESTORE_SHX=YES doesn't correctly repairs it.

[Andrea Giudiceandrea via gdal-dev]

Hi devs,
in a reent QGIS issue report at 
https://github.com/qgis/QGIS/issues/53058 , an user complains about an 
ESRI Shapefile layer that was corrupted after an attribute value was 
changed and the edit was saved. The corrupted layer is opened by QGIS 
without errors or warning being reported, anyway it shows only a subset 
of the original feature geometry: a lot of records have now a null 
geometry associated, so they cannot be displayed.


After some investigations, although I don't know why and how the layer 
was corrupted, it seems to me that the issue is mostly due to a 
corruption of the .idx file: in fact it contains, for various records, 
incorrect value of index and length of the record. This generates the 
incorrect reading of such record and the following ones, until the the 
index in the .idx file and the data in the .shp file line up again.


Running the QGIS "Repair Shapefile" processing algorithm against such 
layer, the algorithm fails while the .idx file is actually updated but 
the layer becomes totally invalid and it is not possible to load it in 
QGIS. The same happens directly using ogrinfo after the .idx file was 
deleted and the SHAPE_RESTORE_SHX variable was set to YES: the .idx file 
was recreated but the layer becomes unreadable by both QGIS and ogrinfo.


Inspecting the .idx file created by ogrinfo with SHAPE_RESTORE_SHX=YES 
(which is the same as the one created by the QGIS tool "Repair 
Shapefile"), it seems to me ogr fails to properly create the .idx file: 
it incorrectly stores, in the index file header, the total length in 
16-bit words of the .shp file instead of the total length in 16-bit 
words of the .idx file itself.

In this particular case,
it stores the incorrect value 00 29 2A C2 = 2697922 16-bit words = 
5395844 bytes
instead of the correct value 00 02 1D 26 = 138534 16-bit words = 277068 
bytes


Changing such incorrect value to the correct one in the repaired .idx 
file, makes the layer valid again and showing again the previously 
missing feature geometries (with only some glitches and a missing record).


This behaviour seems weird to me, as I remember that the Repair 
Shapefile tool or the SHAPE_RESTORE_SHX=YES setting worked well to 
repair Shapefiles with corrupted index in the past.


Maybe the issue in this particular Shapefile prevent ogr to correctly 
repair the index?
For comparison, the old "Shape Checker utility" succeeds to repair the 
.idx file: it creates the same .idx file as the one created by ogr, 
apart from the total file length value which is correct.


Any clue as to what may have gone wrong during the layer editing in QGIS 
that eventually corrupted the layer?



Best regards.

Andrea Giudiceandrea
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Failure to read some geojson files

[Andrea Giudiceandrea via gdal-dev]

Il 28/04/2023 19:08, Joaquim Manuel Freire Luís via gdal-dev ha scritto:
This is with OSGeo4W GDAL but I get the same with my own build. But I 
can read some other geojson files. Why?


ogrinfo 
https://github.com/OSGeo/PROJ/blob/9.1/docs/plot/data/coastline.geojson 



ERROR 4: Failed to read TopoJSON data


Hi Joaquim,
the URL 
https://github.com/OSGeo/PROJ/blob/9.1/docs/plot/data/coastline.geojson 
does not point to the actual coastline.geojson file. It points to a 
GitHub HTML page.


You need to use the URL 
https://github.com/OSGeo/PROJ/raw/9.1/docs/plot/data/coastline.geojson 
instead.


Best regards.

Andrea Giudiceandrea
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] gdal2tiles: RuntimeError: Cannot find coordinate operations from `ENGCRS["WGS_1984_Web_Mercator_Auxiliary_Sphere"

[Andrea Giudiceandrea via gdal-dev]

*Terra Frost*
/Wed Nov 9 15:25:24 PST 2022/

So I just tried the above commands on a brand new Ubuntu 22.04 install.
When I did "sudo add-apt-repository ppa:ubuntugis/ppa" I got this error:

E: The repository 'https://ppa.launchpadcontent.net/ubuntugis/ppa/ubuntu  jammy
Release' does not have a Release file.


Hi Terra,
as the error says, the ubuntugis ppa [1] does not have packages for 
Ubuntu 22.04 Jammy Jellyfish.

GDAL 3.4.3 is available in the ubuntugis-unstable repository for Jammy [2].
GDAL 3.4.1 is available in the ubuntu repository for Jammy [3].

Best regards.

Andrea Giudiceandrea


[1] https://launchpad.net/~ubuntugis/+archive/ubuntu/ppa
[2] https://launchpad.net/~ubuntugis/+archive/ubuntu/ubuntugis-unstable
[3] https://packages.ubuntu.com/jammy/gdal-bin___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] ogr2ogr doesn't connect to MS SQL Server

[Andrea Giudiceandrea via gdal-dev]

Il 21/10/2022 11:48, Reetz, Michael (NLPV) ha scritto:
Since the GDAL/ogr MS SQL Server driver is obviously missing on my 
computer, I will see if I can get a complete installation of the 
current GDAL package. This would be very helpful in many situations, 
especially in scripts that perform transformation tasks.


Hi Michael,
doesn't adding the ogr_MSSQLSpatial.dll file to the 
"\apps\gdal\lib\gdalplugins\" folder (or installing the gdal-mss package 
through OSGeo4W Setup) makes the MSSQLSpatial GDAL/OGR driver available 
to be used with ogr2ogr?


Best regards.

Andrea___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] ogr2ogr doesn't connect to MS SQL Server

[Andrea Giudiceandrea via gdal-dev]

Hi Michael,
the GDAL/OGR library needs the "ogr_MSSQLSpatial.dll" file to properly 
activate the MSSQLSpatial GDAL/OGR driver ("MSSQL:").


You can check if the MSSQLSpatial GDAL/OGR driver is supported by your 
ogr2ogr tool using the following command in the OSGeo4W Shell:


ogr2ogr --formats | find "MSSQL"

If it outputs the string "MSSQLSpatial -vector- (rw+): Microsoft SQL 
Server Spatial Database (BCP)", then it means such driver is supported, 
otherwise not.


If you cannot use the OSGeo4W "Setup" program to install such file, you 
can try to fix the issue just downloading the "gdal-mss" OSGeo4W package 
from [1] and extracting from it the "ogr_MSSQLSpatial.dll" file to the 
"\apps\gdal\lib\gdalplugins\" folder in your QGIS installation dir.
The exact "gdal-mss" OSGeo4W package to download depends on the exact 
GDAL version installed with QGIS.


Best regards.

[1] 
https://download.osgeo.org/osgeo4w/v2/x86_64/release/gdal/gdal-mss/?C=M=D



Il 20/10/2022 16:56, Reetz, Michael (NLPV) ha scritto:


I’ve found msodbcsql17.dll in C:\Program Files\QGIS 3.22.8\bin\1033 
and C:\Program Files\QGIS 3.22.8\bin.


Gdal-mss seems not to be on my computer. But since SQL Server Native 
Client isn’t supported by MS SQL Server 2019 , this shouldn’t be a 
Problem.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Does it hurt to *always* use BIGTIFF when using gdal_translate

[Andrea Giudiceandrea via gdal-dev]

Il 12/10/2022 11:51, Richard Duivenvoorde ha scritto:
One of the 'easy' solutions (on QGIS side) would be to *always* add 
the BIGTIFF=yes option.


Hi all,
while there are also the IF_NEEDED and IF_SAFER values for the BIGTIFF 
creation option, another possibility would be to have an option to let 
GDAL to retry to create the raster file as a BIGTIFF if the the normal 
TIFF creation fails or to let QGIS to inform the user if the normal TIFF 
creation fails and then offer the possibility to retry using the 
BIGTIFF=YES option.


Regards.

Andrea
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Strange (expected?) behaviour exporting multipart geometry to ESRI Shapefile format

[Andrea Giudiceandrea via gdal-dev]

Il 16/02/2022 12:24, Even Rouault ha scritto:

your [2] link contains the shapefile, not the GeoPackage file.


Hi Even,
yes, you are right. The correct URL for the GeoPackage layer is 
https://drive.google.com/file/d/1DMMbFQ922d_ccS00JIt0KaojGk6jMorb


Or you can use the SHAPE_REWIND_ON_WRITE configuration option 
mentioned at 
https://gdal.org/drivers/vector/shapefile.html#advanced-topics :


ogr2ogr out3.shp Issue47288_Polygons.gpkg --config 
SHAPE_REWIND_ON_WRITE OFF


This will keep the original geometry mostly untouched, possibly 
creating a non-conformant shapefile. And on the reading side, the OGR 
shapefile reader might also have issues reconstructing a 
single-feature geometry. On that particular case, it seems to produce 
the "expected" result


Yes, exporting to Shapefile with this option, the visualization issue in 
QGIS doesn't occur in this case and the parts maintain their 
winding/orientation.


Thanks for the insight about the issue and on how the driver works under 
the hood and deal with the (not always clear) specification.


My inclination would be to consider that in the case where we have a 
multipolygon with non -coplanar parts we should probably be avoiding 
considering the ones that are non-coplanar are inner/outer rings of 
others.


I've ticketed that in https://github.com/OSGeo/gdal/issues/5315


Thank you again.

Best regards.

Andrea
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Strange (expected?) behaviour exporting multipart geometry to ESRI Shapefile format

[Andrea Giudiceandrea via gdal-dev]

Hi GDAL devs,
trying find the root cause of a bug [1] reported in the QGIS GitHub 
repository, I've noticed a strange behaviour when a multipart geometry 
is exported to an ESRI Shapefile layer.


I've created a minimal GeoPackage layer file [2] which contains 1 
multipart polygon geometry (MultiPolygon) feature consisting of 3 parts.

The 3 parts are 3 clockwise polygons external rings.

The following command for such GeoPackage layer:
ogrinfo Polygon_3.gpkg Polygon_3 -geom=SUMMARY

reports:
OGRFeature(Polygon_3):0
  MULTIPOLYGON : 3 geometries:
POLYGON : 7 points
POLYGON : 9 points
POLYGON : 21 points

It seems to me something strange happens converting the GeoPackage layer 
to an ESRI Shapefile layer using e.g:

ogr2ogr -f "ESRI Shapefile" Polygon_3.shp Polygon_3.gpkg

ogrinfo for the resulting ESRI Shapefile layer reports:
OGRFeature(Polygon_3):0
  FID (Integer64) = 0
  MULTIPOLYGON : 2 geometries:
POLYGON : 9 points
POLYGON : 21 points, 1 inner rings (7 points)

That is, the 7 points part has been written in the Shapefile layer as a 
counter clockwise polygon and "attached" to the 21 points part (which 
remains a clockwise polygon external ring) as his inner ring. The 9 
points part remains a clockwise polygon external ring.


On the contrary, converting the GeoPackage layer to GML or FlatGeobuf or 
GeoJSON or Spatialite formats, the resulting layer correctly contains 1 
MultiPolygon feature consisting of 3 clockwise Polygon parts like the 
original layer and ogrinfo reports:

OGRFeature(Polygon_3):0
  MULTIPOLYGON : 3 geometries:
POLYGON : 7 points
POLYGON : 9 points
POLYGON : 21 points

I've checked the orientation of the polygons parts using QGIS 3.22.3 for 
all the layers and also ArcMap/ArcGIS 9.3.1 for the Shapefile layer.


QGIS reports 3 parts for the geometry in the GeoPackage, GML, 
FlatGeobuf, GeoJSON or Spatialite layers, while it reports 2 parts for 
the ESRI Shapefile layer.


Is this an expected behaviour of the ESRI Shapefile driver writing 
multipart features geometries?


Best regards.

Andrea Giudiceandrea

[1] https://github.com/qgis/QGIS/issues/47288
[2] https://drive.google.com/file/d/10HiCARTYJqhAXUckK43nU4MzV1wUbLg9
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] TIF + TFW to GeoTIFF without decompression/recompression

[Andrea Giudiceandrea via gdal-dev]

Hi devs,
gdal_transalte has an "hidden" feature (not mentioned in the docs [1]), 
introduced since GDAL 1.10 [2], which allows to convert .JPG + .JPW 
raster layer to GeoTIFF without decompressing and recompressing the 
raster image (a so called "lossless conversion of JPEG into 
JPEG-in-TIFF" [3]).


It seem to me the same is not possible if the source is a .TIF (e.g. a 
JPEG-in-TIFF) + .TFW raster layer. In this case the image is always 
decompressed and then recompressed if a compression method is specified, 
thus adding another unnecessary lossy step.


I think the decompression / recompression cycle should obviously not be 
needed converting a TIFF file to a GeoTIFF file (unless image 
manipulation options are specified) and gdal_transalte should just copy 
the source TIF file and add the proper GeoTIFF metadata tags taken from 
the TFW file and from the -a_srs parameter.


Maybe is there any other "hidden" feature I haven't been able to find yet?

Best regards.

Andrea Giudiceandrea


[1] https://github.com/OSGeo/gdal/issues/4510
[2] 
https://github.com/OSGeo/gdal/commit/a77c726484d508f12b9f5a9a869313092687
[3] 
https://erouault.blogspot.com/2014/04/advanced-jpeg-in-tiff-uses-in-gdal.html



Side notes:

I know gdal_edit could also be used, but it cannot automatically take 
the georeferencing parameters from the TFW file.


It could also be possible to use the geotifcp cli tool from libgeotiff, 
but it seems it (the one shipped by OSGeo4W) has some issues handling 
JPEG-in-TIFF files (while it works well with other compression formats): 
the "Warning, fractional scanline discarded." and "JPEGLib: Application 
transferred too few scanlines." errors are reported unsuccessfully 
trying to convert a JPEG-in-TIFF + TFW to a GeoTIFF.


Instead, the very old and almost nowhere to be found GeoTiffExamin gui 
tool can just add the proper GeoTIFF metadata tags, taken from the TFW 
file, to a TIFF file without modifying the raster image. Anyway the gui 
tool cannot be used for batch conversion.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev