date:20220110


Felipe,

when using PROJ to transform a PROJ.4 string to WKT2, there's (as far as 
I can think of) no loss of information. Loss of information can occur in 
the other way round.


The main source of loss is that proj4 strings have a limited set of 
known datums, so sometimes you'll just have a +ellps without any 
+towgs84 or +nadgrids, which is underspecified.


When you've +ellps with +towgs84, datum transformations will be limited 
to using WGS84 as a pivot, whereas if you had a WKT2 string with the 
original datum name, PROJ could potentially avoid going through WGS84 
when doing datum transformation.


Here's with +datum=WGS84, your PROJ.4 string is fully qualified. You can 
remove the +ellps=WGS84 and +towgs84=0,0,0 which are completely 
redundant here.


Even

Le 10/01/2022 à 21:06, Felipe Matas via gdal-dev a écrit :
Hi hi, I was looking for someplace to ask about this, and I was in 
doubt is was here or in PROJ, but lets do a try.


Actually, probably I'm not the only one, I have some questions about 
how WKT2 and PROJ4Strings are transformed.


From what I read, a PROJ4String don't have enough info to construct a 
precise WKT2, so, actually how PROJ4String is deprecated my main 
question is how to construct the right WKT2, there is a lot o data 
stored in the old format, there is usually no more information, and 
even the software/hardware that get the data in some way save all of 
it with all the errors.


Actually, we can transform a old CRS to a WKT2:

|> st_crs("+type=crs +proj=utm +zone=19 +south +datum=WGS84 +units=m 
+no_defs +ellps=WGS84 +towgs84=0,0,0" + ) Coordinate Reference System: 
User input: +type=crs +proj=utm +zone=19 +south +datum=WGS84 +units=m 
+no_defs +ellps=WGS84 +towgs84=0,0,0 wkt: BOUNDCRS[ SOURCECRS[ 
PROJCRS["unknown", BASEGEOGCRS["unknown", DATUM["World Geodetic System 
1984", ELLIPSOID["WGS 84",6378137,298.257223563, 
LENGTHUNIT["metre",1]], ID["EPSG",6326]], PRIMEM["Greenwich",0, 
ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8901]]], 
CONVERSION["UTM zone 19S", METHOD["Transverse Mercator", 
ID["EPSG",9807]], PARAMETER["Latitude of natural origin",0, 
ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8801]], 
PARAMETER["Longitude of natural origin",-69, 
ANGLEUNIT["degree",0.0174532925199433], ID["EPSG",8802]], 
PARAMETER["Scale factor at natural origin",0.9996, 
SCALEUNIT["unity",1], ID["EPSG",8805]], PARAMETER["False 
easting",50, LENGTHUNIT["metre",1], ID["EPSG",8806]], 
PARAMETER["False northing",1000, LENGTHUNIT["metre",1], 
ID["EPSG",8807]], ID["EPSG",17019]], CS[Cartesian,2], AXIS["(E)",east, 
ORDER[1], LENGTHUNIT["metre",1, ID["EPSG",9001]]], AXIS["(N)",north, 
ORDER[2], LENGTHUNIT["metre",1, ID["EPSG",9001], TARGETCRS[ 
GEOGCRS["WGS 84", DATUM["World Geodetic System 1984", ELLIPSOID["WGS 
84",6378137,298.257223563, LENGTHUNIT["metre",1]]], 
PRIMEM["Greenwich",0, ANGLEUNIT["degree",0.0174532925199433]], 
CS[ellipsoidal,2], AXIS["latitude",north, ORDER[1], 
ANGLEUNIT["degree",0.0174532925199433]], AXIS["longitude",east, 
ORDER[2], ANGLEUNIT["degree",0.0174532925199433]], ID["EPSG",4326]]], 
ABRIDGEDTRANSFORMATION["Transformation from unknown to WGS84", 
METHOD["Geocentric translations (geog2D domain)", ID["EPSG",9603]], 
PARAMETER["X-axis translation",0, ID["EPSG",8605]], PARAMETER["Y-axis 
translation",0, ID["EPSG",8606]], PARAMETER["Z-axis translation",0, 
ID["EPSG",8607|


So, in some way, GDAL is able to cover this "breach" of information 
betwen PROJ4String and WKT2, but is not like the breach does not 
exist, it was just handled in some way, so, the lack of info, also 
means, there can be othe aproximations to that CRS.


Maybe I'm just confused, and mixing things the things I read, I'm just 
trying to figure it out, what and how would be the right 
transformation from proj4string to wkt2.


Thx.


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] [Question] PROJ4String to WKT2 Transformations

Hi,

The more or less same question seems to be asked some time ago in 
gis.stackexchange 
https://gis.stackexchange.com/questions/420378/can-we-construct-wkt2-from-proj4string-correctly.

-Jukka Rahkonen-


Lähettäjä: gdal-dev  Puolesta Felipe Matas 
via gdal-dev
Lähetetty: maanantai 10. tammikuuta 2022 22.07
Vastaanottaja: gdal-dev@lists.osgeo.org
Aihe: [gdal-dev] [Question] PROJ4String to WKT2 Transformations

Hi hi, I was looking for someplace to ask about this, and I was in doubt is was 
here or in PROJ, but lets do a try.

Actually, probably I'm not the only one, I have some questions about how WKT2 
and PROJ4Strings are transformed.

From what I read, a PROJ4String don't have enough info to construct a precise 
WKT2, so, actually how PROJ4String is deprecated my main question is how to 
construct the right WKT2, there is a lot o data stored in the old format, there 
is usually no more information, and even the software/hardware that get the 
data in some way save all of it with all the errors.

Actually, we can transform a old CRS to a WKT2:


> st_crs("+type=crs +proj=utm +zone=19 +south +datum=WGS84 +units=m +no_defs 
> +ellps=WGS84 +towgs84=0,0,0"

+ )

Coordinate Reference System:

  User input: +type=crs +proj=utm +zone=19 +south +datum=WGS84 +units=m 
+no_defs +ellps=WGS84 +towgs84=0,0,0

  wkt:

BOUNDCRS[

SOURCECRS[

PROJCRS["unknown",

BASEGEOGCRS["unknown",

DATUM["World Geodetic System 1984",

ELLIPSOID["WGS 84",6378137,298.257223563,

LENGTHUNIT["metre",1]],

ID["EPSG",6326]],

PRIMEM["Greenwich",0,

ANGLEUNIT["degree",0.0174532925199433],

ID["EPSG",8901]]],

CONVERSION["UTM zone 19S",

METHOD["Transverse Mercator",

ID["EPSG",9807]],

PARAMETER["Latitude of natural origin",0,

ANGLEUNIT["degree",0.0174532925199433],

ID["EPSG",8801]],

PARAMETER["Longitude of natural origin",-69,

ANGLEUNIT["degree",0.0174532925199433],

ID["EPSG",8802]],

PARAMETER["Scale factor at natural origin",0.9996,

SCALEUNIT["unity",1],

ID["EPSG",8805]],

PARAMETER["False easting",50,

LENGTHUNIT["metre",1],

ID["EPSG",8806]],

PARAMETER["False northing",1000,

LENGTHUNIT["metre",1],

ID["EPSG",8807]],

ID["EPSG",17019]],

CS[Cartesian,2],

AXIS["(E)",east,

ORDER[1],

LENGTHUNIT["metre",1,

ID["EPSG",9001]]],

AXIS["(N)",north,

ORDER[2],

LENGTHUNIT["metre",1,

ID["EPSG",9001],

TARGETCRS[

GEOGCRS["WGS 84",

DATUM["World Geodetic System 1984",

ELLIPSOID["WGS 84",6378137,298.257223563,

LENGTHUNIT["metre",1]]],

PRIMEM["Greenwich",0,

ANGLEUNIT["degree",0.0174532925199433]],

CS[ellipsoidal,2],

AXIS["latitude",north,

ORDER[1],

ANGLEUNIT["degree",0.0174532925199433]],

AXIS["longitude",east,

ORDER[2],

ANGLEUNIT["degree",0.0174532925199433]],

ID["EPSG",4326]]],

ABRIDGEDTRANSFORMATION["Transformation from unknown to WGS84",

METHOD["Geocentric translations (geog2D domain)",

ID["EPSG",9603]],

PARAMETER["X-axis translation",0,

ID["EPSG",8605]],

PARAMETER["Y-axis translation",0,

ID["EPSG",8606]],

PARAMETER["Z-axis translation",0,

ID["EPSG",8607

So, in some way, GDAL is able to cover this "breach" of information betwen 
PROJ4String and WKT2, but is not like the breach does not exist, it was just 
handled in some way, so, the lack of info, also means, there can be othe 
aproximations to that CRS.

Maybe I'm just confused, and mixing things the things I read, I'm just trying 
to figure it out, what and how would be the right transformation from 
proj4string to wkt2.

Thx.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Mapsforge binary map format


Hi Matthias,

Le 10/01/2022 à 19:33, Matthias Kuhn a écrit :

Hi,

I just came across the .map format, a tiled vector format (see 
https://www.openandromaps.org/en/ )


I tried opening the files and failed, is there support for this in 
gdal/ogr?


I don't think so.

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Mapsforge binary map format

2022-01-10 Thread Matthias Kuhn

Hi,

I just came across the .map format, a tiled vector format (see
https://www.openandromaps.org/en/)

I tried opening the files and failed, is there support for this in gdal/ogr?

Thanks
Matthias

Previous question on the topic:
https://www.openandromaps.org/en/oam-forums/topic/openandromaps-in-qgis
Specification of the format
https://github.com/mapsforge/mapsforge/blob/master/docs/Specification-Binary-Map-File.md
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] RPCs in GeoTIFF

2022-01-10 Thread Matt.Wilkie

Thanks for the nudge. I implemented that code as a simple command line script a 
couple of months ago but neglected to put it anywhere someone else could use 
it. Fixed that today ;-)

https://gist.github.com/maphew/29a73ee7a4517663cef5081c15799fc3

```
'''Copy RPC metdata from IN raster to OUT raster

Adapted from @user7821537
https://gis.stackexchange.com/questions/264644/transfer-rpc-metadata-from-one-geotiff-to-another
'''
import os
import sys
from osgeo import gdal

gdal.UseExceptions()

if len(sys.argv) < 3:
print(f"Usage: {sys.argv[0]} [in_file] [out_file]")
sys.exit(1)

infile = sys.argv[1]# source filename and path
output = sys.argv[2]# destination file

data_in = gdal.Open(infile, gdal.GA_ReadOnly)
data_out = gdal.Open(output, gdal.GA_Update)

# get the RPCs from the first file ...
rpcs = data_in.GetMetadata('RPC')

# ... write them to the second file
data_out.SetMetadata(rpcs ,'RPC')

# de-reference the datasets, which triggers gdal to save
data_in = None
data_out = None

```

-Matt

From: gdal-dev  On Behalf Of ni hao
Sent: January 4, 2022 3:00 PM
To: gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] RPCs in GeoTIFF


*** External email: Do not click on links or attachments except from trusted 
senders. ***

**



Hi list,

https://gis.stackexchange.com/questions/264644/transfer-rpc-metadata-from-one-geotiff-to-another
The above link showed me how to do it.

thanks.


From: gdal-dev 
mailto:gdal-dev-boun...@lists.osgeo.org>> on 
behalf of ni hao mailto:ni_ha...@hotmail.com>>
Sent: January 3, 2022 2:03 PM
To: gdal-dev@lists.osgeo.org 
mailto:gdal-dev@lists.osgeo.org>>
Subject: [gdal-dev] RPCs in GeoTIFF

Hi list,

I use gdal-2.4.4.
I use gdalwarp to orthorectify RADARSAT-2 images with RPCs (in product.xml).
Now I have done some manipulation and save the result in GTiff (e.g., apply 
Sigma0 LUT).
How can I copy the RPC Metadata from product.xml into GTiff, so I can use 
gdalwarp RPC orthorectification?
What are the Python commands to call ?
Do I have to use VRT?

Thank you!

ds=gdal.Open(r"C:\RADARSAT-2\2008Feb10\product.xml")
ds.GetMetadata(domain='RPC')
{'HEIGHT_OFF': '1.400e+02', 'LINE_NUM_COEFF': 
'1.045003883698053e-03 -3.909147136274753e-05 9.670521993287788e-09 
-1.540774061983927e-08 -1.230034082454030e-01 2.994775580755049e-08 
3.800851793724760e-08 -7.321141763443381e-04 -3.070513868102579e-08 
-3.263789159357597e-08 -1.074590548649705e+00 -1.422870424040082e-07 
2.625729922759320e-07 3.515681824455183e-04 1.737795244565543e-08 
-1.191967212586014e-06 -6.679635986447045e-04 9.917030647189506e-09 
1.183063120029544e-06 9.020179141273994e-07', 'SAMP_SCALE': '1759', 'LONG_OFF': 
'-5.5862000e+00', 'SAMP_DEN_COEFF': '1.000e+00 
-3.873791825160030e-04 -1.654041698986790e-06 1.045768201431545e-08 
-7.357169048971613e-04 -5.663646053748347e-06 -2.270764362304154e-08 
-7.511863551214131e-07 -3.061984244257416e-07 1.247909289740085e-07 
1.555977062556944e-03 2.290649610770613e-06 1.138900631680699e-09 
-4.788174308312700e-07 1.913987966778219e-07 -5.724932691905833e-08 
5.283021826505342e-06 -2.470297385173699e-08 -5.008669473258436e-08 
1.486018900494917e-08', 'ERR_BIAS': '5.000e-01', 'SAMP_NUM_COEFF': 
'1.448914459955711e-03 3.107453606755890e-02 -1.362345055216650e-05 
-5.545316452513627e-08 1.177884096135199e+00 2.873729811978911e-04 
-1.989082475494487e-06 -1.04193883315e-03 8.430382868139787e-06 
-1.570423196668063e-06 -2.556805573684327e-01 -1.724821081109771e-05 
4.713416494967566e-07 -9.676787333902578e-04 -2.144746372053350e-06 
8.115347578419969e-07 1.767758474109092e-04 2.353735909723349e-07 
-6.444353128930045e-06 2.385344654290232e-07', 'LONG_SCALE': 
'1.859e-01', 'LINE_DEN_COEFF': '1.000e+00 
-9.804190622020718e-08 -2.386942243966555e-07 1.801605395896790e-08 
4.160880031382031e-04 -2.095235371483683e-08 -1.659952797934781e-08 
-9.496784414468805e-07 4.758902993568164e-08 5.402628711262573e-08 
4.472124981058622e-04 1.370099419045734e-08 -1.785667248922995e-08 
7.981478756298640e-07 -7.327071351551446e-09 2.253346171933004e-07 
1.637317283006711e-06 -5.178220338393535e-08 1.880261293686099e-06 
5.550513900110941e-06', 'SAMP_OFF': '1705', 'LAT_SCALE': 
'2.073e-01', 'LAT_OFF': '3.5905800e+01', 'LINE_SCALE': 
'4229', 'LINE_OFF': '4227', 'ERR_RAND': '1.000e-01', 
'HEIGHT_SCALE': '5.010e+02'}

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Bug in OGR library

Thanks Even for very quick bug fix in QGIS!
Kind regards,
Marta

From: Even Rouault 
Sent: Monday, January 10, 2022 3:22 PM
To: Kopszak Marta - Partner Hurt ; 
gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] Bug in OGR library

Marta,

I was lucky enough to figure out from the error messages that your issue came 
when using QGIS, but it would have been appropriate to mention it from the 
start so we get the global picture.

This is in fact a QGIS bug, which is only seen with the ODS & XLSX driver since 
they use the OGR Memory driver underneath and it has special behavior when 
creating a feature with a set FID.

I've submitted a QGIS bug fix for that in 
https://github.com/qgis/QGIS/pull/46773

Even
Le 10/01/2022 à 13:34, Kopszak Marta - Partner Hurt a écrit :
Hi Even,
Thanks for your replay. More info about the problem:

Error:
Save error:
Export to vector file failed.
Error: Feature write errors:
Feature creation error (OGR error: negative FID are not supported)

The database has more than 2 bilons of records (record – I mean record in the 
database table). The error occurs when I export a single table witout the 
primary key to .xlsx.

I suppose that when the table doesn’t have the primary key it use tid – the 
tuple identifier, and the size of tid may exceed the size of a 32-bit integer. 
It cause the problem to export this table to .xlsx format by OGR library.

Regards,
Marta Kopszak

From: gdal-dev 
 On 
Behalf Of Even Rouault
Sent: Monday, January 10, 2022 11:42 AM
To: gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] Bug in OGR library

Marta,

What is the bug exactly ?

"2 billion records" : by record, you mean cells, like a sheet with 2048 columns 
and 1048576 rows ? By quickly looking at the code I can't see an obvious 
limitation, but that doesn't mean anything. Closer investigation would be 
needed. Please file an issue at https://github.com/OSGeo/gdal/issues/new with 
exact details how to reproduce (number of rows/features, and columns/fields in 
particular).

But even if the writer side would work, the OGR reader side would probably have 
issues as the whole spreadsheet will be loaded into RAM and for 2 billion 
cells, I would expect at least 32 or more GB of RAM to be needed. Perhaps 
spreadsheet software are able to cope with that in a smarter way.

Even
Le 10/01/2022 à 11:27, Kopszak Marta - Partner Hurt a écrit :
Dear Sirs,
I ran into an error when exporting data through the OGR library to the .xlsx 
and .ods format. The bug occurs when there are more than 2 billion records in 
the database. I know the problem was solved for other formats, e.g. csv. Will 
the problem be solved also for other formats (e.g. xlsx, .ods)? If this is not 
the correct e-mail, please forward this e-mail to the appropriate person.

Kind regards,
Marta Kopszak

Marta Kopszak
Współpracownik

Rozwój Sieci Stacjonarnej, Wydział Rozwoju Planowania
Orange Polska, gen. Romualda Traugutta 55, 50-416 Wrocław | RODO - informacja o 
danych 

___

gdal-dev mailing list

gdal-dev@lists.osgeo.org

https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 

http://www.spatialys.com

My software is free, but my time generally not.

--

http://www.spatialys.com

My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Discrepancy in proj values between ubuntu-small and ubuntu-full images

2022-01-10 Thread Rajsekar Manokaran

Hi Even,

Thanks for explaining the difference between the images.  You're absolutely
right:  I indeed assumed the two were using the same ellipsoids.  If I use
epsg:4269 (nad83 ell.) as the src, all the numbers are consistent.

-
Regards
Rajsekar


On Mon, Jan 10, 2022 at 8:05 PM Even Rouault 
wrote:

> Rajsekar,
>
> One of the reason that the -full image is larger than the -small one is
> that the former includes > 500 MB of geodetic grids, whereas the later only
> includes very basic ones. (I've just amended
> https://github.com/OSGeo/gdal/blob/master/docker/README.md to make it
> clearer)
>
> As you do a WGS 84 to NAD83 transformation, the ubuntu-full image can use
> the HPGN grids, and in particular the Pennsylviana grid
>
> $ echo -77.75 40.16  | PROJ_DEBUG=2 bin/cs2cs
> '+init=epsg:4326' '+to' '+init=epsg:2271'
> [ ... snip ... ]
> Using coordinate operation axis order change (2D) + Inverse of NAD83 to
> WGS 84 (57) + SPCS83 Pennsylvania North zone (US Survey feet)
> pj_open_lib(us_noaa_pahpgn.tif): call
> fopen(/home/even/proj/PROJ-data/us_noaa/us_noaa_pahpgn.tif) - succeeded
> 1968499.39-0.15 0.00
>
> Whereas the -small image has no such grid and will assume that NAD83 ==
> WGS84.
>
> In theory the result with the -full image should be more accurate, but it
> depends if somewhere in your workflow someone has not assumed that NAD83 ==
> WGS84 ...
>
> Anyway WGS 84 is literaly a moving and somewhat fuzzy target, so you
> shouldn't expect in general results to have an accuracy better than ~ 2m in
> any transformation involving it.
>
> Even
> Le 10/01/2022 à 15:21, Rajsekar Manokaran a écrit :
>
> Hi gdal devs and users,
>
> I came across a discrepancy in proj. values between ubuntu-small and
> ubuntu-full docker images even for the same gdal version tag.  I tried to
> convert a few coords from epsg:4326 to epsg:2271.  The coords are:
>
> Input:
> -77.75 40.16  # this is the center meridian, lat of 2271
> -75.18094433657971 40.25904959199275  # another sample point
>
> The outputs of all ubuntu-small-* images seem to be correct, but the
> ubuntu-full-* images have about 1 feet error (2271 uses US_SURVEY_FEET
> units).
>
> Here's how I run it:
>
> $ docker run --rm -i osgeo/gdal:"$tag" cs2cs '+init=epsg:4326' '+to'
> '+init=epsg:2271' < input
> where tag is the docker tag: "ubuntu-full-3.2.2", or "ubuntu-small-3.4.1",
> etc.
>
> Here are the outputs for the ubuntu-full images:
>
> output-ubuntu-full-3.2.2
> 1968499.39  -0.15 0.00
> 2685535.97  44295.90 0.00
> output-ubuntu-full-3.3.3
> 1968499.39  -0.15 0.00
> 2685535.97  44295.90 0.00
> output-ubuntu-full-3.4.1
> 1968499.39  -0.15 0.00
> 2685535.97  44295.90 0.00
>
> Here are the outputs for the small images:
>
> output-ubuntu-small-3.2.2
> 1968500.00  -0.00 0.00
> 2685536.18  44296.27 0.00
> output-ubuntu-small-3.3.3
> 1968500.00  -0.00 0.00
> 2685536.18  44296.27 0.00
> output-ubuntu-small-3.4.1
> 1968500.00  -0.00 0.00
> 2685536.18  44296.27 0.00
>
> The first input is the center and so should translate to "(false east,
> false north)".  Going by that, the small versions seem to be more correct.
> Any idea why this difference in the values?
>
> -
> Rajsekar
>
>
>
> ___
> gdal-dev mailing 
> listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Discrepancy in proj values between ubuntu-small and ubuntu-full images


Rajsekar,

One of the reason that the -full image is larger than the -small one is 
that the former includes > 500 MB of geodetic grids, whereas the later 
only includes very basic ones. (I've just amended 
https://github.com/OSGeo/gdal/blob/master/docker/README.md to make it 
clearer)


As you do a WGS 84 to NAD83 transformation, the ubuntu-full image can 
use the HPGN grids, and in particular the Pennsylviana grid


$ echo -77.75 40.16  | PROJ_DEBUG=2 bin/cs2cs 
'+init=epsg:4326' '+to' '+init=epsg:2271'

[ ... snip ... ]
Using coordinate operation axis order change (2D) + Inverse of NAD83 to 
WGS 84 (57) + SPCS83 Pennsylvania North zone (US Survey feet)
pj_open_lib(us_noaa_pahpgn.tif): call 
fopen(/home/even/proj/PROJ-data/us_noaa/us_noaa_pahpgn.tif) - succeeded

1968499.39    -0.15 0.00

Whereas the -small image has no such grid and will assume that NAD83 == 
WGS84.


In theory the result with the -full image should be more accurate, but 
it depends if somewhere in your workflow someone has not assumed that 
NAD83 == WGS84 ...


Anyway WGS 84 is literaly a moving and somewhat fuzzy target, so you 
shouldn't expect in general results to have an accuracy better than ~ 2m 
in any transformation involving it.


Even

Le 10/01/2022 à 15:21, Rajsekar Manokaran a écrit :

Hi gdal devs and users,

I came across a discrepancy in proj. values between ubuntu-small and 
ubuntu-full docker images even for the same gdal version tag.  I tried 
to convert a few coords from epsg:4326 to epsg:2271.  The coords are:


Input:
-77.75 40.16  # this is the center meridian, lat of 2271
-75.18094433657971 40.25904959199275  # another sample point

The outputs of all ubuntu-small-* images seem to be correct, but the 
ubuntu-full-* images have about 1 feet error (2271 uses US_SURVEY_FEET 
units).


Here's how I run it:

$ docker run --rm -i osgeo/gdal:"$tag" cs2cs '+init=epsg:4326' '+to' 
'+init=epsg:2271' < input
where tag is the docker tag: "ubuntu-full-3.2.2", or 
"ubuntu-small-3.4.1", etc.


Here are the outputs for the ubuntu-full images:

output-ubuntu-full-3.2.2
1968499.39      -0.15 0.00
2685535.97      44295.90 0.00
output-ubuntu-full-3.3.3
1968499.39      -0.15 0.00
2685535.97      44295.90 0.00
output-ubuntu-full-3.4.1
1968499.39      -0.15 0.00
2685535.97      44295.90 0.00

Here are the outputs for the small images:

output-ubuntu-small-3.2.2
1968500.00      -0.00 0.00
2685536.18      44296.27 0.00
output-ubuntu-small-3.3.3
1968500.00      -0.00 0.00
2685536.18      44296.27 0.00
output-ubuntu-small-3.4.1
1968500.00      -0.00 0.00
2685536.18      44296.27 0.00

The first input is the center and so should translate to "(false east, 
false north)".  Going by that, the small versions seem to be more 
correct.  Any idea why this difference in the values?


-
Rajsekar



___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Bug in OGR library


Marta,

I was lucky enough to figure out from the error messages that your issue 
came when using QGIS, but it would have been appropriate to mention it 
from the start so we get the global picture.


This is in fact a QGIS bug, which is only seen with the ODS & XLSX 
driver since they use the OGR Memory driver underneath and it has 
special behavior when creating a feature with a set FID.


I've submitted a QGIS bug fix for that in 
https://github.com/qgis/QGIS/pull/46773


Even

Le 10/01/2022 à 13:34, Kopszak Marta - Partner Hurt a écrit :


Hi Even,

Thanks for your replay. More info about the problem:

*Error:*
/Save error:/

/Export to vector file failed./

/Error: Feature write errors:/

/Feature creation error (OGR error: negative FID are not supported)/

//

The database has more than 2 bilons of records (record – I mean record 
in the database table). The error occurs when I export a single table 
witout the primary key to .xlsx.


I suppose that when the table doesn’t have the primary key it use tid 
– the tuple identifier, and the size of tid may exceed the size of a 
32-bit integer. It cause the problem to export this table to .xlsx 
format by OGR library.


Regards,

Marta Kopszak

*From:*gdal-dev  *On Behalf Of *Even 
Rouault

*Sent:* Monday, January 10, 2022 11:42 AM
*To:* gdal-dev@lists.osgeo.org
*Subject:* Re: [gdal-dev] Bug in OGR library

Marta,

What is the bug exactly ?

"2 billion records" : by record, you mean cells, like a sheet with 
2048 columns and 1048576 rows ? By quickly looking at the code I can't 
see an obvious limitation, but that doesn't mean anything. Closer 
investigation would be needed. Please file an issue at 
https://github.com/OSGeo/gdal/issues/new 
 with exact details how to 
reproduce (number of rows/features, and columns/fields in particular).


But even if the writer side would work, the OGR reader side would 
probably have issues as the whole spreadsheet will be loaded into RAM 
and for 2 billion cells, I would expect at least 32 or more GB of RAM 
to be needed. Perhaps spreadsheet software are able to cope with that 
in a smarter way.


Even

Le 10/01/2022 à 11:27, Kopszak Marta - Partner Hurt a écrit :

Dear Sirs,

I ran into an error when exporting data through the OGR library to
the .xlsx and .ods format. The bug occurs when there are more than
2 billion records in the database. I know the problem was solved
for other formats, e.g. csv. Will the problem be solved also for
other formats (e.g. xlsx, .ods)? If this is not the correct
e-mail, please forward this e-mail to the appropriate person.

Kind regards,

Marta Kopszak


*Marta Kopszak*
Współpracownik 

Rozwój Sieci Stacjonarnej, Wydział Rozwoju Planowania
Orange Polska, gen. Romualda Traugutta 55, 50-416 Wrocław | _RODO
- informacja o danych_ 






___  

gdal-dev mailing list  

_gdal-dev@lists.osgeo.org_  

_https://lists.osgeo.org/mailman/listinfo/gdal-dev_  


--   
_http://www.spatialys.com_  
My software is free, but my time generally not.  


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Discrepancy in proj values between ubuntu-small and ubuntu-full images

2022-01-10 Thread Rajsekar Manokaran

Hi gdal devs and users,

I came across a discrepancy in proj. values between ubuntu-small and
ubuntu-full docker images even for the same gdal version tag.  I tried to
convert a few coords from epsg:4326 to epsg:2271.  The coords are:

Input:
-77.75 40.16  # this is the center meridian, lat of 2271
-75.18094433657971 40.25904959199275  # another sample point

The outputs of all ubuntu-small-* images seem to be correct, but the
ubuntu-full-* images have about 1 feet error (2271 uses US_SURVEY_FEET
units).

Here's how I run it:

$ docker run --rm -i osgeo/gdal:"$tag" cs2cs '+init=epsg:4326' '+to'
'+init=epsg:2271' < input
where tag is the docker tag: "ubuntu-full-3.2.2", or "ubuntu-small-3.4.1",
etc.

Here are the outputs for the ubuntu-full images:

output-ubuntu-full-3.2.2
1968499.39  -0.15 0.00
2685535.97  44295.90 0.00
output-ubuntu-full-3.3.3
1968499.39  -0.15 0.00
2685535.97  44295.90 0.00
output-ubuntu-full-3.4.1
1968499.39  -0.15 0.00
2685535.97  44295.90 0.00

Here are the outputs for the small images:

output-ubuntu-small-3.2.2
1968500.00  -0.00 0.00
2685536.18  44296.27 0.00
output-ubuntu-small-3.3.3
1968500.00  -0.00 0.00
2685536.18  44296.27 0.00
output-ubuntu-small-3.4.1
1968500.00  -0.00 0.00
2685536.18  44296.27 0.00

The first input is the center and so should translate to "(false east,
false north)".  Going by that, the small versions seem to be more correct.
Any idea why this difference in the values?

-
Rajsekar
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Bug in OGR library

Hi,

I asked you to create a GML with SQL query from one feature in a hope that the 
huuuge value from your database would be generated as a FID and appear in GML. 
Did you try what happens?

So if missing primary key is the main problem why not to add a primary key into 
your table? Or did you already try to run ogr2ogr with -unsetFid?

-Jukka Rahkonen-

Lähettäjä: Kopszak Marta - Partner Hurt 
Lähetetty: maanantai 10. tammikuuta 2022 16.09
Vastaanottaja: Rahkonen Jukka (MML) ; Even 
Rouault ; gdal-dev@lists.osgeo.org
Aihe: RE: [gdal-dev] Bug in OGR library

I think a single record wouldn't help you. If I would copy only one record to 
another empty database it could be exported to .xlsx. We have a postgres 
database and a table without any primary key, so QGIS uses a tid. The overall 
size of the database is huuuge (billions of records), so the tid reaches high 
values and apparently overflows int4.

Export to CSV goes fine, so it seems OGR’s CSV driver uses int8, but when 
exporting to XLSX or ODS you get that error. Maybe it’s just as simple as 
change id datatypes in affected drivers?

From: Rahkonen Jukka (MML) 
mailto:jukka.rahko...@maanmittauslaitos.fi>>
Sent: Monday, January 10, 2022 2:24 PM
To: Kopszak Marta - Partner Hurt 
mailto:marta.kops...@orange.com>>; Even Rouault 
mailto:even.roua...@spatialys.com>>; 
gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] Bug in OGR library

Hi,

Now it starts to make sense. Could you paste the contents of a single row that 
you can’t write into xlsx format? You can capture it with ogrinfo “ogrinto 
[database connection] -sql "select * from my_table limit 1". Or maybe better to 
convert that row into GML format and attach that to your mail.

-Jukka Rahkonen-

Lähettäjä: Kopszak Marta - Partner Hurt 
mailto:marta.kops...@orange.com>>
Lähetetty: maanantai 10. tammikuuta 2022 15.07
Vastaanottaja: Rahkonen Jukka (MML) 
mailto:jukka.rahko...@maanmittauslaitos.fi>>;
 Even Rouault mailto:even.roua...@spatialys.com>>; 
gdal-dev@lists.osgeo.org
Aihe: RE: [gdal-dev] Bug in OGR library

I have 2 bilions of rows in the  whole database and I try to export only one 
table which has 56 columns and 423 thousands of rows so it’s not the issue with 
Microsoft Excel limitation. I can’t write even a single row (error Feature 
creation error (OGR error: negative FID are not supported) occurs immediately),
Marta Kopszak

Hi,

Could you clarify a bit? You have billion rows in the database table but xlsx 
and ods can support at maximum a bit more than one million rows (1,048,576). 
You can’t imagine to have success with converting the whole table. Or is your 
problem that you can’t write even a single row from the source table because 
the value of the ID key is too big for xlsx/ods formats?

-Jukka Rahkonen-

Lähettäjä: gdal-dev 
mailto:gdal-dev-boun...@lists.osgeo.org>> 
Puolesta Kopszak Marta - Partner Hurt
Lähetetty: maanantai 10. tammikuuta 2022 14.34
Vastaanottaja: Even Rouault 
mailto:even.roua...@spatialys.com>>; 
gdal-dev@lists.osgeo.org
Aihe: Re: [gdal-dev] Bug in OGR library

Hi Even,
Thanks for your replay. More info about the problem:

Error:
Save error:
Export to vector file failed.
Error: Feature write errors:
Feature creation error (OGR error: negative FID are not supported)

The database has more than 2 bilons of records (record – I mean record in the 
database table). The error occurs when I export a single table witout the 
primary key to .xlsx.

I suppose that when the table doesn’t have the primary key it use tid – the 
tuple identifier, and the size of tid may exceed the size of a 32-bit integer. 
It cause the problem to export this table to .xlsx format by OGR library.

Regards,
Marta Kopszak

From: gdal-dev 
mailto:gdal-dev-boun...@lists.osgeo.org>> On 
Behalf Of Even Rouault
Sent: Monday, January 10, 2022 11:42 AM
To: gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] Bug in OGR library


Marta,

What is the bug exactly ?

"2 billion records" : by record, you mean cells, like a sheet with 2048 columns 
and 1048576 rows ? By quickly looking at the code I can't see an obvious 
limitation, but that doesn't mean anything. Closer investigation would be 
needed. Please file an issue at https://github.com/OSGeo/gdal/issues/new with 
exact details how to reproduce (number of rows/features, and columns/fields in 
particular).

But even if the writer side would work, the OGR reader side would probably have 
issues as the whole spreadsheet will be loaded into RAM and for 2 billion 
cells, I would expect at least 32 or more GB of RAM to be needed. Perhaps 
spreadsheet software are able to cope with that in a smarter way.

Even
Le 10/01/2022 à 11:27, Kopszak Marta - Partner Hurt a écrit :
Dear Sirs,
I ran into an error when exporting data through the OGR library to the .xlsx 
and .ods format. The bug occurs when there are more than 2 billion

Re: [gdal-dev] Bug in OGR library

I think a single record wouldn't help you. If I would copy only one record to 
another empty database it could be exported to .xlsx. We have a postgres 
database and a table without any primary key, so QGIS uses a tid. The overall 
size of the database is huuuge (billions of records), so the tid reaches high 
values and apparently overflows int4.

Export to CSV goes fine, so it seems OGR’s CSV driver uses int8, but when 
exporting to XLSX or ODS you get that error. Maybe it’s just as simple as 
change id datatypes in affected drivers?
From: Rahkonen Jukka (MML) 
Sent: Monday, January 10, 2022 2:24 PM
To: Kopszak Marta - Partner Hurt ; Even Rouault 
; gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] Bug in OGR library

Hi,

Now it starts to make sense. Could you paste the contents of a single row that 
you can’t write into xlsx format? You can capture it with ogrinfo “ogrinto 
[database connection] -sql "select * from my_table limit 1". Or maybe better to 
convert that row into GML format and attach that to your mail.

-Jukka Rahkonen-

Lähettäjä: Kopszak Marta - Partner Hurt 
mailto:marta.kops...@orange.com>>
Lähetetty: maanantai 10. tammikuuta 2022 15.07
Vastaanottaja: Rahkonen Jukka (MML) 
mailto:jukka.rahko...@maanmittauslaitos.fi>>;
 Even Rouault mailto:even.roua...@spatialys.com>>; 
gdal-dev@lists.osgeo.org
Aihe: RE: [gdal-dev] Bug in OGR library

I have 2 bilions of rows in the  whole database and I try to export only one 
table which has 56 columns and 423 thousands of rows so it’s not the issue with 
Microsoft Excel limitation. I can’t write even a single row (error Feature 
creation error (OGR error: negative FID are not supported) occurs immediately),
Marta Kopszak

Hi,

Could you clarify a bit? You have billion rows in the database table but xlsx 
and ods can support at maximum a bit more than one million rows (1,048,576). 
You can’t imagine to have success with converting the whole table. Or is your 
problem that you can’t write even a single row from the source table because 
the value of the ID key is too big for xlsx/ods formats?

-Jukka Rahkonen-

Lähettäjä: gdal-dev 
mailto:gdal-dev-boun...@lists.osgeo.org>> 
Puolesta Kopszak Marta - Partner Hurt
Lähetetty: maanantai 10. tammikuuta 2022 14.34
Vastaanottaja: Even Rouault 
mailto:even.roua...@spatialys.com>>; 
gdal-dev@lists.osgeo.org
Aihe: Re: [gdal-dev] Bug in OGR library

Hi Even,
Thanks for your replay. More info about the problem:

Error:
Save error:
Export to vector file failed.
Error: Feature write errors:
Feature creation error (OGR error: negative FID are not supported)

The database has more than 2 bilons of records (record – I mean record in the 
database table). The error occurs when I export a single table witout the 
primary key to .xlsx.

I suppose that when the table doesn’t have the primary key it use tid – the 
tuple identifier, and the size of tid may exceed the size of a 32-bit integer. 
It cause the problem to export this table to .xlsx format by OGR library.

Regards,
Marta Kopszak

From: gdal-dev 
mailto:gdal-dev-boun...@lists.osgeo.org>> On 
Behalf Of Even Rouault
Sent: Monday, January 10, 2022 11:42 AM
To: gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] Bug in OGR library


Marta,

What is the bug exactly ?

"2 billion records" : by record, you mean cells, like a sheet with 2048 columns 
and 1048576 rows ? By quickly looking at the code I can't see an obvious 
limitation, but that doesn't mean anything. Closer investigation would be 
needed. Please file an issue at https://github.com/OSGeo/gdal/issues/new with 
exact details how to reproduce (number of rows/features, and columns/fields in 
particular).

But even if the writer side would work, the OGR reader side would probably have 
issues as the whole spreadsheet will be loaded into RAM and for 2 billion 
cells, I would expect at least 32 or more GB of RAM to be needed. Perhaps 
spreadsheet software are able to cope with that in a smarter way.

Even
Le 10/01/2022 à 11:27, Kopszak Marta - Partner Hurt a écrit :
Dear Sirs,
I ran into an error when exporting data through the OGR library to the .xlsx 
and .ods format. The bug occurs when there are more than 2 billion records in 
the database. I know the problem was solved for other formats, e.g. csv. Will 
the problem be solved also for other formats (e.g. xlsx, .ods)? If this is not 
the correct e-mail, please forward this e-mail to the appropriate person.

Kind regards,
Marta Kopszak

Marta Kopszak
Współpracownik

Rozwój Sieci Stacjonarnej, Wydział Rozwoju Planowania
Orange Polska, gen. Romualda Traugutta 55, 50-416 Wrocław | RODO - informacja o 
danych 

 



___

gdal-dev mailing list

Re: [gdal-dev] Bug in OGR library

Hi,

Now it starts to make sense. Could you paste the contents of a single row that 
you can’t write into xlsx format? You can capture it with ogrinfo “ogrinto 
[database connection] -sql "select * from my_table limit 1". Or maybe better to 
convert that row into GML format and attach that to your mail.

-Jukka Rahkonen-

Lähettäjä: Kopszak Marta - Partner Hurt 
Lähetetty: maanantai 10. tammikuuta 2022 15.07
Vastaanottaja: Rahkonen Jukka (MML) ; Even 
Rouault ; gdal-dev@lists.osgeo.org
Aihe: RE: [gdal-dev] Bug in OGR library

I have 2 bilions of rows in the  whole database and I try to export only one 
table which has 56 columns and 423 thousands of rows so it’s not the issue with 
Microsoft Excel limitation. I can’t write even a single row (error Feature 
creation error (OGR error: negative FID are not supported) occurs immediately),
Marta Kopszak

Hi,

Could you clarify a bit? You have billion rows in the database table but xlsx 
and ods can support at maximum a bit more than one million rows (1,048,576). 
You can’t imagine to have success with converting the whole table. Or is your 
problem that you can’t write even a single row from the source table because 
the value of the ID key is too big for xlsx/ods formats?

-Jukka Rahkonen-

Lähettäjä: gdal-dev 
mailto:gdal-dev-boun...@lists.osgeo.org>> 
Puolesta Kopszak Marta - Partner Hurt
Lähetetty: maanantai 10. tammikuuta 2022 14.34
Vastaanottaja: Even Rouault 
mailto:even.roua...@spatialys.com>>; 
gdal-dev@lists.osgeo.org
Aihe: Re: [gdal-dev] Bug in OGR library

Hi Even,
Thanks for your replay. More info about the problem:

Error:
Save error:
Export to vector file failed.
Error: Feature write errors:
Feature creation error (OGR error: negative FID are not supported)

The database has more than 2 bilons of records (record – I mean record in the 
database table). The error occurs when I export a single table witout the 
primary key to .xlsx.

I suppose that when the table doesn’t have the primary key it use tid – the 
tuple identifier, and the size of tid may exceed the size of a 32-bit integer. 
It cause the problem to export this table to .xlsx format by OGR library.

Regards,
Marta Kopszak

From: gdal-dev 
mailto:gdal-dev-boun...@lists.osgeo.org>> On 
Behalf Of Even Rouault
Sent: Monday, January 10, 2022 11:42 AM
To: gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] Bug in OGR library


Marta,

What is the bug exactly ?

"2 billion records" : by record, you mean cells, like a sheet with 2048 columns 
and 1048576 rows ? By quickly looking at the code I can't see an obvious 
limitation, but that doesn't mean anything. Closer investigation would be 
needed. Please file an issue at https://github.com/OSGeo/gdal/issues/new with 
exact details how to reproduce (number of rows/features, and columns/fields in 
particular).

But even if the writer side would work, the OGR reader side would probably have 
issues as the whole spreadsheet will be loaded into RAM and for 2 billion 
cells, I would expect at least 32 or more GB of RAM to be needed. Perhaps 
spreadsheet software are able to cope with that in a smarter way.

Even
Le 10/01/2022 à 11:27, Kopszak Marta - Partner Hurt a écrit :
Dear Sirs,
I ran into an error when exporting data through the OGR library to the .xlsx 
and .ods format. The bug occurs when there are more than 2 billion records in 
the database. I know the problem was solved for other formats, e.g. csv. Will 
the problem be solved also for other formats (e.g. xlsx, .ods)? If this is not 
the correct e-mail, please forward this e-mail to the appropriate person.

Kind regards,
Marta Kopszak

Marta Kopszak
Współpracownik

Rozwój Sieci Stacjonarnej, Wydział Rozwoju Planowania
Orange Polska, gen. Romualda Traugutta 55, 50-416 Wrocław | RODO - informacja o 
danych 

 



___

gdal-dev mailing list

gdal-dev@lists.osgeo.org

https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 

http://www.spatialys.com

My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Bug in OGR library

I have 2 bilions of rows in the  whole database and I try to export only one 
table which has 56 columns and 423 thousands of rows so it’s not the issue with 
Microsoft Excel limitation. I can’t write even a single row (error Feature 
creation error (OGR error: negative FID are not supported) occurs immediately),
Marta Kopszak

Hi,

Could you clarify a bit? You have billion rows in the database table but xlsx 
and ods can support at maximum a bit more than one million rows (1,048,576). 
You can’t imagine to have success with converting the whole table. Or is your 
problem that you can’t write even a single row from the source table because 
the value of the ID key is too big for xlsx/ods formats?

-Jukka Rahkonen-

Lähettäjä: gdal-dev 
mailto:gdal-dev-boun...@lists.osgeo.org>> 
Puolesta Kopszak Marta - Partner Hurt
Lähetetty: maanantai 10. tammikuuta 2022 14.34
Vastaanottaja: Even Rouault 
mailto:even.roua...@spatialys.com>>; 
gdal-dev@lists.osgeo.org
Aihe: Re: [gdal-dev] Bug in OGR library

Hi Even,
Thanks for your replay. More info about the problem:

Error:
Save error:
Export to vector file failed.
Error: Feature write errors:
Feature creation error (OGR error: negative FID are not supported)

The database has more than 2 bilons of records (record – I mean record in the 
database table). The error occurs when I export a single table witout the 
primary key to .xlsx.

I suppose that when the table doesn’t have the primary key it use tid – the 
tuple identifier, and the size of tid may exceed the size of a 32-bit integer. 
It cause the problem to export this table to .xlsx format by OGR library.

Regards,
Marta Kopszak

From: gdal-dev 
mailto:gdal-dev-boun...@lists.osgeo.org>> On 
Behalf Of Even Rouault
Sent: Monday, January 10, 2022 11:42 AM
To: gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] Bug in OGR library


Marta,

What is the bug exactly ?

"2 billion records" : by record, you mean cells, like a sheet with 2048 columns 
and 1048576 rows ? By quickly looking at the code I can't see an obvious 
limitation, but that doesn't mean anything. Closer investigation would be 
needed. Please file an issue at https://github.com/OSGeo/gdal/issues/new with 
exact details how to reproduce (number of rows/features, and columns/fields in 
particular).

But even if the writer side would work, the OGR reader side would probably have 
issues as the whole spreadsheet will be loaded into RAM and for 2 billion 
cells, I would expect at least 32 or more GB of RAM to be needed. Perhaps 
spreadsheet software are able to cope with that in a smarter way.

Even
Le 10/01/2022 à 11:27, Kopszak Marta - Partner Hurt a écrit :
Dear Sirs,
I ran into an error when exporting data through the OGR library to the .xlsx 
and .ods format. The bug occurs when there are more than 2 billion records in 
the database. I know the problem was solved for other formats, e.g. csv. Will 
the problem be solved also for other formats (e.g. xlsx, .ods)? If this is not 
the correct e-mail, please forward this e-mail to the appropriate person.

Kind regards,
Marta Kopszak

Marta Kopszak
Współpracownik

Rozwój Sieci Stacjonarnej, Wydział Rozwoju Planowania
Orange Polska, gen. Romualda Traugutta 55, 50-416 Wrocław | RODO - informacja o 
danych 

 



___

gdal-dev mailing list

gdal-dev@lists.osgeo.org

https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 

http://www.spatialys.com

My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Bug in OGR library

Hi,

Could you clarify a bit? You have billion rows in the database table but xlsx 
and ods can support at maximum a bit more than one million rows (1,048,576). 
You can’t imagine to have success with converting the whole table. Or is your 
problem that you can’t write even a single row from the source table because 
the value of the ID key is too big for xlsx/ods formats?

-Jukka Rahkonen-

Lähettäjä: gdal-dev  Puolesta Kopszak Marta - 
Partner Hurt
Lähetetty: maanantai 10. tammikuuta 2022 14.34
Vastaanottaja: Even Rouault ; 
gdal-dev@lists.osgeo.org
Aihe: Re: [gdal-dev] Bug in OGR library

Hi Even,
Thanks for your replay. More info about the problem:

Error:
Save error:
Export to vector file failed.
Error: Feature write errors:
Feature creation error (OGR error: negative FID are not supported)

The database has more than 2 bilons of records (record – I mean record in the 
database table). The error occurs when I export a single table witout the 
primary key to .xlsx.

I suppose that when the table doesn’t have the primary key it use tid – the 
tuple identifier, and the size of tid may exceed the size of a 32-bit integer. 
It cause the problem to export this table to .xlsx format by OGR library.

Regards,
Marta Kopszak

From: gdal-dev 
mailto:gdal-dev-boun...@lists.osgeo.org>> On 
Behalf Of Even Rouault
Sent: Monday, January 10, 2022 11:42 AM
To: gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] Bug in OGR library


Marta,

What is the bug exactly ?

"2 billion records" : by record, you mean cells, like a sheet with 2048 columns 
and 1048576 rows ? By quickly looking at the code I can't see an obvious 
limitation, but that doesn't mean anything. Closer investigation would be 
needed. Please file an issue at https://github.com/OSGeo/gdal/issues/new with 
exact details how to reproduce (number of rows/features, and columns/fields in 
particular).

But even if the writer side would work, the OGR reader side would probably have 
issues as the whole spreadsheet will be loaded into RAM and for 2 billion 
cells, I would expect at least 32 or more GB of RAM to be needed. Perhaps 
spreadsheet software are able to cope with that in a smarter way.

Even
Le 10/01/2022 à 11:27, Kopszak Marta - Partner Hurt a écrit :
Dear Sirs,
I ran into an error when exporting data through the OGR library to the .xlsx 
and .ods format. The bug occurs when there are more than 2 billion records in 
the database. I know the problem was solved for other formats, e.g. csv. Will 
the problem be solved also for other formats (e.g. xlsx, .ods)? If this is not 
the correct e-mail, please forward this e-mail to the appropriate person.

Kind regards,
Marta Kopszak

Marta Kopszak
Współpracownik

Rozwój Sieci Stacjonarnej, Wydział Rozwoju Planowania
Orange Polska, gen. Romualda Traugutta 55, 50-416 Wrocław | RODO - informacja o 
danych 

 



___

gdal-dev mailing list

gdal-dev@lists.osgeo.org

https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 

http://www.spatialys.com

My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Bug in OGR library

Hi Even,
Thanks for your replay. More info about the problem:

Error:
Save error:
Export to vector file failed.
Error: Feature write errors:
Feature creation error (OGR error: negative FID are not supported)

The database has more than 2 bilons of records (record – I mean record in the 
database table). The error occurs when I export a single table witout the 
primary key to .xlsx.

I suppose that when the table doesn’t have the primary key it use tid – the 
tuple identifier, and the size of tid may exceed the size of a 32-bit integer. 
It cause the problem to export this table to .xlsx format by OGR library.

Regards,
Marta Kopszak

From: gdal-dev  On Behalf Of Even Rouault
Sent: Monday, January 10, 2022 11:42 AM
To: gdal-dev@lists.osgeo.org
Subject: Re: [gdal-dev] Bug in OGR library


Marta,

What is the bug exactly ?

"2 billion records" : by record, you mean cells, like a sheet with 2048 columns 
and 1048576 rows ? By quickly looking at the code I can't see an obvious 
limitation, but that doesn't mean anything. Closer investigation would be 
needed. Please file an issue at https://github.com/OSGeo/gdal/issues/new with 
exact details how to reproduce (number of rows/features, and columns/fields in 
particular).

But even if the writer side would work, the OGR reader side would probably have 
issues as the whole spreadsheet will be loaded into RAM and for 2 billion 
cells, I would expect at least 32 or more GB of RAM to be needed. Perhaps 
spreadsheet software are able to cope with that in a smarter way.

Even
Le 10/01/2022 à 11:27, Kopszak Marta - Partner Hurt a écrit :
Dear Sirs,
I ran into an error when exporting data through the OGR library to the .xlsx 
and .ods format. The bug occurs when there are more than 2 billion records in 
the database. I know the problem was solved for other formats, e.g. csv. Will 
the problem be solved also for other formats (e.g. xlsx, .ods)? If this is not 
the correct e-mail, please forward this e-mail to the appropriate person.

Kind regards,
Marta Kopszak

Marta Kopszak
Współpracownik

Rozwój Sieci Stacjonarnej, Wydział Rozwoju Planowania
Orange Polska, gen. Romualda Traugutta 55, 50-416 Wrocław | RODO - informacja o 
danych 

 



___

gdal-dev mailing list

gdal-dev@lists.osgeo.org

https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 

http://www.spatialys.com

My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Bug in OGR library

2022-01-10 Thread Andreas Oxenstierna

Excel limitations, see
https://support.microsoft.com/en-us/office/excel-specifications-and-limits-1672b34d-7043-467e-8e27-269d656771c3

ODS:
https://wiki.openoffice.org/wiki/Documentation/FAQ/Calc/Miscellaneous/What%27s_the_maximum_number_of_rows_and_cells_for_a_spreadsheet_file%3F

Hälsningar

Andreas Oxenstierna
T-Kartor Geospatial AB
Olof Mohlins väg 12 Kristianstad
mobile: +46 733 206831
mailto:
andreas.oxenstie...@t-kartor.com
www.t-kartor.com
On 10 Jan 2022, 11:42 +0100, Even Rouault , wrote:

Marta,

What is the bug exactly ?

"2 billion records" : by record, you mean cells, like a sheet with 2048 columns
and 1048576 rows ? By quickly looking at the code I can't see an obvious
limitation, but that doesn't mean anything. Closer investigation would be
needed. Please file an issue at https://github.com/OSGeo/gdal/issues/new with
exact details how to reproduce (number of rows/features, and columns/fields in
particular).

But even if the writer side would work, the OGR reader side would probably have
issues as the whole spreadsheet will be loaded into RAM and for 2 billion
cells, I would expect at least 32 or more GB of RAM to be needed. Perhaps
spreadsheet software are able to cope with that in a smarter way.

Even

Le 10/01/2022 à 11:27, Kopszak Marta - Partner Hurt a écrit :
Dear Sirs,
I ran into an error when exporting data through the OGR library to the .xlsx
and .ods format. The bug occurs when there are more than 2 billion records in
the database. I know the problem was solved for other formats, e.g. csv. Will
the problem be solved also for other formats (e.g. xlsx, .ods)? If this is not
the correct e-mail, please forward this e-mail to the appropriate person.

Kind regards,
Marta Kopszak
[Logo Orange]
Marta Kopszak
Współpracownik
Rozwój Sieci Stacjonarnej, Wydział Rozwoju Planowania
Orange Polska, gen. Romualda Traugutta 55, 50-416 Wrocław | RODO - informacja o
danych

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Bug in OGR library

Hi,

Excel xlsx format supports at maximum 1,048,576 rows by 16,384 columns 
https://support.microsoft.com/en-us/office/excel-specifications-and-limits-1672b34d-7043-467e-8e27-269d656771c3.
 Do your data fit within these limits?

-Jukka Rahkonen-

Lähettäjä: gdal-dev  Puolesta Kopszak Marta - 
Partner Hurt
Lähetetty: maanantai 10. tammikuuta 2022 12.28
Vastaanottaja: gdal-dev@lists.osgeo.org
Aihe: [gdal-dev] Bug in OGR library

Dear Sirs,
I ran into an error when exporting data through the OGR library to the .xlsx 
and .ods format. The bug occurs when there are more than 2 billion records in 
the database. I know the problem was solved for other formats, e.g. csv. Will 
the problem be solved also for other formats (e.g. xlsx, .ods)? If this is not 
the correct e-mail, please forward this e-mail to the appropriate person.

Kind regards,
Marta Kopszak
[Lähettäjä poisti kuvan. Logo Orange]

Marta Kopszak
Współpracownik

Rozwój Sieci Stacjonarnej, Wydział Rozwoju Planowania
Orange Polska, gen. Romualda Traugutta 55, 50-416 Wrocław | RODO - informacja o 
danych


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Bug in OGR library