Re: [gdal-dev] Ogr2ogr issue with big integers

2022-07-18 Thread Even Rouault 

Moises,

the reason is that autodetection by default only considers the content 
of the first megabyte of the file, and in the first megabyte, only 
integer values that fit on 32 bit integers are present, hence this data 
type is selected. But later in the file, there are values that require a 
64 bit integer.


You can add -oo AUTODETECT_SIZE_LIMIT=0 to mean unlimited size limit. 
Unfortunately I just noticed this got broken in 3.5.1, where 
AUTODETECT_SIZE_LIMIT=0 would disable any autodetection. This will be 
fixed per https://github.com/OSGeo/gdal/pull/6087. So if you've 3.5.1, 
use AUTODETECT_SIZE_LIMIT=-1 instead. Or a large value in bytes not 
greater than 2 billion. In the previously mentioned fix, I've also added 
a warning if integer values that don't fit on 32 bit are set.


Even

Le 18/07/2022 à 15:54, Moises Calzado via gdal-dev a écrit :
test.csv 


Hello everyone!

I'm dealing with some issues with ogr2ogr trying to convert my data to 
a CSV file guessing the data types. I'm using the following command:


|ogr2ogr -f CSV -skipfailures -makevalid /vsistdout/ CSV:test.csv -oo 
AUTODETECT_TYPE=YES -lcoCREATE_CSVT=YES > test.csv|
When I add the AUTODETECT_TYPE=YES flag, ogr2ogr is applying some 
weird transformations on the "uprn" column. If you check the generated 
output and the original file, you'll notice that some 
integers have changed into a negative number. Do you know if I'm doing 
something wrong, or if it's a bug?


Thanks so much for your help,
Regards!



--
*Moises Calzado*

Support Engineer

(US) +1 917 463 3232 | (ES) +34 911 165 823 | mcalz...@carto.com



___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Ogr2ogr issue with big integers

2022-07-18 Thread Moises Calzado via gdal-dev 
 test.csv

Hello everyone!

I'm dealing with some issues with ogr2ogr trying to convert my data to a
CSV file guessing the data types. I'm using the following command:

ogr2ogr -f CSV -skipfailures -makevalid /vsistdout/ CSV:test.csv -oo
AUTODETECT_TYPE=YES -lco CREATE_CSVT=YES > test.csv

When I add the AUTODETECT_TYPE=YES flag, ogr2ogr is applying some weird
transformations on the "uprn" column. If you check the generated output and
the original file, you'll notice that some integers have changed into a
negative number. Do you know if I'm doing something wrong, or if it's a
bug?

Thanks so much for your help,
Regards!



-- 
*Moises Calzado*

Support Engineer

(US) +1 917 463 3232 | (ES) +34 911 165 823 | mcalz...@carto.com

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev