Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

2021-07-16 Thread Andrea Giudiceandrea
Il 16/07/2021 11:51, Andreas Neumann ha scritto: Hi all and thanks for the discussion, Even and I discovered that the .xls files are loading with correct encoding in Linux but not on Windows which leads to the assumption that freexl on Windows might not have been correctly compiled/linked agai

Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

2021-07-16 Thread Andreas Neumann
Hi all and thanks for the discussion, Even and I discovered that the .xls files are loading with correct encoding in Linux but not on Windows which leads to the assumption that freexl on Windows might not have been correctly compiled/linked against iconv. Something that maybe Jürgen could cl

Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

2021-07-15 Thread Andrea Giudiceandrea
Il 15/07/2021 22:30, Nyall Dawson ha scritto: I don't think that's completely correct -- looking at the freexl documentation it seems that only some xls file versions are utf8, and others have a codepage indicating the encoding which needs to be read from the xls metadata: Yes, of course BIFF

Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

2021-07-15 Thread Andrea Giudiceandrea
Il 15/07/2021 20:51, Andrea Giudiceandrea ha scritto: Instead, QGIS imports correctly XLSX files as UTF-8 encoded, while XLS files are wrongly imported as "system" encoded, even selecting UTF-8 [3] encoding in the Data Source Manager vector import window. Looking at GDA/OGR code, I see that th

Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

2021-07-15 Thread Andrea Giudiceandrea
Isn't this limitation ultimately that GDAL isn't reading the encoding correctly? (Or perhaps it's a limitation in the underlying freexl library...) Nyall Hi Nyall and Andreas, it seems to me GDAL/OGR [1] reads XLS and XLSX files with the relative proper encoding [2] and ogrinfo outputs the te

Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

2021-07-15 Thread Even Rouault
Le 15/07/2021 à 22:30, Nyall Dawson a écrit : On Fri, 16 Jul 2021, 4:52 am Andrea Giudiceandrea, mailto:andreaer...@libero.it>> wrote: > Isn't this limitation ultimately that GDAL isn't reading the encoding > correctly? (Or perhaps it's a limitation in the underlying freexl

Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

2021-07-15 Thread Nyall Dawson
On Fri, 16 Jul 2021, 4:52 am Andrea Giudiceandrea, wrote: > > Isn't this limitation ultimately that GDAL isn't reading the encoding > > correctly? (Or perhaps it's a limitation in the underlying freexl > > library...) > > > > Nyall > > > Hi Nyall and Andreas, > it seems to me GDAL/OGR [1] reads X

Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

2021-07-15 Thread Even Rouault
Le 15/07/2021 à 21:51, Andrea Giudiceandrea a écrit : Il 15/07/2021 20:51, Andrea Giudiceandrea ha scritto: Instead, QGIS imports correctly XLSX files as UTF-8 encoded, while XLS files are wrongly imported as "system" encoded, even selecting UTF-8 [3] encoding in the Data Source Manager vector

Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

2021-07-15 Thread Andreas Neumann
Hi all, Thank you Nyall for checking the situation around codepage and freexl/OGR. Let's plan to improve the situation around spreadsheet data loading in future OGR and QGIS versions. We can start with a QEP collecting all the requirements around the spreadsheet data loading and an easy co

Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

2021-07-15 Thread Andreas Neumann
Isn't this limitation ultimately that GDAL isn't reading the encoding correctly? (Or perhaps it's a limitation in the underlying freexl library...) Yes - I also assume it is a limitation of OGR or FreeXL. I was hoping that some secret OGR opening option, environment variable or sidecar file co

Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

2021-07-15 Thread Nyall Dawson
On Thu, 15 Jul 2021 at 17:12, Andreas Neumann wrote: > > Hi, > > Seems like when I load the same file from the newer .xlsx format then the > encoding is fine. > > Now I need to find a way how to batch-convert all of the many .xls to .xlsx > files automatically. > > Still I find it disturbing tha

Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

2021-07-15 Thread Andreas Neumann
Hi, Seems like when I load the same file from the newer .xlsx format then the encoding is fine. Now I need to find a way how to batch-convert all of the many .xls to .xlsx files automatically. Still I find it disturbing that QGIS doesn't allow me to load the older .xls files with correct e

Re: [QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

2021-07-14 Thread Andreas Neumann
Hi again, I tried to set the layer encoding after the layer merging, but it fails ;-( And before the layer merging I cannot set the encoding in the model, because it only accepts a single layer as input, and not multiple layers. So it seems that I will have to set the correct encoding righ

[QGIS-Developer] Loading many Excel-Files in QGIS with correct encoding

2021-07-14 Thread Andreas Neumann
Hi, I have a bunch of MS Excel .xls files in the old format (hundreds of files), not the new .xlsx format. The content of the .xls files seems to be UTF-8 encoded, but QGIS loads them with some LATIN-x encoding - all of the non-ASCII-characters look garbled. I want to process these files w