Hi again,
MiraMon files have had INI files containing metadata information for ages.
To read and write sections and key/values from them, we use specific
Windows functions
(https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-getprivateprofilestring).
When I started programming the driver, I had to create a function to
do that as I couldn't use the Windows one.
I suspect that these functions are not accurate and are failing.
Is there any function in GDAL that can read INI files (at least to
check if a section exists and to get the value of a key in an existing
section)? If not, perhaps using the following could be considered:
https://github.com/brofield/simpleini
It seems to have an MIT license... If it's possible to use this
parser, what would be the procedure? I noticed that Even added a
parser for some command line arguments and perhaps something similar
can be done with that.
If not, is it possible for me to add some files in the MiraMonVector
driver and use them?
It's better to use something existing and multiplatform than to
"reinvent the wheel" with some new function that can fail (in fact, is
failing now).
Thanks.
*De:*gdal-dev <gdal-dev-boun...@lists.osgeo.org> *En nombre de *Abel
Pau via gdal-dev
*Enviado el:* dimecres, 13 de març de 2024 12:38
*Para:* Even Rouault <even.roua...@spatialys.com>;
gdal-dev@lists.osgeo.org
*Asunto:* Re: [gdal-dev] There's something strange about two tests
that are very similar
Hi Even,
thanks for your effort on that.
I’ll spend some time analyzing that but I can imagine where the
problem is.
For your information our .REL files are generated in ANSI codification
and when this kind of files are read I forgot to translate to UTF-8.
It’s very probable that this is the mistake.
I’ll tell you when I solve that!
About
1)it’s not my intention to output part of the content file (I think
that is part of the name which is bad composed because is read from
this REL file)
2)Due to codification, certainly
3)It’s odd and I didn’t want to do that, of course! I’ll review
4)I used Valgrind some time, And I’ll do again, thanks
Thanks for the report!
*De:*Even Rouault <even.roua...@spatialys.com>
*Enviado el:* dimecres, 13 de març de 2024 11:47
*Para:*Abel Pau <a....@creaf.uab.cat>; gdal-dev@lists.osgeo.org
*Asunto:* Re: [gdal-dev] There's something strange about two tests
that are very similar
Hi,
I've checkout'ed your branch locally and I can reproduce the error
when running the Python tests. Some of the CI checks give an
interesting hint about this being related to a UTF8 issue, and I
wondered why
But running "ogrinfo
autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol"
outputs:
"""
ERROR 4: Error pMMPolygonLayer.MMArc.pF: Cannot open file
autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.arc
[QUALITY:LINEAGE:PROCESS1]
nOrganismes=1
history=C:\MiraMon\MM64.exe
date=20230628 16204988+0200
[QUALITY:LINEAGE:PROCESS1:ORGANISME_1]
IndividualName=Abel Pau
PositionName=T�cnic SIG
OrganisationName=Students and educational institutions
[...]
""""
So there are several problems:
1) On failure, you shouldn't output the content of the file in the
error message, to keep it brief the UTF-8 error comes from the fact
that the file content is not UTF-8 encoded)
2) There is an error opening the file.
3) Interestingly something in your code *deletes* file
autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol !
4) You should really invest in running Valgrind. After restoring the
deleted file, I can see:
$ valgrind ogrinfo
autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol
==1737168== Memcheck, a memory error detector
==1737168== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward
et al.
==1737168== Using Valgrind-3.15.0 and LibVEX; rerun with -h for
copyright info
==1737168== Command: ogrinfo
autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol
==1737168==
==1737168== Conditional jump or move depends on uninitialised value(s)
==1737168== at 0x4843C6F: strstr (in
/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1737168== by 0x635FC44: MMReturnValueFromSectionINIFile
(mm_wrlayr.c:5019)
==1737168== by 0x636219A: MMCheck_REL_FILE (mm_wrlayr.c:5785)
==1737168== by 0x63586B0: MMInitPolygonLayer (mm_wrlayr.c:1665)
==1737168== by 0x6358BC6: MMInitLayerByType (mm_wrlayr.c:1761)
==1737168== by 0x636B818: MMInitLayerToRead (mm_rdlayr.c:142)
==1737168== by 0x634D6A7: OGRMiraMonLayer::OGRMiraMonLayer(char
const*, VSIVirtualHandle*, OGRSpatialReference const*, int, char**,
MiraMonVectMapInfo*) (ogrmiramonlayer.cpp:219)
==1737168== by 0x634C03E: OGRMiraMonDataSource::Open(char const*,
VSIVirtualHandle*, OGRSpatialReference const*, int, char**)
(ogrmiramondatasource.cpp:71)
==1737168== by 0x634C997: OGRMiraMonDriverOpen(GDALOpenInfo*)
(ogrmiramondriver.cpp:87)
==1737168== by 0x67DD68C: GDALDriver::Open(GDALOpenInfo*, bool)
(gdaldriver.cpp:116)
==1737168== by 0x67F61F4: GDALOpenEx (gdaldataset.cpp:3745)
==1737168== by 0x10AC95: GDALDataset::Open(char const*, unsigned
int, char const* const*, char const* const*, char const* const*)
(gdal_priv.h:835)
==1737168==
==1737168== Conditional jump or move depends on uninitialised value(s)
==1737168== at 0x483EF58: strlen (in
/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1737168== by 0x5262132: CPLStrdup (cpl_conv.cpp:316)
==1737168== by 0x635FC6E: MMReturnValueFromSectionINIFile
(mm_wrlayr.c:5023)
==1737168== by 0x636219A: MMCheck_REL_FILE (mm_wrlayr.c:5785)
==1737168== by 0x63586B0: MMInitPolygonLayer (mm_wrlayr.c:1665)
==1737168== by 0x6358BC6: MMInitLayerByType (mm_wrlayr.c:1761)
==1737168== by 0x636B818: MMInitLayerToRead (mm_rdlayr.c:142)
==1737168== by 0x634D6A7: OGRMiraMonLayer::OGRMiraMonLayer(char
const*, VSIVirtualHandle*, OGRSpatialReference const*, int, char**,
MiraMonVectMapInfo*) (ogrmiramonlayer.cpp:219)
==1737168== by 0x634C03E: OGRMiraMonDataSource::Open(char const*,
VSIVirtualHandle*, OGRSpatialReference const*, int, char**)
(ogrmiramondatasource.cpp:71)
==1737168== by 0x634C997: OGRMiraMonDriverOpen(GDALOpenInfo*)
(ogrmiramondriver.cpp:87)
==1737168== by 0x67DD68C: GDALDriver::Open(GDALOpenInfo*, bool)
(gdaldriver.cpp:116)
==1737168== by 0x67F61F4: GDALOpenEx (gdaldataset.cpp:3745)
==1737168==
[...]
Le 12/03/2024 à 20:43, Abel Pau a écrit :
>>> Are sure all needed files have been committed and pushed for the
failing test?
Yes, sure. 9 files are needed for oine polygon layer. Absolutely sure.
>>> Perhaps an issue Windows CR-LF vs Unix LF end of line characters ?
The default settings of git on Windows may mess with binary files.
cf
https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings
I’ll take a look but in binary files the format are bytes of
information or blanks (there is no line endings or something
similar). The drivers Writes and reads bytes and it goes to
different features using offsets of 4 or 8 bytes depending on the
version. 4 in this case. Here is the format description for more
information.
https://www.miramon.cat/new_note/eng/notes/MiraMon_structured_vectors_file_format.pdf
Taking a look to the log I can see an advice about GDALOpenEx (See
below).
Thinking about differences between extension POL and the other two
(PNT and ARC) is that is the main.
poDriver->SetMetadataItem(GDAL_DMD_EXTENSION, "pol");
poDriver->SetMetadataItem(GDAL_DMD_EXTENSIONS, "pol arc pnt");
I look into multiple examples where ds = gdal.OpenEx() is called
with one, two or three arguments.
So, could it be related with this issue?
Thanks!!!
THE LOG:
/home/runner/work/gdal/gdal/build-ubuntu_22.04/autotest/ogr/ogr_miramon_vector.py:175:
17281
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17282>_
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _ _ _
17282
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17283>
17283
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17284>args
= ('data/miramon/Polygons/SimplePolygons/SimplePolFile.pol',),
kwargs = {}
17284
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17285>
17285
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17286>
def OpenEx(*args, **kwargs) -> "GDALDatasetShadow *":
17286
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17287>
r"""
17287
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17288>
OpenEx(char const * utf8_path, unsigned int nOpenFlags=0, char **
allowed_drivers=None, char ** open_options=None, char **
sibling_files=None) -> Dataset
17288
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17289>
17289
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17290>
17290
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17291>
Open a raster or vector file as a :py:class:`Dataset`.
17291
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17292>
See :cpp:func:`GDALOpenEx`.
17292
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17293>
17293
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17294>
Parameters
17294
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17295>
----------
17295
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17296>
utf8_path : str
17296
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17297>
name of the file to open
17297
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17298>
flags : int
17298
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17299>
Flags controlling how the Dataset is opened. Multiple
``gdal.OF_XXX`` flags
17299
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17300>
may be combined using the ``|`` operator. See :cpp:func:`GDALOpenEx`.
17300
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17301>
allowed_drivers : list, optional
17301
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17302>
A list of the names of drivers that may attempt to open the dataset.
17302
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17303>
open_options : dict/list, optional
17303
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17304>
A dict or list of name=value driver-specific opening options.
17304
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17305>
sibling_files: list, optional
17305
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17306>
A list of filenames that are auxiliary to the main filename
17306
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17307>
17307
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17308>
Returns
17308
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17309>
-------
17309
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17310>
Dataset, or ``None`` on failure.
17310
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17311>
17311
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17312>
See Also
17312
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17313>
--------
17313
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17314>
:py:func:`Open`
17314
<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17315>
:py:func:`OpenShared`
Le 12/03/2024 à 17:22, Abel Pau via gdal-dev a écrit :
Hi,
I have in a single test python file with 3 tests inside:
gdal/autotest/ogr/ogr_miramon_vector.py at master ·
AbelPau/gdal (github.com)
<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L134>
In some actions the first and second are Ok but the third is
not. Example of failure:
Some improvements for empty layers · AbelPau/gdal@90a289c
(github.com)
<https://github.com/AbelPau/gdal/actions/runs/8251854865/job/22569933640#step:12:4863>
Here you have the log (return _gdal.OpenEx(*args, **kwargs)
4912
<https://github.com/AbelPau/gdal/actions/runs/8251854865/job/22569933640#step:12:4913>23:
E RuntimeError: Cannot open file %s.) <--- This happen in the
driver when open fails.
The first two are OK:
gdal/autotest/ogr/ogr_miramon_vector.py at master ·
AbelPau/gdal (github.com)
<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L48>
gdal/autotest/ogr/ogr_miramon_vector.py at master ·
AbelPau/gdal (github.com)
<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L99>
But this is not OK:
gdal/autotest/ogr/ogr_miramon_vector.py at master ·
AbelPau/gdal (github.com)
<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L134>
And the file are all on the correct place (I am sure about that):
For the second one, for example:
gdal/autotest/ogr/data/miramon/Arcs/SimpleArcs at master ·
AbelPau/gdal (github.com)
<https://github.com/AbelPau/gdal/tree/master/autotest/ogr/data/miramon/Arcs/SimpleArcs>
And for the third one:
gdal/autotest/ogr/data/miramon/Polygons/SimplePolygons at
master · AbelPau/gdal (github.com)
<https://github.com/AbelPau/gdal/tree/master/autotest/ogr/data/miramon/Polygons/SimplePolygons>
On local tests are perfectly OK (see below). So, after some
days of trying to figure out WHY and what is de difference
between 2 and 3th test I have to ask you because it must be
obvious but not to me.
Any idea? Some permissions stuff?? THANKS!
root@f5b8c1fb00ae:/gdal/build/autotest# python3 -m pytest -vv
-p no:sugar --color=no ogr/ogr_miramon_vector.py
================================================= test session
starts ==================================================
platform linux -- Python 3.10.12, pytest-8.1.1, pluggy-1.4.0
-- /usr/bin/python3
cachedir: .pytest_cache
benchmark: 4.0.0 (defaults: timer=time.perf_counter
disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0
calibration_precision=10 warmup=False warmup_iterations=100000)
GDAL Build Info:
PAM_ENABLED: YES
OGR_ENABLED: YES
CURL_ENABLED: YES
CURL_VERSION: 7.81.0
GEOS_ENABLED: YES
GEOS_VERSION: 3.10.2-CAPI-1.16.0
PROJ_BUILD_VERSION: 8.2.1
PROJ_RUNTIME_VERSION: 8.2.1
COMPILER: GCC 11.4.0
GDAL_DOWNLOAD_TEST_DATA: undefined (tests relying on
downloaded data may be skipped)
GDAL_RUN_SLOW_TESTS: undefined (tests marked as "slow" will be
skipped)
rootdir: /gdal/build/autotest
configfile: pytest.ini
plugins: env-1.1.3, benchmark-4.0.0
collected 3 items
ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_point
PASSED [ 33%]
ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_arc
PASSED [ 66%]
ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_polygon
PASSED [100%]
================================================== 3 passed in
0.11s ===================================================
*Abel Pau Garcia*
*GIS developer*
https://www.creaf.cat/sites/default/files/creaf-signature.png
*a....@creaf.uab.cat* <mailto:a....@creaf.uab.cat>
*Let's chat on Teams!*
<https://teams.microsoft.com/l/chat/0/0?users=a....@creaf.uab.cat>
*Tel. +34 934814277*
https://www.creaf.cat/sites/default/files/so-en-signature.png
https://www.creaf.cat/sites/default/files/twitter-icon-signature.png
<https://twitter.com/CREAF_ecologia>https://www.creaf.cat/sites/default/files/linkedin-icon-signature.png
<https://www.linkedin.com/company/1363052?trk=tyah&trkInfo=clickedVertical:company,clickedEntityId:1363052,idx:2-1-2,tarId:1465807877789,tas:creaf>https://www.creaf.cat/sites/default/files/youtube-icon-signature.png
<https://www.youtube.com/c/creafecologia>https://www.creaf.cat/sites/default/files/instagram-icon-signature.png
<https://www.instagram.com/CREAF_ecologia/>
*www.creaf.cat* <http://www.creaf.cat>*|
**http://blog.creaf.cat* <http://blog.creaf.cat>
https://www.creaf.cat/sites/default/files/uab_logo_signatura.png
CREAF. Campus UAB. Edifici C. 08193 Bellaterra (Barcelona)
Before printing this electronic message, think about the
environment.
http://www.creaf.uab.cat/_signatura/line.gif
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
--
http://www.spatialys.com
My software is free, but my time generally not.
_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
