Hi Even, thanks for your effort on that. I’ll spend some time analyzing that but I can imagine where the problem is. For your information our .REL files are generated in ANSI codification and when this kind of files are read I forgot to translate to UTF-8.
It’s very probable that this is the mistake. I’ll tell you when I solve that! About 1) it’s not my intention to output part of the content file (I think that is part of the name which is bad composed because is read from this REL file) 2) Due to codification, certainly 3) It’s odd and I didn’t want to do that, of course! I’ll review 4) I used Valgrind some time, And I’ll do again, thanks Thanks for the report! De: Even Rouault <even.roua...@spatialys.com> Enviado el: dimecres, 13 de març de 2024 11:47 Para: Abel Pau <a....@creaf.uab.cat>; gdal-dev@lists.osgeo.org Asunto: Re: [gdal-dev] There's something strange about two tests that are very similar Hi, I've checkout'ed your branch locally and I can reproduce the error when running the Python tests. Some of the CI checks give an interesting hint about this being related to a UTF8 issue, and I wondered why But running "ogrinfo autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol" outputs: """ ERROR 4: Error pMMPolygonLayer.MMArc.pF: Cannot open file autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.arc [QUALITY:LINEAGE:PROCESS1] nOrganismes=1 history=C:\MiraMon\MM64.exe date=20230628 16204988+0200 [QUALITY:LINEAGE:PROCESS1:ORGANISME_1] IndividualName=Abel Pau PositionName=T�cnic SIG OrganisationName=Students and educational institutions [...] """" So there are several problems: 1) On failure, you shouldn't output the content of the file in the error message, to keep it brief the UTF-8 error comes from the fact that the file content is not UTF-8 encoded) 2) There is an error opening the file. 3) Interestingly something in your code *deletes* file autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol ! 4) You should really invest in running Valgrind. After restoring the deleted file, I can see: $ valgrind ogrinfo autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol ==1737168== Memcheck, a memory error detector ==1737168== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==1737168== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info ==1737168== Command: ogrinfo autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol ==1737168== ==1737168== Conditional jump or move depends on uninitialised value(s) ==1737168== at 0x4843C6F: strstr (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) ==1737168== by 0x635FC44: MMReturnValueFromSectionINIFile (mm_wrlayr.c:5019) ==1737168== by 0x636219A: MMCheck_REL_FILE (mm_wrlayr.c:5785) ==1737168== by 0x63586B0: MMInitPolygonLayer (mm_wrlayr.c:1665) ==1737168== by 0x6358BC6: MMInitLayerByType (mm_wrlayr.c:1761) ==1737168== by 0x636B818: MMInitLayerToRead (mm_rdlayr.c:142) ==1737168== by 0x634D6A7: OGRMiraMonLayer::OGRMiraMonLayer(char const*, VSIVirtualHandle*, OGRSpatialReference const*, int, char**, MiraMonVectMapInfo*) (ogrmiramonlayer.cpp:219) ==1737168== by 0x634C03E: OGRMiraMonDataSource::Open(char const*, VSIVirtualHandle*, OGRSpatialReference const*, int, char**) (ogrmiramondatasource.cpp:71) ==1737168== by 0x634C997: OGRMiraMonDriverOpen(GDALOpenInfo*) (ogrmiramondriver.cpp:87) ==1737168== by 0x67DD68C: GDALDriver::Open(GDALOpenInfo*, bool) (gdaldriver.cpp:116) ==1737168== by 0x67F61F4: GDALOpenEx (gdaldataset.cpp:3745) ==1737168== by 0x10AC95: GDALDataset::Open(char const*, unsigned int, char const* const*, char const* const*, char const* const*) (gdal_priv.h:835) ==1737168== ==1737168== Conditional jump or move depends on uninitialised value(s) ==1737168== at 0x483EF58: strlen (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so) ==1737168== by 0x5262132: CPLStrdup (cpl_conv.cpp:316) ==1737168== by 0x635FC6E: MMReturnValueFromSectionINIFile (mm_wrlayr.c:5023) ==1737168== by 0x636219A: MMCheck_REL_FILE (mm_wrlayr.c:5785) ==1737168== by 0x63586B0: MMInitPolygonLayer (mm_wrlayr.c:1665) ==1737168== by 0x6358BC6: MMInitLayerByType (mm_wrlayr.c:1761) ==1737168== by 0x636B818: MMInitLayerToRead (mm_rdlayr.c:142) ==1737168== by 0x634D6A7: OGRMiraMonLayer::OGRMiraMonLayer(char const*, VSIVirtualHandle*, OGRSpatialReference const*, int, char**, MiraMonVectMapInfo*) (ogrmiramonlayer.cpp:219) ==1737168== by 0x634C03E: OGRMiraMonDataSource::Open(char const*, VSIVirtualHandle*, OGRSpatialReference const*, int, char**) (ogrmiramondatasource.cpp:71) ==1737168== by 0x634C997: OGRMiraMonDriverOpen(GDALOpenInfo*) (ogrmiramondriver.cpp:87) ==1737168== by 0x67DD68C: GDALDriver::Open(GDALOpenInfo*, bool) (gdaldriver.cpp:116) ==1737168== by 0x67F61F4: GDALOpenEx (gdaldataset.cpp:3745) ==1737168== [...] Le 12/03/2024 à 20:43, Abel Pau a écrit : >>> Are sure all needed files have been committed and pushed for the failing >>> test? Yes, sure. 9 files are needed for oine polygon layer. Absolutely sure. >>> Perhaps an issue Windows CR-LF vs Unix LF end of line characters ? The >>> default settings of git on Windows may mess with binary files. cf >>> https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings I’ll take a look but in binary files the format are bytes of information or blanks (there is no line endings or something similar). The drivers Writes and reads bytes and it goes to different features using offsets of 4 or 8 bytes depending on the version. 4 in this case. Here is the format description for more information. https://www.miramon.cat/new_note/eng/notes/MiraMon_structured_vectors_file_format.pdf Taking a look to the log I can see an advice about GDALOpenEx (See below). Thinking about differences between extension POL and the other two (PNT and ARC) is that is the main. poDriver->SetMetadataItem(GDAL_DMD_EXTENSION, "pol"); poDriver->SetMetadataItem(GDAL_DMD_EXTENSIONS, "pol arc pnt"); I look into multiple examples where ds = gdal.OpenEx() is called with one, two or three arguments. So, could it be related with this issue? Thanks!!! THE LOG: /home/runner/work/gdal/gdal/build-ubuntu_22.04/autotest/ogr/ogr_miramon_vector.py:175: 17281<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17282>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 17282<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17283> 17283<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17284>args = ('data/miramon/Polygons/SimplePolygons/SimplePolFile.pol',), kwargs = {} 17284<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17285> 17285<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17286> def OpenEx(*args, **kwargs) -> "GDALDatasetShadow *": 17286<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17287> r""" 17287<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17288> OpenEx(char const * utf8_path, unsigned int nOpenFlags=0, char ** allowed_drivers=None, char ** open_options=None, char ** sibling_files=None) -> Dataset 17288<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17289> 17289<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17290> 17290<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17291> Open a raster or vector file as a :py:class:`Dataset`. 17291<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17292> See :cpp:func:`GDALOpenEx`. 17292<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17293> 17293<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17294> Parameters 17294<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17295> ---------- 17295<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17296> utf8_path : str 17296<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17297> name of the file to open 17297<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17298> flags : int 17298<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17299> Flags controlling how the Dataset is opened. Multiple ``gdal.OF_XXX`` flags 17299<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17300> may be combined using the ``|`` operator. See :cpp:func:`GDALOpenEx`. 17300<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17301> allowed_drivers : list, optional 17301<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17302> A list of the names of drivers that may attempt to open the dataset. 17302<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17303> open_options : dict/list, optional 17303<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17304> A dict or list of name=value driver-specific opening options. 17304<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17305> sibling_files: list, optional 17305<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17306> A list of filenames that are auxiliary to the main filename 17306<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17307> 17307<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17308> Returns 17308<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17309> ------- 17309<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17310> Dataset, or ``None`` on failure. 17310<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17311> 17311<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17312> See Also 17312<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17313> -------- 17313<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17314> :py:func:`Open` 17314<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17315> :py:func:`OpenShared` Le 12/03/2024 à 17:22, Abel Pau via gdal-dev a écrit : Hi, I have in a single test python file with 3 tests inside: gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal (github.com)<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L134> In some actions the first and second are Ok but the third is not. Example of failure: Some improvements for empty layers · AbelPau/gdal@90a289c (github.com)<https://github.com/AbelPau/gdal/actions/runs/8251854865/job/22569933640#step:12:4863> Here you have the log (return _gdal.OpenEx(*args, **kwargs) 4912<https://github.com/AbelPau/gdal/actions/runs/8251854865/job/22569933640#step:12:4913>23: E RuntimeError: Cannot open file %s.) <--- This happen in the driver when open fails. The first two are OK: gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal (github.com)<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L48> gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal (github.com)<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L99> But this is not OK: gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal (github.com)<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L134> And the file are all on the correct place (I am sure about that): For the second one, for example: gdal/autotest/ogr/data/miramon/Arcs/SimpleArcs at master · AbelPau/gdal (github.com)<https://github.com/AbelPau/gdal/tree/master/autotest/ogr/data/miramon/Arcs/SimpleArcs> And for the third one: gdal/autotest/ogr/data/miramon/Polygons/SimplePolygons at master · AbelPau/gdal (github.com)<https://github.com/AbelPau/gdal/tree/master/autotest/ogr/data/miramon/Polygons/SimplePolygons> On local tests are perfectly OK (see below). So, after some days of trying to figure out WHY and what is de difference between 2 and 3th test I have to ask you because it must be obvious but not to me. Any idea? Some permissions stuff?? THANKS! root@f5b8c1fb00ae:/gdal/build/autotest# python3 -m pytest -vv -p no:sugar --color=no ogr/ogr_miramon_vector.py ================================================= test session starts ================================================== platform linux -- Python 3.10.12, pytest-8.1.1, pluggy-1.4.0 -- /usr/bin/python3 cachedir: .pytest_cache benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) GDAL Build Info: PAM_ENABLED: YES OGR_ENABLED: YES CURL_ENABLED: YES CURL_VERSION: 7.81.0 GEOS_ENABLED: YES GEOS_VERSION: 3.10.2-CAPI-1.16.0 PROJ_BUILD_VERSION: 8.2.1 PROJ_RUNTIME_VERSION: 8.2.1 COMPILER: GCC 11.4.0 GDAL_DOWNLOAD_TEST_DATA: undefined (tests relying on downloaded data may be skipped) GDAL_RUN_SLOW_TESTS: undefined (tests marked as "slow" will be skipped) rootdir: /gdal/build/autotest configfile: pytest.ini plugins: env-1.1.3, benchmark-4.0.0 collected 3 items ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_point PASSED [ 33%] ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_arc PASSED [ 66%] ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_polygon PASSED [100%] ================================================== 3 passed in 0.11s =================================================== Abel Pau Garcia GIS developer [https://www.creaf.cat/sites/default/files/creaf-signature.png] a....@creaf.uab.cat<mailto:a....@creaf.uab.cat> Let's chat on Teams!<https://teams.microsoft.com/l/chat/0/0?users=a....@creaf.uab.cat> Tel. +34 934814277 [https://www.creaf.cat/sites/default/files/so-en-signature.png] [https://www.creaf.cat/sites/default/files/twitter-icon-signature.png]<https://twitter.com/CREAF_ecologia>[https://www.creaf.cat/sites/default/files/linkedin-icon-signature.png]<https://www.linkedin.com/company/1363052?trk=tyah&trkInfo=clickedVertical:company,clickedEntityId:1363052,idx:2-1-2,tarId:1465807877789,tas:creaf>[https://www.creaf.cat/sites/default/files/youtube-icon-signature.png]<https://www.youtube.com/c/creafecologia>[https://www.creaf.cat/sites/default/files/instagram-icon-signature.png]<https://www.instagram.com/CREAF_ecologia/> www.creaf.cat<http://www.creaf.cat> | http://blog.creaf.cat [https://www.creaf.cat/sites/default/files/uab_logo_signatura.png] CREAF. Campus UAB. Edifici C. 08193 Bellaterra (Barcelona) Before printing this electronic message, think about the environment. _______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org<mailto:gdal-dev@lists.osgeo.org> https://lists.osgeo.org/mailman/listinfo/gdal-dev -- http://www.spatialys.com My software is free, but my time generally not. -- http://www.spatialys.com My software is free, but my time generally not.
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev