Re: [gdal-dev] Call to GDALDestroy results in occasional core dump, GDAL 3.4.2

[Andrew Bell via gdal-dev]

On Wed, Sep 18, 2024 at 8:32 PM Fox, Shawn D (US) via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> In our case we have a singleton class that acts as a façade and all of our
> calls to GDAL Apis are done by the methods of this class.  The rest of our
> code base only interacts with the singleton so that we only have one
> project that actually depends directly on the GDAL library.  Since the
> _instance member is a static smart pointer the destructor of our class and
> the GDALDestroy function is being called after the main function exits.
>

This is not a great plan unless you understand exactly the order of things
being destroyed, which is not well-defined between compilation units unless
you have done things to guarantee it. It seems likely that GDALDestroy() is
attempting to free things already destroyed during program tear-down. Since
your program is exiting, I can't imagine you need to call GDALDestroy() at
all.  You could also eliminate this issue by instantiating your GDAL
class-thingee as the first line of your program rather than as a static.

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call to GDALDestroy results in occasional core dump, GDAL 3.4.2

[Andrew Bell via gdal-dev]

There are many potential causes. Providing code and a backtrace would allow
someone else to look.

On Wed, Sep 18, 2024, 7:28 PM Fox, Shawn D (US) via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Could someone help me understand this sentence from the documentation at 
> gdal.h:
> Raster C API — GDAL documentation
>  for the GDALDestroy
> function?
>
>
>
> “Since GDAL 2.4.0, this function may be called by application code, since
> it is no longer called automatically, on non-MSVC builds, due to ordering
> problems with respect to automatic destruction of global C++ objects.”
>
>
>
> What are the ordering problems and why might it be called due to ordering
> problems?  The statement does state some kind of difference between MSVC
> and non-MSVC builds. The sentence doesn’t clearly state to me whether I
> should or shouldn’t call that function to cleanup. The language in the
> documentation is ambiguous, and it is not clear to me how important it is
> to call this method.  Cleaning up seems like a good idea but I could simply
> free the memory via delete for any object that is returned instead of
> calling GDALDestroy.
>
>
>
> On a RHEL8 build, which is non-MSVC, I am observing segmentation faults.
> The errors vary from one execution of the program to another.
>
>
>
> Sometimes I observe this error.
>
> corrupted size vs. prev_size in fastbins
>
> Aborted (core dumped)
>
>
>
> Other times I observe this one
>
> corrupted double-linked list
>
> Aborted (core dumped)
>
>
>
> Our code is not very complex.  It calls GDALRegister in the main thread at
> the beginning, we perform some CRS transformations during the lifetime of
> the program, and GDALDestroy from the main thread before exiting.  I do not
> observe problems with an MSVC build using Visual Studio 2019. If others
> using GDAL 2.4.x or later on Linux systems could share with me how you
> handle shutdown and cleanup, I’d appreciate it.
>
>
>
> Thanks,
>
> Shawn Fox
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Band Initialization

[Andrew Bell via gdal-dev]

Hi,

Is a newly created raster band initialized to zero? If not, is there some
trivial way to perform such an initialization (this is a MEM driver raster,
if that matters).

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] gdal2xyz

[Andrew Bell via gdal-dev]

Hi,

The documentation describes an application `gdal2xyz`, but I don't see one
built in my development environment. I also don't see anything in the
CMakeLists.txt that refers to it. Is it named something I'm not expecting?
Does it still exist?

Thanks,

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] pre-commit hook failure

[Andrew Bell via gdal-dev]

Hi,

I'm getting an error when flake8 runs as a pre-commit hook and it won't
allow me to commit code. My understanding was that the pre-commit stuff was
supposed to make sure the proper versions were found. Ideas?

(gdal) [viewshed-oor] $ git commit
blackPassed
isortPassed
flake8...Failed
- hook id: flake8
- exit code: 1

Traceback (most recent call last):
  File
"/Users/abell/.cache/pre-commit/repowsfp6jbc/py_env-python3.12/bin/flake8",
line 8, in 
sys.exit(main())
 ^^
  File
"/Users/abell/.cache/pre-commit/repowsfp6jbc/py_env-python3.12/lib/python3.12/site-packages/flake8/main/cli.py",
line 22, in main
app.run(argv)
  File
"/Users/abell/.cache/pre-commit/repowsfp6jbc/py_env-python3.12/lib/python3.12/site-packages/flake8/main/application.py",
line 363, in run
self._run(argv)
  File
"/Users/abell/.cache/pre-commit/repowsfp6jbc/py_env-python3.12/lib/python3.12/site-packages/flake8/main/application.py",
line 350, in _run
self.initialize(argv)
  File
"/Users/abell/.cache/pre-commit/repowsfp6jbc/py_env-python3.12/lib/python3.12/site-packages/flake8/main/application.py",
line 330, in initialize
self.find_plugins(config_finder)
  File
"/Users/abell/.cache/pre-commit/repowsfp6jbc/py_env-python3.12/lib/python3.12/site-packages/flake8/main/application.py",
line 153, in find_plugins
self.check_plugins = plugin_manager.Checkers(local_plugins.extension)
 
  File
"/Users/abell/.cache/pre-commit/repowsfp6jbc/py_env-python3.12/lib/python3.12/site-packages/flake8/plugins/manager.py",
line 356, in __init__
self.manager = PluginManager(
   ^^
  File
"/Users/abell/.cache/pre-commit/repowsfp6jbc/py_env-python3.12/lib/python3.12/site-packages/flake8/plugins/manager.py",
line 238, in __init__
self._load_entrypoint_plugins()
  File
"/Users/abell/.cache/pre-commit/repowsfp6jbc/py_env-python3.12/lib/python3.12/site-packages/flake8/plugins/manager.py",
line 254, in _load_entrypoint_plugins
eps = importlib_metadata.entry_points().get(self.namespace, ())
  ^
AttributeError: 'EntryPoints' object has no attribute 'get'

--
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] get_cli_utility_path_internal

[Andrew Bell via gdal-dev]

The function in the subject returns a path to a utility on which to run a
test. But in the process, it tries to run the utility itself. This seems
strange. In my case, the test itself failed, but the code reported "Could
not find {file}", which is incorrect. It also causes the actual tests to
get skipped, when they should fail.

I can fix this, but I don't understand why there's an attempt to run a
utility in order to determine its path. I would think that checking that
the file exists would be sufficient (and this already occurs), but I'm not
privy to the history.

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Thread-safe raster access

[Andrew Bell via gdal-dev]

() time, that will basically applies strategy (1) by
> protecting all calls with a mutex). Perhaps using thread-specific instances
> of VSILFILE* and third-party "reader" objects could be a way of solving
> this. But realistically doing a pass in all GDAL drivers would be a
> multi-month-man to multi-year-man type of effort. A realistic plan should
> be designed to allow combining (1) and (2): (2) for a few select drivers,
> and (1) as a fallback for most drivers that wouldn't be updated.
> >
> > Even
> >
> > Le 03/06/2024 à 15:44, Andrew Bell via gdal-dev a écrit :
> >
> > Hi,
> >
> > I am aware that there isn't thread-safe raster access with the current
> GDAL interface for various reasons. Given the state of processors, I was
> wondering if it would be valuable to take a look at providing the ability
> to do Raster I/O (at least reads) in a thread-safe way. This could be done
> through a new set of API calls or perhaps by modifications to what
> currently exists -- I don't know what makes sense at this point. I would be
> happy to spend some time looking at this if there is interest, but I would
> also like to learn from existing experience as to what kinds of things that
> I'm surely not considering would have to be dealt with.
> >
> > Thanks,
> >
> > --
> > Andrew Bell
> > andrew.bell...@gmail.com
> >
> > ___
> > gdal-dev mailing list
> > gdal-dev@lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
> >
> > --
> > http://www.spatialys.com
> > My software is free, but my time generally not.
> >
> > ___
> > gdal-dev mailing list
> > gdal-dev@lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Thread-safe raster access

[Andrew Bell via gdal-dev]

Hi,

I am aware that there isn't thread-safe raster access with the current GDAL
interface for various reasons. Given the state of processors, I was
wondering if it would be valuable to take a look at providing the ability
to do Raster I/O (at least reads) in a thread-safe way. This could be done
through a new set of API calls or perhaps by modifications to what
currently exists -- I don't know what makes sense at this point. I would be
happy to spend some time looking at this if there is interest, but I would
also like to learn from existing experience as to what kinds of things that
I'm surely not considering would have to be dealt with.

Thanks,

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Threading and Datasets

[Andrew Bell via gdal-dev]

Hi,

Are there any issues with opening the same input multiple times as distinct
datasets so that simultaneous read access can be done by threads?

Thanks,

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Data Type Conversion

[Andrew Bell via gdal-dev]

Hi,

The documentation for GDALRasterBand::RasterIO says that it will convert
data from the source data type to the destination data type. I searched the
source code a bit I'm not sure I located the bit that does this conversion.
I'm trying to understand what happens in this conversion process. Is this a
numeric conversion as would be done with a C/C++ cast or is this conversion
simply add or truncate bytes in order to fit in a destination data type. Or
perhaps something else?

Thanks,

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Doxygen Arguments

[Andrew Bell via gdal-dev]

Thanks to all who looked at this and Even for resolution.

On Thu, May 9, 2024 at 12:11 PM Even Rouault 
wrote:

> Andrew,
>
> this is now fixed per
> https://github.com/OSGeo/gdal/commit/393eee77b40ae16751e4d2c0ad3e53386b22d025
> . Doxygen apparently takes the function/method at declaration time, and not
> definition time, and we had a number of unnamed parameters in declarations.
> (I also struggled with weird things when trying to fix that locally due not
> using a too recent Doxygen version. The HTML output was fine, but once
> integrated with Sphinx/Breathe, the parameters were still missing. Updating
> to Doxygen 1.9.8 as now used by our CI fixed things)
>
> Even
> Le 09/05/2024 à 15:20, Andrew Bell via gdal-dev a écrit :
>
> Hi,
>
> I'm looking at the documentation for GDALRasterBand::RasterIO and it's
> surprising that the names of the arguments aren't included in the function
> prototype despite them being in the doxygen function definition. The
> arguments are described well below the function prototype, making it hard
> to match things up and you have to assume that the order matches.
>
> Does some doxygen expert know how the argument names could somehow be
> included in the function prototype?
>
> Web page:
> https://gdal.org/api/gdalrasterband_cpp.html#classGDALRasterBand_1a75d4af97b3436a4e79d9759eedf89af4
>
> Source: https://fossies.org/linux/gdal/gcore/gdalrasterband.cpp (line 110)
>
> --
> Andrew Bell
> andrew.bell...@gmail.com
>
> ___
> gdal-dev mailing 
> listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Doxygen Arguments

[Andrew Bell via gdal-dev]

Hi,

I'm looking at the documentation for GDALRasterBand::RasterIO and it's
surprising that the names of the arguments aren't included in the function
prototype despite them being in the doxygen function definition. The
arguments are described well below the function prototype, making it hard
to match things up and you have to assume that the order matches.

Does some doxygen expert know how the argument names could somehow be
included in the function prototype?

Web page:
https://gdal.org/api/gdalrasterband_cpp.html#classGDALRasterBand_1a75d4af97b3436a4e79d9759eedf89af4

Source: https://fossies.org/linux/gdal/gcore/gdalrasterband.cpp (line 110)

--
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Warning/Error Update?

[Andrew Bell via gdal-dev]

Hi,

I don't know the history but I'm getting failures from warnings like this:

[ 75%] Building CXX object
frmts/raw/CMakeFiles/gdal_raw.dir/dipxdataset.cpp.obj
1169In file included from ../../alg/viewshed.cpp:36:
1170../../alg/viewshed.h: In constructor 'gdal::Viewshed::Viewshed(const
Options&)':
1171../../alg/viewshed.h:106:22: error: 'gdal::Viewshed::vFirstLineVal'
should be initialized in the member initialization list [-Werror=effc++]
1172 106 | CPL_DLL explicit Viewshed(const Options &opts)
1173 | ^~~~

This complaint is about a std::vector, whose initialization is
automatically addressed by its default constructor. The same kind of thing
happens with std::string. It's tedious (and IMO error-prone) to have to
initialize things that are already initialized.  This was recognized long
ago but perhaps never addressed:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55837

Thoughts on turning off such checks as they're not sufficiently specific?

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Mass operations to raster block?

[Andrew Bell via gdal-dev]

Hi,

Is there any support in the library for applying some arithmetic operations
to all entries in a block? For example, can I optimally subtract or add
some scalar to every entry?

Thanks,

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Test problem After Merge

[Andrew Bell via gdal-dev]

On Tue, Apr 30, 2024 at 9:18 AM Even Rouault 
wrote:

Even,

> Also try in a Python interpreter "from osgeo import gdal" to see if the
> exception is more verbose. If your GDAL lib links against libraries that
> are not in the default library search path, GDAL command line utilities
> might still be able to find them, but loading libgdal through the Python
> bindings might fail (at least that happens to me on Linux, cf
> https://github.com/OSGeo/gdal/pull/9783)
>
Also try otool -L on libgdal.dylib and on _gdal.cpython-312-darwin.so
>
This was indeed the issue. I didn't think there was a version of GDAL in my
environment, but it must have gotten in by error at some point. When the
SWIG library _gdal...so was created, a dependency was created for the
system-located library (version 34) rather than the one in the build
(version 35). I'd have to do more research to figure out how to rectify
this, if desired.

Thanks for your help. The python failure message wasn't too helpful on its
own.

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Test problem After Merge

[Andrew Bell via gdal-dev]

Hi all,

I removed all files from the build and started over, building using the
same procedure that I used with the source from a couple of weeks ago
without problem and now I get the following error:

(gdal) [master] $ pytest -v autotest/utilities/test_gdal_viewshed.py
ImportError while loading conftest
'/Users/abell/gdal.2/build/autotest/conftest.py'.
autotest/conftest.py:9: in 
from osgeo import gdal, ogr, osr
swig/python/osgeo/__init__.py:35: in 
_gdal = swig_import_helper()
swig/python/osgeo/__init__.py:32: in swig_import_helper
return importlib.import_module('_gdal')
E   ModuleNotFoundError: No module named '_gdal'


PYTHONPATH is set as follows:

(gdal) [master] $ env | grep PYTHONPATH
PYTHONPATH=/Users/abell/gdal.2/build/swig/python:


The osgeo directory referenced from PYTHONPATH contains the following:

(gdal) [master] $ ls -l /Users/abell/gdal.2/build/swig/python/osgeo
total 6136
-rw-r--r--  1 abell  staff4758 Apr 30 08:00 __init__.py
drwxr-xr-x  3 abell  staff  96 Apr 30 08:01 __pycache__
-rwxr-xr-x  1 abell  staff  970248 Apr 30 08:00 _gdal.cpython-312-darwin.so
-rwxr-xr-x  1 abell  staff  136376 Apr 30 08:00 _
gdal_array.cpython-312-darwin.so
-rwxr-xr-x  1 abell  staff   54352 Apr 30 08:00 _
gdalconst.cpython-312-darwin.so
-rwxr-xr-x  1 abell  staff  104072 Apr 30 08:00 _gnm.cpython-312-darwin.so
-rwxr-xr-x  1 abell  staff  717640 Apr 30 08:00 _ogr.cpython-312-darwin.so
-rwxr-xr-x  1 abell  staff  416120 Apr 30 08:00 _osr.cpython-312-darwin.so
-rw-r--r--  1 abell  staff  336675 Apr 30 08:00 gdal.py
-rw-r--r--  1 abell  staff   33746 Apr 30 08:00 gdal_array.py
-rw-r--r--  1 abell  staff   12517 Apr 30 08:00 gdalconst.py
-rw-r--r--  1 abell  staff 214 Apr 30 08:00 gdalnumeric.py
-rw-r--r--  1 abell  staff   14413 Apr 30 08:00 gnm.py
-rw-r--r--  1 abell  staff  221167 Apr 30 08:00 ogr.py
-rw-r--r--  1 abell  staff   84266 Apr 30 08:00 osr.py

I would have thought the file _gdal.cpython-312-darwin.so would have been
found and loaded, but my expertise in this is limited. I tried tracing with
PYTHONVERBOSE set, but the only references to _gdal I saw were in the same
error message quoted above.

Thanks for any help,

On Mon, Apr 29, 2024 at 5:37 PM Even Rouault 
wrote:

> Andrew,
>
> Perhaps you're running updated GDAL Python bindings against a libgdal that
> hasn't been rebuilt recently? OSRIsDerivedProjected() has been added
> recently both in libgdal and the bindings
>

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Test problem After Merge

[Andrew Bell via gdal-dev]

Hi,

I just merged master into my branch and I'm now getting an error when
trying to run a test. If someone might know what happened, I'd appreciate a
tip.

(gdal) [master] $ pytest -v autotest/utilities/test_gdal_viewshed.py
ImportError while loading conftest
'/Users/abell/gdal.2/build/autotest/conftest.py'.
autotest/conftest.py:9: in 
from osgeo import gdal, ogr, osr
swig/python/osgeo/gdal.py:3942: in 
from . import ogr
swig/python/osgeo/ogr.py:608: in 
from . import osr
swig/python/osgeo/osr.py:13: in 
from . import _osr
E   ImportError: dlopen(/Users/abell/gdal.2/build/swig/python/osgeo/_
osr.cpython-312-darwin.so, 0x0002): symbol not found in flat namespace
'_OSRIsDerivedProjected'

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Hungarian Notation

[Andrew Bell via gdal-dev]

Hi,

I was going to make some changes to some GDAL code and I generally want to
follow along. I see that most of the code is still using Hungarian notation
for identifier names. I haven't seen this anywhere else in like 20 years.

Is it OK to submit code without this naming convention or do we still think
it's important. I didn't see an RFC or documentation that covers this, but
there's lots so I may have missed it.

Thanks,

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev