[gdal-dev] Some Tests fail in Actions tab of my branch

[Abel Pau via gdal-dev]

Hi,
lately I've been solving some issues in my code to finally do a pull request of 
the MiraMon vector driver.

There some things about test in the action I don't understand. I haven't commit 
any test yet. So I assume it's not my fault.
Anyone knows why some of them are failing?

At macos_build action:
https://github.com/AbelPau/gdal/actions/runs/7978287443/job/21783093990#step:6:2449
There are some === FAILURES 
===


At build-linux-ubuntu-focal
The following tests FAILED:
https://github.com/AbelPau/gdal/actions/runs/7978287434/job/21783086492#step:15:5119
54 - autotest_ogr (Failed)  <-- Remember: there are no tests on my side already.

At build-windows-msys2-mingw
Final version 1 * AbelPau/gdal@3a8d91e 
(github.com)

There are other but I think that all are in the same problem.

[cid:image009.png@01DA6494.12729ED0]


Thanks for any clarification!!

Abel Pau Garcia
GIS developer
[https://www.creaf.cat/sites/default/files/creaf-signature.png]
a@creaf.uab.cat
Let's chat on 
Teams!
Tel. +34 934814277
[https://www.creaf.cat/sites/default/files/so-en-signature.png]
[https://www.creaf.cat/sites/default/files/twitter-icon-signature.png][https://www.creaf.cat/sites/default/files/linkedin-icon-signature.png][https://www.creaf.cat/sites/default/files/youtube-icon-signature.png][https://www.creaf.cat/sites/default/files/instagram-icon-signature.png]
www.creaf.cat | http://blog.creaf.cat
[https://www.creaf.cat/sites/default/files/uab_logo_signatura.png]
CREAF. Campus UAB. Edifici C. 08193 Bellaterra (Barcelona)

Before printing this electronic message, think about the environment.



___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Assert due to stack corruption in FlatGeoBuf export

[Robert Coup via gdal-dev]

Hi,

On Tue, 20 Feb 2024 at 21:44, Robert Coup 
wrote:

> Hi Simon,
>
> On Tue, 20 Feb 2024 at 21:11, Simon Eves  wrote:
>
>> Here's the stack trace for the original assert. Something is stepping on
>> scratch_ to make it 0x10 instead of null, which it starts out as
>> when the flatbuffer object is created, but by the time it gets to
>> allocating memory, it's broken.
>>
>
> What happens if you set a watchpoint in gdb when the flatbuffer is created?
>
> watch -l myfb->scratch
> or watch *0x1234c0ffee
>

Or I've also had success with Mozilla's rr: https://rr-project.org/ — you
can run to a point where scratch is wrong, set a watchpoint on it, and then
run the program backwards to find out what touched it.

Rob :)
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Assert due to stack corruption in FlatGeoBuf export

[Simon Eves via gdal-dev]

gdb) p m_fbb
$5 = (flatbuffers::FlatBufferBuilder &) @0x7fff9070: {static
kFileIdentifierLength = 4, buf_ = {allocator_ = 0x0, own_allocator_ =
false, initial_size_ = 1024, buffer_minalign_ = 8, reserved_ = 0, size_ =
0, buf_ = 0x0, cur_ = 0x0,
scratch_ = 0x1 }, num_field_loc = 8, max_voffset_ = 0, nested = false,
finished = false, minalign_ = 8, force_defaults_ = false, dedup_vtables_ =
true, string_pool = 0x0}

On Tue, Feb 20, 2024 at 1:10 PM Simon Eves  wrote:

> Here's the stack trace for the original assert. Something is stepping on
> scratch_ to make it 0x10 instead of null, which it starts out as
> when the flatbuffer object is created, but by the time it gets to
> allocating memory, it's broken.
>
> On Tue, Feb 20, 2024 at 1:05 PM Simon Eves  wrote:
>
>> (starting a new thread to avoid derailing the static-build one any
>> further)
>>
>> Totally agreed on the mismatch idea, but the code in question is all
>> self-contained down in *ogr/ogrsf_frmts/flatgeobuf* and the *flatbuffers*
>> sub-project (which is a snapshot of a Google OSS project) so I'm struggling
>> to see how there could be a mismatch.
>>
>> Also, although we're building on CentOS 7, we're using relatively new
>> compilers (GCC 11.4 and Clang 14.0.6), and we bundle the matching newer
>> runtimes.
>>
>> We don't have a full static build stack on our normal dev platform
>> (Ubuntu 22.04) so I haven't been able to repro the problem there.
>>
>> I should have mentioned the first time that we have tried using ASAN, and
>> it definitely catches something wrong, but the behavior is different, and
>> varies if you add more debug printfs. For example:
>>
>> DEBUG: vector_downward::push() num = 16
>> DEBUG: about to reallocate, buf_ = 0, cur_ = 0, scratch = 0
>> DEBUG: reallocated, buf_ = 0x61900062d380, cur_ = 0x61900062cf80, scratch
>> = 0
>> DEBUG: vector_downward::push() ptr = 0x61900062cf70, about to do memcpy
>> =
>> ==25459==ERROR: AddressSanitizer: heap-buffer-overflow on address
>> 0x61900062cf70 at pc 0x7f8933eb87f6 bp 0x7fffa7aa0e70 sp 0x7fffa7aa0620
>> WRITE of size 16 at 0x61900062cf70 thread T0
>>
>> ...but it's still not obvious what exactly is going wrong. The code and
>> data flow makes perfect sense when you step through it in a dynamic build
>> that doesn't fail.
>>
>> Like I said, the frustrating part is that a simple test program
>> (attached) compiled against the same set of static libs works fine.
>>
>> S
>>
>> On Tue, Feb 20, 2024 at 12:33 PM Robert Coup 
>> wrote:
>>
>>> Hi Simon,
>>>
>>> On Tue, 20 Feb 2024 at 18:58, Simon Eves via gdal-dev <
>>> gdal-dev@lists.osgeo.org> wrote:
>>>
 We still have one VERY strange issue whereby FlatGeoBuf export fails in
 a very consistent and reproducible form down in the flatbuffer code, but
 only in the static build, and only in the full system. I have written a
 simple test harness that links the very same static libgdal and does a
 simple GDAL startup and FGB export of a single feature and that works fine.
 It's some kind of data/stack corruption when it first tries to write to the
 flatbuffer on the first feature, which results in a pointer member of the
 buffer class becoming 0x1000 (always) instead of null, and then it
 stops on an assert. There is also one private function in the
 vector_downward class which the debugger won't even step into in that
 build.  I can even put printfs in that function and they don't come out.
 I've tried it on CentOS and on Ubuntu, with GCC and Clang, and it's always
 the same. Everything else in GDAL works just fine (we have LOTS of
 import/export unit tests). This makes zero sense as all the FGB code is
 internal to GDAL and compiled together. I've been poking at it for over a
 week and it's doing my head in.

>>>
>>> One cause of this sort of crash is a header/library mismatch somewhere
>>> where a function is expecting different parameters/types than the caller is
>>> actually providing. Otherwise, maybe a bug in glibc/libstdc++/gcc/something
>>> that's been fixed in the intervening ten years since CentOS 7 was released?
>>>
>>>
>>> If you run your *build* on a modern distro/libc/gcc/etc does it change
>>> things? If it's the same, maybe hints more towards the former.
>>>
>>> ASAN (https://github.com/google/sanitizers/wiki/AddressSanitizer) might
>>> help track down stack/heap corruption.
>>>
>>> Rob :)
>>>
>>>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Assert due to stack corruption in FlatGeoBuf export

[Simon Eves via gdal-dev]

Here's the stack trace for the original assert. Something is stepping on
scratch_ to make it 0x10 instead of null, which it starts out as
when the flatbuffer object is created, but by the time it gets to
allocating memory, it's broken.

On Tue, Feb 20, 2024 at 1:05 PM Simon Eves  wrote:

> (starting a new thread to avoid derailing the static-build one any further)
>
> Totally agreed on the mismatch idea, but the code in question is all
> self-contained down in *ogr/ogrsf_frmts/flatgeobuf* and the *flatbuffers*
> sub-project (which is a snapshot of a Google OSS project) so I'm struggling
> to see how there could be a mismatch.
>
> Also, although we're building on CentOS 7, we're using relatively new
> compilers (GCC 11.4 and Clang 14.0.6), and we bundle the matching newer
> runtimes.
>
> We don't have a full static build stack on our normal dev platform (Ubuntu
> 22.04) so I haven't been able to repro the problem there.
>
> I should have mentioned the first time that we have tried using ASAN, and
> it definitely catches something wrong, but the behavior is different, and
> varies if you add more debug printfs. For example:
>
> DEBUG: vector_downward::push() num = 16
> DEBUG: about to reallocate, buf_ = 0, cur_ = 0, scratch = 0
> DEBUG: reallocated, buf_ = 0x61900062d380, cur_ = 0x61900062cf80, scratch
> = 0
> DEBUG: vector_downward::push() ptr = 0x61900062cf70, about to do memcpy
> =
> ==25459==ERROR: AddressSanitizer: heap-buffer-overflow on address
> 0x61900062cf70 at pc 0x7f8933eb87f6 bp 0x7fffa7aa0e70 sp 0x7fffa7aa0620
> WRITE of size 16 at 0x61900062cf70 thread T0
>
> ...but it's still not obvious what exactly is going wrong. The code and
> data flow makes perfect sense when you step through it in a dynamic build
> that doesn't fail.
>
> Like I said, the frustrating part is that a simple test program (attached)
> compiled against the same set of static libs works fine.
>
> S
>
> On Tue, Feb 20, 2024 at 12:33 PM Robert Coup 
> wrote:
>
>> Hi Simon,
>>
>> On Tue, 20 Feb 2024 at 18:58, Simon Eves via gdal-dev <
>> gdal-dev@lists.osgeo.org> wrote:
>>
>>> We still have one VERY strange issue whereby FlatGeoBuf export fails in
>>> a very consistent and reproducible form down in the flatbuffer code, but
>>> only in the static build, and only in the full system. I have written a
>>> simple test harness that links the very same static libgdal and does a
>>> simple GDAL startup and FGB export of a single feature and that works fine.
>>> It's some kind of data/stack corruption when it first tries to write to the
>>> flatbuffer on the first feature, which results in a pointer member of the
>>> buffer class becoming 0x1000 (always) instead of null, and then it
>>> stops on an assert. There is also one private function in the
>>> vector_downward class which the debugger won't even step into in that
>>> build.  I can even put printfs in that function and they don't come out.
>>> I've tried it on CentOS and on Ubuntu, with GCC and Clang, and it's always
>>> the same. Everything else in GDAL works just fine (we have LOTS of
>>> import/export unit tests). This makes zero sense as all the FGB code is
>>> internal to GDAL and compiled together. I've been poking at it for over a
>>> week and it's doing my head in.
>>>
>>
>> One cause of this sort of crash is a header/library mismatch somewhere
>> where a function is expecting different parameters/types than the caller is
>> actually providing. Otherwise, maybe a bug in glibc/libstdc++/gcc/something
>> that's been fixed in the intervening ten years since CentOS 7 was released?
>>
>>
>> If you run your *build* on a modern distro/libc/gcc/etc does it change
>> things? If it's the same, maybe hints more towards the former.
>>
>> ASAN (https://github.com/google/sanitizers/wiki/AddressSanitizer) might
>> help track down stack/heap corruption.
>>
>> Rob :)
>>
>>
Thread 1 "ImportExportTes" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) where
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x75ec6859 in __GI_abort () at abort.c:79
#2  0x75ec6729 in __assert_fail_base (fmt=0x7605c588 "%s%s%s:%u: 
%s%sAssertion `%s' failed.\n%n", assertion=0x13bf5a98 "cur_ >= scratch_ && 
scratch_ >= buf_", 
file=0x13bf5a40 
"/build/scripts/gdal-3.7.3/ogr/ogrsf_frmts/flatgeobuf/flatbuffers/vector_downward.h",
 line=136, function=) at assert.c:92
#3  0x75ed7fd6 in __GI___assert_fail (assertion=0x13bf5a98 "cur_ >= 
scratch_ && scratch_ >= buf_", 
file=0x13bf5a40 
"/build/scripts/gdal-3.7.3/ogr/ogrsf_frmts/flatgeobuf/flatbuffers/vector_downward.h",
 line=136, 
function=0x13bf5a00 "size_t 
flatbuffers::vector_downward::ensure_space(size_t)") at assert.c:101
#4  0x0a240bec in flatbuffers::vector_downward::ensure_space 

[gdal-dev] Assert due to stack corruption in FlatGeoBuf export

[Simon Eves via gdal-dev]

(starting a new thread to avoid derailing the static-build one any further)

Totally agreed on the mismatch idea, but the code in question is all
self-contained down in *ogr/ogrsf_frmts/flatgeobuf* and the *flatbuffers*
sub-project (which is a snapshot of a Google OSS project) so I'm struggling
to see how there could be a mismatch.

Also, although we're building on CentOS 7, we're using relatively new
compilers (GCC 11.4 and Clang 14.0.6), and we bundle the matching newer
runtimes.

We don't have a full static build stack on our normal dev platform (Ubuntu
22.04) so I haven't been able to repro the problem there.

I should have mentioned the first time that we have tried using ASAN, and
it definitely catches something wrong, but the behavior is different, and
varies if you add more debug printfs. For example:

DEBUG: vector_downward::push() num = 16
DEBUG: about to reallocate, buf_ = 0, cur_ = 0, scratch = 0
DEBUG: reallocated, buf_ = 0x61900062d380, cur_ = 0x61900062cf80, scratch =
0
DEBUG: vector_downward::push() ptr = 0x61900062cf70, about to do memcpy
=
==25459==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x61900062cf70 at pc 0x7f8933eb87f6 bp 0x7fffa7aa0e70 sp 0x7fffa7aa0620
WRITE of size 16 at 0x61900062cf70 thread T0

...but it's still not obvious what exactly is going wrong. The code and
data flow makes perfect sense when you step through it in a dynamic build
that doesn't fail.

Like I said, the frustrating part is that a simple test program (attached)
compiled against the same set of static libs works fine.

S

On Tue, Feb 20, 2024 at 12:33 PM Robert Coup 
wrote:

> Hi Simon,
>
> On Tue, 20 Feb 2024 at 18:58, Simon Eves via gdal-dev <
> gdal-dev@lists.osgeo.org> wrote:
>
>> We still have one VERY strange issue whereby FlatGeoBuf export fails in a
>> very consistent and reproducible form down in the flatbuffer code, but only
>> in the static build, and only in the full system. I have written a simple
>> test harness that links the very same static libgdal and does a simple GDAL
>> startup and FGB export of a single feature and that works fine. It's some
>> kind of data/stack corruption when it first tries to write to the
>> flatbuffer on the first feature, which results in a pointer member of the
>> buffer class becoming 0x1000 (always) instead of null, and then it
>> stops on an assert. There is also one private function in the
>> vector_downward class which the debugger won't even step into in that
>> build.  I can even put printfs in that function and they don't come out.
>> I've tried it on CentOS and on Ubuntu, with GCC and Clang, and it's always
>> the same. Everything else in GDAL works just fine (we have LOTS of
>> import/export unit tests). This makes zero sense as all the FGB code is
>> internal to GDAL and compiled together. I've been poking at it for over a
>> week and it's doing my head in.
>>
>
> One cause of this sort of crash is a header/library mismatch somewhere
> where a function is expecting different parameters/types than the caller is
> actually providing. Otherwise, maybe a bug in glibc/libstdc++/gcc/something
> that's been fixed in the intervening ten years since CentOS 7 was released?
>
>
> If you run your *build* on a modern distro/libc/gcc/etc does it change
> things? If it's the same, maybe hints more towards the former.
>
> ASAN (https://github.com/google/sanitizers/wiki/AddressSanitizer) might
> help track down stack/heap corruption.
>
> Rob :)
>
>
#include 
#include 
#include 

#include 
#include 
#include 

#include 

#define GDAL_DATA_PATH DEPS "/share/gdal"
#define PROJ_DATA_PATH DEPS "/share/proj"

void gdal_error_handler(CPLErr err_class, int err_no, const char* err_msg) {
  static constexpr std::array err_class_strings{
  "Info",
  "Debug",
  "Warning",
  "Failure",
  "Fatal",
  };
  std::string log_msg = std::string("GDAL ") + err_class_strings[err_class] + ": " +
err_msg + " (" + std::to_string(err_no) + ")";
std::cout << log_msg << std::endl;
}

int main(int argc, char** argv)
{
	// parameters
	if (argc != 3) {
		std::cout << "Usage: gdal_export_test  " << std::endl;
		std::cout << " must be one of the names returned by ogrinfo --formats, e.g. GeoJSON, FlatGeobuf, \"ESRI Shapefile\"" << std::endl;
		exit(1);
	}
	const char* output_file = argv[1];
	const char* driver_name = argv[2];

	// init GDAL
	std::cout << "Initializing GDAL" << std::endl;
	std::cout << "GDAL_DATA_PATH = " << GDAL_DATA_PATH << std::endl;
	std::cout << "PROJ_DATA_PATH = " << PROJ_DATA_PATH << std::endl;
	setenv("GDAL_DATA", GDAL_DATA_PATH, true);
	setenv("PROJ_DATA", PROJ_DATA_PATH, true);
	GDALAllRegister();
	OGRRegisterAll();
	CPLSetErrorHandler(*gdal_error_handler);

	// get driver
	std::cout << "Getting driver" << std::endl;
	auto* gdal_driver = GetGDALDriverManager()->GetDriverByName(driver_name);
	assert(gdal_driver);

	// open 

Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps

[Robert Coup via gdal-dev]

Hi Simon,

On Tue, 20 Feb 2024 at 18:58, Simon Eves via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> We still have one VERY strange issue whereby FlatGeoBuf export fails in a
> very consistent and reproducible form down in the flatbuffer code, but only
> in the static build, and only in the full system. I have written a simple
> test harness that links the very same static libgdal and does a simple GDAL
> startup and FGB export of a single feature and that works fine. It's some
> kind of data/stack corruption when it first tries to write to the
> flatbuffer on the first feature, which results in a pointer member of the
> buffer class becoming 0x1000 (always) instead of null, and then it
> stops on an assert. There is also one private function in the
> vector_downward class which the debugger won't even step into in that
> build.  I can even put printfs in that function and they don't come out.
> I've tried it on CentOS and on Ubuntu, with GCC and Clang, and it's always
> the same. Everything else in GDAL works just fine (we have LOTS of
> import/export unit tests). This makes zero sense as all the FGB code is
> internal to GDAL and compiled together. I've been poking at it for over a
> week and it's doing my head in.
>

One cause of this sort of crash is a header/library mismatch somewhere
where a function is expecting different parameters/types than the caller is
actually providing. Otherwise, maybe a bug in glibc/libstdc++/gcc/something
that's been fixed in the intervening ten years since CentOS 7 was released?

If you run your *build* on a modern distro/libc/gcc/etc does it change
things? If it's the same, maybe hints more towards the former.

ASAN (https://github.com/google/sanitizers/wiki/AddressSanitizer) might
help track down stack/heap corruption.

Rob :)
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps

[Simon Eves via gdal-dev]

One of our build variants is a fully-static (or as-static-as-possible) done
on CentOS 7 for old-Linux compatibility.

In reverse-dependency order we have static builds of

SQLite 3450100
Expat 2.5.0
KML 1.2.0
HDF5 1.12.1
NetCDF 4.8.1
TIFF 4.5.1
Proj 9.3.0
GeoTIFF 1.7.1
LittleCMS 2.15
WebP 1.3.2
Zstd 1.4.8
OpenJPEG 2.5.0
GDAL 3.7.3

We use GEOS as well, but directly, not linked into GDAL.

We do our own builds of TIFF and GeoTIFF since they are also dependencies
of other libs we use (e.g. pdal for LIDAR import) and the rest are to
activate desirable GDAL drivers.

LittleCMS, WebP, Zstd, and OpenJPEG were added recently in order to
activate the GDAL JPEG2000 driver (for Sentinel2 satellite raster data) and
it took some time to get all the options right so the static stack would
work.

I can provide the build options we use for the static, if that would be
helpful.

I never got the GDAL tools compiling as full-static, but that's not
important for us, as we don't bundle those with our product. Our
development systems are Ubuntu with more conventional dynamic builds.

We still have one VERY strange issue whereby FlatGeoBuf export fails in a
very consistent and reproducible form down in the flatbuffer code, but only
in the static build, and only in the full system. I have written a simple
test harness that links the very same static libgdal and does a simple GDAL
startup and FGB export of a single feature and that works fine. It's some
kind of data/stack corruption when it first tries to write to the
flatbuffer on the first feature, which results in a pointer member of the
buffer class becoming 0x1000 (always) instead of null, and then it
stops on an assert. There is also one private function in the
vector_downward class which the debugger won't even step into in that
build.  I can even put printfs in that function and they don't come out.
I've tried it on CentOS and on Ubuntu, with GCC and Clang, and it's always
the same. Everything else in GDAL works just fine (we have LOTS of
import/export unit tests). This makes zero sense as all the FGB code is
internal to GDAL and compiled together. I've been poking at it for over a
week and it's doing my head in.

Not meaning to derail this thread, but just wondering if anyone else had
encountered any strange run-time behavior with static builds.


On Mon, Feb 12, 2024 at 4:02 AM Michael Otto via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hello,
>
> I have been trying for some time to compile GDAL and the GDAL apps as a
> static library / executable programs.
>
> The goal is to cast GDAL and all its dependencies (PROJ / GEOS / all
> dependencies to system libraries / ...) into a static library and to create
> the GDAL apps as static executable programs.
> There should be no dynamic dependencies.
>
> Unfortunately, I have not had any success so far. The library is created
> statically but the apps are not yet.
> Does anyone have experience with this topic or possibly a procedure that
> leads to success?
>
> Best regards.
> Michael Otto
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps

[Robert Coup via gdal-dev]

Hi,

On Tue, 20 Feb 2024 at 12:50, Michael Otto 
wrote:

>
> I have now tried it with 'vcpkg install gdal[tools]'. The tools/apps are
> there, but they are not statically linked!
>
> See for example gdalinfo:
>
> root@vmuser-VirtualBox:/home/vmuser/Git/vcpkg/installed/x64-linux/tools/gdal#
> ldd gdalinfo
> linux-vdso.so.1 (0x7ffcc0fb3000)
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7fe3a31ae000)
> libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6
> (0x7fe3a2f82000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7fe3a2d59000)
> /lib64/ld-linux-x86-64.so.2 (0x7fe3a6b09000)
> libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> (0x7fe3a2d39000)


You're closer! GDAL itself is statically linked into those executables, as
are its dependencies...

So, I did a test, and I think this is what you need:

$ mkdir custom-triplets
$ cp vcpkg/triplets/x64-linux.cmake custom-triplets/x64-linux-static.cmake

(a "triplet" is a vcpkg configuration for a particular system/cpu/build
configuration — linux vs windows vs android, static vs dynamic, x64 vs arm,
etc)

Then edit custom-triplets/x64-linux-static.cmake and change the following
to tell it to statically link the system libraries as well:

- set(VCPKG_CRT_LINKAGE dynamic)
+ set(VCPKG_CRT_LINKAGE static)

Then build GDAL using your new triplet:

$ vcpkg/vcpkg --overlay-triplets=custom-triplets --triplet=x64-linux-static
install gdal[tools]

But of course it's not quite that simple... libtiff needs a fix too:

$ mkdir custom-ports
$ cp -a vcpkg/ports/tiff custom-ports/tiff

Then edit custom-ports/tiff/FindCMath.patch and change the following line:

- +find_library(CMath_LIBRARY NAMES m PATHS
${CMAKE_C_IMPLICIT_LINK_DIRECTORIES})
+ +find_library(CMath_LIBRARY NAMES libm.a m PATHS
${CMAKE_C_IMPLICIT_LINK_DIRECTORIES})

(this is super-hacky, there will definitely be a cleaner fix than this, but
it works for this purpose right now)

And there's a static linking issue with something SQLite-plugin-related in
GDAL. So lets reduce GDAL to just the core dependencies and the tools:

$ vcpkg/vcpkg --overlay-triplets=custom-triplets --triplet=x64-linux-static
--overlay-ports=custom-ports install gdal[core,tools]
... go and play in the sunshine ...
$ vcpkg/installed/x64-linux-static/tools/gdal/gdalinfo --version
GDAL 3.8.3, released 2024/01/04
$ ldd vcpkg/installed/x64-linux-static/tools/gdal/gdalinfo
not a dynamic executable

Victory! Now you can hopefully add any other GDAL features/dependencies via
the gdal[core,tools,...] vcpkg feature options. *Fully* static binaries
aren't really a normal tested output, so you might run into further issues.

Hope that helps,

Rob :)
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Question about PG_LIST_ALL_TABLES in the documentation

[Even Rouault via gdal-dev]

Jukka,

I would probably completely remove that paragraph, or at least remove 
the wrong hint on how to check if permissions are correct. Correct 
permission settings of a PostgreSQL/PostGIS database is probably out of 
scope of the driver documentation. But your suggestion for an alternate 
method sounds OK too.


spatial_ref_sys and geometry_columns table/views are always removed from 
the listing due 
https://github.com/OSGeo/gdal/blob/e7f14b7ff025bea54b9fd2363cda5eeae6d5ade6/ogr/ogrsf_frmts/pg/ogrpgdatasource.cpp#L1357 
. The weird thing is that looking in the history that filtering was 
already done before the FAQ entry was written, so I'm not sure to 
remember what I had in mind 16 years ago when I wrote it :-)


Even

Le 20/02/2024 à 15:37, Rahkonen Jukka via gdal-dev a écrit :


Hi,

I don’t undestand this part of the PosgGIS driver documentation 
https://gdal.org/drivers/vector/pg.html#faqs


“Permission issues on geometry_columns and/or spatial_ref_sys tables 
can be generally confirmed if you can see the tables by setting the 
configuration option PG_LIST_ALL_TABLES to YES. (e.g. ogrinfo - 
-config PG_LIST_ALL_TABLES YES PG:x)”


For the first, was the meaning to say the opposite, that there are 
permission issues if you cannot see the tables? And for the second, 
with current GDAL and PostGIS versions user does not see 
geometry_columns and spatial_ref_sys on the ogrinfo layer list even 
with this config option. It seems that they are filtered away from the 
result. I do see that the config option has an effect and for example 
raster_columns and raster_overviews are added to the layer list.


An easy way to check the permissions is to query those views/tables 
directly.

ogrinfo PG:x spatial_ref_sys

ogrinfo PG:x geometry_columns

If ogrinfo lists something else than an error, then permissions are 
OK. Is this method good enough so I could edit the documentation?


-Jukka Rahkonen-


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Question about PG_LIST_ALL_TABLES in the documentation

[Rahkonen Jukka via gdal-dev]

Hi,

I don't undestand this part of the PosgGIS driver documentation 
https://gdal.org/drivers/vector/pg.html#faqs
"Permission issues on geometry_columns and/or spatial_ref_sys tables can be 
generally confirmed if you can see the tables by setting the configuration 
option PG_LIST_ALL_TABLES to YES. (e.g. ogrinfo - -config PG_LIST_ALL_TABLES 
YES PG:x)"

For the first, was the meaning to say the opposite, that there are permission 
issues if you cannot see the tables? And for the second, with current GDAL and 
PostGIS versions user does not see geometry_columns and spatial_ref_sys on the 
ogrinfo layer list even with this config option. It seems that they are 
filtered away from the result. I do see that the config option has an effect 
and for example raster_columns and raster_overviews are added to the layer list.

An easy way to check the permissions is to query those views/tables directly.
ogrinfo PG:x spatial_ref_sys
ogrinfo PG:x geometry_columns

If ogrinfo lists something else than an error, then permissions are OK. Is this 
method good enough so I could edit the documentation?

-Jukka Rahkonen-


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Antwort: Re: Build static GDAL-Lib and static GDAL-Apps

[Michael Otto via gdal-dev]

Thanks to all who have responded.

I have now tried it with 'vcpkg install gdal[tools]'. The tools/apps are 
there, but they are not statically linked!

See for example gdalinfo:

root@vmuser-VirtualBox:/home/vmuser/Git/vcpkg/installed/x64-linux/tools/gdal# 
ldd gdalinfo
linux-vdso.so.1 (0x7ffcc0fb3000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7fe3a31ae000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 
(0x7fe3a2f82000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7fe3a2d59000)
/lib64/ld-linux-x86-64.so.2 (0x7fe3a6b09000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
(0x7fe3a2d39000)

Michael


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps

[Kai Pastor via gdal-dev]

> > The directory './vcpkg/packages/_x64-linux' now contains the 
> > compiled static versions of the libraries. But where can I find the GDAL 
> > apps? These are not contained in this directory.

Ignore packages. It is a staging area. `installed` is the real thing.

Kai
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps

[Robert Coup via gdal-dev]

Hi Michael,

On Tue, 20 Feb 2024 at 11:22, Michael Otto 
wrote:

> The directory './vcpkg/packages/_x64-linux' now contains the
> compiled static versions of the libraries. But where can I find the GDAL
> apps? These are not contained in this directory.


Great! So if you change your vcpkg installation to use `gdal[tools]`
instead of `gdal`, it will build the applications too.

If you need other dependencies, see https://vcpkg.link/ports/gdal then you
can install eg: `gdal[tools,curl]` to add curl too. The options shown under
`default-features` are automatically installed.

Rob :)

>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Antwort: Re: Build static GDAL-Lib and static GDAL-Apps

[Kai Pastor, DG0YT via gdal-dev]

When you install gdal[tools], the apps are installed to 
`/tools/gdal`.

(The vcpkg focus is on providing libs build from source.)

To explore port features, you can look at the port's vcpkg.json (aka 
manifest), or start from

https://vcpkg.link/ports/gdal

Regards,
Kai


Am 20.02.24 um 12:22 schrieb Michael Otto via gdal-dev:

Hi Robert,

because I'm not getting anywhere with the "standard" CMake solution, 
I've now used the vcpkg you mentioned and am using it in a Virtualbox 
VM with Linux Mint.


I got the installation instructions from here: 
https://github.com/microsoft/vcpkg?tab=readme-ov-file#quick-start-unix 
. 
I have installed the Linux developer tools (see instructions).


I have installed vcpkg under 
'root@vmuser-VirtualBox:/home/vmuser/Git/vcpkg'.


When installing the libraries, it seems that a lot of dependencies are 
also loaded, compiled and installed in the now large set of directories.


The directory './vcpkg/packages/_x64-linux' now contains 
the compiled static versions of the libraries. But where can I find 
the GDAL apps? These are not contained in this directory.


I am still new to the topic of "self-compiling".
Do you have any tips on what else I need to do?

Michael




Von: "Robert Coup" 
An: "Michael Otto" 
Kopie: gdal-dev@lists.osgeo.org
Datum: 12.02.2024 15:43
Betreff: Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps




Hi Michael,

On Mon, 12 Feb 2024 at 12:02, Michael Otto via gdal-dev 
<_gdal-dev@lists.osgeo.org_ > wrote:


The goal is to cast GDAL and all its dependencies (PROJ / GEOS / all 
dependencies to system libraries / ...) into a static library and to 
create the GDAL apps as static executable programs.


Which platform are you working on? If you need a fully static 
compilation of everything /all/ the way down including openssl + zlib 
+ zstd + libtiff + all the other libraries, using vcpkg[1] might be a 
reasonable approach. By default it does static compilation on macOS & 
Linux, and it has a static compilation option on Windows (the default 
on Windows is dynamic). It doesn't support every compile option 
though, but it's customisable and if you're after a reasonably vanilla 
GDAL it should work ok.



There should be no dynamic dependencies.

Note that GDAL + Proj (maybe others), have data files they rely on, so 
while you may be able to get a single executable, you'll still need to 
distribute these files. I can imagine ways around that, but they're 
getting fairly complex.


Unfortunately, I have not had any success so far. The library is 
created statically but the apps are not yet.
Does anyone have experience with this topic or possibly a procedure 
that leads to success?


Can you share the process you've currently got to?

Rob :)

[1] _https://github.com/microsoft/vcpkg_ 





___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GeoTiff with ColorInterp=Palette

[Even Rouault via gdal-dev]

Le 20/02/2024 à 11:15, Elena Ruiz via gdal-dev a écrit :
Hello, I have version 3.6.2 of GDAL and I need to crop an image using 
a contour from a GML file, so far there is no problem.


The problem begins when the original image is a GEOTIFF with 
transparency and it generates a cropped image, the information of both 
images is a little lower.


The color table changes from having 2 values to having 256, this 
causes it to lose transparency although it has the label "Mask Flags: 
PER_DATASET ALPHA" which indicates that it has an alpha channel.


My question is, why does the color table change?, and is there any 
parameter that can be used with gdalwarp or gdal_translate to avoid 
this and keep the image transparent?


Strictly speaking your input image has no transparency, just black & 
white colors. Not sure which one you interpret as transparent?



You could force a 1-bit output image (ie with a color palette of 2 
colors) by adding -co NBITS=1 . But that will likely won't play well 
with -dstalpha.  You'd then want to use instead -dstnodata 0 if black is 
the transparent color or -dstnodata 1 if white is the transparent color



Even

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Antwort: Re: Build static GDAL-Lib and static GDAL-Apps

[Michael Otto via gdal-dev]

Hi Robert,

because I'm not getting anywhere with the "standard" CMake solution, I've 
now used the vcpkg you mentioned and am using it in a Virtualbox VM with 
Linux Mint.

I got the installation instructions from here: 
https://github.com/microsoft/vcpkg?tab=readme-ov-file#quick-start-unix. I 
have installed the Linux developer tools (see instructions).

I have installed vcpkg under 
'root@vmuser-VirtualBox:/home/vmuser/Git/vcpkg'.

When installing the libraries, it seems that a lot of dependencies are 
also loaded, compiled and installed in the now large set of directories.

The directory './vcpkg/packages/_x64-linux' now contains the 
compiled static versions of the libraries. But where can I find the GDAL 
apps? These are not contained in this directory. 

I am still new to the topic of "self-compiling".
Do you have any tips on what else I need to do?

Michael




Von:"Robert Coup" 
An: "Michael Otto" 
Kopie:  gdal-dev@lists.osgeo.org
Datum:  12.02.2024 15:43
Betreff:Re: [gdal-dev] Build static GDAL-Lib and static GDAL-Apps



Hi Michael,

On Mon, 12 Feb 2024 at 12:02, Michael Otto via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

The goal is to cast GDAL and all its dependencies (PROJ / GEOS / all 
dependencies to system libraries / ...) into a static library and to 
create the GDAL apps as static executable programs. 

Which platform are you working on? If you need a fully static compilation 
of everything all the way down including openssl + zlib + zstd + libtiff + 
all the other libraries, using vcpkg[1] might be a reasonable approach. By 
default it does static compilation on macOS & Linux, and it has a static 
compilation option on Windows (the default on Windows is dynamic). It 
doesn't support every compile option though, but it's customisable and if 
you're after a reasonably vanilla GDAL it should work ok.
 

There should be no dynamic dependencies.

Note that GDAL + Proj (maybe others), have data files they rely on, so 
while you may be able to get a single executable, you'll still need to 
distribute these files. I can imagine ways around that, but they're 
getting fairly complex.
 
Unfortunately, I have not had any success so far. The library is created 
statically but the apps are not yet.
Does anyone have experience with this topic or possibly a procedure that 
leads to success?

Can you share the process you've currently got to?

Rob :) 

[1] https://github.com/microsoft/vcpkg



___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GeoTiff with ColorInterp=Palette

[Elena Ruiz via gdal-dev]

Hello, I have version 3.6.2 of GDAL and I need to crop an image using a contour 
from a GML file, so far there is no problem.

The problem begins when the original image is a GEOTIFF with transparency and 
it generates a cropped image, the information of both images is a little lower.

The color table changes from having 2 values to having 256, this causes it to 
lose transparency although it has the label "Mask Flags: PER_DATASET ALPHA" 
which indicates that it has an alpha channel.

My question is, why does the color table change?, and is there any parameter 
that can be used with gdalwarp or gdal_translate to avoid this and keep the 
image transparent?



I use this command:

gdalwarp.exe -cutline fichero.gml -crop_to_cutline -dstalpha ficheroIN 
ficheroOUT --config GDAL_DATA rutaGDAL


ORIGINAL IMAGE

Driver: GTiff/GeoTIFF

Size is 5616, 3655

…

Metadata:

  TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)

  TIFFTAG_XRESOLUTION=300

  TIFFTAG_YRESOLUTION=300

Image Structure Metadata:

  INTERLEAVE=BAND

Corner Coordinates:

…

Band 1 Block=5616x1493 Type=Byte, ColorInterp=Palette

  Image Structure Metadata:

NBITS=1

  Color Table (RGB with 2 entries)

0: 0,0,0,255

1: 255,255,255,255



CROPPED IMAGE

Driver: GTiff/GeoTIFF

Size is 5605, 2689

….

Metadata:

  TIFFTAG_DATETIME=2024:02:09 08:49:12

  TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)

  TIFFTAG_SOFTWARE=Microsoft Windows Photo Viewer 10.0.19041.1

  TIFFTAG_XRESOLUTION=300

  TIFFTAG_YRESOLUTION=300

Image Structure Metadata:

  COMPRESSION=LZW

  INTERLEAVE=PIXEL

Corner Coordinates:

….

Band 1 Block=5605x1 Type=Byte, ColorInterp=Palette

  Mask Flags: PER_DATASET ALPHA

  Color Table (RGB with 256 entries)

0: 0,0,0,255

1: 255,255,255,255

2: 0,0,0,255

3: 0,0,0,255

….

  252: 0,0,0,255

  253: 0,0,0,255

  254: 0,0,0,255

  255: 0,0,0,255



I await your response, thank you and regards.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev