Re: [gdal-dev] Show isBigTIFF in image metadata

[Kurt Schwehr via gdal-dev]

Jukka,

What's the exact use case for needing to know if a tiff is traditional or
BigTIFF? Is there a key tool that doesn't understand BigTIFF? The only one
I know of is Autodesk Civil3D.

Thanks,
-Kurt

On Thu, Nov 2, 2023 at 2:49 PM Even Rouault via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Jukka,
>
>
> Does it feel reasonable? I know that overviews may be standard TIFFs while
> the main image is BigTIFF but maybe the information from the header would
> be enough. Or have I missed some existing tool? I thought that tiffinfo at
> least would report TIFF/BigTIFF but it doesn’t.
>
> tiffdump does:
>
> $ tiffdump byte.tif
> byte.tif:
> Magic: 0x4949  Version: 0x2a 
>
> vs
>
> $ tiffdump bigtiff.tif
> bigtiff.tif:
> Magic: 0x4949  Version: 0x2b 
>
> Even
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Show isBigTIFF in image metadata

[Even Rouault via gdal-dev]

Jukka,



Does it feel reasonable? I know that overviews may be standard TIFFs 
while the main image is BigTIFF but maybe the information from the 
header would be enough. Or have I missed some existing tool? I thought 
that tiffinfo at least would report TIFF/BigTIFF but it doesn’t.



tiffdump does:

$ tiffdump byte.tif
byte.tif:
Magic: 0x4949  Version: 0x2a 

vs

$ tiffdump bigtiff.tif
bigtiff.tif:
Magic: 0x4949  Version: 0x2b 

Even

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Show isBigTIFF in image metadata

[Rahkonen Jukka via gdal-dev]

Hi,

I tried to find some easy way for checking with GDAL if an image is a standard 
TIFF or BigTIFF but I could not find any. It seems that "more big.tiff" on 
Windows finds "II+" from the beginning of the file if the image is BigTIFF and 
"II*" otherwise. Or then I can open the image with a hex editor for seeing if 
the third byte is 2A or 2B. However, I would like to see the information with 
gdalinfo for example as

Image Structure Metadata:
  INTERLEAVE=BAND
  VERSION=BigTIFF

Does it feel reasonable? I know that overviews may be standard TIFFs while the 
main image is BigTIFF but maybe the information from the header would be 
enough. Or have I missed some existing tool? I thought that tiffinfo at least 
would report TIFF/BigTIFF but it doesn't.

-Jukka Rahkonen-
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

[Even Rouault via gdal-dev]

Howard,


I would like to see some language that describes the expectations for 
version compatibility of plugins (ie, what happens with a plugin built 
against x.0 is used against a main library of y.0).


This RFC actually doesn't change anything regarding this. See the 
mention of the GDAL_CHECK_VERSION macro in 
https://gdal.org/tutorials/raster_driver_tut.html#the-driver which 
covers this topic (I've just slightly tuned it to the below version 
which will be online soon):


"Note the use of GDAL_CHECK_VERSION macro. This is a macro that should 
be used by drivers that can be built as a plugin. As the GDAL C++ ABI 
may, and will, change between GDAL feature releases (for example from 
GDAL 3.x.0 to 3.y.0), it is necessary to recompile your driver against 
the header files of the GDAL feature version with which you want to make 
it work. The GDAL_CHECK_VERSION macro will check that the GDAL version 
with which the driver was compiled and the version against which it is 
running are compatible (checking that the major and minor version 
numbers are equal). The C++ ABI will however remain stable for releases 
of the same release branch (that is for bug fixes releases x.y.z of a 
given feature release x.y.0)."


Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

[Howard Butler via gdal-dev]

> On Nov 2, 2023, at 6:59 AM, Even Rouault via gdal-dev 
>  wrote:
> 
> Hi,
> 
> I'm seeking for feedback and review on a new RFC (RFC 96: Deferred in-tree 
> C++ plugin loading),
> detailed in https://github.com/OSGeo/gdal/pull/8648, whose summary is:
> 
> This RFC adds a mechanism to defer the loading of in-tree C++ plugin drivers 
> to
> the point where their executable code is actually needed, and converts a 
> number
> of relevant drivers to use that mechanism. The aim is to allow for more 
> modular
> GDAL builds, while improving the performance of plugin loading.
> 
> (This is material only for GDAL 3.9 of course)


IMO this has been needed for a while. I'm glad you've been able to come to a 
solution that doesn't significantly impact behavior or performance. 

I would like to see some language that describes the expectations for version 
compatibility of plugins (ie, what happens with a plugin built against x.0 is 
used against a main library of y.0).

I'm very interested to hear from packagers about this topic and whether or not 
a plugin model like this is desirable to address GDAL's large dependency 
challenge.

Howard___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.8.0beta1 available for testing

[Even Rouault via gdal-dev]

Hi Sean,
Rasterio's CI picked up a change to the AAIGrid driver in 3.8. The 3.7 
version driver used to have whitespace before a row and no whitespace 
after. It looks like this has flipped in 3.8. Is it intentional?


Yes: https://github.com/OSGeo/gdal/pull/8362

Even

--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: Adopt GDAL 3.7.3RC1 as 3.7.3 release

[Sean Gillies via gdal-dev]

I'm changing Rasterio's CI to test against both the head of the master
branch and the head of the current release branch, so in the future I'll
likely only speak up if something is broken right before the release.
3.7.3RC1 looks good to me. I can't think of anything missing.

On Wed, Nov 1, 2023 at 5:14 AM Even Rouault via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi,
>
> Motion:
>
> Adopt GDAL 3.7.3RC1 as 3.7.3 release
>
> Starting with my +1
>
> Even
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL 3.8.0beta1 available for testing

[Sean Gillies via gdal-dev]

Hi Even,

Rasterio's CI picked up a change to the AAIGrid driver in 3.8. The 3.7
version driver used to have whitespace before a row and no whitespace
after. It looks like this has flipped in 3.8. Is it intentional? I only
noticed because one of my tests is parsing the file as text. It's certainly
not a big deal. Hopefully this format has mostly expired on the internet :)

On Tue, Oct 31, 2023 at 8:34 AM Even Rouault via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi,
>
> I've prepared a beta1 of GDAL 3.8.0 to get feedback from earlier testers.
>
> Sorry no updated NEWS.md file yet, but I'd in particular be interesting
> by testing of ogr2ogr workflows, since they have underwent significant
> changes in the underlying implementation:
>
> - when the source layer is a layer implementing the ArrowStream API
> (that is  GeoPackage, FlatGeoBuf, Arrow or Parquet), and when no ogr2ogr
> options than -of, -where, -spat, -lco, -dsco, -gt,
> -append/-overwrite/-update are used (and -sql as well for GeoPackage).
> When enabling CPL_DEBUG, you'll see a "OGR2OGR: Using WriteArrowBatch()"
> trace when that new code path is taken. If specifying other options, the
> feature-iteration-based traditional implementation is used
>
> - and/or when the output layer is GeoPackage (new layer), due to the
> revamped much faster spatial index creation.  This enhanced spatial
> index creation is not ogr2ogr specific and is actually available more
> generally for CreateLayer() + CreateFeature() or CreateLayer() +
> WriteArrowBatch() scenarios.
>
> Point of attention would be when in situations with large files and/or
> with low RAM.
>
> The ghcr.io/osgeo/gdal:ubuntu-small-latest,
> ghcr.io/osgeo/gdal:ubuntu-full-latest,
> ghcr.io/osgeo/gdal:alpine-normal-latest Docker images have been
> refreshed with 3.8.0beta1 (ghcr.io/osgeo/gdal:alpine-small-latest still
> building at time of writing).
>
> (Note: the GDAL master conda builds mentioned at
> https://gdal.org/download.html#gdal-master-conda-builds have been broken
> for a couple weeks and are thus not usable to test beta1 currently. I'm
> investigating)
>
> Source snapshots at:
>
> - https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0beta1.tar.gz
>
> - https://download.osgeo.org/gdal/3.8.0/gdal-3.8.0beta1.tar.xz
>
> - https://download.osgeo.org/gdal/3.8.0/gdal380beta1.zip
>
> Autotest snapshots:
>
> - https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0beta1.tar.gz
>
> - https://download.osgeo.org/gdal/3.8.0/gdalautotest-3.8.0beta1.zip
>
> Even
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>


-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

[Kurt Schwehr via gdal-dev]

Thanks Even for the RFC!

After a quick read, this seems reasonable. I was mostly concerned about the
impact on folks who statically build everything (my biggest use case), but
that is completely addressed in the doc.

On Thu, Nov 2, 2023 at 5:00 AM Even Rouault via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

> Hi,
>
> I'm seeking for feedback and review on a new RFC (RFC 96: Deferred
> in-tree C++ plugin loading),
> detailed in https://github.com/OSGeo/gdal/pull/8648, whose summary is:
>
> This RFC adds a mechanism to defer the loading of in-tree C++ plugin
> drivers to
> the point where their executable code is actually needed, and converts a
> number
> of relevant drivers to use that mechanism. The aim is to allow for more
> modular
> GDAL builds, while improving the performance of plugin loading.
>
> (This is material only for GDAL 3.9 of course)
>
> Even
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

[Even Rouault via gdal-dev]

Hi Rob,


This looks great from my perspective. Are there any downsides?


- A bit of additional coding complexity for driver development, but not 
that much


- As mentioned in the backwards compatibility paragraph, for people 
doing multi-step builds to build first libgdal and then plugins, that 
will require extra care and defining the 
-DGDAL/OGR_DECLARE_DRIVER__FOR_LATER_PLUGIN. Like the condaforge 
build recipee, but I would expect this to be more the exception, than 
what most distributions do. The recommended approach is still to do a 
single build and dispatch the artifacts in separate installable packages 
afterwards




I guess conceptually "Driver X depending on LibW clashes with Driver Z 
depending on LibY" cases are less likely to be hit during unit 
testing, since a particular test-runner/process won't (eventually) be 
loading the full set of drivers + dependencies? Usually that's a LibW 
vs LibY problem though, not a GDAL issue.


Yes that could somehow hide such issues, but as you said, there are 
integration issues, and GDAL can't be expected to reliably detect those. 
Clashes also happen generally at runtime, when you used clashing 
drivers. Just loading the libraries isn't sufficient per se to run into 
the clashes. At least in most situations on Linux.


I've considered yet-another config option like 
GDAL_DEFERRED_PLUGIN_LOADING=NO to disable the new behaviour (meaning to 
fully load the drivers at GDALAllRegister() time), but didn't go for it 
for now as the use cases for it are not obvious. People wanting to 
emulate it can just loop through drivers and call GetMetadata() on them, 
which will force plugin loading,  to simulate that.




Rob :)


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

[Robert Coup via gdal-dev]

Hi Even,

On Thu, 2 Nov 2023 at 11:59, Even Rouault via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:

>
> I'm seeking for feedback and review on a new RFC (RFC 96: Deferred
> in-tree C++ plugin loading), detailed in
> https://github.com/OSGeo/gdal/pull/8648, whose summary is:
>
> This RFC adds a mechanism to defer the loading of in-tree C++ plugin
> drivers to the point where their executable code is actually needed, and
> converts a
> number of relevant drivers to use that mechanism. The aim is to allow for
> more
> modular GDAL builds, while improving the performance of plugin loading.
>

This looks great from my perspective. Are there any downsides?

I guess conceptually "Driver X depending on LibW clashes with Driver Z
depending on LibY" cases are less likely to be hit during unit testing,
since a particular test-runner/process won't (eventually) be loading the
full set of drivers + dependencies? Usually that's a LibW vs LibY problem
though, not a GDAL issue.

Rob :)
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Call for review and discussion on RFC96: Deferred in-tree C++ plugin loading

[Even Rouault via gdal-dev]

Hi,

I'm seeking for feedback and review on a new RFC (RFC 96: Deferred 
in-tree C++ plugin loading),

detailed in https://github.com/OSGeo/gdal/pull/8648, whose summary is:

This RFC adds a mechanism to defer the loading of in-tree C++ plugin 
drivers to
the point where their executable code is actually needed, and converts a 
number
of relevant drivers to use that mechanism. The aim is to allow for more 
modular

GDAL builds, while improving the performance of plugin loading.

(This is material only for GDAL 3.9 of course)

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] WFS: How to use feature ordering?

[Even Rouault via gdal-dev]

Craig,

remove the leading end-of-line and spaces in your ExecuteSQL

l = ds.ExecuteSQL(
"""SELECT *
FROM "open-data-platform:v_s_parcel_proposed"
ORDER BY "parcel_pfi" ASC
"""
)

Also fixed 
perhttps://github.com/OSGeo/gdal/commit/f6c7d95e2c66ba1f62f6ff17e31af37f7f8f6bc8

Even

Le 02/11/2023 à 02:50, Craig de Stigter via gdal-dev a écrit :

Hey folks

We're having a lot of trouble getting data out of a WFS server that 
doesn't have primary keys. (previous work on this) 




The URL that's causing us grief is this one: 
https://opendata.maps.vic.gov.au/geoserver/wfs?REQUEST=GetFeature=WFS=2.0.0=open-data-platform:v_s_parcel_proposed=0=2 



* we can't disable pagination, because the server only returns 5000 
records even if we ask for more.

* we can't paginate without ordering, because there's no PK.

So, adding a `=parcel_pfi` to the querystring fixes the issue. 
But I can't seem to get GDAL to do that for us.


There does seem to be code for applying an "ORDER BY" clause from 
ExecuteSQL to the WFS 1.1+ SORTBY field. However, I can't get it to work.


I made a test script 
 
to demonstrate the problem. It uses ExecuteSQL to apply an ORDER BY 
clause, but the resulting curl queries don't have a SORTBY parameter. 
Then it crashes after the server returns a 400 error due to the 
missing SORTBY.


I assume there's something I need to do to cause GDAL to use the 
SORTBY parameter but I'm not sure what it is. Any tips would be 
appreciated.


--
Regards,
Craig

Platform Engineer
Koordinates

+64 21 256 9488  / koordinates.com 
 / @koordinates 


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev