Re: [gdal-dev] Considering drivers removal ?

[Stephen Woodbridge]

On 1/12/2021 5:36 PM, Even Rouault wrote:

On mardi 12 janvier 2021 12:56:13 CET Frank Warmerdam wrote:

On Tue, Jan 12, 2021 at 12:38 PM Howard Butler  wrote:

The only question that matters here is "Who is going to maintain it?" and
if the answer to that is "no one", it should be removed. There doesn't
need
to be any meetings because the only criteria that matters is if someone is
willing to maintain it. We should provide the list of drivers and assign
the GitHub handles that step forward to be responsible for each. If
obscure
government one-offs formats have an audience of downtrodden government
users forced to use them, they need to put their handle in and take
ownership. They then need to find the time, money, or attention to carry
things forward.

Howard / Even,

I'd be willing to commit to maintaining some of the archaic drivers that
meet the conditions I mentioned (buildable, testable in core build).  If
Even would like I can provide a sublist of those he proposed i'd be willing
to be responsible for.

NTv1: this is the perfect example of a driver of absolutely no use in 2021.
Unless I'm wrong, there was only one single public dataset for that format,
ntv1_can.dat, and it is now available as GeoTIFF in
https://github.com/OSGeo/PROJ-data/blob/master/ca_nrc/ca_nrc_ntv1_can.tif

My current plan is:

- for BPG, E00GRID, EPSILON, IGNFHeightASCIIGrid, ISG, Aeronav FAA, BNA, HTF,
OpenAir, SEG-P1, SEG-Y, SUA, X-Plane, move *now* driver code, documentation
and tests to https://github.com/OSGeo/gdal-extra-drivers which is a slightly
improved version of a cemetery repository, since it includes a build script to
create a plugin. I have no plan to maintain that repository after that initial
move (that means I won't merge pull requests unless someone else steps up for
the role) and it will likely break in the future. I'd wish we would agree to
move more drivers there. And probably most future drivers for esoteric formats
should go there.
Those drivers are ones I've authored, that received no significant
contribution from anyone else AFAIR, no-one paid development for and I suspect
are close to be unused. So hopefully no one should have bad feelings with them
going away. It was a bad taste of mine to have put them in GDAL to start with.
Why did I picked up the extra repository after all ? A tiny fraction of them
might be useful like ISG or IGNFHeightASCIIGrid in some contexts (to create
grids for PROJ), but definitely not for general purpose. So as far as I'm
concerned, I'll go through the extra step of building the extra repository or
a subset of it if I've an occasional need for them.

- proceed as I mentionned initially for other drivers I listed and no-one
steps up to maintain, with the variation of moving the code to gdal-extra-
drivers instead of just removing them (but potentially not including a build
recipee for them in the build script, if that proves to be too complex).

The issue with esoteric/legacy drivers is not that much maintenance of the
actual code of the drivers, in the sense of dealing with bug reports,
questions, etc. (pretty sure they are none for the ones I listed). Most of
them must work reasonably well and be feature complete, and most
vulnerabilities have now been fixed. My concern is that this legacy code has
indirect costs on other GDAL developers and users. The psychological cost I
mentionned. Let's say someone want to turn on higher warning levels, and that
this breaks in tens of drivers. Would he have to ping every maintainer and
wait for them to address the issue ? Or maybe he will just give up. Similarly
for breaking changes in the driver API. As Sean mentionned, this is probably a
serious obstacle to growing up the core development team.


Even


Even,

Just want to say this sounds like good plan to me, not that my input 
means a lot. I also want to say Thank You! for all your hard work 
supporting this and other projects, but answering my questions through 
the years. I've had a lot of roles in my career in open source and 
industry and can appreciate the difficult balance between compatibility 
with legacy code and the need to break free of it to move forward. It's 
hard and I never enjoyed having to make those decisions, but you have my 
respect and support whatever you decide.


Thank You! again for your efforts and support!

-Steve W
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Even Rouault]

On mardi 12 janvier 2021 12:56:13 CET Frank Warmerdam wrote:
> On Tue, Jan 12, 2021 at 12:38 PM Howard Butler  wrote:
> > The only question that matters here is "Who is going to maintain it?" and
> > if the answer to that is "no one", it should be removed. There doesn't
> > need
> > to be any meetings because the only criteria that matters is if someone is
> > willing to maintain it. We should provide the list of drivers and assign
> > the GitHub handles that step forward to be responsible for each. If
> > obscure
> > government one-offs formats have an audience of downtrodden government
> > users forced to use them, they need to put their handle in and take
> > ownership. They then need to find the time, money, or attention to carry
> > things forward.
> 
> Howard / Even,
> 
> I'd be willing to commit to maintaining some of the archaic drivers that
> meet the conditions I mentioned (buildable, testable in core build).  If
> Even would like I can provide a sublist of those he proposed i'd be willing
> to be responsible for.

NTv1: this is the perfect example of a driver of absolutely no use in 2021. 
Unless I'm wrong, there was only one single public dataset for that format, 
ntv1_can.dat, and it is now available as GeoTIFF in
https://github.com/OSGeo/PROJ-data/blob/master/ca_nrc/ca_nrc_ntv1_can.tif

My current plan is:

- for BPG, E00GRID, EPSILON, IGNFHeightASCIIGrid, ISG, Aeronav FAA, BNA, HTF, 
OpenAir, SEG-P1, SEG-Y, SUA, X-Plane, move *now* driver code, documentation  
and tests to https://github.com/OSGeo/gdal-extra-drivers which is a slightly 
improved version of a cemetery repository, since it includes a build script to 
create a plugin. I have no plan to maintain that repository after that initial 
move (that means I won't merge pull requests unless someone else steps up for 
the role) and it will likely break in the future. I'd wish we would agree to 
move more drivers there. And probably most future drivers for esoteric formats 
should go there.
Those drivers are ones I've authored, that received no significant 
contribution from anyone else AFAIR, no-one paid development for and I suspect 
are close to be unused. So hopefully no one should have bad feelings with them 
going away. It was a bad taste of mine to have put them in GDAL to start with.
Why did I picked up the extra repository after all ? A tiny fraction of them 
might be useful like ISG or IGNFHeightASCIIGrid in some contexts (to create 
grids for PROJ), but definitely not for general purpose. So as far as I'm 
concerned, I'll go through the extra step of building the extra repository or 
a subset of it if I've an occasional need for them.

- proceed as I mentionned initially for other drivers I listed and no-one 
steps up to maintain, with the variation of moving the code to gdal-extra-
drivers instead of just removing them (but potentially not including a build 
recipee for them in the build script, if that proves to be too complex).

The issue with esoteric/legacy drivers is not that much maintenance of the 
actual code of the drivers, in the sense of dealing with bug reports, 
questions, etc. (pretty sure they are none for the ones I listed). Most of 
them must work reasonably well and be feature complete, and most 
vulnerabilities have now been fixed. My concern is that this legacy code has 
indirect costs on other GDAL developers and users. The psychological cost I 
mentionned. Let's say someone want to turn on higher warning levels, and that 
this breaks in tens of drivers. Would he have to ping every maintainer and 
wait for them to address the issue ? Or maybe he will just give up. Similarly 
for breaking changes in the driver API. As Sean mentionned, this is probably a 
serious obstacle to growing up the core development team.


Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Frank Warmerdam]

On Tue, Jan 12, 2021 at 12:38 PM Howard Butler  wrote:

> The only question that matters here is "Who is going to maintain it?" and
> if the answer to that is "no one", it should be removed. There doesn't need
> to be any meetings because the only criteria that matters is if someone is
> willing to maintain it. We should provide the list of drivers and assign
> the GitHub handles that step forward to be responsible for each. If obscure
> government one-offs formats have an audience of downtrodden government
> users forced to use them, they need to put their handle in and take
> ownership. They then need to find the time, money, or attention to carry
> things forward.
>

Howard / Even,

I'd be willing to commit to maintaining some of the archaic drivers that
meet the conditions I mentioned (buildable, testable in core build).  If
Even would like I can provide a sublist of those he proposed i'd be willing
to be responsible for.

Best regards,
-- 
---+--
I set the clouds in motion - turn up   | Frank Warmerdam,
warmer...@pobox.com
light and sound - activate the windows | +1 650-701-7823
and watch the world go round - Rush| Geospatial Software Developer
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Howard Butler]

> On Jan 12, 2021, at 8:48 AM, Jonathan Gale  wrote:
> 
> Hi,
>  
> Looking at the list, we use/support SDTS Raster import. As a US government 
> format with the use case mentioned below, I’d prefer to not see the format 
> removed from the software.  We have some evidence that the format is still 
> used by some of our users.

It would be helpful for Mathworks to provide the GDAL project a list of "used" 
formats in its software. GDAL doesn't get that recon, and it can't put phone 
home capability into the software to get it either.

>  The users of the software I help develop and maintain expect a high level of 
> compatibility across releases.  A commonly discussed use case involves 
> rerunning the same code with the same data across many years to ensure 
> reproducibility of results.  While we see the need to keep code maintainable, 
> it feels important to consider this use case. 

So the open source project your commercial software product depends upon is 
supposed to worry about forever version compatibility to protect your product? 
You may not have said it this way, but this is what I hear when I read this 
statement. It is especially frustrating to hear given the many commercial 
entities like Mathworks that have leveraged GDAL for years without providing 
non-targeted financial and developer resources back to the project to carry 
this kind of maintenance burden. 

The community doesn't maintain GDAL – Even Rouault does (right now, just like 
Frank Warmerdam did before him). The community doesn't really share the burden, 
but it does share its rewards (1000 line NEWS posts of updates to the project 
every release). Even is now searching for ways to make it more maintainable 
given that the resources to sustain the current trajectory have not 
materialized. This includes culling drivers that merely add burden and evolving 
old stuff where appropriate (code, configuration, APIs).  It isn't on us to say 
whether something is or isn't a burden either – we need to step into the breach 
if we need to keep something around. 

A problem with GDAL is the breach is a 1 million line behemoth with homegrown 
test suites, build setups, software layout, and type system. It is not a 
comfortable project to make casual contributions because it requires a lot of 
investment to get up to speed. You end up having to carry the rock of 20 years 
of software (mis)behavior as you do it. For anything other than adding another 
driver, this burden ends up being too great. So while we want more maintainers 
(working free to us!), the software itself makes it very difficult for them to 
grow into the role. Attempts to relax the dough to allow that to happen are met 
with, well, this mailing list thread.  
 
> Not specific to the software I work on, I think of GDAL as a “swiss army 
> knife” of geospatial format support.  It is the FOSS way to access many 
> formats both new and old.  I do wonder whether alternatives available should 
> be considered when evaluating a given driver for removal.  It would be 
> unfortunate if there was no way to read a format anymore with available 
> software.

The old versions are still available, but I understand as a commercial product, 
you want BOTH the maintenance and bug fixing of existing software plus all of 
the goodie features of a new release. 

> Overall, we like the idea of introducing a process for this.  Thinking about 
> next steps, it is important to consider what the criterion should be as Even 
> mentioned.  How many users?  Who are these users?  How do we estimate the 
> downstream impact on users who don’t know that their format is being 
> considered for removal?  As others have mentioned, how does maintainability 
> factor in?  I hope that if there was a process to do this, that it would be 
> codified and visible.

The only question that matters here is "Who is going to maintain it?" and if 
the answer to that is "no one", it should be removed. There doesn't need to be 
any meetings because the only criteria that matters is if someone is willing to 
maintain it. We should provide the list of drivers and assign the GitHub 
handles that step forward to be responsible for each. If obscure government 
one-offs formats have an audience of downtrodden government users forced to use 
them, they need to put their handle in and take ownership. They then need to 
find the time, money, or attention to carry things forward. 

Howard

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Jonathan Gale]

Hi,

Looking at the list, we use/support SDTS Raster import. As a US government 
format with the use case mentioned below, I'd prefer to not see the format 
removed from the software.  We have some evidence that the format is still used 
by some of our users.

The users of the software I help develop and maintain expect a high level of 
compatibility across releases.  A commonly discussed use case involves 
rerunning the same code with the same data across many years to ensure 
reproducibility of results.  While we see the need to keep code maintainable, 
it feels important to consider this use case.

Not specific to the software I work on, I think of GDAL as a "swiss army knife" 
of geospatial format support.  It is the FOSS way to access many formats both 
new and old.  I do wonder whether alternatives available should be considered 
when evaluating a given driver for removal.  It would be unfortunate if there 
was no way to read a format anymore with available software.

Overall, we like the idea of introducing a process for this.  Thinking about 
next steps, it is important to consider what the criterion should be as Even 
mentioned.  How many users?  Who are these users?  How do we estimate the 
downstream impact on users who don't know that their format is being considered 
for removal?  As others have mentioned, how does maintainability factor in?  I 
hope that if there was a process to do this, that it would be codified and 
visible.

Best,
Jonathan


From: gdal-dev  On Behalf Of Even Rouault
Sent: Sunday, January 10, 2021 6:02 PM
To: gdal-dev@lists.osgeo.org
Subject: [gdal-dev] Considering drivers removal ?

Hi,

It's not spring yet, but I'm in a mood lately of axing useless things, and we
probably have tons of candidate for that in GDAL, especially in drivers.
I was going to just axe the DB2 driver
(https://github.com/OSGeo/gdal/pull/3366)
 but the issue is more general.

Any idea how we can know what is used and what isn't ? A "call-home"
functionality where we would track driver usage would only be acceptable if
people enable it and have network connectivity, so we won't probably get lots
of feedback. Having a spreadsheet with the driver list and asking people to
fill it would probably also receive little feedback. So the idea I had was to
do something like the following in the Open() method of a candidate for
removal:

GDALDataset* FooDriver::Open(  )
{
if( !Identify(poOpenInfo) )
return nullptr;

if( !CPLTestBool(CPLGetConfigOption("GDAL_ENABLE_DRIVER_FOO", "NO") )
{
CPLError(CE_Failure, CPLE_AppDefined,
"Driver FOO is considered for removal in GDAL 3.5. You are invited "
"to convert any dataset in that format to another more common one ."
"If you need this driver in future GDAL versions, create a ticket at "
"https://github.com/OSGeo/gdal (look first for 
an existing one first) to "
"explain how critical it is for you (but the GDAL project may still "
"remove it), and to enable it now, set the GDAL_ENABLE_DRIVER_FOO "
"configuration option / environment variable to YES");
return nullptr;
}
...
}

That is, when we detect a file to be handled by the driver, emit the above
error message and do not open the dataset, unless the user defines the
environment variable.
Similarly in the Create()/CreateCopy() methods.
If we ship this in 3.3, with a 3.5 milestone for removal, this would offer a
feedback period of one year / 2 feature versions.

Here's my own list of candidates for retirement (probably over-conservative).
Mostly based on gut feeling. None of them are particularly bad citizens, but I
have no indication that they are still used, which doesn't mean they aren't.

* Raster side:
BPG
DB2Raster
DOQ1
DOQ2
E00GRID
Epsilon
FujiBAS
GS7BG
GSAG
IDA
JDEM
JPEG2000 (Jasper): JP2OpenJPEG is a better replacement
JPEGLS
LAN
MFF
MG4Lidar ?
NDF
NTv1
SDTS Raster
SGI
XPM
ZMap

* Vector side:
AERONAVFAA
ESRI ArcObjects
ARCGEN
BNA
Cloudant
CouchDB
DB2
DODS
FMEObjects Gateway
Geomedia MDB
GMT ASCII Vectors
GTM
HTF
INGRES
MongoDB (the old one, superseded by MongoDBv3)
OpenAIR
REC
SDTS
SUA
SVG
TIGER
WALK


Anything you'd add / remove ?

What is not obvious is what would be the criterion for keeping a driver: 1,
10, 100 users asking for the driver to be kept ?
If a GDAL developer contributing to the overall good of the project needs the
preservation of a driver to be able to justify its continued involvement, I'd
tend to think it to be enough to keep it.


Even

--
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Considering drivers removal ?

[Jan Heckman]

In good company when pruning the source trees:
https://www.phoronix.com/scan.php?page=news_item=2021-Linux-Drop-Old-CPUs
Apparently 68000 may not be supported any longer in linux...
Just joking. Hope you don't mind...

On Tue, Jan 12, 2021 at 3:46 AM Jed O. Kaplan 
wrote:

> I for one am using the GMT vector driver for GDAL on a regular basis,
> e.g., for integration between GRASS GIS and GMT for cartography. I am
> grateful to the developers for their continued support for this driver.
>
> Thanks, Jed
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Catching many errors of the same record (ogr2ogr)

[Even Rouault]

On vendredi 8 janvier 2021 15:58:39 CET matteo wrote:
> Hi all,
> 
> I've 2 small questions during a syncing from a GPKG to a PG DB with
> constraints using -skipfailures:
> 
> * is it possible to show only the constraint error and not also the
> insert one? Even if there is only a single error, reading the log it
> seems that there are 2 of them (I know a person should only read)
> * is there are 2 errors (e.g. constraints error) BUT in the same record,
> ogr2ogr shows only the first one and not all of them

Why not experimenting outside GDAL a bit ?

$ psql

# create table foo(bar integer check (bar = 1), baz integer check (baz = 1));
# insert into foo values (2,2);
ERROR:  new row for relation "foo" violates check constraint "foo_bar_check"
DETAIL:  Failing row contains (2, 2).


and asking your favorite search engine about
"postgresql get all constraint violations"

==>
https://stackoverflow.com/questions/23822162/error-message-with-all-constraint-violations-in-postgres


Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Registering C++ Pixel Functions for Command-Line Utilities

[Conor McIndoe]

Wonderful, thank you Thomas, that has indeed resolve the issue

Best,

Conor

On 12/01/2021 12:02, thomas bonfort wrote:

Conor,
I suspect that you should add an
extern "C" {
void GDALRegisterMe();
}
line at the beginning of your cpp file.

--
Thomas

On Tue, Jan 12, 2021 at 12:56 PM Conor McIndoe 
mailto:conormcind...@gmail.com>> wrote:


Hello developers,

I am just learning the ropes with GDAL so apologies if some of
this is
quite far off. Also apologies in advance for the essay :)

I am trying to register a pixel function, written in C++, in order to
produce a VRT and eventually a TIF from some input TIFs. I am
trying to
follow an thread from this mailing list
(https://lists.osgeo.org/pipermail/gdal-dev/2011-May/028742.html
) to
make GDAL aware of a shared library which contains and registers the
pixel function. Hopefully there is some clear error here which one of
you can spot?

I have adapted the example found at the VRT reference page
(https://gdal.org/drivers/raster/vrt.html
) to produce a function
which
sets every pixel equal to 1 (just to test the workflow):

// src/gdalPixelFunctions.cpp

#include "gdal.h"
#include "cpl_conv.h"
#include "gdal_priv.h"

#include 

CPLErr OneFunction(
 void **papoSources, int nSources, void *pData,
 int nXSize, int nYSize, GDALDataType eSrcType,
 GDALDataType eBufType, int nPixelSpace, int nLineSpace
)
{
 int ii, iLine, iCol;
 double pix_val;
 double x0, x3, x4, x8;

 if(nSources != 4) return CE_Failure;

 for(iLine = 0; iLine < nYSize; iLine++)
 {
 for(iCol = 0; iCol < nXSize; iCol++)
 {
 ii = iLine * nXSize + iCol;

 /* Source raster pixels may be obtained with SRCVAL
macro */
 x0 = SRCVAL(papoSources[0], eSrcType, ii);
 x3 = SRCVAL(papoSources[1], eSrcType, ii);
 x4 = SRCVAL(papoSources[2], eSrcType, ii);
 x8 = SRCVAL(papoSources[3], eSrcType, ii);

 pix_val = 1;

 GDALCopyWords(
 _val,
 GDT_Float64,
 0,
 ((GByte *)pData) + nLineSpace * iLine + iCol *
nPixelSpace,
 eBufType,
 nPixelSpace,
 1
 );
 }
 }

 return CE_None;
}

void GDALRegisterMe()
{
 std::cout << "Inside GDALRegisterME()" << std::endl;
 GDALAddDerivedBandPixelFunc("OneFunction", OneFunction);
}


Compilation:

g++ -std=c++20 -fPIC -lgdal -c src/gdalPixelFunctions.cpp -o
build/gdal_pixelFunctions.o

g++ -lgdal -shared build/gdal_pixelFunctions.o -o
build/gdal_pixelFunctions.so

export GDAL_DRIVER_PATH="/path/to/build"


Attempting to call:

gdal_translate testVrt.vrt testVrt.tif

(where testVrt.vrt references the OneFunction pixel function)
results in
the following error messages:

ERROR 1: /path/to/build/gdal_pixelFunctions.so: undefined symbol:
GDALRegisterMe
ERROR 1: /path/to/build/gdal_pixelFunctions.so: undefined symbol:
GDALRegister_pixelFunctions

I have also tried registering by defining a void
GDALRegister_pixelFunctions() but with the same issue. I assume this
error means that GDAL is looking in the shared library for these two
register functions and failing to find them?

Thank you for your help and sorry for the long post!

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org 
https://lists.osgeo.org/mailman/listinfo/gdal-dev


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] NetCDF on S3?

[dun...@gmail.com]

Hello,
I'm trying to open a NetCDF file from S3 with Gdal and I'm hitting some
problems and I'm not sure whether they are limitations of the driver or I'm
just not using it right.

I have two NetCDF files, one is (small) sample file from the Copernicus
File Danger Index, the other is one month of data (1.5GB) from the ECMWF
Reanalysis 2m Temperature variable. The first file opens file from my local
file system and using the /vsis3 virtual filesystem but the second one
doesn't.

With CPL_DEBUG=ON, successfully opening the file with gdalinfo looks like
this:

GNM: GNMRegisterAllInternal
GNM: RegisterGNMFile
GNM: RegisterGNMdatabase
GDAL_netCDF: driver detected file type=2, libnetcdf detected type=2
GDAL_netCDF: dim_count = 3
GDAL_netCDF: var_count = 4
GDAL_netCDF:
=
SetProjectionFromVar( 3)
GDAL_netCDF: bIsGdalFile=0 bIsGdalCfFile=0 bBottomUp=1
GDAL_netCDF: set bBottomUp = 0 from Y axis
GDAL_netCDF: xdim: 1440 nSpacingBegin: 250 nSpacingMiddle: 250 nSpacingLast: 250
GDAL_netCDF: ydim: 721 nSpacingBegin: -250 nSpacingMiddle: -250
nSpacingLast: -250
GDAL_netCDF: 
SetGeoTransform(-0.125000,0.25,0.00,90.125000,0.00,-0.25)
GDAL_netCDF: bGotGeogCS=0 bGotCfSRS=0 bGotCfGT=1 bGotGdalSRS=0 bGotGdalGT=0
GDAL_netCDF: netcdf type=3 gdal type=3 signedByte=1
...

The same file when uploaded to S3 outputs the following using the
vsis3 virtual filesystem:

GNM: GNMRegisterAllInternal
GNM: RegisterGNMFile
GNM: RegisterGNMdatabase
HTTP: libcurl/7.69.1 OpenSSL/1.1.1g zlib/1.2.5 libssh2/1.8.2
HTTP: GDAL was built against curl 7.64.1, but is running against 7.69.1.
S3: Switching to region eu-west-1
VSICURL: Downloading 0-16383
(https://some-s3-bucket.s3.amazonaws.com/era5-reanalysis/2m_temperature/2020-01.nc)...
VSICURL: Got response_code=206
ERROR 4: `/vsis3/some-s3-bucket/era5-reanalysis/2m_temperature/2020-01.nc'
not recognized as a supported file format.
gdalinfo failed - unable to open
'/vsis3/some-s3-bucket/era5-reanalysis/2m_temperature/2020-01.nc'.


Is it clear from this output whether there is an issue with the 2020-01.nc file?

Thanks,

Duncan Walker
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Registering C++ Pixel Functions for Command-Line Utilities

[thomas bonfort]

Conor,
I suspect that you should add an
extern "C" {
void GDALRegisterMe();
}
line at the beginning of your cpp file.

--
Thomas

On Tue, Jan 12, 2021 at 12:56 PM Conor McIndoe 
wrote:

> Hello developers,
>
> I am just learning the ropes with GDAL so apologies if some of this is
> quite far off. Also apologies in advance for the essay :)
>
> I am trying to register a pixel function, written in C++, in order to
> produce a VRT and eventually a TIF from some input TIFs. I am trying to
> follow an thread from this mailing list
> (https://lists.osgeo.org/pipermail/gdal-dev/2011-May/028742.html) to
> make GDAL aware of a shared library which contains and registers the
> pixel function. Hopefully there is some clear error here which one of
> you can spot?
>
> I have adapted the example found at the VRT reference page
> (https://gdal.org/drivers/raster/vrt.html) to produce a function which
> sets every pixel equal to 1 (just to test the workflow):
>
> // src/gdalPixelFunctions.cpp
>
> #include "gdal.h"
> #include "cpl_conv.h"
> #include "gdal_priv.h"
>
> #include 
>
> CPLErr OneFunction(
>  void **papoSources, int nSources, void *pData,
>  int nXSize, int nYSize, GDALDataType eSrcType,
>  GDALDataType eBufType, int nPixelSpace, int nLineSpace
> )
> {
>  int ii, iLine, iCol;
>  double pix_val;
>  double x0, x3, x4, x8;
>
>  if(nSources != 4) return CE_Failure;
>
>  for(iLine = 0; iLine < nYSize; iLine++)
>  {
>  for(iCol = 0; iCol < nXSize; iCol++)
>  {
>  ii = iLine * nXSize + iCol;
>
>  /* Source raster pixels may be obtained with SRCVAL macro */
>  x0 = SRCVAL(papoSources[0], eSrcType, ii);
>  x3 = SRCVAL(papoSources[1], eSrcType, ii);
>  x4 = SRCVAL(papoSources[2], eSrcType, ii);
>  x8 = SRCVAL(papoSources[3], eSrcType, ii);
>
>  pix_val = 1;
>
>  GDALCopyWords(
>  _val,
>  GDT_Float64,
>  0,
>  ((GByte *)pData) + nLineSpace * iLine + iCol *
> nPixelSpace,
>  eBufType,
>  nPixelSpace,
>  1
>  );
>  }
>  }
>
>  return CE_None;
> }
>
> void GDALRegisterMe()
> {
>  std::cout << "Inside GDALRegisterME()" << std::endl;
>  GDALAddDerivedBandPixelFunc("OneFunction", OneFunction);
> }
>
>
> Compilation:
>
> g++ -std=c++20 -fPIC -lgdal -c src/gdalPixelFunctions.cpp -o
> build/gdal_pixelFunctions.o
>
> g++ -lgdal -shared build/gdal_pixelFunctions.o -o
> build/gdal_pixelFunctions.so
>
> export GDAL_DRIVER_PATH="/path/to/build"
>
>
> Attempting to call:
>
> gdal_translate testVrt.vrt testVrt.tif
>
> (where testVrt.vrt references the OneFunction pixel function) results in
> the following error messages:
>
> ERROR 1: /path/to/build/gdal_pixelFunctions.so: undefined symbol:
> GDALRegisterMe
> ERROR 1: /path/to/build/gdal_pixelFunctions.so: undefined symbol:
> GDALRegister_pixelFunctions
>
> I have also tried registering by defining a void
> GDALRegister_pixelFunctions() but with the same issue. I assume this
> error means that GDAL is looking in the shared library for these two
> register functions and failing to find them?
>
> Thank you for your help and sorry for the long post!
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Registering C++ Pixel Functions for Command-Line Utilities

[Conor McIndoe]

Hello developers,

I am just learning the ropes with GDAL so apologies if some of this is 
quite far off. Also apologies in advance for the essay :)


I am trying to register a pixel function, written in C++, in order to 
produce a VRT and eventually a TIF from some input TIFs. I am trying to 
follow an thread from this mailing list 
(https://lists.osgeo.org/pipermail/gdal-dev/2011-May/028742.html) to 
make GDAL aware of a shared library which contains and registers the 
pixel function. Hopefully there is some clear error here which one of 
you can spot?


I have adapted the example found at the VRT reference page 
(https://gdal.org/drivers/raster/vrt.html) to produce a function which 
sets every pixel equal to 1 (just to test the workflow):


// src/gdalPixelFunctions.cpp

#include "gdal.h"
#include "cpl_conv.h"
#include "gdal_priv.h"

#include 

CPLErr OneFunction(
    void **papoSources, int nSources, void *pData,
    int nXSize, int nYSize, GDALDataType eSrcType,
    GDALDataType eBufType, int nPixelSpace, int nLineSpace
)
{
    int ii, iLine, iCol;
    double pix_val;
    double x0, x3, x4, x8;

    if(nSources != 4) return CE_Failure;

    for(iLine = 0; iLine < nYSize; iLine++)
    {
    for(iCol = 0; iCol < nXSize; iCol++)
    {
    ii = iLine * nXSize + iCol;

    /* Source raster pixels may be obtained with SRCVAL macro */
    x0 = SRCVAL(papoSources[0], eSrcType, ii);
    x3 = SRCVAL(papoSources[1], eSrcType, ii);
    x4 = SRCVAL(papoSources[2], eSrcType, ii);
    x8 = SRCVAL(papoSources[3], eSrcType, ii);

    pix_val = 1;

    GDALCopyWords(
    _val,
    GDT_Float64,
    0,
    ((GByte *)pData) + nLineSpace * iLine + iCol * nPixelSpace,
    eBufType,
    nPixelSpace,
    1
    );
    }
    }

    return CE_None;
}

void GDALRegisterMe()
{
    std::cout << "Inside GDALRegisterME()" << std::endl;
    GDALAddDerivedBandPixelFunc("OneFunction", OneFunction);
}


Compilation:

g++ -std=c++20 -fPIC -lgdal -c src/gdalPixelFunctions.cpp -o 
build/gdal_pixelFunctions.o


g++ -lgdal -shared build/gdal_pixelFunctions.o -o 
build/gdal_pixelFunctions.so


export GDAL_DRIVER_PATH="/path/to/build"


Attempting to call:

gdal_translate testVrt.vrt testVrt.tif

(where testVrt.vrt references the OneFunction pixel function) results in 
the following error messages:


ERROR 1: /path/to/build/gdal_pixelFunctions.so: undefined symbol: 
GDALRegisterMe
ERROR 1: /path/to/build/gdal_pixelFunctions.so: undefined symbol: 
GDALRegister_pixelFunctions


I have also tried registering by defining a void 
GDALRegister_pixelFunctions() but with the same issue. I assume this 
error means that GDAL is looking in the shared library for these two 
register functions and failing to find them?


Thank you for your help and sorry for the long post!

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] TargetAlignedPixels for Python-GdalWarp when reprojecting

[Felix Keßler]

Hey Guys,
I am trying to reproject different satellite images to the same UTM-Zone
using gdalwarp in Python
It works so far, but I can not align the pixels. Is there a possibility?

I also opened a question on stackexchange, there are some images and my
current code:

https://gis.stackexchange.com/questions/384012/how-can-i-align-the-pixels-with-using-gdal-and-python



Thanks,
Felix

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] netCDF chunk fetch failed

[Julien Demaria]

Kurt,

I tried the latest version of NetCDF’s nccopy utility on your files and it also 
failed.  Same with an old NetCDF v4.2/HDF5 v1.8.9 version.
So the files seem wrong/corrupted for an unknow reason, or created with a 
specific NetCDF/HDF5 version which include a compatibility break.
Did you get answer from NOAA?
I’m not sure if GDAL should handle this case (skip failed chunks and set them 
to nodata).

Best,
Julien


De : gdal-dev  De la part de Kurt Schwehr
Envoyé : mardi 12 janvier 2021 00:52
À : gdal dev 
Objet : [gdal-dev] netCDF chunk fetch failed

Hi all,

In a build of gdal 3.2.0, I'm seeing this error: "netCDF chunk fetch failed" 
for 2 out of 5 subdatasets.  Is there an easy way to handle this so the bad 
data is returned as the nodata value?  And also, any idea what is actually 
wrong with the netcdf files?

I am in contact with the NOAA folks who made this data.

Thanks!
-kurt



gsutil cp 
gs://gcp-public-data-goes-16/ABI-L2-FDCF/2017/263/21/OR_ABI-L2-FDCF-M3_G16_s20172632115407_e20172632126173_c20172632126283.nc
 .
gsutil cp 
gs://gcp-public-data-goes-16/ABI-L2-FDCF/2018/123/12/OR_ABI-L2-FDCF-M3_G16_s20181231215382_e20181231226149_c20181231226258.nc
 .
gsutil cp 
gs://gcp-public-data-goes-16/ABI-L2-FDCF/2018/324/18/OR_ABI-L2-FDCF-M3_G16_s20183241845341_e20183241856108_c20183241856213.nc
 .
gsutil cp 
gs://gcp-public-data-goes-16/ABI-L2-FDCF/2020/247/16/OR_ABI-L2-FDCF-M6_G16_s20202471650186_e20202471659494_c20202471700318.nc
 .


for subdataset in Area Temp Mask Power DQF; do
echo $subdataset
(gdalinfo -stats 
NETCDF:OR_ABI-L2-FDCF-M3_G16_s20183241845341_e20183241856108_c20183241856213.nc:$subdataset
 | grep 'Band 1');
done

Area
Warning 1: netCDFDataset::valid_range: min > max:
  min: 0.00
  max: -6.00

ERROR 1: netCDF chunk fetch failed: #-101 (NetCDF: HDF error)
ERROR 1: 
NETCDF:OR_ABI-L2-FDCF-M3_G16_s20183241845341_e20183241856108_c20183241856213.nc:Area,
 band 1: IReadBlock failed at X offset 19, Y offset 10: netCDF chunk fetch 
failed: #-101 (NetCDF: HDF error)
Band 1 Block=226x226 Type=Int16, ColorInterp=Undefined
Temp
Warning 1: netCDFDataset::valid_range: min > max:
  min: 0.00
  max: -6.00
Band 1 Block=226x226 Type=Int16, ColorInterp=Undefined
Mask
ERROR 1: netCDF chunk fetch failed: #-101 (NetCDF: HDF error)
ERROR 1: 
NETCDF:OR_ABI-L2-FDCF-M3_G16_s20183241845341_e20183241856108_c20183241856213.nc:Mask,
 band 1: IReadBlock failed at X offset 4, Y offset 11: netCDF chunk fetch 
failed: #-101 (NetCDF: HDF error)
Band 1 Block=226x226 Type=Int16, ColorInterp=Undefined
Power
Band 1 Block=226x226 Type=Float32, ColorInterp=Undefined
DQF
Band 1 Block=226x226 Type=Byte, ColorInterp=Undefined
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev