Re: [gdal-dev] Gdal doesn't see NaN values as NoData?

[Matt.Wilkie]

>> a) Why does gdalinfo not report the presence of nan? Meaning: how do I know 
>> when I will need to invoke (b)?
>
>It doesn't report nan as the nodata value, because the file lacks the 
>GDAL_NODATA tiff tag set at "nan". So the issue is on the producer side.

Ok, noted. I will try trace this backward and report there.


>> b) What do I need to do so that applications will use nan as nodata?
>
>   gdalwarp -dstnodata nan ...
>
>or if you want to remap nan to some other value
>
>   gdalwarp -dstnodata - ...

Wonderful, thank you.

>In most cases, you'd need to explicitly add the srcnodata value with 
>-srcnodata nan , but nan is a special case. When gdalwarp finds a pixel 
>at NaN value, be it the declared nodata value or not, it assumes this is 
>nodata, hence you can omit -srcnodata nan. Wouldn't be true if 0 or some 
>other value was the undeclared nodata  value.

Hmmm. Part of me expects gdalwarp to then record it found and used undeclared 
nodata by setting the "nodata=nan" in the metadata. Regardless thanks for this 
important detail.

-Matt
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Gdal doesn't see NaN values as NoData?

[Even Rouault]

Matt,


a) Why does gdalinfo not report the presence of nan? Meaning: how do I know 
when I will need to invoke (b)?
It doesn't report nan as the nodata value, because the file lacks the 
GDAL_NODATA tiff tag set at "nan". So the issue is on the producer side.

b) What do I need to do so that applications will use nan as nodata?


gdalwarp -dstnodata nan ...

or if you want to remap nan to some other value

gdalwarp -dstnodata - ...

In most cases, you'd need to explicitly add the srcnodata value with 
-srcnodata nan , but nan is a special case. When gdalwarp finds a pixel 
at NaN value, be it the declared nodata value or not, it assumes this is 
nodata, hence you can omit -srcnodata nan. Wouldn't be true if 0 or some 
other value was the undeclared nodata  value.



I've some doubt that it is produced by GEE. Seems move to come from Esri 
software seeing:


$ tiffinfo NDVI1984.tif
[...]

  Tag 42112: 
  NEAREST
  NDVI


Even

--

http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Gdal doesn't see NaN values as NoData?

[Matt.Wilkie]

Hi Folks,

I have a 32bit NDVI image that I'm told was exported from Google Earth Engine, 
with possibly a trip through R along the way. In ArcGIS desktop and Qgis there 
are "nan" values reported for the image collar, however gdalinfo doesn't say 
anything about them. If I project the image in gdalwarp the output image 
contains both nan AND nodata. This is rather confusing.

a) Why does gdalinfo not report the presence of nan? Meaning: how do I know 
when I will need to invoke (b)?
b) What do I need to do so that applications will use nan as nodata? 

Fig1: Projected output. Pink == nodata, red == 0, black == nan.


Here's the file  https://sft.gov.yk.ca/link/61czGFXOyyFsh387CxbTEF
(255 MB, link valid for 3 months. The image is free and libre.)

Gdalinfo report:

Size is 14742, 6761
Coordinate System is:
GEOGCRS["WGS 84",
    DATUM["World Geodetic System 1984",
    ELLIPSOID["WGS 84",6378137,298.257223563,
    LENGTHUNIT["metre",1]]],
    PRIMEM["Greenwich",0,
    ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
    AXIS["geodetic latitude (Lat)",north,
    ORDER[1],
    ANGLEUNIT["degree",0.0174532925199433]],
    AXIS["geodetic longitude (Lon)",east,
    ORDER[2],
    ANGLEUNIT["degree",0.0174532925199433]],
    ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (-137.935952550438884,61.086337635411589)
Pixel Size = (0.000269494585236,-0.000269494585236)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (-137.9359526,  61.0863376) (137d56' 9.43"W, 61d 5'10.82"N)
Lower Left  (-137.9359526,  59.2642847) (137d56' 9.43"W, 59d15'51.43"N)
Upper Right (-133.9630634,  61.0863376) (133d57'47.03"W, 61d 5'10.82"N)
Lower Right (-133.9630634,  59.2642847) (133d57'47.03"W, 59d15'51.43"N)
Center  (-135.9495080,  60.1753112) (135d56'58.23"W, 60d10'31.12"N)
Band 1 Block=256x256 Type=Float32, ColorInterp=Gray
  Description = NDVI
  Min=-0.888 Max=0.992
  Minimum=-0.888, Maximum=0.992, Mean=0.103, StdDev=0.141
  Overviews: 7371x3381, 3686x1691, 1843x846, 922x423, 461x212
  Metadata:
    STATISTICS_COVARIANCES=0.01992952634076865
    STATISTICS_MAXIMUM=0.9921875
    STATISTICS_MEAN=0.10253506184277
    STATISTICS_MINIMUM=-0.88755017518997
    STATISTICS_SKIPFACTORX=1
    STATISTICS_SKIPFACTORY=1
    STATISTICS_STDDEV=0.14117197434607


Gdalwarp command:

set _opt=-co compress=zstd -co level=17 -co num_threads=all_cpus -co 
predictor=yes -co bigtiff=yes

gdalwarp -wm 512 %_opt% -of cog -t_srs epsg:3579 -r med NDVI1984.tif 
t\NDVI1984.tif


GDAL 3.1.4, released 2020/10/20

Thanks!

Matt Wilkie
Geomatics Analyst
Environment | Technology, Innovation and Mapping
T 867-667-8133 | http://yukon.ca/
Hours: 08:30-16:30, Tue-Wed: Office, Thu-Fri: Remote.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] vsicurl and an issue with http 302 redirect in the middle of the file

[Even Rouault]

Jukka,

Running with CPL_CURL_VERBOSE=YES shows the following

> GET 
/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t15z.wrfsfcf01.grib2 
HTTP/1.1

Host: ftpprd.ncep.noaa.gov
Accept: */*
Range: bytes=84819968-84836351

* Mark bundle as not supporting multiuse
< HTTP/1.1  302 Your allowed limit has been reached. Please go to 
https://www.weather.gov/abusive-user-block  for more info

* no chunk, no close, no size. Assume close to signal end
<

So it seems that after a number of quickly consecutive GET requests to 
the server as here, the server no longer want to see us. The 302 HTTP 
error it sends however is not appropriate. 429 "Too many requests" would 
be more appropriate


GRIB files with lots of "messages" / bands like that one aren't really 
cloud friendly, as you have to skip in many parts of the file to get an 
inventory of it.


That said, I see that in that instance, the .grib files are accompanied 
by a sidecar file, like 
https://ftpprd.ncep.noaa.gov/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t15z.wrfsfcf01.grib2.idx, 
which I believe is produced by the wgrib utility. We could conceptually 
try to make use of it, however it doesn't contain all metadata that are 
typically retrieved by the driver, so for gdalinfo itself that wouldn't 
be sufficient, but for just a Open() call that might be enough.


I've noted that https://github.com/OSGeo/gdal/issues/3799

Even


Le 05/05/2021 à 22:04, Rahkonen Jukka (MML) a écrit :


Hi,

Have a look at 
https://gis.stackexchange.com/questions/395867/opening-a-grib-from-the-web-with-gdal-in-python-using-vsicurl-throws-error-on-m 
<https://gis.stackexchange.com/questions/395867/opening-a-grib-from-the-web-with-gdal-in-python-using-vsicurl-throws-error-on-m>. 
There gdalinfo fails with /vsicurl/ somewhere in the middle of the 
grib2 file when the server sends http 302 as a response for a range 
request. The file is OK as was tested by downloading it first totally 
with curl. Could it be that the http “moved permanently” at that stage 
makes /vsicurl/ to fail?


I made also a few tests and it seems that the error may happen in 
another http range, but it is always similar:

VSICURL: Got response_code=302

GRIB: ERROR: Ran out of file in Section 7

-Jukka Rahkonen-


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] vsicurl and an issue with http 302 redirect in the middle of the file

[Rahkonen Jukka (MML)]

Hi,

Have a look at 
https://gis.stackexchange.com/questions/395867/opening-a-grib-from-the-web-with-gdal-in-python-using-vsicurl-throws-error-on-m.
 There gdalinfo fails with /vsicurl/ somewhere in the middle of the grib2 file 
when the server sends http 302 as a response for a range request. The file is 
OK as was tested by downloading it first totally with curl. Could it be that 
the http "moved permanently" at that stage makes /vsicurl/ to fail?

I made also a few tests and it seems that the error may happen in another http 
range, but it is always similar:
VSICURL: Got response_code=302
GRIB: ERROR: Ran out of file in Section 7

-Jukka Rahkonen-
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] About packages auto-detection

[Javier Jimenez Shaw]

Hi
>From time to time I have problems with the options that are automatically
selected by configure script on Unix systems (Linux and macOS). The current
behaviour is that if nothing is explicitly specified in the command line
(like --without-geos or --with-zstd=X), the configure script checks if that
particular package is installed in the build machine and links with it if
it is present.
The problem with that approach comes if you later run on a different
machine that might not have that package installed: Then the execution will
fail due to missing libraries. This whole procedure makes the compilation
dependent on the build machine configuration and it is more difficult to
have reproducible builds which are defined solely by code, since you must
remember to add any new option that may appear explicitly in your build
setup. Is there any possibility to disable the auto-detection globally,
i.e. have an option that disables all features that are not explicitly
enabled regardless of the build machine's configuration?

Thanks
.___ ._ ..._ .. . ._.  .___ .. __ . _. . __..  ...  ._ .__
Entre dos pensamientos racionales
hay infinitos pensamientos irracionales.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] gdal_priv.h and CPL_DLL

[Even Rouault]

Le 05/05/2021 à 15:23, Andrew Bell a écrit :

Hi,

I don't understand function declarations in gdal_priv.h being marked 
CPL_DLL. I would think that if the functions are exposed, they 
wouldn't be exposed from the GDAL shared library. Is this a mistake? 
Is this for testing?
Mostly because they are needed by drivers built as plugins. Anyway, the 
_priv.h is somewhat misleading, as they C++ classes are also exported to 
be usable by C++ users


Thanks,

--
Andrew Bell
andrew.bell...@gmail.com 

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] gdal_priv.h and CPL_DLL

[Andrew Bell]

Hi,

I don't understand function declarations in gdal_priv.h being marked
CPL_DLL. I would think that if the functions are exposed, they wouldn't be
exposed from the GDAL shared library. Is this a mistake? Is this for
testing?

Thanks,

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL /vsistdin/ 1MB limit

[Robert Coup]

On Wed, 5 May 2021 at 03:02, Romeo Alvaraz  wrote:
> I'm just wondering  if there is another handler/method you could suggest
> that could feed the input directly to gdal_translate in order to bypass use
> of file system?

If your data easily fits in memory, write it to a tmpfs filesystem
(eg: to /dev/shm/my.pdf)? Doesn't touch the disk so you get the same
performance without the downside of it needing to work as a stream.

Rob :)
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GDAL /vsistdin/ 1MB limit

[Andrew C Aitchison]

On Tue, 4 May 2021, Romeo Alvaraz wrote:


Thanks Even for the response and clarifications on /vsistdin/.  Our reasons
for attempting to use /vsistdin/ is to avoid having to read a file from the
local file system (which we would have to write to first by other means) on
our remote server. We've been using this in conjunction with /vsistdout/.

/vsicurl/  was another option we considered for the input but it's not ideal
to our situation.

I'm just wondering  if there is another handler/method you could suggest
that could feed the input directly to gdal_translate in order to bypass use
of file system?


A python script could store the incoming data in /vsimem/ and call 
gdal_translate internally


--
Andrew C. Aitchison Kendal, UK
and...@aitchison.me.uk
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev