Re: [gdal-dev] vsicurl and an issue with http 302 redirect in the middle of the file

[Even Rouault]

Jukka,

Running with CPL_CURL_VERBOSE=YES shows the following

> GET 
/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t15z.wrfsfcf01.grib2 
HTTP/1.1

Host: ftpprd.ncep.noaa.gov
Accept: */*
Range: bytes=84819968-84836351

* Mark bundle as not supporting multiuse
< HTTP/1.1  302 Your allowed limit has been reached. Please go to 
https://www.weather.gov/abusive-user-block  for more info

* no chunk, no close, no size. Assume close to signal end
<

So it seems that after a number of quickly consecutive GET requests to 
the server as here, the server no longer want to see us. The 302 HTTP 
error it sends however is not appropriate. 429 "Too many requests" would 
be more appropriate


GRIB files with lots of "messages" / bands like that one aren't really 
cloud friendly, as you have to skip in many parts of the file to get an 
inventory of it.


That said, I see that in that instance, the .grib files are accompanied 
by a sidecar file, like 
https://ftpprd.ncep.noaa.gov/data/nccf/com/hrrr/prod/hrrr.20210505/conus/hrrr.t15z.wrfsfcf01.grib2.idx, 
which I believe is produced by the wgrib utility. We could conceptually 
try to make use of it, however it doesn't contain all metadata that are 
typically retrieved by the driver, so for gdalinfo itself that wouldn't 
be sufficient, but for just a Open() call that might be enough.


I've noted that https://github.com/OSGeo/gdal/issues/3799

Even


Le 05/05/2021 à 22:04, Rahkonen Jukka (MML) a écrit :


Hi,

Have a look at 
https://gis.stackexchange.com/questions/395867/opening-a-grib-from-the-web-with-gdal-in-python-using-vsicurl-throws-error-on-m 
. 
There gdalinfo fails with /vsicurl/ somewhere in the middle of the 
grib2 file when the server sends http 302 as a response for a range 
request. The file is OK as was tested by downloading it first totally 
with curl. Could it be that the http “moved permanently” at that stage 
makes /vsicurl/ to fail?


I made also a few tests and it seems that the error may happen in 
another http range, but it is always similar:

VSICURL: Got response_code=302

GRIB: ERROR: Ran out of file in Section 7

-Jukka Rahkonen-


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] vsicurl and an issue with http 302 redirect in the middle of the file

[Rahkonen Jukka (MML)]

Hi,

Have a look at 
https://gis.stackexchange.com/questions/395867/opening-a-grib-from-the-web-with-gdal-in-python-using-vsicurl-throws-error-on-m.
 There gdalinfo fails with /vsicurl/ somewhere in the middle of the grib2 file 
when the server sends http 302 as a response for a range request. The file is 
OK as was tested by downloading it first totally with curl. Could it be that 
the http "moved permanently" at that stage makes /vsicurl/ to fail?

I made also a few tests and it seems that the error may happen in another http 
range, but it is always similar:
VSICURL: Got response_code=302
GRIB: ERROR: Ran out of file in Section 7

-Jukka Rahkonen-
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev