Re: [gdal-dev] cURL response content output

[Daniel Evans]

Hi again,

To loop back in the mailing list on the conversation Even and I had on the
PR, the answer to my original question (at least in the /vsis3/ use case)
was to enable CPL_DEBUG, which outputs the raw response in these sorts of
error cases, e.g.:

"CPLE_None in S3: \nNoSuchBucketThe
specified bucket does not
exista-very-very-fake-s3-bucket.."

I'd been thrown off by the fact that rasterio captures GDAL's CPL_DEBUG
logs and re-emits them in Python, but CPL_CURL_VERBOSE messages aren't
captured in the same way. Some quirks in the configuration of the logging
on my end meant the DEBUG log messages were being discarded when I thought
they weren't.

Regards,
Daniel

On Wed, 6 Jul 2022 at 18:55, Even Rouault 
wrote:

> Daniel,
>
> https://github.com/OSGeo/gdal/pull/6028 should help, but with the
> restriction indicated in the commit message.
>
> Even
> Le 06/07/2022 à 13:13, Daniel Evans a écrit :
>
> Hi,
>
> I'm trying to diagnose some issues with accessing remote datasets in GDAL
> - I believe the problem is unrelated to GDAL itself, but it's proving hard
> to debug.
>
> By enabling CPL_CURL_VERBOSE, I can see that the server is kicking back a
> 400 Bad Request status, but the debug output doesn't include the actual
> response content, which would allow me to understand the problem in more
> detail. Is there a way to get GDAL/cURL to output the response content for
> failed requests?
>
>
> By way of example, if I attempt to access a nonexistent file on S3 with
> cURL directly, I get the 404 error code and the descriptive XML error
> message at the end:
>
> $ curl -v https://a-very-very-fake-s3-bucket.s3.amazonaws.com/example.tif
> [...]
> < HTTP/1.1 404 Not Found
> < Content-Type: application/xml
> < Transfer-Encoding: chunked
> < Date: Wed, 06 Jul 2022 11:09:09 GMT
> < Server: AmazonS3
> --- these lines ---
> <
> 
> NoSuchBucketThe specified bucket does not
> exista-very-very-fake-s3-bucket[...]
> --- these lines ---
>
>
> In contrast, the output from GDAL with CPL_CURL_VERBOSE excludes the XML
> response:
>
> < HTTP/1.1 404 Not Found
> < Content-Type: application/xml
> < Transfer-Encoding: chunked
> < Date: Wed, 06 Jul 2022 11:04:16 GMT
> < Server: AmazonS3
>
> --- no XML output here ---
>
> * Connection #1 to host a-very-very-fake-s3-bucket.s3.amazonaws.com left
> intact
> * Couldn't find host a-very-very-fake-s3-bucket.s3.amazonaws.com in the
> .netrc file; using defaults
> * Found bundle for host a-very-very-fake-s3-bucket.s3.amazonaws.com:
> 0x25b5080 [can pipeline]
> * Re-using existing connection! (#1) with host
> a-very-very-fake-s3-bucket.s3.amazonaws.com
> * Connected to a-very-very-fake-s3-bucket.s3.amazonaws.com
> (52.216.207.115) port 443 (#1)
> > GET /?delimiter=%2F=100=example.tif%2F HTTP/1.1
> [...]
>
> Regards,
> Daniel
>
> ___
> gdal-dev mailing 
> listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] cURL response content output

[Even Rouault]

Daniel,

https://github.com/OSGeo/gdal/pull/6028 should help, but with the 
restriction indicated in the commit message.


Even

Le 06/07/2022 à 13:13, Daniel Evans a écrit :

Hi,

I'm trying to diagnose some issues with accessing remote datasets in 
GDAL - I believe the problem is unrelated to GDAL itself, but it's 
proving hard to debug.


By enabling CPL_CURL_VERBOSE, I can see that the server is kicking 
back a 400 Bad Request status, but the debug output doesn't include 
the actual response content, which would allow me to understand the 
problem in more detail. Is there a way to get GDAL/cURL to output the 
response content for failed requests?



By way of example, if I attempt to access a nonexistent file on S3 
with cURL directly, I get the 404 error code and the descriptive 
XML error message at the end:


$ curl -v https://a-very-very-fake-s3-bucket.s3.amazonaws.com/example.tif
[...]
< HTTP/1.1 404 Not Found
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Wed, 06 Jul 2022 11:09:09 GMT
< Server: AmazonS3
--- these lines ---
<

NoSuchBucketThe specified bucket does not 
exista-very-very-fake-s3-bucket[...]

--- these lines ---


In contrast, the output from GDAL with CPL_CURL_VERBOSE excludes the 
XML response:


< HTTP/1.1 404 Not Found
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Wed, 06 Jul 2022 11:04:16 GMT
< Server: AmazonS3

--- no XML output here ---

* Connection #1 to host a-very-very-fake-s3-bucket.s3.amazonaws.com 
 left intact
* Couldn't find host a-very-very-fake-s3-bucket.s3.amazonaws.com 
 in the .netrc 
file; using defaults
* Found bundle for host a-very-very-fake-s3-bucket.s3.amazonaws.com 
: 0x25b5080 [can 
pipeline]
* Re-using existing connection! (#1) with host 
a-very-very-fake-s3-bucket.s3.amazonaws.com 

* Connected to a-very-very-fake-s3-bucket.s3.amazonaws.com 
 (52.216.207.115) 
port 443 (#1)

> GET /?delimiter=%2F=100=example.tif%2F HTTP/1.1
[...]

Regards,
Daniel

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] cURL response content output

[Daniel Evans]

Hi,

I'm trying to diagnose some issues with accessing remote datasets in GDAL -
I believe the problem is unrelated to GDAL itself, but it's proving hard to
debug.

By enabling CPL_CURL_VERBOSE, I can see that the server is kicking back a
400 Bad Request status, but the debug output doesn't include the actual
response content, which would allow me to understand the problem in more
detail. Is there a way to get GDAL/cURL to output the response content for
failed requests?


By way of example, if I attempt to access a nonexistent file on S3 with
cURL directly, I get the 404 error code and the descriptive XML error
message at the end:

$ curl -v https://a-very-very-fake-s3-bucket.s3.amazonaws.com/example.tif
[...]
< HTTP/1.1 404 Not Found
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Wed, 06 Jul 2022 11:09:09 GMT
< Server: AmazonS3
--- these lines ---
<

NoSuchBucketThe specified bucket does not
exista-very-very-fake-s3-bucket[...]
--- these lines ---


In contrast, the output from GDAL with CPL_CURL_VERBOSE excludes the XML
response:

< HTTP/1.1 404 Not Found
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Wed, 06 Jul 2022 11:04:16 GMT
< Server: AmazonS3

--- no XML output here ---

* Connection #1 to host a-very-very-fake-s3-bucket.s3.amazonaws.com left
intact
* Couldn't find host a-very-very-fake-s3-bucket.s3.amazonaws.com in the
.netrc file; using defaults
* Found bundle for host a-very-very-fake-s3-bucket.s3.amazonaws.com:
0x25b5080 [can pipeline]
* Re-using existing connection! (#1) with host
a-very-very-fake-s3-bucket.s3.amazonaws.com
* Connected to a-very-very-fake-s3-bucket.s3.amazonaws.com (52.216.207.115)
port 443 (#1)
> GET /?delimiter=%2F=100=example.tif%2F HTTP/1.1
[...]

Regards,
Daniel
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev