Re: [gdal-dev] errors using IAM instance profile auth in s3

michael . smith . erdc Sat, 19 Nov 2022 07:00:39 -0800

Correct, not a public bucket, which is why the IAM credentials are needed. If I set them manually, it all works fine.

[ u02]$ export AWS_ACCESS_KEY_ID=xxxxx

[ u02]$ gdalinfo /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif

HTTP: libcurl/7.86.0 OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0 nghttp2/1.47.0

S3: Downloading 0-16383 (https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif)...

S3: Got response_code=206

GDAL: GDALOpen(/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif, this=0x5645e67d7f30) succeeds as GTiff.

Driver: GTiff/GeoTIFF

GDAL: GDALDefaultOverviews::OverviewScan()

Files: /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif

Size is 75000, 52000

Coordinate System is:

PROJCRS["Estonian Coordinate System of 1997",

….

Mike

On Nov 19, 2022, at 9:48 AM, Even Rouault <even.roua...@spatialys.com> wrote:

Yes, a 206 response code means success here as we are requesting only bytes 0-16383. So maybe the file is not a valid TIFF ?

( "grid-dev-publiclidar" must not be so public I guess, because when trying with my credentials, I get a Access Denied)

Le 19/11/2022 à 15:40, michael.smith.e...@gmail.com a écrit :

I’m seeing that it’s getting a 206 response code, so wouldn’t that indicate auth is working?

gdalinfo /vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif

HTTP: Fetch(http://169.254.169.254/latest/api/token)

HTTP: libcurl/7.86.0 OpenSSL/3.0.7 zlib/1.2.13 libssh2/1.10.0 nghttp2/1.47.0

HTTP: These HTTP headers were set: X-aws-ec2-metadata-token-ttl-seconds: 10

HTTP: Fetch(http://169.254.169.254/latest/meta-data/iam/security-credentials/)

HTTP: Fetch(http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3)

AWS: Storing AIM credentials until 2022-11-19T20:42:58Z

S3: Downloading 0-16383 (https://grid-dev-publiclidar.s3.amazonaws.com/estonia/dtm/estonia_dtm_5m.tif)...

S3: Got response_code=206

gdalinfo failed - unable to open '/vsis3/grid-dev-publiclidar/estonia/dtm/estonia_dtm_5m.tif'.

Mike

On Nov 19, 2022, at 9:26 AM, Even Rouault <even.roua...@spatialys.com> wrote:

Hi Mike,

could you send the output of

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-grid-s3

Slightly redacted of course, but with the exact formatting. This part of thee code currently uses a "simple JSON parser" (https://github.com/OSGeo/gdal/blob/c61d116a469821b769630a112dee7f1a61fed885/port/cpl_aws.cpp#L554), which is actually just a non JSON-aware string tokenizer, and I suspect it could be defeated by a new formatting of S3 or something specific to your credentials.

It could also be that something unhandled by that parser appears inside quoted strings, like an escaped double quote or some other JSON escaped character (like an escaped forward slash \/ )

If that was the case we should likely switch to proper JSON deserialization (that part of the code must predate libjson-c being a build requirement of GDAL).

Even

--
http://www.spatialys.com
My software is free, but my time generally not.
-- 
http://www.spatialys.com
My software is free, but my time generally not.

_______________________________________________
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] errors using IAM instance profile auth in s3

Reply via email to