markhoerth opened a new issue, #14744:
URL: https://github.com/apache/gravitino/issues/14744
### Version
main branch, commit 463296caca4082133bf12bad2bd4ed5d2572933e (2.0.0-SNAPSHOT)
### Describe what's wrong
A user calls the Iceberg REST `registerTable` endpoint (`POST
/v1/namespaces/{namespace}/register`) with a `metadata-location` that cannot be
read as a metadata file: a directory, or a path whose scheme the server has no
file system for, such as `s3://` on a catalog with local storage. The location
comes from the request, so the user expects a 400 that names the problem.
Instead the request is rejected with a 500. A location that does not exist
already returns 400 since #14628, so only these two variants are left.
With authorization enabled, a directory returns 500 with `Authorization
failed due to system internal error`, which reads as an authorization fault
although the request is only malformed. An unconfigured scheme returns 500
`RuntimeIOException`. No table is created in either case.
Cause (Gravitino code):
-
`iceberg/iceberg-common/src/main/java/org/apache/gravitino/iceberg/common/ops/IcebergCatalogWrapper.java:345-353`:
`loadTableMetadataFromLocation` reads the client-supplied location and
converts only `NotFoundException` to `BadRequestException`.
`TableMetadataParser.read` at line 347 raises `IllegalArgumentException` (`...
is not a valid metadata file`) for a directory and `RuntimeIOException`
(`Failed to get file system for path`) for a scheme with no file system.
-
`iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/server/web/filter/RegisterTableAuthzHandler.java:162-163`:
the authorization handler calls that method while authorizing, so these
exceptions are raised inside the authorization interceptor.
-
`iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/server/web/filter/IcebergMetadataAuthorizationMethodInterceptor.java:191-193`
propagates only exceptions from the `org.apache.iceberg.exceptions` package.
`server-common/src/main/java/org/apache/gravitino/server/web/filter/BaseMetadataAuthorizationMethodInterceptor.java:389-404`
therefore wraps the `IllegalArgumentException` in a `RuntimeException` with
the "system internal error" message, which becomes a 500.
-
`iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/IcebergExceptionMapper.java:106-118`:
`RuntimeIOException` is an Iceberg exception, so the interceptor passes it
through, but the mapper has no entry for it and returns the default 500.
### Error message and/or stacktrace
Directory as `metadata-location`:
```
HTTP 500
{"error":{"message":"Authorization failed due to system internal error,
User: 'admin', Operation:
'registerTable'","type":"RuntimeException","code":500,"stack":[
"java.lang.RuntimeException: Authorization failed due to system internal
error, User: 'admin', Operation: 'registerTable'",
"at
org.apache.gravitino.server.web.filter.BaseMetadataAuthorizationMethodInterceptor.authorizeMethodInScope(BaseMetadataAuthorizationMethodInterceptor.java:404)",
...
"Caused by: java.lang.IllegalArgumentException:
/home/ubuntu/gvt/data/irc_warehouse/repro_reg/src/metadata is not a valid
metadata file",
"at
org.apache.gravitino.iceberg.common.ops.IcebergCatalogWrapper.loadTableMetadataFromLocation(IcebergCatalogWrapper.java:347)",
"at
org.apache.gravitino.server.web.filter.RegisterTableAuthzHandler.authorizeMetadataSources(RegisterTableAuthzHandler.java:163)",
```
Unconfigured scheme as `metadata-location`:
```
HTTP 500
{"error":{"message":"Failed to get file system for path:
s3://no-such-bucket/x/metadata/00000-x.metadata.json","type":"RuntimeIOException","code":500,"stack":[
"org.apache.iceberg.exceptions.RuntimeIOException: Failed to get file
system for path: s3://no-such-bucket/x/metadata/00000-x.metadata.json",
"at org.apache.iceberg.hadoop.Util.getFs(Util.java:57)",
"at
org.apache.iceberg.hadoop.HadoopInputFile.fromLocation(HadoopInputFile.java:56)",
...
"at
org.apache.gravitino.iceberg.common.ops.IcebergCatalogWrapper.loadTableMetadataFromLocation(IcebergCatalogWrapper.java:347)",
"at
org.apache.gravitino.server.web.filter.RegisterTableAuthzHandler.authorizeMetadataSources(RegisterTableAuthzHandler.java:163)",
"Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
FileSystem for scheme \"s3\"",
```
A location that does not exist, for comparison:
```
HTTP 400
{"error":{"message":"Metadata location
/home/ubuntu/gvt/data/irc_warehouse/repro_reg/src/metadata/missing.metadata.json
does not exist","type":"BadRequestException","code":400,
```
### How to reproduce
1. Start the server with these settings, without any S3 file system on the
Iceberg REST classpath, and a JDBC driver for the catalog backend (the check
ran with H2):
```
gravitino.authorization.enable = true
gravitino.authorization.serviceAdmins = admin
gravitino.auxService.names = iceberg-rest
gravitino.iceberg-rest.classpath = iceberg-rest-server/libs,
iceberg-rest-server/conf
gravitino.iceberg-rest.httpPort = 9001
gravitino.iceberg-rest.catalog-config-provider = dynamic-config-provider
gravitino.iceberg-rest.gravitino-metalake = repro_ml
gravitino.iceberg-rest.default-catalog-name = irc_cat
```
2. Set the shell variables, then create the clean metalake and the catalog:
```
AUTH="Authorization: Basic $(printf admin:x | base64)"
API=http://127.0.0.1:8090/api
IRC=http://127.0.0.1:9001/iceberg/v1
curl -s -X POST -H "$AUTH" -H 'Content-Type: application/json'
$API/metalakes \
-d '{"name":"repro_ml","properties":{}}'
curl -s -X POST -H "$AUTH" -H 'Content-Type: application/json'
$API/metalakes/repro_ml/catalogs \
-d
'{"name":"irc_cat","type":"RELATIONAL","provider":"lakehouse-iceberg","comment":"c","properties":{"catalog-backend":"jdbc","uri":"jdbc:h2:file:/tmp/repro_irc_h2;AUTO_SERVER=TRUE;DATABASE_TO_LOWER=TRUE","jdbc-driver":"org.h2.Driver","jdbc-user":"sa","jdbc-password":"sa","jdbc-initialize":"true","warehouse":"/home/ubuntu/gvt/data/irc_warehouse"}}'
```
3. Create a namespace and a table, and take the table's metadata directory:
```
curl -s -X POST -H "$AUTH" -H 'Content-Type: application/json'
$IRC/namespaces \
-d '{"namespace":["repro_reg"]}'
curl -s -X POST -H "$AUTH" -H 'Content-Type: application/json'
$IRC/namespaces/repro_reg/tables \
-d
'{"name":"src","schema":{"type":"struct","fields":[{"id":1,"name":"id","type":"long","required":false}]}}'
LOC=$(curl -s -H "$AUTH" $IRC/namespaces/repro_reg/tables/src | python3
-c 'import json,sys;print(json.load(sys.stdin)["metadata-location"])')
DIR=${LOC%/*}
```
4. Register from the directory. The response is 500:
```
curl -s -w '\n%{http_code}\n' -X POST -H "$AUTH" -H 'Content-Type:
application/json' $IRC/namespaces/repro_reg/register \
-d "{\"name\":\"r1\",\"metadata-location\":\"$DIR\"}"
```
5. Register from an unconfigured scheme. The response is 500:
```
curl -s -w '\n%{http_code}\n' -X POST -H "$AUTH" -H 'Content-Type:
application/json' $IRC/namespaces/repro_reg/register \
-d
'{"name":"r2","metadata-location":"s3://no-such-bucket/x/metadata/00000-x.metadata.json"}'
```
6. Register from a missing file, which returns 400 as expected:
```
curl -s -w '\n%{http_code}\n' -X POST -H "$AUTH" -H 'Content-Type:
application/json' $IRC/namespaces/repro_reg/register \
-d
"{\"name\":\"r3\",\"metadata-location\":\"$DIR/missing.metadata.json\"}"
```
### Additional context
Related issues: #14302 and #14628 fixed the missing-file case by converting
`NotFoundException`, and #14432 covered other invalid register requests. Those
fixes do not reach the exceptions described here, so these two cases are
reported separately.
The 2.0 OSS test suite check
`test_register_bad_metadata_location_is_client_error`, cases `directory` and
`unconfigured_scheme`, found it.
The cause is in Gravitino's own code: the exceptions come from the Iceberg
library, but Gravitino is what reads a client-supplied location and decides how
to report the failure.
Also present in v1.3.1 for the unconfigured scheme: `RuntimeIOException` has
no entry in `IcebergExceptionMapper` at that tag, so the same read returns 500.
The directory case returning 500 is new on main, since
`RegisterTableAuthzHandler` does not exist at v1.3.1. The v1.3.1 statement
comes from reading the code at the tag, not from running that release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
