David-N-Perkins opened a new issue, #11836:
URL: https://github.com/apache/iceberg/issues/11836
### Apache Iceberg version
1.7.1 (latest release)
### Query engine
Flink
### Please describe the bug 🐞
We attempted to upgrade Iceberg `1.6.1` and Flink `1.18.1` to Iceberg
`1.7.1` and Flink `1.20.0`. When creating the Iceberg catalog in Flink SQL, we
now encounter this exception.
```
Could not execute SQL statement. Reason:
org.apache.iceberg.exceptions.ForbiddenException: Forbidden:
```
Here is the create catalog statement.
```
CREATE CATALOG c_iceberg WITH (
'type'='iceberg',
'catalog-type'='rest',
'uri'='http://polaris:8181/api/catalog',
'credential'='xxx:xxx',
'warehouse'='azure-catalog',
'scope'='PRINCIPAL_ROLE:ALL'
);
```
I'm not positive, but believe the issue is with Iceberg and not Polaris
since we didn't change the Polaris version and it still works fine with the
older version.
Here are the logs from Flink.
```
2024-12-19 21:09:16,010 WARN org.apache.iceberg.rest.RESTSessionCatalog
[] - Iceberg REST client is m
issing the OAuth2 server URI configuration and defaults to
http://polaris:8181/api/catalogv1/oauth/tokens. This automati
c fallback will be removed in a future Iceberg release.It is recommended to
configure the OAuth2 endpoint using the 'oau
th2-server-uri' property to be prepared. This warning will disappear if the
OAuth2 endpoint is explicitly configured. Se
e https://github.com/apache/iceberg/issues/10537
2024-12-19 21:09:16,017 WARN org.apache.iceberg.rest.ErrorHandlers
[] - Unable to parse error re
sponse
java.io.UncheckedIOException:
org.apache.iceberg.shaded.com.fasterxml.jackson.databind.exc.MismatchedInputException:
No
content to map due to end-of-input
at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION`
disabled); line: 1]
at org.apache.iceberg.util.JsonUtil.parse(JsonUtil.java:101)
~[iceberg-flink-runtime-1.20-1.7.1.jar:?]
at
org.apache.iceberg.rest.responses.ErrorResponseParser.fromJson(ErrorResponseParser.java:71)
~[iceberg-flink-r
untime-1.20-1.7.1.jar:?]
at
org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.parseResponse(ErrorHandlers.java:194)
~[iceberg-fli
nk-runtime-1.20-1.7.1.jar:?]
at
org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:181)
~[iceberg-flink-runtime-1.20-1.7.1.jar:?
]
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:323)
~[iceberg-flink-runtime-1.20-1.7.1.jar:?]
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:262)
~[iceberg-flink-runtime-1.20-1.7.1.jar:?]
at org.apache.iceberg.rest.HTTPClient.get(HTTPClient.java:358)
~[iceberg-flink-runtime-1.20-1.7.1.jar:?]
at
org.apache.iceberg.rest.RESTSessionCatalog.fetchConfig(RESTSessionCatalog.java:1045)
~[iceberg-flink-runtime-
1.20-1.7.1.jar:?]
at
org.apache.iceberg.rest.RESTSessionCatalog.initialize(RESTSessionCatalog.java:251)
~[iceberg-flink-runtime-1.
20-1.7.1.jar:?]
at
org.apache.iceberg.rest.RESTCatalog.initialize(RESTCatalog.java:78)
~[iceberg-flink-runtime-1.20-1.7.1.jar:?]
at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:274)
~[iceberg-flink-runtime-1.20-1.7.1.jar:?]
at
org.apache.iceberg.flink.CatalogLoader$RESTCatalogLoader.loadCatalog(CatalogLoader.java:163)
~[iceberg-flink-
runtime-1.20-1.7.1.jar:?]
at
org.apache.iceberg.flink.FlinkCatalog.<init>(FlinkCatalog.java:114)
~[iceberg-flink-runtime-1.20-1.7.1.jar:?]
at
org.apache.iceberg.flink.FlinkCatalogFactory.createCatalog(FlinkCatalogFactory.java:166)
~[iceberg-flink-runt
ime-1.20-1.7.1.jar:?]
at
org.apache.iceberg.flink.FlinkCatalogFactory.createCatalog(FlinkCatalogFactory.java:139)
~[iceberg-flink-runt
ime-1.20-1.7.1.jar:?]
at
org.apache.flink.table.factories.FactoryUtil.createCatalog(FactoryUtil.java:492)
~[flink-table-api-java-uber-
1.20.0.jar:1.20.0]
at
org.apache.flink.table.catalog.CatalogManager.initCatalog(CatalogManager.java:368)
~[flink-table-api-java-ube
r-1.20.0.jar:1.20.0]
at
org.apache.flink.table.catalog.CatalogManager.createCatalog(CatalogManager.java:322)
~[flink-table-api-java-u
ber-1.20.0.jar:1.20.0]
at
org.apache.flink.table.operations.ddl.CreateCatalogOperation.execute(CreateCatalogOperation.java:88)
~[flink-
table-api-java-uber-1.20.0.jar:1.20.0]
```
It appears that Flink is able to authenticate and get an access token from
Polaris. But when attempting to fetch the Polaris configuration, a 403 response
is received, and the body can't be parsed into JSON.
I also checked the Polaris logs.
```
WARN [2024-12-19 21:09:16,008 - 171371485] [pool-3-thread-11] []
o.a.p.s.c.DefaultContextResolver: Failed to parse prin
cipal from headers ({x-request-id=a5a35de0-dcb6-40d0-8cc4-a66cba92f077,
x-b3-parentspanid=730e3221997844e4, Accept=appli
cation/json, x-client-version=Apache Iceberg 1.7.1 (commit
4a432839233f2343a9eae8255532f911f06358ef), x-b3-sampled=0, Us
er-Agent=Apache-HttpClient/5.4 (Java/11.0.25), X-Forwarded-Proto=http,
Host=polaris:8181, Accept-Encoding=gzip, x-gzip,
deflate, x-envoy-attempt-count=1, x-client-git-commit-short=4a43283,
x-b3-traceid=43748bd2c2e73bc3730e3221997844e4, x-b3
-spanid=4f6cc905a0d0dbe1,
x-forwarded-client-cert=By=spiffe://cluster.local/ns/data-platform/sa/polaris;Hash=55045c2d814
4e5d0718e2706502ac713201f42ec44b799fd508674c475b49465;Subject="";URI=spiffe://cluster.local/ns/data-platform/sa/flink,
C
ontent-Length=116, Content-Type=application/x-www-form-urlencoded}); using
default-principal
INFO [2024-12-19 21:09:16,008 - 171371485] [pool-3-thread-11] []
o.a.p.s.tracing.TracingFilter: Started span with paren
t spanId="208f3f42abfc42ee" traceId="170feb2185813eb2f25a5c3306ced564"
parentContext="{}"
WARN [2024-12-19 21:09:16,008 - 171371485] [pool-3-thread-11 - POST
/api/catalog/v1/oauth/tokens] [] o.g.j.servlet.WebC
omponent: A servlet request to the URI
http://polaris:8181/api/catalog/v1/oauth/tokens contains form parameters in the
r
equest body but the request body has been consumed by the servlet or a
servlet filter accessing the request parameters.
Only resource methods using @FormParam will work as expected. Resource
methods consuming the request body by other means
will not work as expected.
[EL Fine]: sql: 2024-12-19
21:09:16.008--ServerSession(214651918)--Connection(166093349)--SELECT
PRINCIPALCLIENTID, MAIN
SECRET, MAINSECRETHASH, PRINCIPALID, SECONDARYSECRET, SECONDARYSECRETHASH,
SECRETSALT, VERSION FROM PRINCIPAL_SECRETS WH
ERE (PRINCIPALCLIENTID = ?)
bind => [root]
[EL Fine]: sql: 2024-12-19
21:09:16.009--ServerSession(214651918)--Connection(166093349)--SELECT
CATALOGID, ID, CREATETI
MESTAMP, DROPTIMESTAMP, ENTITYVERSION, GRANTRECORDSVERSION,
INTERNALPROPERTIES, LASTUPDATETIMESTAMP, NAME, PARENTID, PRO
PERTIES, PURGETIMESTAMP, SUBTYPECODE, TOPURGETIMESTAMP, TYPECODE, VERSION
FROM ENTITIES WHERE ((CATALOGID = ?) AND (ID =
?))
bind => [0, 151]
INFO [2024-12-19 21:09:16,010 - 171371487] [pool-3-thread-11] []
i.o.e.l.LoggingSpanExporter: 'POST /api/catalog/v1/oau
th/tokens' : 170feb2185813eb2f25a5c3306ced564 208f3f42abfc42ee SERVER
[tracer: /api/catalog/v1/oauth/tokens:] Attributes
Map{data={url.path=/api/catalog/v1/oauth/tokens, server.address=polaris,
http.request.method=POST, realm=default-realm,
url.scheme=http}, capacity=128, totalAddedValues=5}
```
I don't see anything in the Polaris logs related to fetching the
configuration, which makes me think there is something wrong with the request
that causes the HTTP server to reject it.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [X] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
