danhuawang opened a new issue, #11077: URL: https://github.com/apache/gravitino/issues/11077
### Version main branch ### Describe what's wrong ``` trino> show schemas in gravitino_irc; Query 20260513_083429_00003_nsf8h, FAILED, 1 node http://35.190.166.193:8080/ui/query.html?20260513_083429_00003_nsf8h Splits: 5 total, 4 done (80.00%) CPU Time: 2.4s total, 0 rows/s, 0B/s, 3% active Per Node: 0.0 parallelism, 0 rows/s, 0B/s Parallelism: 0.0 Peak Memory: 48B 1:19 [0 rows, 0B] [0 rows/s, 0B/s] Query 20260513_083429_00003_nsf8h failed: Delayed StackOverflowError due to ReservedStackAccess annotated method java.lang.StackOverflowError: Delayed StackOverflowError due to ReservedStackAccess annotated method at java.base/sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:821) at java.base/java.net.Socket$SocketOutputStream.implWrite(Socket.java:1086) at java.base/java.net.Socket$SocketOutputStream.write(Socket.java:1076) at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection$1.write(DefaultBHttpClientConnection.java:259) at org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117) at org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flush(SessionOutputBufferImpl.java:126) at org.apache.hc.core5.http.impl.io.ContentLengthOutputStream.flush(ContentLengthOutputStream.java:98) at org.apache.hc.core5.http.io.entity.StringEntity.writeTo(StringEntity.java:189) at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:285) at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:162) at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.lambda$execute$0(InternalExecRuntime.java:236) at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:851) at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:233) at org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:120) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:192) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:195) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:138) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:112) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:110) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:106) at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:318) at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:297) at org.apache.iceberg.rest.BaseHTTPClient.postForm(BaseHTTPClient.java:136) at org.apache.iceberg.rest.auth.OAuth2Util.fetchToken(OAuth2Util.java:291) at org.apache.iceberg.rest.auth.OAuth2Util$AuthSession.fromCredential(OAuth2Util.java:734) at org.apache.iceberg.rest.auth.OAuth2Manager.newSessionFromCredential(OAuth2Manager.java:266) at org.apache.iceberg.rest.auth.OAuth2Manager.lambda$maybeCreateChildSession$4(OAuth2Manager.java:240) at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$0(BoundedLocalCache.java:2707) at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1932) at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2705) at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2686) at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112) at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:63) at org.apache.iceberg.rest.auth.AuthSessionCache.cachedSession(AuthSessionCache.java:88) at org.apache.iceberg.rest.auth.OAuth2Manager.maybeCreateChildSession(OAuth2Manager.java:238) at org.apache.iceberg.rest.auth.OAuth2Manager.contextualSession(OAuth2Manager.java:140) at org.apache.iceberg.rest.auth.OAuth2Manager.contextualSession(OAuth2Manager.java:40) at org.apache.iceberg.rest.RESTSessionCatalog.listNamespaces(RESTSessionCatalog.java:577) at io.trino.plugin.iceberg.catalog.rest.TrinoRestCatalog.collectNamespaces(TrinoRestCatalog.java:206) at io.trino.plugin.iceberg.catalog.rest.TrinoRestCatalog.lambda$collectNamespaces$0(TrinoRestCatalog.java:209) ``` ### Error message and/or stacktrace ``` 2026-05-13T08:32:09.925Z INFO Query-20260513_083208_00000_nsf8h-139 org.apache.iceberg.rest.auth.AuthManagers Loading AuthManager implementation: org.apache.iceberg.rest.auth.OAuth2Manager 2026-05-13T08:33:00.226Z INFO SplitRunner-20260513_083208_00000_nsf8h.2.0.0-1-170 org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec ex-0000001898 recoverable I/O exception (org.apache.hc.core5.http.NoHttpResponseException) caught when sending request to http://gravitino.irc-e2e-env1:9001;request will be automatically re-executed in 0 MILLISECONDS (exec count 2) 2026-05-13T08:33:00.738Z INFO dispatcher-query-1 io.trino.event.QueryMonitor TIMELINE: Query 20260513_083208_00000_nsf8h :: FAILED (ICEBERG_CATALOG_ERROR) :: elapsed 50913ms :: planning 453ms :: waiting 652ms :: scheduling 1743ms :: running 48277ms :: finishing 440ms :: begin 2026-05-13T08:32:09.785Z :: end 2026-05-13T08:33:00.698Z 2026-05-13T08:33:25.135Z INFO dispatcher-query-4 io.trino.event.QueryMonitor TIMELINE: Query 20260513_083324_00001_nsf8h :: FAILED (ICEBERG_CATALOG_ERROR) :: elapsed 191ms :: planning 38ms :: waiting 4ms :: scheduling 68ms :: running 31ms :: finishing 54ms :: begin 2026-05-13T08:33:24.878Z :: end 2026-05-13T08:33:25.069Z 2026-05-13T08:33:31.312Z INFO dispatcher-query-4 io.trino.event.QueryMonitor TIMELINE: Query 20260513_083330_00002_nsf8h :: FAILED (ICEBERG_CATALOG_ERROR) :: elapsed 177ms :: planning 35ms :: waiting 4ms :: scheduling 29ms :: running 69ms :: finishing 44ms :: begin 2026-05-13T08:33:31.047Z :: end 2026-05-13T08:33:31.224Z OpenJDK 64-Bit Server VM warning: Potentially dangerous stack overflow in ReservedStackAccess annotated method sun.nio.ch.NioSocketImpl.write([BII)V [1] 2026-05-13T08:35:49.374Z INFO dispatcher-query-9 io.trino.event.QueryMonitor TIMELINE: Query 20260513_083429_00003_nsf8h :: FAILED (GENERIC_INTERNAL_ERROR) :: elapsed 79009ms :: planning 24ms :: waiting 4ms :: scheduling 29ms :: running 78911ms :: finishing 45ms :: begin 2026-05-13T08:34:30.298Z :: end 2026-05-13T08:35:49.307Z 2026-05-13T08:36:40.043Z WARN gravitino_irc-token-refresh-0 org.apache.iceberg.util.Tasks Retrying task after failure: sleepTimeMs=101 Connection pool shut down java.lang.IllegalStateException: Connection pool shut down at org.apache.hc.core5.util.Asserts.check(Asserts.java:38) at org.apache.hc.core5.pool.StrictConnPool.lease(StrictConnPool.java:170) at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.lease(PoolingHttpClientConnectionManager.java:366) at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.acquireEndpoint(InternalExecRuntime.java:105) at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:127) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:195) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:138) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:112) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:110) at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:106) at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:318) at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:297) at org.apache.iceberg.rest.BaseHTTPClient.postForm(BaseHTTPClient.java:136) at org.apache.iceberg.rest.auth.OAuth2Util.fetchToken(OAuth2Util.java:291) at org.apache.iceberg.rest.auth.OAuth2Util.fetchToken(OAuth2Util.java:326) at org.apache.iceberg.rest.auth.OAuth2Util.refreshToken(OAuth2Util.java:188) at org.apache.iceberg.rest.auth.OAuth2Util$AuthSession.refreshCurrentToken(OAuth2Util.java:626) at org.apache.iceberg.rest.auth.OAuth2Util$AuthSession.lambda$refresh$1(OAuth2Util.java:596) at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413) at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219) at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203) at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196) at org.apache.iceberg.rest.auth.OAuth2Util$AuthSession.refresh(OAuth2Util.java:596) at org.apache.iceberg.rest.auth.OAuth2Util$AuthSession.lambda$scheduleTokenRefresh$2(OAuth2Util.java:670) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:545) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:309) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) at java.base/java.lang.Thread.run(Thread.java:1474) ``` ### How to reproduce 1. Create Iceberg catalog for IRC ``` { "code": 0, "catalog": { "name": "catalog_1", "type": "relational", "provider": "lakehouse-iceberg", "comment": "env1 catalog", "properties": { "jdbc-password": "****", "jdbc-driver": "org.postgresql.Driver", "jdbc-schema-version": "V1", "io-impl": "org.apache.iceberg.aws.s3.S3FileIO", "s3-secret-access-key": "XXXX, "warehouse": "s3://irc-e2e-env1-metalake-b-warehouse/env1/metalake_b/catalog_1", "credential-providers": "s3-token", "uri": "jdbc:postgresql://gravitino-env1-metalake-b-postgresql.irc-e2e-env1:5432/gravitinoIRC", "in-use": "true", "catalog-backend": "jdbc", "s3-access-key-id": "XXXXX", "s3-token-expire-in-secs": "900", "jdbc-user": "gravitino", "jdbc-initialize": "true", "s3-region": "us-east-1", "s3-role-arn": "arn:aws:iam::XXXX:role/sts_s3_role" }, "audit": { "creator": "service-account-postman-client", "createTime": "2026-04-27T11:54:16.594716071Z", "lastModifier": "admin", "lastModifiedTime": "2026-05-06T04:26:01.170748872Z" } } } ``` 2. Trino (480) catalog proporty as following ``` /etc/trino/catalog/gravitino_irc.properties connector.name=iceberg iceberg.catalog.type=rest iceberg.rest-catalog.uri=http://gravitino.irc-e2e-env1:9001/iceberg/ iceberg.rest-catalog.prefix=catalog_1 iceberg.rest-catalog.warehouse=catalog_1 iceberg.rest-catalog.security=OAUTH2 iceberg.rest-catalog.oauth2.server-uri=http://XXXXXX:8080/realms/myrealm/protocol/openid-connect/token iceberg.rest-catalog.oauth2.credential=trino-client:XXXXXXX iceberg.rest-catalog.oauth2.scope=openid profile email iceberg.rest-catalog.oauth2.token-exchange-enabled=false iceberg.rest-catalog.vended-credentials-enabled=true iceberg.rest-catalog.nested-namespace-enabled=true fs.native-gcs.enabled=false fs.native-s3.enabled=true s3.region=us-east-1 ``` 3. run sql in Trino CLI trino> show schemas in gravitino_irc; ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
