gsparre opened a new issue, #13030:
URL: https://github.com/apache/iceberg/issues/13030
### Apache Iceberg version
1.9.0 (latest release)
### Query engine
Flink
### Please describe the bug 🐞
Experiencing authentication failures due to expired tokens when streaming
from Kafka to Iceberg via PyFlink, running on Azure Blob Storage with Polaris
catalog.
- Flink Version: 1.18
- Iceberg Version: 1.9.0
- Polaris Version: 0.11.0
- Storage: Azure Blob Storage
- Streaming Framework: PyFlink
The streaming job runs successfully for approximately 1 hour, then fails
with an authentication error indicating expired tokens. The job needs to be
manually restarted to continue processing.
```
<Code>AuthenticationFailed</Code><Message>Server failed to authenticate the
request. Make sure the value of Authorization header is formed correctly
including the signature.
Time:2025-04-29T10:04:32.3424886Z</Message><AuthenticationErrorDetail>Signed
expiry time [Tue, 29 Apr 2025 09:39:38 GMT] must be after signed start time
[Tue, 29 Apr 2025 10:04:32 GMT]</AuthenticationErrorDetail></Error>
```
Polaris-side logs showing successful reauthentication:
```
2025-04-29 09:31:31,217 INFO [io.qua.htt.access-log] [,POLARIS] [,,,]
(executor-thread-1) 10.244.4.122 - - [29/April/2025:09:31:31 +0000] "POST
/api/catalog/v1/oauth/tokens HTTP/1.1" 200 770
2025-04-29 09:31:33,661 INFO [com.azu.ide.ChainedTokenCredential] [,] [,,,]
(azure-sdk-global-thread-1) Azure Identity => Attempted credential
EnvironmentCredential returns a token
```
The catalog is created using Flink SQL
tbl_env.execute_sql(f"""
CREATE CATALOG {catalog_name} WITH (
'type' = 'iceberg',
'catalog-type' = 'rest',
'uri' = '{catalog_uri}',
'credential' = '{client_id}:{client_secret}',
'scope' = 'PRINCIPAL_ROLE:ALL',
'warehouse' = '{warehouse}',
'token-refresh-enabled' = 'true',
'header.Polaris-realm' = 'POLARIS',
'header.X-Iceberg-Access-Delegation' =
'vended-credentials',
'io-impl' = 'org.apache.iceberg.io.ResolvingFileIO'
);
""")
**Expected Behavior**
The streaming job should automatically refresh authentication tokens before
they expire, allowing for continuous operation without manual intervention.
**Actual Behavior**
The job fails after ~1 hour with authentication errors and requires manual
restart to resume processing.
Additional context
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
