chandrasekhar-188k opened a new issue, #17285:
URL: https://github.com/apache/iceberg/issues/17285
### Feature Request / Improvement
Background
Iceberg currently validates metadata timestamps using a hardcoded one-minute
tolerance (TableMetadata.ONE_MINUTE) when constructing new table metadata
during a commit.
This validation helps detect situations where metadata timestamps are
significantly ahead of the local system clock, which could indicate incorrect
system time or metadata corruption.
However, in deployments where multiple compute engines (for example, Spark,
Flink, Trino, or custom applications) operate on the same Iceberg table from
different clusters, temporary clock skew greater than one minute can occur
despite otherwise healthy deployments. In such cases, valid commits may fail
because the metadata timestamp appears to be too far in the future relative to
the committing process.
Although proper clock synchronization (for example, via NTP) is recommended,
operators may still encounter larger clock skew in distributed or multi-region
environments.
Problem
The allowable clock skew is currently hardcoded as one minute and cannot be
configured.
As a result:
Valid commits may fail when clocks differ by more than one minute.
Operators have no mechanism to tune this behavior for their deployment.
The only current workaround is improving clock synchronization across all
participating systems.
Proposed Solution
Introduce a JVM system property to configure the maximum allowable clock
skew used during metadata timestamp validation.
System Property
iceberg.commit.allowed-clock-skew-ms
Default Value
60000
(Existing behavior remains unchanged.)
Example
-Diceberg.commit.allowed-clock-skew-ms=300000
This would allow metadata timestamps to be up to five minutes ahead of the
local system clock before the commit is rejected.
Behavior
The validation logic would determine the allowed clock skew as follows:
Read the JVM system property iceberg.commit.allowed-clock-skew-ms.
If the property is present and valid, use its value.
Otherwise, fall back to the current default of 60,000 milliseconds.
This proposal does not change the default behavior for existing deployments.
Compatibility
This proposal is fully backward compatible.
Existing applications continue to use the current one-minute tolerance.
No table metadata changes are required.
No catalog changes are required.
No API changes are required.
Future Work
A future enhancement could introduce catalog-level configuration, allowing
operators to configure the allowable clock skew once per catalog instead of per
JVM.
One possible configuration hierarchy would be:
Catalog property
↓
System property
↓
Default (60000 ms)
The current proposal intentionally limits the scope to a JVM system property
to minimize implementation complexity while addressing the immediate
operational need.
### Query engine
None
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with
guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
