yihua opened a new issue, #19562:
URL: https://github.com/apache/hudi/issues/19562
A table created below table version 8 gets `hoodie.timeline.path=timeline`
and `hoodie.timeline.history.path=history` written into its
`hoodie.properties`, even though it does not use the layout those configs
describe.
Both configs were introduced with timeline layout version 2 in 1.0.0 and are
read only by `TimelinePathProviderV2`, which is selected exclusively at layout
version 2 (table version 8 and above). A table below that version keeps its
timeline directly under `.hoodie` (`TimelinePathProviderV1` hardcodes it) and
resolves its archived timeline through `hoodie.archivelog.folder`. Neither
config has any meaning there.
The cause is that `HoodieTableConfig.dropInvalidConfigs()` only strips a
config that declares a `sinceVersion`:
```java
if (!configProperty.getSinceVersion().isPresent()) {
return true; // never dropped
}
```
`RECORD_MERGE_MODE` declares `.sinceVersion("1.0.0")` and is correctly
dropped from a version 6 table; `TIMELINE_PATH` and `TIMELINE_HISTORY_PATH`
declare nothing and survive.
The immediate effect is cosmetic, since every consumer resolves these
through `getStringOrDefault`. It matters because table version 6 is the
interoperable format used when writing with a 1.x binary for a 0.x reader, and
the 0.x line has no definition for either key. Recording storage layout that
the table does not have is also a trap for anything that later starts honouring
the values.
Reproduce by creating a table with `hoodie.write.table.version=6` and
reading `hoodie.properties`.
Fix: declare the introducing version on both so the existing gate drops them
below table version 8.
Note that `hoodie.table.format` looks similar but must not be handled this
way: the table format SPI is orthogonal to the table version, nothing gates a
custom format on table version 9, and no upgrade handler restores the config,
so gating it on `sinceVersion` would permanently discard a custom format on any
table below version 9.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
