codope opened a new pull request, #20304:
URL: https://github.com/apache/hudi/pull/20304
### Describe the issue this Pull Request addresses
Closes #20300
Closes #20301
**#20300:** a DataFrame CDC read with
`hoodie.datasource.read.begin.instanttime=earliest` returns
0 rows, while `begin=000` and `hudi_table_changes(..., 'cdc', 'earliest')`
return the expected rows.
`HoodieCDCFileIndex` passes the option value straight into
`InstantRange.startInstant`.
`HoodieCDCExtractor` then filters instants with `isInRange(requestedTime)`,
a string comparison, and
`"earliest"` sorts after every numeric instant time, so every commit is
excluded. The SQL path works
because `HoodieTableChanges` rewrites `earliest` to `000` before building
the relation.
**#20301:** the `hoodie.datasource.read.begin.instanttime` doc says
`completion_time >= START_COMMIT`, but the begin bound is exclusive. Both
incremental relations
build an `OPEN_CLOSED` range: V2 by default, and V1 through
`findInstantsInRange`, which is
`(start, end]`. On a two-commit table, begin = commit 1's completion time
returns only commit 2's
rows. Begin 1 ms earlier returns both commits.
### Summary and Changelog
DataFrame CDC reads now accept `earliest` (case-insensitive) as the begin
instant and return the
same rows as the SQL function.
- `HoodieCDCFileIndex`: map `IncrementalQueryAnalyzer.START_COMMIT_EARLIEST`
(case-insensitive) to
`HoodieTimeline.INIT_INSTANT_TS` before building the `InstantRange`. This
matches what
`hudi_table_changes` does for `earliest`, so both paths behave the same.
That includes the
existing check that the start instant is not older than the archived
timeline.
- `DataSourceReadOptions.START_COMMIT`: the doc now says `completion_time >
START_COMMIT` and that
the start is exclusive while `hoodie.datasource.read.end.instanttime` is
inclusive. The example
sentence is updated to match.
- `TestCDCDataFrameSuite#testCDCReadFromEarliest`: COW and MOR, with
`earliest` and `EARLIEST`.
Asserts the op counts, and that the rows equal a read whose begin instant
is just before the
first commit.
### Impact
DataFrame CDC queries with `begin.instanttime=earliest` now return data
instead of an empty
result. The other change is to config documentation only.
### Risk Level
low. The new branch only runs when the begin option is `earliest`, which
previously matched no
commit and returned an empty result.
Verification (JDK 11, `-Pspark3.5`):
- The new test fails on master in all 4 cases (`expected: <100> but was:
<0>`) and passes with the
fix.
- `TestCDCDataFrameSuite`, `TestCDCStreamingSuite` and `TestCDCForSparkSQL`
all pass.
- The #20300 repro on a bundle built from this branch:
| Read | Before | After |
|---|---|---|
| DataFrame CDC, begin `earliest` | 0 rows | 4 rows |
| DataFrame CDC, begin `000` | 4 rows | 4 rows |
| `hudi_table_changes(..., 'cdc', 'earliest')` | 4 rows | 4 rows |
The results are the same for COW and MOR.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
