rangareddy opened a new pull request, #19510:
URL: https://github.com/apache/hudi/pull/19510
### Describe the issue this Pull Request addresses
Closes #15676 (HUDI-5526).
`SELECT COUNT(*)` on a Hudi **bootstrap** table fails under Hive 3 while the
same query works on Hive 2.
Reported against every EMR 6.x release:
```
java.io.IOException: cannot find dir =
s3://.../parquet-source-tables/hive_style_partitioned_tb/event_type=two/part-....parquet
in pathToPartitionInfo:
[s3://.../test_bootstrap_hive_partitionedrt/event_type=one,
s3://.../test_bootstrap_hive_partitionedrt/event_type=two]
```
The path Hive cannot resolve is the **external source file**, and
`pathToPartitionInfo` only holds the
**Hudi table's** partition directories.
`HoodieParquetInputFormat#createBootstrappingRecordReader` picks one of the
bootstrap split's two files
whenever only one is needed. The split itself is the skeleton file, which
lives inside the table root;
`getBootstrapFileSplit()` is the external source file, which does not.
Before this change the two
single-file conditions were tested in this order:
```java
if (hoodieColsProjected.isEmpty()) { // -> external source file
...
} else if (externalColsProjected.isEmpty()) { // -> skeleton file
```
`COUNT(*)` projects no columns at all, so **both** conditions hold and the
first one won: the split handed
to Hive pointed outside the table root. Hive 3's
`VectorizedParquetRecordReader` derives partition values
by looking the split path up in `pathToPartitionInfo` and throws. Hive 2
never vectorized this path, which
is why the same query succeeded there.
### Summary and Changelog
- `HoodieParquetInputFormat`: test the external-columns condition first, so
a query projecting no columns
reads the **skeleton** file. Bootstrap keeps a one-to-one row
correspondence between skeleton and
external file, so the count is identical — and the skeleton is both inside
the table root and far
smaller (5 meta columns vs. the full row).
- Extracted the selection into `@VisibleForTesting static Option<FileSplit>
resolveSingleFileSplit(...)`,
returning empty when both files are needed, so the rule can be asserted
directly. The call site is
otherwise unchanged; the stitching branch is untouched.
- New `TestHoodieParquetInputFormatBootstrapSplitSelection` pins all four
projection combinations. This
legacy bootstrap reader path previously had **no** test coverage — nothing
in the tree referenced
`BootstrapBaseFileSplit`. It is a standalone class rather than an addition
to `TestHoodieParquetInputFormat`
because these are pure assertions on a static method and would otherwise
be re-run pointlessly by
`TestGloballyConsistentTimeStampFilteringInputFormat`, which extends that
class.
Only one projection shape changes which file is read:
| query | any meta col projected | any external col projected | before |
after |
| --- | --- | --- | --- | --- |
| `COUNT(*)` (no columns) | no | no | external | **skeleton** |
| meta columns only | yes | no | skeleton | skeleton |
| data columns only | no | yes | external | external |
| meta + data columns | yes | yes | stitch | stitch |
### Impact
Hive 3 `COUNT(*)` (and any other projection that names no column) on a
bootstrap table stops failing, and
reads the smaller skeleton file instead of the external one. Applies to both
COW and MOR bootstrap tables:
`HoodieRealtimeBootstrapBaseFileSplit extends BootstrapBaseFileSplit`, and
`HoodieParquetRealtimeInputFormat#getRecordReader` delegates here, so both
reach the same branch.
`shouldUseFilegroupReader` excludes bootstrap splits, so this legacy path is
still the live one in 1.x.
One further behaviour change worth calling out: a query projecting **only**
Hive virtual columns
(`INPUT__FILE__NAME` and friends, which `externalColsProjected` deliberately
filters out) also lands in the
no-columns case, so `INPUT__FILE__NAME` now reports the skeleton path rather
than the external path. That is
consistent with a non-bootstrap table, where the value is the Hudi base
file, and with the requirement that
the split path stay inside the table root.
No public API, config key or default is changed.
### Risk Level
low
Confined to one branch of the bootstrap record-reader selection, and the
only projection whose behaviour
changes is one that reads no column from either file. Row counts are
unaffected because bootstrap skeleton
and external file hold the same rows in the same order — the same invariant
`BootstrapColumnStichingRecordReader`
already relies on when it advances both readers in lockstep.
**Verification caveat, stated plainly:** the four-way selection rule is
covered by the new unit test, which
fails on master and passes here. I do **not** have a Hive 3 + Tez +
bootstrap-table environment, so the
reporter's end-to-end query has not been re-run; the link from "split path
outside the table root" to the
reported `pathToPartitionInfo` failure is established by reading Hive's
`VectorizedParquetRecordReader#initPartitionValues`, not by executing it. An
end-to-end confirmation from
anyone with an EMR bootstrap table would be welcome.
### Documentation Update
None required — no configuration, default or public API changes.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly above
- [x] Adequate tests were added if applicable
- [x] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
