nsivabalan commented on PR #19205: URL: https://github.com/apache/hudi/pull/19205#issuecomment-5226044864
The third follow-up listed in this PR's description — the CoW incremental read bug found while writing the incremental tests here — is now fixed in **#19568**, on its own branch off current upstream master. It is independent of this PR and reviewable separately. Two things about it worth noting for anyone who read the earlier description here: **It is broader than `count()`.** Any incremental query whose projection excludes `_hoodie_commit_time` returns zero rows — `select(subset)` too. `collect()` worked only incidentally, because it happens to project every column including the filtered one. I had described it as a `count()` bug; that was the symptom, not the shape. **My original mechanism guess was wrong in an important way.** I had blamed filter pushdown. But as @nsivabalan pointed out, an incremental read *always* pushes those filters — including on the path that works — so pushdown cannot be the anomaly on its own. The real defect is narrower: the reader evaluates a pushed predicate against the schema it was asked to read, and nothing guarantees the filtered column is in that schema. Also confirmed along the way that two things which look like the bug are in fact correct and deliberate: `isCount` excluding incremental (you cannot row-count when only some rows are in range) and the column-less fallthrough to the plain Parquet reader (HUDI-8079). The tests in this PR are unaffected — they read whole rows, which sidesteps the issue, and the comment in `TestMetaFieldsModeE2E` explaining why can be removed once #19568 lands. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
