prashantwason commented on PR #20029: URL: https://github.com/apache/hudi/pull/20029#issuecomment-6020282168
@yihua @voonhous thanks for the reviews. I've addressed all the comments, please take another look. - **yihua: assert on the nested predicates.** The tests now check that `EqualTo(item.name,a1)` and `GreaterThan(item.price,5)` show up in `PushedFilters`, with a negative leg (empty allowlist) showing they're absent. `shortName()` now returns the base file format instead of mutating session config (0c24ffd). - **voonhous: non-ASCII columns under schema-on-read.** You were right. Once nested pushdown is on, Spark backtick-quotes non-identifier names (`` `名字` ``, ``item.`my-name` ``), and `rebuildFilterFromParquet` threw `cannot find filter col name`. The fix parses the name into its parts for the internal-schema lookup, then quotes the file-side name back the way `ParquetFilters` expects (08101e3). I added `WHERE` filters to `TestSpark3DDL` "Test Chinese table" plus a COW/MOR test with renamed non-ASCII top-level and nested columns. Both failed with that error before the fix. - **voonhous: rewrite vs. drop on rename.** Added a nested case to `TestParquetSchemaEvolutionUtils` asserting `item.title` → `item.name`, not `AlwaysTrue`. It also covers quoted, dotted and hyphenated names. - **voonhous: MOR without log files.** The MOR legs now assert log files exist. The pushdown test includes a log update that moves a record off the filtered value. - **Also:** Vortex now reports its own `shortName` instead of falling through to `parquet` (f75f8b4). I fixed the description too: EXPLAIN's `Format` comes from `toString`, so it doesn't change. I didn't add an ORC end-to-end test (your nit), per your note that it can wait. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
