Davis-Zhang-Onehouse opened a new pull request, #20688:
URL: https://github.com/apache/hudi/pull/20688

   ### Describe the issue this Pull Request addresses
   
   issue: #20687
   Part of #20683, RFC-118 (#20684).
   Stacked on #20686: review only the last commit. Until #20686 merges, this PR 
also shows its commit.
   
   Defects in the batched BLOB read path:
   
   - `read_blob()` over a join whose two sides have same-named BLOB columns 
resolves the column by name (`schema.fieldIndex`), so it can read the other 
side's bytes.
   - Managed out-of-line BLOB references are relative to the table, and the 
batched reader is not given the table base path.
   - The broadcast storage configuration is a `HadoopStorageConfiguration` with 
a transient configuration; under Kryo without Hudi's registrator executors see 
null.
   - A scan that opens one reader per file keeps every exhausted reader 
reachable from the task's completion listeners until the task ends.
   
   ### Summary and Changelog
   
   - `BatchedBlobReadExec` / `BatchedBlobReader`: resolve the BLOB column by 
expression id and ordinal (`processRDDAtOrdinal`) instead of by name.
   - New `BlobTableBasePath`: finds the base path of the table a BLOB attribute 
comes from; `BatchedBlobReaderStrategy` passes it to the reader, which resolves 
managed relative references against it.
   - `BatchedBlobReader.broadcastStorageConf`: broadcast a Spark 
`SerializableConfiguration`, which works with any `spark.serializer`, and 
rebuild the storage configuration on the executor.
   - `CloseableIteratorListener.addListenerReleasedOnExhaustion`, used by 
`HoodieFileGroupReaderFunction`: release a file's reader once its iterator is 
exhausted or closed, not at task end.
   - Tests: same-named join columns in `TestReadBlobSQL` (fails on master: it 
reads the wrong bytes), `TestBatchedBlobReaderBasePath`, 
`TestBatchedBlobReaderKryo`, `TestFileScanReaderRelease`, 
`TestCloseableIteratorListener`, `TestBlobFileGroupReaderOutputType`, and 
`TestBlobPrunedProjection` (pruned BLOB projections on MOR, using master's 
`SparkSchemaTransformUtils` nested pruning). The scan test builds its files 
through the existing `SparkAdapter` factories, so it compiles on every 
supported Spark version.
   
   Master's general nested pruning and its identical-descriptor batching and 
join handling are kept unchanged. No code was copied from other projects.
   
   ### Impact
   
   No public API change. `read_blob()` returns the right bytes for same-named 
columns in a join, managed references resolve, Kryo jobs read out-of-line 
BLOBs, and long scans release readers as they go.
   
   ### Risk Level
   
   low: changes are confined to the BLOB read path and the per-file listener. 
Verified on JDK 11 / Spark 3.5 / Scala 2.12: 30 hudi-spark-client tests 
(listener, schema transform) and 30 JUnit plus 8 ScalaTest hudi-spark tests, 
including the whole `TestReadBlobSQL`, `TestHoodieFileGroupReaderFunction` and 
`TestSparkSchemaTransformUtils`; checkstyle, scalastyle and RAT pass. The 
changed reader sources compile on JDK 17 / Spark 4.0 / Scala 2.13. Spark 3.3 
was not checked: test compilation on master already stops at 
`InputFormatTestUtil.java:584` (`AvroWriteSupport.WRITE_FIXED_AS_INT96` 
unavailable), which this PR does not touch.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] fix(spark): pruned BLOB... Davis-Zhang-Onehouse (via GitHub) via commits

Reply via email to