hudi-agent commented on code in PR #19617: URL: https://github.com/apache/hudi/pull/19617#discussion_r3772993411
########## website/docs/quick-start-guide.md: ########## @@ -28,6 +28,38 @@ Hudi works with Spark 3.3 and above versions. You can follow instructions [here] The *default build* Spark version indicates how we build `hudi-spark3-bundle`. ::: +### Reading Hudi tables on the Databricks runtime + +The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a modified Spark, and a few of the +internals Hudi's Spark datasource builds on differ there. Reading a Hudi table on DBR needs **no +Hudi-specific configuration** — Hudi detects those differences at runtime and adapts, rather than requiring +you to set a flag: + +* **`FileStatusCache`** — DBR changed this API. Hudi checks reflectively for the + `putLeafFiles(Path, FileStatus[])` signature before using it and falls back when it is absent, instead of + failing with `NoSuchMethodError`. +* **`PartitionDirectory`** — DBR's Spark 3.4 runtime backports `FileStatusWithMetadata` from Spark 3.5, so + `PartitionDirectory` takes a `Seq[FileStatusWithMetadata]` rather than a `Seq[FileStatus]`. That type wraps + `FileStatus` by composition instead of extending it, so the elements cannot simply be cast; Hudi constructs + the wrapper reflectively. This adaptation ships in **1.2.0**, so a DBR Spark 3.4 runtime needs 1.2.0 or + later. + +Snapshot, read-optimized and incremental queries all go through these paths, including incremental queries in Review Comment: 🤖 This sentence makes a fairly broad guarantee — that snapshot, read-optimized, and incremental queries (including MoR full-scan incremental) *all* route through the adapted `FileStatusCache`/`PartitionDirectory` paths. The two adaptations verify cleanly, but it would help to confirm this coverage claim is exhaustive: the `PartitionDirectory` fix lives only in the `hudi-spark3.4.x` module (`HoodieSpark34PartitionedFileUtils`), so it applies specifically to a DBR Spark 3.4 runtime, whereas the `FileStatusCache` guard is in the shared common module. Could a committer confirm every listed query path actually exercises these code paths on DBR? @yihua you know the Spark read path best — does this hold for MoR full-scan incremental as well? <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## website/docs/quick-start-guide.md: ########## @@ -28,6 +28,38 @@ Hudi works with Spark 3.3 and above versions. You can follow instructions [here] The *default build* Spark version indicates how we build `hudi-spark3-bundle`. ::: +### Reading Hudi tables on the Databricks runtime + +The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a modified Spark, and a few of the +internals Hudi's Spark datasource builds on differ there. Reading a Hudi table on DBR needs **no +Hudi-specific configuration** — Hudi detects those differences at runtime and adapts, rather than requiring +you to set a flag: + +* **`FileStatusCache`** — DBR changed this API. Hudi checks reflectively for the + `putLeafFiles(Path, FileStatus[])` signature before using it and falls back when it is absent, instead of + failing with `NoSuchMethodError`. Review Comment: 🤖 Minor completeness point: the `FileStatusCache` reflective guard actually predates the `PartitionDirectory` adaptation — it landed earlier (in the 1.1.0 line) in the shared `hudi-spark-common` module, while the `PartitionDirectory` reflective construction is the piece that ships in 1.2.0. The current wording scopes the "ships in 1.2.0" note only to the PartitionDirectory bullet, which is accurate — just flagging in case you want to note that the FileStatusCache handling has been available since an earlier release so 1.1.x DBR users aren't left wondering. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
