hudi-agent commented on code in PR #19617:
URL: https://github.com/apache/hudi/pull/19617#discussion_r3946556342
##########
website/versioned_docs/version-1.2.0/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
+2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These
are the same four values the
+ quick start passes with `--conf`, in the `key value` form the Databricks UI
expects:
+
+ ```
+ spark.serializer org.apache.spark.serializer.KryoSerializer
Review Comment:
🤖 This copy still presents all four configs as required cluster setup,
including `spark.sql.catalog.spark_catalog
org.apache.spark.sql.hudi.catalog.HoodieCatalog`. The `docs/` version was
revised to split them by purpose and to add the Unity Catalog caution — on a
UC-enabled cluster (the default on current DBR) overriding `spark_catalog` can
break UC access or prevent the session from starting. Following the
instructions as they stand on this page can leave a user with a cluster that
will not start, for configs a DataFrame read does not need. Worth mirroring the
revised split and caution here.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
+2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These
are the same four values the
+ quick start passes with `--conf`, in the `key value` form the Databricks UI
expects:
+
+ ```
+ spark.serializer org.apache.spark.serializer.KryoSerializer
+ spark.sql.catalog.spark_catalog
org.apache.spark.sql.hudi.catalog.HoodieCatalog
+ spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.kryo.registrator org.apache.spark.HoodieSparkKryoRegistrar
+ ```
+
+3. Read as usual — no extra option is required:
+
+ ```python
+ spark.read.format("hudi").load(basePath)
+ ```
+
+#### What Hudi adapts, and in which release
+
+* **`FileStatusCache`** — DBR changed this API. `SparkHoodieTableFileIndex`
checks reflectively for the
+ `putLeafFiles(Path, FileStatus[])` signature before using it and falls back
when it is absent, rather than
+ failing with `NoSuchMethodError`. This guard lives in the shared Spark
module and has been present since
+ 0.15.x, so it applies on the 1.0.x and 1.1.x lines as well.
Review Comment:
🤖 The versioned 1.2.0 copy still carries the pre-revision text here: "has
been present since 0.15.x, so it applies on the 1.0.x and 1.1.x lines as well."
The `docs/` copy in this same PR was corrected to say the `FileStatusCache`
guard is absent in 0.15.0, 1.0.0 and 1.0.1 and present in 0.15.1 and 1.0.2+. As
written, this page tells a 1.0.0/1.0.1 user that they are covered when they are
not, and they will hit the `NoSuchMethodError` on `putLeafFiles(Path,
FileStatus[])` the section says is guarded against. Since the two files were
meant to be identical in this region, it looks like only `docs/` got the fix —
could the corrected release table be mirrored here?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
##########
website/versioned_docs/version-1.2.0/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
+2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These
are the same four values the
+ quick start passes with `--conf`, in the `key value` form the Databricks UI
expects:
+
+ ```
+ spark.serializer org.apache.spark.serializer.KryoSerializer
+ spark.sql.catalog.spark_catalog
org.apache.spark.sql.hudi.catalog.HoodieCatalog
+ spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.kryo.registrator org.apache.spark.HoodieSparkKryoRegistrar
+ ```
+
+3. Read as usual — no extra option is required:
+
+ ```python
+ spark.read.format("hudi").load(basePath)
+ ```
+
+#### What Hudi adapts, and in which release
+
+* **`FileStatusCache`** — DBR changed this API. `SparkHoodieTableFileIndex`
checks reflectively for the
+ `putLeafFiles(Path, FileStatus[])` signature before using it and falls back
when it is absent, rather than
+ failing with `NoSuchMethodError`. This guard lives in the shared Spark
module and has been present since
+ 0.15.x, so it applies on the 1.0.x and 1.1.x lines as well.
+* **`PartitionDirectory`** — DBR's Spark 3.4 runtime backports
`FileStatusWithMetadata` from Spark 3.5, so
+ `PartitionDirectory` takes a `Seq[FileStatusWithMetadata]` rather than a
`Seq[FileStatus]`. That type wraps
+ `FileStatus` by composition instead of extending it, so the elements cannot
be cast; Hudi constructs the
+ wrapper reflectively. This adaptation lives in the Spark 3.4 module and
**ships in 1.2.0**, so a DBR release
+ built on Spark 3.4 needs Hudi 1.2.0 or later.
+
+:::caution
+Community guides for older Hudi releases tell you to set
`hoodie.file.index.enable=false` when reading on
+Databricks. That was the workaround before the adaptations above existed: it
falls back from `HoodieFileIndex`
+to `HoodieROTablePathFilter`, which sidesteps the incompatible Spark internals
but disables the file index for
+**every** table in the session, losing the listing optimisation it exists for.
The config is also deprecated
+(since 0.11.0). On 1.2.0 and later you should not need it; if a read still
fails, it remains a fallback, and
+please report the failure.
+:::
Review Comment:
🤖 Same drift here: "On 1.2.0 and later you should not need it; if a read
still fails, it remains a fallback." The `docs/` copy was corrected to state
that on 1.2.0 the flag is a no-op — the default read path no longer consults
`hoodie.file.index.enable`, and the only remaining consumer is the
`.hoodie/metadata` read path. Offering it as a fallback on the page that is
specifically pinned to 1.2.0 will send users down a dead end on a failing read.
Could this caution be replaced with the corrected wording from
`docs/quick-start-guide.md`? The metadata-table caution (#16951 /
`hoodie.metadata.enable=false`) also appears to be missing from this copy, and
that one is the actionable workaround for the DBR-on-S3 failure.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
