voonhous commented on code in PR #19617:
URL: https://github.com/apache/hudi/pull/19617#discussion_r3920793235
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
+2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These
are the same four values the
+ quick start passes with `--conf`, in the `key value` form the Databricks UI
expects:
+
+ ```
+ spark.serializer org.apache.spark.serializer.KryoSerializer
+ spark.sql.catalog.spark_catalog
org.apache.spark.sql.hudi.catalog.HoodieCatalog
+ spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.kryo.registrator org.apache.spark.HoodieSparkKryoRegistrar
+ ```
+
+3. Read as usual — no extra option is required:
+
+ ```python
+ spark.read.format("hudi").load(basePath)
+ ```
+
+#### What Hudi adapts, and in which release
+
+* **`FileStatusCache`** — DBR changed this API. `SparkHoodieTableFileIndex`
checks reflectively for the
+ `putLeafFiles(Path, FileStatus[])` signature before using it and falls back
when it is absent, rather than
+ failing with `NoSuchMethodError`. This guard lives in the shared Spark
module and has been present since
+ 0.15.x, so it applies on the 1.0.x and 1.1.x lines as well.
+* **`PartitionDirectory`** — DBR's Spark 3.4 runtime backports
`FileStatusWithMetadata` from Spark 3.5, so
+ `PartitionDirectory` takes a `Seq[FileStatusWithMetadata]` rather than a
`Seq[FileStatus]`. That type wraps
+ `FileStatus` by composition instead of extending it, so the elements cannot
be cast; Hudi constructs the
+ wrapper reflectively. This adaptation lives in the Spark 3.4 module and
**ships in 1.2.0**, so a DBR release
+ built on Spark 3.4 needs Hudi 1.2.0 or later.
+
+:::caution
+Community guides for older Hudi releases tell you to set
`hoodie.file.index.enable=false` when reading on
+Databricks. That was the workaround before the adaptations above existed: it
falls back from `HoodieFileIndex`
+to `HoodieROTablePathFilter`, which sidesteps the incompatible Spark internals
but disables the file index for
+**every** table in the session, losing the listing optimisation it exists for.
The config is also deprecated
+(since 0.11.0). On 1.2.0 and later you should not need it; if a read still
fails, it remains a fallback, and
+please report the failure.
+:::
Review Comment:
**major:** On 1.2.0 `hoodie.file.index.enable=false` is not a fallback: the
default read path never consults it. At `release-1.2.0`,
`DefaultSource.scala:324-331` sends every COW snapshot / RO read of a user
table to `HoodieCopyOnWriteSnapshotHadoopFsRelationFactory`; the only remaining
consumer is `BaseFileOnlyRelation.toHadoopFsRelation`
(`BaseFileOnlyRelation.scala:140`), reached only when reading
`.hoodie/metadata` itself. The `DefaultSource` consumer went away in #14061; on
1.1.1 it was already gated behind `hoodie.file.group.reader.enabled=false`.
Also "every table in the session" only holds for a session conf; the flag was
read via `HoodieSparkConfUtils.getConfigValue(optParams, sessionConf, ...)`, so
a per-read `.option()` was possible. Could we say it is a no-op on 1.2.0 and
drop the fallback advice?
```suggestion
:::caution
Community guides for older Hudi releases tell you to set
`hoodie.file.index.enable=false` when reading on
Databricks. That was the workaround before the adaptations above existed: on
0.x it swapped `HoodieFileIndex`
for a plain file listing filtered by `HoodieROTablePathFilter`, which
sidestepped the incompatible Spark
internals but lost the listing optimisation the file index exists for. The
config is deprecated (since 0.11.0)
and on 1.2.0 the default read path no longer consults it, so it is a no-op
there rather than a fallback. If a
read fails on 1.2.0 or later, please report it (see the note below) instead
of setting this config.
:::
```
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
+2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These
are the same four values the
+ quick start passes with `--conf`, in the `key value` form the Databricks UI
expects:
+
+ ```
+ spark.serializer org.apache.spark.serializer.KryoSerializer
+ spark.sql.catalog.spark_catalog
org.apache.spark.sql.hudi.catalog.HoodieCatalog
+ spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.kryo.registrator org.apache.spark.HoodieSparkKryoRegistrar
+ ```
+
+3. Read as usual — no extra option is required:
Review Comment:
**major:** Open #16951 (HUDI-9305, the follow-up #13129 itself flagged;
re-triaged as still open on 2026-08-24) reports that with
`hoodie.metadata.enable=true` a DBR read on S3 fails with
`InconsistentReadException` / `RemoteFileChangedException`, and works with it
set to false. The metadata table is on by default on the write side, so a
typical table hits this even on 1.2.0, which makes "no extra option is
required" too strong. Could we add a caution after this step pointing at #16951
and naming `.option("hoodie.metadata.enable", "false")` as the current
workaround?
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
+2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These
are the same four values the
+ quick start passes with `--conf`, in the `key value` form the Databricks UI
expects:
+
+ ```
+ spark.serializer org.apache.spark.serializer.KryoSerializer
+ spark.sql.catalog.spark_catalog
org.apache.spark.sql.hudi.catalog.HoodieCatalog
+ spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.kryo.registrator org.apache.spark.HoodieSparkKryoRegistrar
+ ```
+
+3. Read as usual — no extra option is required:
+
+ ```python
+ spark.read.format("hudi").load(basePath)
+ ```
+
+#### What Hudi adapts, and in which release
+
+* **`FileStatusCache`** — DBR changed this API. `SparkHoodieTableFileIndex`
checks reflectively for the
+ `putLeafFiles(Path, FileStatus[])` signature before using it and falls back
when it is absent, rather than
+ failing with `NoSuchMethodError`. This guard lives in the shared Spark
module and has been present since
+ 0.15.x, so it applies on the 1.0.x and 1.1.x lines as well.
+* **`PartitionDirectory`** — DBR's Spark 3.4 runtime backports
`FileStatusWithMetadata` from Spark 3.5, so
+ `PartitionDirectory` takes a `Seq[FileStatusWithMetadata]` rather than a
`Seq[FileStatus]`. That type wraps
+ `FileStatus` by composition instead of extending it, so the elements cannot
be cast; Hudi constructs the
+ wrapper reflectively. This adaptation lives in the Spark 3.4 module and
**ships in 1.2.0**, so a DBR release
+ built on Spark 3.4 needs Hudi 1.2.0 or later.
Review Comment:
**major:** The release claims do not match the tags. The `FileStatusCache`
guard is absent at `release-0.15.0`, `release-1.0.0` and `release-1.0.1`,
present from `release-0.15.1` and `release-1.0.2` (#13129 merged 2025-04-11),
so "since 0.15.x ... 1.0.x" overclaims; #12944 is a Hudi 1.0 user hitting
exactly this `putLeafFiles` `NoSuchMethodError`. `DatabricksRuntimeHelper` and
its `HoodieSpark34PartitionedFileUtils.scala:53` call site are also in
`release-0.15.1` (#18291), so "needs 1.2.0 or later" misses the 0.x line. The
helper itself lives in `hudi-spark-common`; only the call site is in the 3.4
module. Could we state the exact versions?
```suggestion
failing with `NoSuchMethodError`. This guard lives in the shared Spark
module and ships in 0.15.1 and in
1.0.2 or later; it is absent in 0.15.0, 1.0.0 and 1.0.1.
* **`PartitionDirectory`**: DBR's Spark 3.4 runtime backports
`FileStatusWithMetadata` from Spark 3.5, so
`PartitionDirectory` takes a `Seq[FileStatusWithMetadata]` rather than a
`Seq[FileStatus]`. That type wraps
`FileStatus` by composition instead of extending it, so the elements
cannot be cast; Hudi constructs the
wrapper reflectively. The helper lives in the shared Spark module and is
wired in from the Spark 3.4 module;
it **ships in 1.2.0** and in 0.15.1, so a DBR release built on Spark 3.4
needs one of those or later.
```
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
+2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These
are the same four values the
+ quick start passes with `--conf`, in the `key value` form the Databricks UI
expects:
+
+ ```
+ spark.serializer org.apache.spark.serializer.KryoSerializer
+ spark.sql.catalog.spark_catalog
org.apache.spark.sql.hudi.catalog.HoodieCatalog
+ spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.kryo.registrator org.apache.spark.HoodieSparkKryoRegistrar
+ ```
+
+3. Read as usual — no extra option is required:
+
+ ```python
+ spark.read.format("hudi").load(basePath)
+ ```
+
+#### What Hudi adapts, and in which release
+
+* **`FileStatusCache`** — DBR changed this API. `SparkHoodieTableFileIndex`
checks reflectively for the
+ `putLeafFiles(Path, FileStatus[])` signature before using it and falls back
when it is absent, rather than
Review Comment:
**nit:** The fallback is `BaseHoodieTableFileIndex.NoopCache` with a WARN
(`SparkHoodieTableFileIndex.scala:674-697` on master), so on DBR file listings
are not cached across queries. Feel free to ignore, but could "falls back when
it is absent" say "falls back to a no-op cache (logged as a WARN), so listings
are not cached across queries", since that is a perf implication the reader
would want?
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
Review Comment:
**minor:** Adding evidence to this: open #12985 is DBR 13.3 LTS (Spark 3.4),
multi-node, with the bundle installed as a cluster library and
`spark.kryo.registrator` set exactly as step 2 says, failing with `Failed to
serialize task 0 ... HoodieSparkKryoRegistrar not found`. Not blocking, but
since a DataFrame read needs neither Kryo line, could we mark
`spark.serializer` / `spark.kryo.registrator` as optional for reads and point
at #12985 for the registrator failure?
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
Review Comment:
**nit:** Placement: 55 lines of DBR cluster setup sit inside `## Setup`
ahead of `### Spark Shell/SQL`, so every quick-start reader scrolls past
Databricks instructions to reach the spark-shell command. Feel free to ignore,
but would it read better at the end of `## Setup` (after "Setup project"), or
as its own page next to `azure_hoodie` under Storage Configurations?
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
Review Comment:
**minor:** `azure_hoodie.md:27-49` already hosts a Databricks recipe
("Databricks Spark2.4 on ADLS Gen2": `.format("org.apache.hudi")`, mount
points, no bundle or config steps) and `ecosystem.md:34` links the Databricks
row to that page, so the site now carries two conflicting Databricks recipes
and neither points here. Not blocking. Could we add a one-line link from that
subsection to this section and retarget the ecosystem row?
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
+2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These
are the same four values the
+ quick start passes with `--conf`, in the `key value` form the Databricks UI
expects:
+
+ ```
+ spark.serializer org.apache.spark.serializer.KryoSerializer
+ spark.sql.catalog.spark_catalog
org.apache.spark.sql.hudi.catalog.HoodieCatalog
+ spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.kryo.registrator org.apache.spark.HoodieSparkKryoRegistrar
+ ```
+
+3. Read as usual — no extra option is required:
+
+ ```python
+ spark.read.format("hudi").load(basePath)
+ ```
+
+#### What Hudi adapts, and in which release
Review Comment:
**major:** This lists two of the four Databricks adaptations and only covers
a DBR built on Spark 3.4. Missing: #18003
(`FileStatusWithMetadata#toFileStatus` and `PartitionedFile.locations`
Seq-vs-Array on DBR's Spark 3.5, in the 3.5 and 4.0 modules; MOR incremental
full-scan, #18002) and #14059 (`InterpretedPredicate` 2-arg constructor,
partition-predicate pruning, #14058). By tag: #18003 is in 1.2.0 and 0.15.1 but
not 1.1.1; #14059 is in 1.1.0+ and 0.15.1 but not 1.0.2. DBR 14.x-16.x run
Spark 3.5 and get no guidance here. Could we add those two bullets, or a small
adaptation / affects / ships-in table, so the heading's completeness holds?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
