rangareddy commented on code in PR #19617:
URL: https://github.com/apache/hudi/pull/19617#discussion_r3946514899
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
+2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These
are the same four values the
+ quick start passes with `--conf`, in the `key value` form the Databricks UI
expects:
+
+ ```
+ spark.serializer org.apache.spark.serializer.KryoSerializer
+ spark.sql.catalog.spark_catalog
org.apache.spark.sql.hudi.catalog.HoodieCatalog
+ spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.kryo.registrator org.apache.spark.HoodieSparkKryoRegistrar
+ ```
+
+3. Read as usual — no extra option is required:
+
+ ```python
+ spark.read.format("hudi").load(basePath)
+ ```
+
+#### What Hudi adapts, and in which release
+
+* **`FileStatusCache`** — DBR changed this API. `SparkHoodieTableFileIndex`
checks reflectively for the
+ `putLeafFiles(Path, FileStatus[])` signature before using it and falls back
when it is absent, rather than
+ failing with `NoSuchMethodError`. This guard lives in the shared Spark
module and has been present since
+ 0.15.x, so it applies on the 1.0.x and 1.1.x lines as well.
+* **`PartitionDirectory`** — DBR's Spark 3.4 runtime backports
`FileStatusWithMetadata` from Spark 3.5, so
+ `PartitionDirectory` takes a `Seq[FileStatusWithMetadata]` rather than a
`Seq[FileStatus]`. That type wraps
+ `FileStatus` by composition instead of extending it, so the elements cannot
be cast; Hudi constructs the
+ wrapper reflectively. This adaptation lives in the Spark 3.4 module and
**ships in 1.2.0**, so a DBR release
+ built on Spark 3.4 needs Hudi 1.2.0 or later.
Review Comment:
Fixed, and your tag-by-tag reading is what the checkout shows. The
reflective guard marker is absent at release-0.15.0, release-1.0.0 and
release-1.0.1 and present from release-0.15.1 and release-1.0.2 onward;
DatabricksRuntimeHelper is used by the 3.4 module in release-0.15.1 and
release-1.2.0 only, absent in 1.0.2 and 1.1.1. The text now gives those exact
versions and says the helper lives in `hudi-spark-common` with only the call
site in the 3.4 module.
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
+2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These
are the same four values the
+ quick start passes with `--conf`, in the `key value` form the Databricks UI
expects:
+
+ ```
+ spark.serializer org.apache.spark.serializer.KryoSerializer
+ spark.sql.catalog.spark_catalog
org.apache.spark.sql.hudi.catalog.HoodieCatalog
+ spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.kryo.registrator org.apache.spark.HoodieSparkKryoRegistrar
+ ```
+
+3. Read as usual — no extra option is required:
+
+ ```python
+ spark.read.format("hudi").load(basePath)
+ ```
+
+#### What Hudi adapts, and in which release
+
+* **`FileStatusCache`** — DBR changed this API. `SparkHoodieTableFileIndex`
checks reflectively for the
+ `putLeafFiles(Path, FileStatus[])` signature before using it and falls back
when it is absent, rather than
+ failing with `NoSuchMethodError`. This guard lives in the shared Spark
module and has been present since
+ 0.15.x, so it applies on the 1.0.x and 1.1.x lines as well.
+* **`PartitionDirectory`** — DBR's Spark 3.4 runtime backports
`FileStatusWithMetadata` from Spark 3.5, so
+ `PartitionDirectory` takes a `Seq[FileStatusWithMetadata]` rather than a
`Seq[FileStatus]`. That type wraps
+ `FileStatus` by composition instead of extending it, so the elements cannot
be cast; Hudi constructs the
+ wrapper reflectively. This adaptation lives in the Spark 3.4 module and
**ships in 1.2.0**, so a DBR release
+ built on Spark 3.4 needs Hudi 1.2.0 or later.
+
+:::caution
+Community guides for older Hudi releases tell you to set
`hoodie.file.index.enable=false` when reading on
+Databricks. That was the workaround before the adaptations above existed: it
falls back from `HoodieFileIndex`
+to `HoodieROTablePathFilter`, which sidesteps the incompatible Spark internals
but disables the file index for
+**every** table in the session, losing the listing optimisation it exists for.
The config is also deprecated
+(since 0.11.0). On 1.2.0 and later you should not need it; if a read still
fails, it remains a fallback, and
+please report the failure.
+:::
Review Comment:
Fixed, and confirmed: at release-1.2.0 `hoodie.file.index.enable` appears
only in its own `DataSourceOptions` definition with no consumer left on the
read path, and on master the constant is gone entirely. The caution now says it
is a no-op and not to reach for it, rather than offering it as a fallback.
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
+2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These
are the same four values the
+ quick start passes with `--conf`, in the `key value` form the Databricks UI
expects:
+
+ ```
+ spark.serializer org.apache.spark.serializer.KryoSerializer
+ spark.sql.catalog.spark_catalog
org.apache.spark.sql.hudi.catalog.HoodieCatalog
+ spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.kryo.registrator org.apache.spark.HoodieSparkKryoRegistrar
+ ```
+
+3. Read as usual — no extra option is required:
+
+ ```python
+ spark.read.format("hudi").load(basePath)
+ ```
+
+#### What Hudi adapts, and in which release
Review Comment:
Added both, and turned the section into a table of adaptation / symptom /
DBR Spark version, so DBR 14.x-16.x is covered. One caveat on the ships-in
column: I independently verified the FileStatusCache and PartitionDirectory
rows against the tags, but could not cleanly pin per-release availability for
#18003 and #14059, so those two rows cite the PRs rather than assert versions.
If you have those to hand I will fill them in.
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
+2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These
are the same four values the
+ quick start passes with `--conf`, in the `key value` form the Databricks UI
expects:
+
+ ```
+ spark.serializer org.apache.spark.serializer.KryoSerializer
+ spark.sql.catalog.spark_catalog
org.apache.spark.sql.hudi.catalog.HoodieCatalog
+ spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.kryo.registrator org.apache.spark.HoodieSparkKryoRegistrar
+ ```
+
+3. Read as usual — no extra option is required:
Review Comment:
Added a caution naming #16951 and `hoodie.metadata.enable=false` as the
current workaround, and noting the metadata table is a write-side default so an
ordinary table can hit it on 1.2.0. Agreed that "no extra option is required"
was too strong.
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
Review Comment:
Fixed. The configs are now split by purpose: a DataFrame read needs none of
them, the Kryo pair is a performance choice, and the catalog and extensions
pair is only for Hudi Spark SQL.
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
Review Comment:
Marked both Kryo lines optional for reads and pointed at #12985 for the
registrator failure, noting that dropping them is a workaround since a read
does not need them.
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
+2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These
are the same four values the
+ quick start passes with `--conf`, in the `key value` form the Databricks UI
expects:
+
+ ```
+ spark.serializer org.apache.spark.serializer.KryoSerializer
+ spark.sql.catalog.spark_catalog
org.apache.spark.sql.hudi.catalog.HoodieCatalog
Review Comment:
Added a caution: overriding `spark.sql.catalog.spark_catalog` replaces the
session catalog, which on a Unity Catalog enabled cluster can break UC access
or stop the session starting, so a reader should leave both SQL configs unset.
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
+
+The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a
modified Spark, and a couple of the
+internals Hudi's Spark datasource builds on differ there. Hudi detects those
differences at runtime and adapts,
+so there is no Databricks-specific Hudi config to set.
+
+#### Cluster setup
+
+1. **Install the Hudi bundle as a cluster library.** In the cluster's
**Libraries** tab, add
+
`org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>`
as Maven coordinates, or
+ upload the jar directly. Pick the bundle that matches the Spark version
your DBR release ships — see the
+ support matrix above.
+2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These
are the same four values the
+ quick start passes with `--conf`, in the `key value` form the Databricks UI
expects:
+
+ ```
+ spark.serializer org.apache.spark.serializer.KryoSerializer
+ spark.sql.catalog.spark_catalog
org.apache.spark.sql.hudi.catalog.HoodieCatalog
+ spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.kryo.registrator org.apache.spark.HoodieSparkKryoRegistrar
+ ```
+
+3. Read as usual — no extra option is required:
+
+ ```python
+ spark.read.format("hudi").load(basePath)
+ ```
+
+#### What Hudi adapts, and in which release
+
+* **`FileStatusCache`** — DBR changed this API. `SparkHoodieTableFileIndex`
checks reflectively for the
+ `putLeafFiles(Path, FileStatus[])` signature before using it and falls back
when it is absent, rather than
Review Comment:
Taken, it now says the fallback is a no-op cache logged as a WARN, so
listings are not cached across queries on DBR. That is worth stating rather
than a nit.
##########
website/docs/quick-start-guide.md:
##########
@@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow
instructions [here]
The *default build* Spark version indicates how we build `hudi-spark3-bundle`.
:::
+### Reading Hudi tables on the Databricks runtime
Review Comment:
Done both: the older Spark 2.4 ADLS recipe now links here, and the ecosystem
Databricks row pointed at that recipe and now points at this section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
