rangareddy commented on code in PR #19617: URL: https://github.com/apache/hudi/pull/19617#discussion_r4002789352
########## website/docs/quick-start-guide.md: ########## @@ -158,6 +158,104 @@ basePath = "file:///tmp/trips_table" </Tabs > +### Reading Hudi tables on the Databricks runtime + +The Spark support matrix at the top of this page is for Apache Spark. The Databricks Runtime (DBR) ships a +modified Spark, and several of the internals Hudi's Spark datasource builds on differ there. Hudi detects those +differences at runtime and adapts, so there is no Databricks-specific Hudi config to set. Review Comment: Fixed, and a fair catch: the intro contradicted both the table and the caution below it. It now says "in the common case" and names the two qualifications explicitly, the per-release table and the `hoodie.metadata.enable=false` workaround, so a reader who stops after the intro is not misled. ########## website/src/pages/ecosystem.md: ########## @@ -31,7 +31,7 @@ In such cases, you can leverage another tool like Apache Spark or Apache Flink t | Google DataProc | [Read + Write](https://cloud.google.com/blog/products/data-analytics/getting-started-with-new-table-formats-on-dataproc) | | | Azure Synapse | [Read + Write](https://www.onehouse.ai/blog/apache-hudi-on-microsoft-azure) | | | Azure HDInsight | [Read + Write](https://www.onehouse.ai/blog/apache-hudi-on-microsoft-azure) | | -| Databricks | [Read + Write](https://hudi.apache.org/docs/azure_hoodie/) | | +| Databricks | [Read](https://hudi.apache.org/docs/quick-start-guide#reading-hudi-tables-on-the-databricks-runtime) | | Review Comment: Reverted to Read + Write, thank you. You are right that narrowing it retracted a capability the site still documents: `azure_hoodie` shows `inputDF.write.format("org.apache.hudi")` against an abfss path. The row now links Read at the new section and Write at that recipe, so neither is lost. ########## website/versioned_docs/version-1.2.0/quick-start-guide.md: ########## @@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow instructions [here] The *default build* Spark version indicates how we build `hudi-spark3-bundle`. ::: +### Reading Hudi tables on the Databricks runtime + +The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a modified Spark, and a couple of the +internals Hudi's Spark datasource builds on differ there. Hudi detects those differences at runtime and adapts, +so there is no Databricks-specific Hudi config to set. + +#### Cluster setup + +1. **Install the Hudi bundle as a cluster library.** In the cluster's **Libraries** tab, add + `org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>` as Maven coordinates, or + upload the jar directly. Pick the bundle that matches the Spark version your DBR release ships — see the + support matrix above. +2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These are the same four values the + quick start passes with `--conf`, in the `key value` form the Databricks UI expects: + + ``` + spark.serializer org.apache.spark.serializer.KryoSerializer + spark.sql.catalog.spark_catalog org.apache.spark.sql.hudi.catalog.HoodieCatalog + spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension + spark.kryo.registrator org.apache.spark.HoodieSparkKryoRegistrar + ``` + +3. Read as usual — no extra option is required: + + ```python + spark.read.format("hudi").load(basePath) + ``` + +#### What Hudi adapts, and in which release + +* **`FileStatusCache`** — DBR changed this API. `SparkHoodieTableFileIndex` checks reflectively for the + `putLeafFiles(Path, FileStatus[])` signature before using it and falls back when it is absent, rather than + failing with `NoSuchMethodError`. This guard lives in the shared Spark module and has been present since + 0.15.x, so it applies on the 1.0.x and 1.1.x lines as well. Review Comment: Already mirrored in 5680690f4719, which pushed after this was written: the versioned 1.2.0 copy now carries the corrected release table, so the 0.15.0/1.0.0/1.0.1 gap is stated there too. ########## website/versioned_docs/version-1.2.0/quick-start-guide.md: ########## @@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow instructions [here] The *default build* Spark version indicates how we build `hudi-spark3-bundle`. ::: +### Reading Hudi tables on the Databricks runtime + +The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a modified Spark, and a couple of the +internals Hudi's Spark datasource builds on differ there. Hudi detects those differences at runtime and adapts, +so there is no Databricks-specific Hudi config to set. + +#### Cluster setup + +1. **Install the Hudi bundle as a cluster library.** In the cluster's **Libraries** tab, add + `org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>` as Maven coordinates, or + upload the jar directly. Pick the bundle that matches the Spark version your DBR release ships — see the + support matrix above. +2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These are the same four values the + quick start passes with `--conf`, in the `key value` form the Databricks UI expects: + + ``` + spark.serializer org.apache.spark.serializer.KryoSerializer + spark.sql.catalog.spark_catalog org.apache.spark.sql.hudi.catalog.HoodieCatalog + spark.sql.extensions org.apache.spark.sql.hudi.HoodieSparkSessionExtension + spark.kryo.registrator org.apache.spark.HoodieSparkKryoRegistrar + ``` + +3. Read as usual — no extra option is required: + + ```python + spark.read.format("hudi").load(basePath) + ``` + +#### What Hudi adapts, and in which release + +* **`FileStatusCache`** — DBR changed this API. `SparkHoodieTableFileIndex` checks reflectively for the + `putLeafFiles(Path, FileStatus[])` signature before using it and falls back when it is absent, rather than + failing with `NoSuchMethodError`. This guard lives in the shared Spark module and has been present since + 0.15.x, so it applies on the 1.0.x and 1.1.x lines as well. +* **`PartitionDirectory`** — DBR's Spark 3.4 runtime backports `FileStatusWithMetadata` from Spark 3.5, so + `PartitionDirectory` takes a `Seq[FileStatusWithMetadata]` rather than a `Seq[FileStatus]`. That type wraps + `FileStatus` by composition instead of extending it, so the elements cannot be cast; Hudi constructs the + wrapper reflectively. This adaptation lives in the Spark 3.4 module and **ships in 1.2.0**, so a DBR release + built on Spark 3.4 needs Hudi 1.2.0 or later. + +:::caution +Community guides for older Hudi releases tell you to set `hoodie.file.index.enable=false` when reading on +Databricks. That was the workaround before the adaptations above existed: it falls back from `HoodieFileIndex` +to `HoodieROTablePathFilter`, which sidesteps the incompatible Spark internals but disables the file index for +**every** table in the session, losing the listing optimisation it exists for. The config is also deprecated +(since 0.11.0). On 1.2.0 and later you should not need it; if a read still fails, it remains a fallback, and +please report the failure. +::: Review Comment: Same, mirrored in 5680690f4719: the file-index caution now says the flag is a no-op on 1.2.0, and the #16951 metadata-table caution with `hoodie.metadata.enable=false` is present in that copy as well. ########## website/versioned_docs/version-1.2.0/quick-start-guide.md: ########## @@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow instructions [here] The *default build* Spark version indicates how we build `hudi-spark3-bundle`. ::: +### Reading Hudi tables on the Databricks runtime + +The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a modified Spark, and a couple of the +internals Hudi's Spark datasource builds on differ there. Hudi detects those differences at runtime and adapts, +so there is no Databricks-specific Hudi config to set. + +#### Cluster setup + +1. **Install the Hudi bundle as a cluster library.** In the cluster's **Libraries** tab, add + `org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>` as Maven coordinates, or + upload the jar directly. Pick the bundle that matches the Spark version your DBR release ships — see the + support matrix above. +2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These are the same four values the + quick start passes with `--conf`, in the `key value` form the Databricks UI expects: + + ``` + spark.serializer org.apache.spark.serializer.KryoSerializer Review Comment: Same, mirrored in 5680690f4719: the versioned copy has the split-by-purpose configs and the Unity Catalog caution. Worth adding, since it is the part that was genuinely still wrong: that mirror replaced the section text in place but left it ahead of Spark Shell/SQL, so the copy the site actually serves still had the placement the earlier round asked to change. Just moved it to the end of Setup, and the two copies are now byte-identical through this section. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
