hudi-agent commented on code in PR #19617: URL: https://github.com/apache/hudi/pull/19617#discussion_r3776362289
########## website/docs/quick-start-guide.md: ########## @@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow instructions [here] The *default build* Spark version indicates how we build `hudi-spark3-bundle`. ::: +### Reading Hudi tables on the Databricks runtime + +The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a modified Spark, and a couple of the +internals Hudi's Spark datasource builds on differ there. Hudi detects those differences at runtime and adapts, +so there is no Databricks-specific Hudi config to set. + +#### Cluster setup + +1. **Install the Hudi bundle as a cluster library.** In the cluster's **Libraries** tab, add + `org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>` as Maven coordinates, or + upload the jar directly. Pick the bundle that matches the Spark version your DBR release ships — see the + support matrix above. Review Comment: 🤖 The four configs are introduced as 'the same four values the quick start passes with --conf' needed to read. Since step 3 only exercises the DataFrame read path, only `spark.serializer`/`spark.kryo.registrator` are strictly relevant to reads (and even those are for Kryo performance, not correctness), while the catalog/extensions configs matter for Spark SQL. It could clarify which configs are required for a plain read vs. for Spark SQL usage, so DBR users don't set catalog/extensions unnecessarily. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## website/docs/quick-start-guide.md: ########## @@ -28,6 +28,61 @@ Hudi works with Spark 3.3 and above versions. You can follow instructions [here] The *default build* Spark version indicates how we build `hudi-spark3-bundle`. ::: +### Reading Hudi tables on the Databricks runtime + +The matrix above is for Apache Spark. The Databricks Runtime (DBR) ships a modified Spark, and a couple of the +internals Hudi's Spark datasource builds on differ there. Hudi detects those differences at runtime and adapts, +so there is no Databricks-specific Hudi config to set. + +#### Cluster setup + +1. **Install the Hudi bundle as a cluster library.** In the cluster's **Libraries** tab, add + `org.apache.hudi:hudi-spark<spark.version>-bundle_<scala.version>:<hudi.version>` as Maven coordinates, or + upload the jar directly. Pick the bundle that matches the Spark version your DBR release ships — see the + support matrix above. +2. **Set Hudi's Spark configs** in the cluster's **Spark config** box. These are the same four values the + quick start passes with `--conf`, in the `key value` form the Databricks UI expects: + + ``` + spark.serializer org.apache.spark.serializer.KryoSerializer + spark.sql.catalog.spark_catalog org.apache.spark.sql.hudi.catalog.HoodieCatalog Review Comment: 🤖 It might help to add a caveat here about Unity Catalog. Setting `spark.sql.catalog.spark_catalog` to `HoodieCatalog` at the cluster level overrides the session catalog, and most current DBR clusters run with Unity Catalog enabled, which reserves `spark_catalog` and generally rejects/conflicts with replacing it. As written, a user who copies all four configs onto a UC-enabled cluster could break UC access or fail to start the session, even though they only wanted to *read* a Hudi table. Could we either note that this override is only needed for Hudi Spark SQL / catalog operations (not for the DataFrame read shown in step 3), or add a caution for UC-enabled clusters? @yihua could you confirm the recommended Databricks cluster-config guidance here, particularly the spark_catalog override under Unity Catalog? <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
