hudi-agent commented on code in PR #19617: URL: https://github.com/apache/hudi/pull/19617#discussion_r3947962474
########## website/docs/quick-start-guide.md: ########## @@ -158,6 +158,104 @@ basePath = "file:///tmp/trips_table" </Tabs > +### Reading Hudi tables on the Databricks runtime + +The Spark support matrix at the top of this page is for Apache Spark. The Databricks Runtime (DBR) ships a +modified Spark, and several of the internals Hudi's Spark datasource builds on differ there. Hudi detects those +differences at runtime and adapts, so there is no Databricks-specific Hudi config to set. Review Comment: 🤖 This opening claim reads as unconditional — "Hudi detects those differences at runtime and adapts, so there is no Databricks-specific Hudi config to set" — but the rest of the section contradicts it in two ways that matter for correctness: 1. The adaptation table below states the `PartitionDirectory` / `FileStatusWithMetadata` handling is **absent in 1.0.2 and 1.1.1**, so a user on those releases with a DBR built on Spark 3.4 will hit a hard cast failure, not a graceful adaptation. 2. The metadata-table caution further down *does* prescribe a Databricks-specific option (`hoodie.metadata.enable=false`) as the current workaround for #16951. A reader who stops after this paragraph will conclude any Hudi version works on any DBR with no options, and only find out otherwise from a stack trace. It might help to scope the sentence to the versions the table confirms, e.g. "On 1.2.0 and later, Hudi detects those differences at runtime and adapts; see the table below for which adaptations landed in which release, and the cautions for the cases that still need an option." <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## website/src/pages/ecosystem.md: ########## @@ -31,7 +31,7 @@ In such cases, you can leverage another tool like Apache Spark or Apache Flink t | Google DataProc | [Read + Write](https://cloud.google.com/blog/products/data-analytics/getting-started-with-new-table-formats-on-dataproc) | | | Azure Synapse | [Read + Write](https://www.onehouse.ai/blog/apache-hudi-on-microsoft-azure) | | | Azure HDInsight | [Read + Write](https://www.onehouse.ai/blog/apache-hudi-on-microsoft-azure) | | -| Databricks | [Read + Write](https://hudi.apache.org/docs/azure_hoodie/) | | +| Databricks | [Read](https://hudi.apache.org/docs/quick-start-guide#reading-hudi-tables-on-the-databricks-runtime) | | Review Comment: 🤖 This changes the Databricks row from "Read + Write" to "Read" and drops the link to `azure_hoodie`. The new quick-start section is read-focused, but the existing Databricks recipe in `azure_hoodie.md` (which this PR keeps, and which now links back here) documents writing Hudi tables from Databricks. Narrowing the ecosystem entry to "Read" reads as a statement that writes are no longer supported on Databricks, which is a stronger claim than the PR verifies. Would it be better to keep "Read + Write" and point the link at the new section (or list both links), so the matrix does not silently retract a documented capability? <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
