rangareddy opened a new pull request, #19617:
URL: https://github.com/apache/hudi/pull/19617
### Describe the issue this Pull Request addresses
Closes #15666 (HUDI-5498): *"We need to document how users can read Hudi
tables on Databricks Spark runtime."*
The Spark Support Matrix in the quick start covers Apache Spark only.
Nothing in the docs said what happens on
the **Databricks Runtime (DBR)**, which ships a modified Spark — so a user
hitting a `NoSuchMethodError` or a
`PartitionDirectory` signature mismatch there had no page to land on, and
the workaround that circulates for it
is a config that has been deprecated since 0.11.0.
Support itself has landed over several PRs (#13129, #18256, #18291/#18292,
#18003/#18258, #14059/#18257); only
the documentation was missing.
### Summary and Changelog
Adds a **Reading Hudi tables on the Databricks runtime** subsection to
`quick-start-guide.md`, immediately
after the Spark Support Matrix, stating what Hudi actually does — and the
headline is that **no Hudi-specific
configuration is needed**, because the adaptation is at runtime:
* **`FileStatusCache`** — DBR changed the API, so
`SparkHoodieTableFileIndex.adapt` checks reflectively for the
`putLeafFiles(Path, FileStatus[])` signature before using it and falls
back when absent, instead of failing
with `NoSuchMethodError`.
* **`PartitionDirectory`** — DBR's Spark 3.4 runtime backports
`FileStatusWithMetadata` from Spark 3.5, so
`PartitionDirectory` takes `Seq[FileStatusWithMetadata]`. That type wraps
`FileStatus` by composition rather
than extending it, so the elements cannot be cast and
`DatabricksRuntimeHelper` constructs the wrapper
reflectively.
### Verification
Docs change, so no test to add. Every claim was read off master
(`3ba31dd37fff`):
| claim | source |
| --- | --- |
| DBR changed `FileStatusCache`; guarded reflectively |
`SparkHoodieTableFileIndex.scala:674-690` — the comment says so outright, then
`ReflectionUtils.getMethod(classOf[FileStatusCache],
PUT_LEAF_FILES_METHOD_NAME, …)` gates the branch |
| DBR 3.4 backports `FileStatusWithMetadata`; `PartitionDirectory` signature
differs | `DatabricksRuntimeHelper.scala` javadoc and implementation |
| `FileStatusWithMetadata` does not extend `FileStatus`, so casts are
impossible | same file — it is why the helper reflects instead of casting |
| the helper is reached from the Spark 3.4 adapter |
`hudi-spark3.4.x/.../HoodieSpark34PartitionedFileUtils.scala:53` calls
`DatabricksRuntimeHelper.newPartitionDirectory` |
| no user-facing flag gates any of it | both paths are unconditional at
runtime; the only decision input is class/method presence |
**Two things called out deliberately.**
1. **The config the issue points at is a trap.** #15666 cites #7088
(HUDI-5104), which added
`hoodie.file.index.enable` to fall back from `HoodieFileIndex` to
`HoodieROTablePathFilter`. That config
carries `deprecatedAfter("0.11.0")` in `DataSourceOptions.scala:97-103`,
and it disables the file index for
every table in the session, losing the listing optimisation it exists
for. The page therefore warns against
reaching for it rather than presenting it as the answer.
2. **Version scope, checked at the tags rather than assumed.**
`DatabricksRuntimeHelper.scala` is present at
`release-1.2.0` and **absent** at `release-1.1.1` and `release-1.0.2`. So
the Spark 3.4 adaptation is
documented as shipping in **1.2.0**, and the section is applied to
`website/docs/` and
`versioned_docs/version-1.2.0/` **only**. Adding it to the 1.1.1 copy
would claim compatibility that
release does not have.
Markdown checked by running `markdownlint` over both files before and after,
comparing rule *classes* rather
than counts (this page already carries ten classes): **no new class of
finding**. The added region is
byte-identical between the two files, `:::` admonitions are balanced, and
nothing in the prose puts a `<` or
`{` outside inline code.
**Scope I deliberately did not claim:** individual DBR versions are not in
Hudi's CI matrix and I have no
Databricks environment, so the section says it documents the compatibility
Hudi *implements* rather than a
certified DBR version list, and asks for an issue with the DBR version and
stack trace on failure. I would
rather that be explicit than imply a support guarantee nobody tests.
**Not done:** the Docusaurus build (`website/node_modules` absent, full
install heavy). The section adds a
bullet list and two admonitions, both constructs already used on this page.
### Impact
Documentation only — no code, config or format change. Users on the
Databricks runtime get a definitive answer
(reads work, nothing to configure, 1.2.0+ for DBR Spark 3.4) and are steered
away from a deprecated config that
would quietly disable the file index across their session.
### Risk Level
none
### Documentation Update
This is the documentation update. Targets `asf-site`; applied to `docs/` and
`versioned_docs/version-1.2.0/`
for the version reason given above.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable — n/a for docs; verification
table above instead
- [ ] CI passes on my PR — `asf-site` PRs do not run the `master` gates;
`markdownlint` parity checked locally
as described
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
