deepakpanda93 opened a new pull request, #19627: URL: https://github.com/apache/hudi/pull/19627
### Describe the issue this Pull Request addresses Closes #16626. (JIRA: [HUDI-8226](https://issues.apache.org/jira/browse/HUDI-8226).) Hudi has recognised the GooseFS `gfs` scheme for years, but the **Storage Configurations** section never documented it. Every other supported cloud filesystem has a page — S3, GCS, OSS, Azure, COS, IBM COS, BOS, JuiceFS, OCI, KS3 — and GooseFS had none, so nothing told a reader which filesystem implementations to register or which client jar to put on the classpath. Confirmed the gap rather than assuming it: `goosefs` appears nowhere under `website/docs`, `website/versioned_docs` or `website/learn`, and `git log --diff-filter=A -- '*goosefs*'` returns nothing, so no such page has ever existed on the site. ### Why the previous attempt did not land There is a prior attempt, #4018, opened in 2021 and closed unmerged in 2024. It is worth being explicit about what went wrong there, since this PR is meant to be the version that does not repeat it: - It was **written in Chinese** (`language: cn` in the front matter). @leesf asked "would you use english please?" - The same review asked "and here not just spark job?", objecting to framing the page around Spark specifically. - Earlier feedback told the author to place it under `website/docs/` and to update `cloud.md`. - The author replied once and then never revised. @leesf and @nsivabalan both followed up; the PR sat for three years. This page is in English, is written engine-neutrally ("your Hudi jobs" rather than "your Hudi spark job"), and updates `cloud.md` and both sidebars — none of which the original did. ### Summary and Changelog Six files: | File | Change | |---|---| | `website/docs/goosefs_hoodie.md` | **new** | | `website/versioned_docs/version-1.2.0/goosefs_hoodie.md` | **new**, byte-identical (md5 `9c55ed389557`) | | `website/docs/cloud.md` | one list entry, `last_modified_at` bumped per `AGENTS.md` | | `website/versioned_docs/version-1.2.0/cloud.md` | same | | `website/sidebars.js` | `'goosefs_hoodie'` after `'ks3_hoodie'` | | `website/versioned_sidebars/version-1.2.0-sidebars.json` | same, edited as JSON (a no-op round-trip was checked byte-identical first, so the diff is two lines rather than a reformat) | The page follows the existing `cos_hoodie.md` / `ks3_hoodie.md` template: front matter, a short intro, a "two configurations required" list, the `core-site.xml` block, and a Libs section. ### Every value verified, not carried over | Documented | Verified against | |---|---| | the `gfs` scheme is genuinely supported | `StorageSchemes.GOOSEFS("gfs", null, null, null)` — present in all five 1.x tags, so this documents real behaviour rather than an aspiration | | `fs.gfs.impl` = `com.qcloud.cos.goosefs.hadoop.FileSystem` | Tencent product documentation (matched #4018) | | `fs.AbstractFileSystem.gfs.impl` = `com.qcloud.cos.goosefs.hadoop.GooseFileSystem` | Tencent product documentation (matched #4018) | | base path `gfs://<master>:9200/...` | Tencent docs; 9200 is the default master RPC port | | `com.qcloud.cos:goosefs-client:1.4.9.1` | Maven Central. #4018 pinned `1.1.0`, which is five years stale | **One claim was dropped before it reached the page**, and I would rather record it than quietly omit it. Maven Central's class index lists the two Hadoop classes only for `goosefs-client` 1.3.0 and later, which would have meant #4018 told users to configure classes its own recommended jar did not contain — a tidy finding, and wrong. Downloading the 1.1.0 and 1.3.0 jars shows `GooseFileSystem.class` and `FileSystem.class` in **both**, so that was a gap in the index, not a real absence. No such claim appears in the page. ### One caveat no sibling storage page carries `GOOSEFS` is declared with `supportAtomicCreation = null`, so `isAtomicCreationSupported("gfs")` returns false and `FileSystemBasedLockProvider` refuses to start: ```java if (!customSupportedFSs.contains(this.storage.getScheme()) && !StorageSchemes.isAtomicCreationSupported(this.storage.getScheme())) { throw new HoodieLockException("Unsupported scheme :" + this.storage.getScheme() + ", since this fs can not support atomic creation"); } ``` The page documents the escape hatch, `hoodie.fs.atomic_creation.support=gfs`, which exists on all 1.x releases (since 0.14.0) and is read as a comma-separated list of schemes. This is the sort of thing a user would otherwise only discover by hitting the exception. Relatedly, `fs.defaultFS` is presented as **optional** rather than required: Tencent does not prescribe it for GooseFS, and a fully qualified `gfs://` base path is sufficient. #4018 set it unconditionally. ### Version scope `next` and 1.2.0, so the default `/docs` serves it immediately rather than only `/docs/next`. For the record, the established precedent for adding a *new* cloud storage page is `website/docs` only — commit `ca1d36b71d90`, which introduced `ks3_hoodie.md`, touched no `versioned_docs`, which is why `ks3_hoodie.md` is absent from 1.0.x today. Including 1.2.0 here is a deliberate step beyond that so the live default docs pick it up; happy to drop the versioned copy if reviewers would rather stick strictly to the precedent. `docusaurus.config.js` is deliberately untouched. Its footer "Hudi On Cloud" list is one entry per cloud *provider*, Tencent already points at `cos_hoodie`, and KS3, BOS and JuiceFS are not in it either. ### Verification `npm run build` passes with the warning block **byte-identical** to a baseline built from the same base commit (`d11a5b0adee4`), 13,265 lines both ways. Worth noting the one intermediate result, since it looked like a regression: before committing, the build reported one extra warning, `Cannot infer the update date for some files, as they are not tracked by git`. That is `showLastUpdateTime: true` reacting to the two new files having no git history yet — it survives `git add` and clears on commit, which the post-commit build confirms. It was not a content problem. Rendering confirmed on `/docs/goosefs_hoodie` and `/docs/next/goosefs_hoodie`: both `fs.*.impl` values, the `gfs://` form, port 9200, the Maven coordinate, `hoodie.fs.atomic_creation.support` and the note admonition all present. `cloud.md` links the page on both versions, the sidebar entry appears, the `concurrency_control` link resolves per-version (`/docs/concurrency_control` and `/docs/next/concurrency_control`), and `/docs/1.1.1/goosefs_hoodie` correctly does not exist. **Limitation, stated plainly:** I do not have a GooseFS cluster, so this is verified by reading `StorageSchemes` and `FileSystemBasedLockProvider`, by confirming the class names inside the published `goosefs-client` jar, and against Tencent's own documentation — not by performing a live read/write against GooseFS. Someone with a GooseFS deployment confirming the `core-site.xml` block end to end would be a genuine improvement on that. ### Impact Documentation only. No code, config, or behaviour change. ### Risk Level none ### Documentation Update This PR is the documentation update — a new Storage Configurations page, `/docs/goosefs_hoodie` and `/docs/next/goosefs_hoodie`, linked from `/docs/cloud`. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
