cshuo opened a new pull request, #20621: URL: https://github.com/apache/hudi/pull/20621
### Describe the issue this Pull Request addresses Closes #20620. Part of #20619 and RFC-117 (#20575). Add a Docker Compose installation for users who want to try the Hudi lakehouse without setting up a host JDK, Maven, Kubernetes, or Helm. From a checkout, `./hudi-lakehouse/docker/lakehouse.sh --build --mcp-only` starts the stack, loads sample data, and verifies the MCP query path. ### Summary and Changelog - Package MinIO, persistent Derby-backed Hive Metastore, Spark Connect, sample-data initialization, and the existing agent gateway. Preserve the Spark Connect → HMS-backed HoodieCatalog query path and read-only gateway tools. - Add installer preflight checks, readiness checks, diagnostics, and lifecycle commands. Support host Ollama, optional containerized Ollama with model caching, and API providers. - Keep image consumption in `compose.yaml` and opt-in source builds in `compose.build.yaml`. Consolidate HMS and S3A configuration in `hive-site.xml` and document each service's role. - Preserve data across container recreation, make sample initialization repeatable, and recover an initialized HMS schema when its completion marker is missing. - Add regression tests and an opt-in Docker live acceptance workflow for MCP queries, persistence without reseeding, and repeated installation. ### Impact Adds a local Docker installation with localhost-bound Chat UI, MCP, Spark Connect, and MinIO console endpoints. Existing Helm charts, gateway interfaces, and the Hudi storage format are unchanged. This is the checkout-based Compose installation and contributor build path. Publishing release images, versioned downloads, the public bootstrap endpoint, catalog adapters, and semantic-layer integration remain separate work under the umbrella issue. Default application image tags are local build tags, not published ASF artifacts. ### Risk Level Low: changes are confined to the new Docker installation, its documentation, and tests. The local demo uses fixed storage credentials and unauthenticated services bound to localhost; it is not a production deployment. Validation completed locally: - All 28 Docker regression tests passed; shell syntax and diff checks passed. - Fresh image builds and installation with new data volumes passed. - MCP discovery, schema inspection, and joins through Spark Connect/HMS/MinIO passed. - Real local Ollama chat (`qwen3:0.6b`) invoked the SQL tool and returned expected city totals of 33, 55, and 77. - HMS loaded S3A settings from `hive-site.xml` and created, inspected, and deleted a test directory in MinIO. - Container recreation without seeding, missing-schema-marker recovery, cached-model reuse, and repeated installation passed; sample counts remained 9 trips and 3 drivers. The new GitHub Actions workflow has not yet been run on GitHub. Hosted-provider authentication/inference and the default 8B model were not exercised by these local acceptance runs. ### Documentation Update Add the Docker installation guide, environment example, and Compose service comments, and link the new installation path from the lakehouse README. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
