cshuo opened a new issue, #20620:
URL: https://github.com/apache/hudi/issues/20620

   ### Task Description
   
   Implement the Docker Compose packaging portion of RFC-117 so users can run a 
local Hudi lakehouse without installing a host JDK, Maven, Kubernetes, or Helm.
   
   Package the existing agent gateway and Spark Connect endpoint with Hive 
Metastore, MinIO, and sample data. Preserve the current catalog and query path:
   
   Chat / MCP → Gateway → Spark Connect → HMS-backed HoodieCatalog → Hudi data 
in MinIO
   
   This task retains the existing read-only gateway tools and does not 
introduce a catalog adapter or semantic layer.
   
   ### Scope
   
   - Add a Compose stack containing:
     - MinIO for Hudi storage and a one-shot bucket initialization job.
     - Hive Metastore with persistent Derby metadata.
     - Spark Connect running in local mode with HoodieCatalog.
     - A one-shot Python Connect client to seed and verify the trips/drivers 
tables.
     - The agent gateway exposing the chat UI, HTTP API, and MCP endpoint.
     - An optional Ollama service for local inference.
   - Add an installation script with Docker/Compose and resource checks, 
startup ordering, readiness checks, diagnostics, and lifecycle commands.
   - Support host Ollama, containerized Ollama, and configured API providers.
   - Provide an MCP-only installation mode that does not require a gateway LLM 
or API key.
   - Keep the base Compose file image-based and provide a separate opt-in 
source-build override.
   - Bind published ports to localhost and persist table data, metastore 
metadata, and downloaded models in named volumes.
   - Document service roles, configuration, client connections, and data 
lifecycle.
   
   ### User Experience
   
   From a repository checkout:
   
   ```bash
   ./hudi-lakehouse/docker/lakehouse.sh --build --mcp-only
   ```
   
   For built-in chat using containerized Ollama:
   
   ```bash
   ./hudi-lakehouse/docker/lakehouse.sh --build --local-llm
   ```
   
   Default endpoints:
   
   - Chat UI: http://localhost:8000/ui/
   - MCP: http://localhost:8000/mcp/
   - Spark Connect: sc://localhost:15002
   - MinIO console: http://localhost:9001
   
   The initial prerequisite is Docker Engine/Desktop with Compose 2.27.2 or 
newer. Source builds run inside Docker.
   
   ### Acceptance Criteria
   
   - [ ] A clean installation starts successfully and creates readable Hudi 
sample tables.
   - [ ] MCP table discovery, schema inspection, and SQL queries work through 
Spark Connect.
   - [ ] A configured local LLM can answer a sample question using the SQL tool.
   - [ ] Missing local models and unavailable providers produce actionable 
readiness failures.
   - [ ] Repeated installation preserves existing sample rows without 
introducing duplicates.
   - [ ] Container recreation without reseeding preserves tables, partitions, 
and data.
   - [ ] HMS can recover a missing initialization marker by validating the 
existing schema; invalid databases are preserved and startup fails.
   - [ ] Cached Ollama models are reused on subsequent installations.
   - [ ] Automated regression tests and an opt-in Docker acceptance workflow 
cover the installation lifecycle.
   
   ### Boundaries and Follow-ups
   
   This issue delivers the checkout-based Compose installation and contributor 
build path. Default application image tags are local build tags.
   
   Publishing release images, hosting versioned downloads, and providing the 
public `hudi.apache.org/lakehouse.sh` bootstrap require separate 
release-infrastructure work tracked by the umbrella issue.
   
   Catalog abstractions, semantic models, optional BI/Trino profiles, and 
Kubernetes installation changes are outside this task.
   
   ### Related Issues
   
   Parent feature issue: #20619
   RFC PR: https://github.com/apache/hudi/pull/20575
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to