cshuo opened a new issue, #20234: URL: https://github.com/apache/hudi/issues/20234
## Summary Provide an optional Helm-deployable Spark Connect endpoint for Hudi queries and writes, with ADBC access from hudi-agent-gateway, Superset, and Zeppelin. This issue tracks the implementation discussed in [#19843](https://github.com/apache/hudi/discussions/19843). Local testing demonstrated Hudi reads and writes through Foundry ADBC with Superset and Zeppelin; the next step is to package the endpoint and integrations and validate them through Helm. ## Scope - Provide a Hudi-enabled Spark Connect image and Helm chart, reusing the lakehouse catalog and object storage. - Integrate hudi-agent-gateway through ADBC while preserving its existing tool interfaces and SELECT-only guardrails. - Provide Superset and Zeppelin examples supporting direct queries and writes. - Add repeatable end-to-end validation and a complete quickstart. Direct-client write coverage includes table creation, INSERT, UPDATE, DELETE, and MERGE INTO. Agent gateway write tools are out of scope. ## Sub-issues - [ ] #20231 — Add a Helm-deployable Spark Connect endpoint - [ ] #20232 — Integrate hudi-agent-gateway and BI clients with Spark Connect - [ ] #20233 — Add end-to-end validation and documentation for Spark Connect ## Completion Criteria - A clean local Kubernetes environment can deploy the endpoint using the quickstart. - Agent/MCP queries work through Gateway → ADBC → Spark Connect → Hudi, with read-only guardrails enforced. - Superset and Zeppelin can query and write Hudi data, with result readback and rendering. - Tests cover the scoped SQL operations, external-commit visibility, and persistence across endpoint restarts. - Documentation records tested versions, setup, cleanup, and known limitations. ## Follow-up Work Broader JDBC compatibility, gateway write tools, and production-scale security, concurrency, and isolation validation are separate follow-up work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
