GitHub user danny0405 added a comment to the discussion: Spark query and write serving endpoint for hudi-lakehouse
I would start with **Option A (Spark Thrift Server) for v1**, while keeping the Hudi-enabled image and catalog/storage configuration reusable for Spark Connect later. If remote DataFrame applications are a day-one requirement, my preference would change to **Option B**. To make the proposal easier to read and respond to, I suggest: 1. **Lead with the user problem and the decision requested:** which clients need a persistent endpoint, which operations they need, and whether v1 must support remote DataFrame APIs. Move the architecture-layer explanation after the proposed scope. 2. **Separate v1 requirements from future extensions.** Propose an initial client/operation set and ask the community to adjust it. Keep Flight SQL and Substrait in future work until there is a concrete consumer. 3. **Use one comparison table with consistent dimensions:** client coverage, gateway integration, isolation, operational cost, and compatibility gaps. Consolidate shared validation concerns into one checklist. 4. **Treat Option D as an incremental expansion strategy.** It combines the other endpoints rather than representing a separate server choice. Similarly, distinguish the client-protocol decision from the engine-management/isolation decision that motivates Kyuubi. 5. **Focus the community questions on required clients, required Hudi operations, and required isolation.** These answers should determine the implementation. My reasons for preferring A initially are: - #19411 already proposes gateway integration through Spark Thrift Server. Although that PR is still open, deploying the server would complement a concrete integration path and should reduce the first release's scope. - The listed operations—table creation, `INSERT`, `UPDATE`, `DELETE`, and `MERGE INTO`—can be expressed through SQL. These requirements alone do not establish a need for remote DataFrame APIs; Hudi correctness still needs to be validated through the chosen endpoint. - ADBC does not require choosing Connect or implementing Flight SQL. The [linked Spark ADBC driver](https://adbc-drivers.org/drivers/spark/v0.1.0.html) supports both Thrift and Connect backends. We should state support in terms of a tested driver/backend/version combination, rather than assuming driver availability establishes Hudi write compatibility. One scope clarification would help: **a write-capable Spark endpoint does not automatically make the agent gateway write-capable**. #19411 retains SELECT-only guardrails. Are the proposed writes intended for direct clients, or should this work also include gateway write tools? For validation, I would explicitly include external-commit visibility, concurrent writers, and the outcome of a write when the client times out or cancels. Session separation and engine resource isolation should also be evaluated separately. I would choose **B** first if notebooks or applications need remote DataFrame transformations immediately, and **C** if separately managed tenant engines are required from the beginning. Otherwise, one optional Thrift endpoint, a pinned Hudi/Spark combination, and a query/write quickstart seem like a focused first deliverable, with additional endpoints introduced as client demand is demonstrated. This is an architectural preference; I have not validated Hudi compatibility against these server options. GitHub link: https://github.com/apache/hudi/discussions/19843#discussioncomment-18340900 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
