GitHub user cshuo edited a discussion: Spark query and write serving endpoint for hudi-lakehouse
## Summary We would like to discuss adding a persistent Spark serving endpoint under `hudi-lakehouse`, providing interactive query and write access to Hudi tables. The initial lakehouse implementation provides Trino-based query serving and Spark writer examples. A shared Spark endpoint would expose Hudi's Spark SQL and DataFrame capabilities without requiring each client to configure and launch its own Spark application. Rather than simply adding a Spark Thrift Server chart, we should evaluate the available serving approaches and keep the design friendly to Spark Connect, ADBC, and potentially Substrait. The goal of this discussion is to collect requirements and compare options, not to require every interface in the first release. ## Goals - Provide an optional Spark serving component and Helm chart under `hudi-lakehouse`. - Support interactive queries and Hudi writes, including table creation, `INSERT`, `UPDATE`, `DELETE`, and `MERGE INTO`. - Support writes through direct Spark endpoint clients. Agent gateway write tools are out of scope. - Reuse the catalog and object-store configuration of the lakehouse deployment. The exact operation set and production requirements for the first version remain open for discussion. ## Architecture layers The relevant technologies address different layers and are not necessarily mutually exclusive: | Layer | Possible interfaces | |---|---| | Client API | JDBC/ODBC, Spark Connect clients, ADBC | | Transport | HiveServer2/Thrift, Spark Connect gRPC, Arrow Flight SQL | | Query/plan representation | SQL text, Spark logical plans, optional Substrait plans | | Execution | Spark SQL with the Hudi bundle and Spark SQL extensions | | Data and catalog | Hudi, HMS/Glue-compatible catalog, object storage | In particular: - Spark Thrift Server exposes SQL through HiveServer2 and provides broad JDBC/ODBC compatibility. - Spark Connect provides a decoupled Spark client/server model, using protobuf logical plans over gRPC and Arrow for result transfer. - ADBC is a client API rather than a server protocol. An ADBC client can connect through an ADBC driver for Spark, or through the ADBC Flight SQL driver when the server exposes a Flight SQL endpoint. - Substrait is a portable plan representation rather than a transport or server implementation. Supporting one does not automatically provide the others, but they are not necessarily mutually exclusive. ## Options to consider ### Option A: Spark Thrift Server Deploy Spark Thrift Server with the Hudi bundle, Spark SQL extensions, and shared catalog/storage configuration. Advantages: - Mature JDBC/ODBC and SQL-tool compatibility. - Straightforward deployment using an existing Spark server implementation. Concerns: - SQL-only interface; does not expose Spark Connect DataFrame APIs. - Does not directly address Flight SQL or Substrait. - Session isolation, metadata freshness, dependency configuration, and long-running server stability require careful validation. ### Option B: Spark Connect Server Deploy Spark Connect as the primary Spark endpoint. Advantages: - Supports remote SQL and DataFrame applications. - Provides client/server separation and Arrow-based result transport. - Can also serve JDBC/ADBC clients through a compatible Spark driver. Concerns: - ODBC clients need an additional frontend or adapter. - Hudi writes, SQL extensions, and Spark/client version compatibility must be validated specifically in Connect mode. - Authentication, session routing, and multi-tenant resource isolation require an explicit deployment design. ### Option C: Kyuubi or another Spark SQL gateway Use a gateway to manage Spark engines and client sessions. Advantages: - Established JDBC/ODBC integration. - Provides authentication, multi-tenancy, and engine lifecycle management. - Supports resource isolation through separately managed Spark engines. Concerns: - Adds another major runtime dependency and operational layer. - Spark Connect support and its maturity must be evaluated separately. - Does not automatically provide Flight SQL, ADBC compatibility, or Substrait support. ### Option D: Multi-protocol Spark serving Share a Hudi-enabled Spark image and catalog/storage configuration, while exposing independently enabled endpoints: - Spark Connect for remote SQL and DataFrame clients. - Optional Thrift/HS2 for JDBC/ODBC tools. - ADBC access through an existing [Spark ADBC driver](https://adbc-drivers.org/drivers/spark/v0.1.0.html), without requiring a new Flight SQL server. This option combines existing serving interfaces rather than introducing a new server implementation. Advantages: - Supports different client ecosystems without committing to a single protocol. - Reuses common runtime packaging and catalog/storage configuration. - Allows interfaces to be introduced incrementally using existing servers and drivers. Concerns: - Multiple endpoints increase deployment, testing, and maintenance costs. - Separate Spark applications increase runtime overhead and do not share sessions, temporary views, or caches. - Sharing one driver instead requires additional compatibility and lifecycle validation. - Flight SQL/Substrait adapters remain additional work. The [Spark Substrait Gateway](https://github.com/voltrondata/spark-substrait-gateway) is an architecture reference, not a drop-in Hudi solution. ## Initial validation Before selecting an approach, we should validate: - Hudi query and write correctness through direct clients. - Spark/Hudi/client version compatibility. - Concurrent sessions, cancellation, and visibility of external commits. - Authentication, resource isolation, and long-running stability. The first deliverable could then be one optional endpoint with a query/write quickstart, while keeping additional interfaces independently extensible. ## Questions for the community 1. **What should the first version support?** Which clients and workloads are essential: JDBC/ODBC tools, remote SQL, or Spark Connect DataFrame applications? Which Hudi write operations are required on day one? 2. **Which serving approach best fits these requirements?** Should we start with Spark Connect, Spark Thrift Server, or a gateway such as Kyuubi? Experience from existing deployments would be helpful. 3. **What isolation and security model is needed initially?** Is a shared Spark application with multiple sessions sufficient, or do we need per-user engines, stronger resource isolation, and multi-tenant authentication from the beginning? 4. **What level of ADBC support should we target?** Is a compatible driver or adapter sufficient initially, or is there a concrete need for an Arrow Flight SQL endpoint? 5. **What concrete Hudi use case would benefit from accepting Substrait plans?** Feedback on requirements, protocol preferences, and operational experience would be very helpful. ## Related work - #19256 — Agentic lakehouse initiative. - #19265 — Initial `hudi-lakehouse` implementation. - #19411 — Gateway connectivity to an existing Spark Thrift Server; this does not deploy the server itself. - #19484 — Native SQL DDL support across engines. - #15527 — Trino Hudi connector DDL/DML support. GitHub link: https://github.com/apache/hudi/discussions/19843 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
