cshuo commented on issue #20605:
URL: https://github.com/apache/hudi/issues/20605#issuecomment-7002357990

   I propose using **Spark Connect’s built-in bearer-token authentication**, 
with TLS terminated by a gRPC sidecar or an existing gRPC gateway. Spark 4.1.3 
already checks `Authorization: Bearer <token>` and returns `UNAUTHENTICATED` 
for a missing or invalid token. This gives every Spark client path the same 
authentication check and avoids maintaining a custom interceptor or token 
validation rules in the proxy. [Spark 
interceptor](https://github.com/apache/spark/blob/v4.1.3/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/PreSharedKeyAuthenticationInterceptor.scala#L22-L43)
   
   **Chart and secrets.** The chart would populate 
`SPARK_CONNECT_AUTHENTICATE_TOKEN` from an existing Kubernetes Secret. With the 
sidecar enabled, the chart would mount an existing TLS Secret in the sidecar, 
bind Spark to loopback, and expose only the sidecar’s TLS port through the 
Service. Operators with an existing gRPC gateway could terminate TLS there and 
forward the `authorization` header to a private Spark Service over a trusted or 
encrypted backend connection. Authentication and TLS would be opt-in so 
`quickstart.sh --spark-connect` continues to work with its defaults. [Spark 
token 
configuration](https://github.com/apache/spark/blob/v4.1.3/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala#L352-L357)
   
   **Client compatibility.** I’d verify the following before calling the 
feature ready for 2.0.
   
   - **Foundry ADBC 0.2.1** is the first check because both Superset and 
Zeppelin depend on it. The pinned driver’s tests cover `auth_type=none` and 
`auth_type=token` for `api=connect` and show how to pass credentials through 
ADBC `db_kwargs`. The successful upstream TLS test disables certificate 
verification. I’d verify that `SELECT 1` succeeds against our Helm deployment 
with `auth_type=token`, `tls=true`, and certificate verification enabled 
against a trusted CA. I’d flag any interoperability issue early so we can 
assess its impact on 2.0. [Foundry 0.2.1 authentication 
tests](https://github.com/adbc-drivers/spark/blob/go/v0.2.1/go/validation/tests/test_auth.py)
   - **Superset and Zeppelin** each need a live query with credentials supplied 
outside the saved URI or notebook. One detail from the current code: the 
Superset dialect defaults to `auth_type=none`, forwards an explicit 
`auth_type`, and rejects username/password in the URL. The Zeppelin live test 
explicitly requires `auth_type=none` because it stores the URI in the notebook. 
I’d verify passing Superset’s stored credential through `db_kwargs` and loading 
Zeppelin’s token from the interpreter environment at runtime.
   - **PySpark Connect and `hudi-agent-gateway`** each need an end-to-end query 
with `token`, `use_ssl=true`, and a trusted certificate. The chart’s readiness 
probe also opens a PySpark session, so I’d verify that it authenticates when 
the feature is enabled.
   
   **Tests and docs.** I’d add offline Helm tests for the default, sidecar, and 
external-gateway configurations, covering Secret references, bind addresses, 
and exposed ports. I’d extend the existing live workflow with an auth-enabled 
run that verifies a successful query, `UNAUTHENTICATED` for missing and 
incorrect tokens, and TLS failure for an untrusted certificate.
   
   The chart README would include a working token + TLS values example and 
explain the trust model: every token holder still uses the server’s storage 
credentials. The 2.0 known-limitations notes would list per-user identity, 
authorization, multi-tenant isolation, and access control for the Spark UI and 
History Server as follow-up work.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to