L. C. Hsieh created SPARK-59349:
-----------------------------------
Summary: Override the gRPC authority for UDF worker Unix socket
channels
Key: SPARK-59349
URL: https://issues.apache.org/jira/browse/SPARK-59349
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
{{GrpcWorkerChannel}} builds its gRPC channel without calling
{{overrideAuthority}}, so gRPC derives the HTTP/2 {{:authority}} pseudo-header
from the target address. For a Unix domain socket that address is the socket
path, which is not a valid HTTP authority (and is percent-encoded where the
path needs escaping, e.g. {{var%2Ffolders%2F...%2Frw.sock}}).
A conforming HTTP/2 server validates the pseudo-headers while decoding the
HEADERS frame and resets the stream:
{noformat}
h2::server: malformed headers: malformed authority (...): invalid authority
h2::proto::streams::send: send_reset(..., reason=PROTOCOL_ERROR,
initiator=Library, ...)
{noformat}
The client sees only:
{noformat}
io.grpc.StatusRuntimeException: INTERNAL: RST_STREAM closed stream. HTTP/2
error code: PROTOCOL_ERROR
{noformat}
The rejection happens before any worker application code runs -- not the
service method, not an interceptor -- so the worker logs nothing and Init is
never observed. The failure points at the wrong side of the connection, and the
worker has no way to report why.
This is not reachable through any in-tree code path today. grpc-java's own
server tolerates the malformed authority, and both Spark's client and the only
in-tree worker ({{EchoGrpcWorkerMain}}) are grpc-java, so the pair works and
the header is never validated. That is why it went unnoticed in SPARK-56922.
It does affect any worker written against a conforming HTTP/2 stack, which is
most non-JVM stacks. worker_spec.proto invites workers in any language and the
SPARK-55278 SPIP names onboarding a Go/Rust/Swift worker as its final exam;
such a worker hits this on its very first RPC. Found while building a Rust
worker, and confirmed three ways: the same client succeeds against the same
worker binary when only the authority changes; driving the worker through
Spark's own DirectGrpcDispatcher reproduces the identical PROTOCOL_ERROR (so it
is not a grpc-python quirk); and the server-side h2 trace names the rejected
header.
Fix: call {{overrideAuthority}} on the channel builder with a placeholder
authority. A placeholder is correct rather than a workaround -- for a Unix
socket peer the authority carries no information, because the socket path
already identifies the peer completely. There is no name to resolve and no
virtual host to select.
Note this cannot be fixed on the worker side: a tonic/tower layer rewriting the
request URI runs after header decode, by which point the stream is already
reset.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
