L. C. Hsieh created SPARK-59908:
-----------------------------------
Summary: Sync the vendored Spark Connect protos to v4.2.0
Key: SPARK-59908
URL: https://issues.apache.org/jira/browse/SPARK-59908
Project: Spark
Issue Type: Sub-task
Components: Connect
Affects Versions: connect-gateway-0.1.0
Reporter: L. C. Hsieh
Assignee: L. C. Hsieh
SPARK-59856 recorded that the vendored Spark Connect protos were behind
upstream and added dev/sync-protos.sh, but deliberately left the drift in
place: re-syncing moves the protocol the gateway speaks, so it wanted its own
review. This does that sync.
dev/sync-protos.sh --sync --ref v4.2.0
The result is +31 lines in relations.proto and +42 in pipelines.proto, with no
deletions -- the drift was one-directional, so closing it is purely additive:
relations.proto the NearestByJoin message, and its entry in the
Relation.rel_type oneof (tag 47)
pipelines.proto the AutoCdcFlowDetails message, and the import of
spark/connect/expressions.proto it needs
v4.2.0 was chosen because it is the newest released Spark tag. v4.3.0 exists
only as release candidates; syncing to an rc would pin the gateway to a
protocol that can still change before release. For reference, the tree is 4
files away from v4.3.0-rc1, so there will be another sync to make once 4.3.0
ships.
Verified beyond the build, because a compiling binding is weak evidence for a
protocol change:
- The vendored files are byte-identical to upstream v4.2.0, all 11 of 11,
compared file by file rather than trusting the script that wrote them.
- cargo build -p scg-genproto regenerates the bindings, and the two new
messages are present in the generated Rust as real structs -- not merely
compiled past.
- The new rel_type variant cannot break an exhaustive match: grepping the
workspace outside crates/genproto for rel_type / RelType returns nothing.
The one place that does inspect message contents,
crates/proxy/src/config_filter.rs, matches on
ConfigRequest.operation.op_type, which this change does not touch.
- End to end against a real Spark backend on kind, deployed via the Helm
chart: AnalyzePlan returns the correct schema for a range plan and
ExecutePlan streams 15 response messages back, with the gateway's audit log
confirming both RPCs passed through it. The client encoded its requests from
the newly synced .proto files, which is what makes this a test of the sync
rather than of the gateway alone -- and it confirms the v4.2.0 protocol
remains compatible with a Spark 4.0 backend.
- cargo test --workspace is 204 passed, 0 failed, 10 ignored, matching main;
clippy --workspace --all-targets -D warnings and cargo fmt --check are
clean.
proto/PROVENANCE.md is updated to match the new state. The revision is now
recorded as an exact match rather than a diff baseline, and the uncertainty the
old text carried -- that the previous copy came from an untagged 4.2
development commit that no release tag matched -- moves into a History section,
since it still documents where the original snapshot came from. The
explanation of why protocol drift degrades gracefully is kept, rewritten as
general guidance for judging how urgent a future sync is rather than as a
description of this particular gap. UPSTREAM_REF in dev/sync-protos.sh is
unchanged at v4.2.0; its comment is corrected, as it now names the revision the
files are at rather than one to diff against.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
