[
https://issues.apache.org/jira/browse/SPARK-60942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-60942.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 164
[https://github.com/apache/spark-connect-rust/pull/164]
> Fix artifact request attribution and async hashing in Spark Connect Rust
> ------------------------------------------------------------------------
>
> Key: SPARK-60942
> URL: https://issues.apache.org/jira/browse/SPARK-60942
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: DB Tsai
> Assignee: DB Tsai
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Spark Connect Rust artifact upload frames are built with the default protobuf
> user_context and client_type, even when ChannelBuilder has a configured user
> ID and agent. gRPC headers carry these values, but servers that use protobuf
> request identity cannot associate uploads with the same user as artifact
> status and query requests. Every Batch, BeginChunk and Chunk frame needs the
> configured context. ArtifactStatusesRequest should carry the same client_type.
> cache_artifacts also computes SHA-256 for all IPC blobs synchronously before
> its first await. A large input can occupy an async worker for tens of
> milliseconds even with accelerated SHA. Hash owned blobs in one Tokio
> blocking task, then return the original allocations for lazy upload framing.
> Preserve hash order, duplicate handling, cache checks, CRC and
> acknowledgements.
> The shared Bytes slices should continue to avoid copying. Their source
> allocation is retained until its last slice is dropped; document this
> expected lifetime. The one-shot stream replay contract is documented
> separately in PR #163.
> Regression coverage should inspect every upload frame and status request for
> configured/default users, cover cache/file/named uploads, and
> deterministically prove hashing yields while retaining the original payload
> buffers.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
