L. C. Hsieh created SPARK-61425:
-----------------------------------

             Summary: Frame artifact uploads in one place and reject a chunked 
artifact that changed size during its upload
                 Key: SPARK-61425
                 URL: https://issues.apache.org/jira/browse/SPARK-61425
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: L. C. Hsieh


In spark-connect-rust, crates/spark-connect-core/src/artifact.rs frames 
artifact uploads in three places: build_artifact_request_stream (public, 
eager), cache_artifact_requests (in-memory cache artifacts) and ArtifactFrames 
(the lazy frames of a file upload). Each has its own state machine for the same 
policy: batch small artifacts up to CHUNK_SIZE, and send a larger one as a 
BeginChunk followed by Chunks. They produce the same requests today, but their 
conditions are written differently and nothing keeps them the same.

A chunked artifact whose data is not of the size it declared is also not 
detected. The size is taken before the data is read, and total_bytes and 
num_chunks of the BeginChunk come from it. If the file is truncated in between 
and nothing can be read, the frames end before any BeginChunk: the server 
receives an empty stream, answers with success, and the upload returns Ok with 
nothing uploaded. If it is shorter or longer, the frames contradict their 
BeginChunk and no error is raised on the client.

Proposed: frame artifacts in one place, without changing the requests on the 
wire, and make a chunked artifact whose data ends before or goes on past its 
declared size an error that names the artifact and both sizes.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to