andygrove opened a new issue, #2870:
URL: https://github.com/apache/datafusion-ballista/issues/2870

   ### Describe the bug
   
   `stream_sort_shuffle_block` in `ballista/executor/src/flight_service.rs` 
serves a sort-shuffle partition over `IO_BLOCK_TRANSPORT` as the schema-header 
stream followed by the partition's byte range. It reads the two through a 
`tokio::fs::File` and its `try_clone()`, and seeks the clone to the partition's 
start.
   
   A cloned handle shares its file cursor with the original (it is a `dup` of 
the same open file), so the seek moves the header read too. The response is 
then `[start, end + header_end)` instead of `[0, header_end) + [start, end)`: 
no header, and `header_end` stray bytes from after the partition at the end.
   
   It has gone unnoticed because every partition stream starts with its own 
schema message, and the stray bytes at the end are a partial message that 
`BlockDataStream`'s `StreamDecoder` drops silently when the input ends.
   
   Introduced in #1615, so 53.0.0 and later are affected.
   
   ### To Reproduce
   
   Write a data file with bytes `0..30` and a `ShuffleIndex` with header `[0, 
10)`, partition 0 `[10, 20)` and partition 1 `[20, 30)`, then collect 
`stream_sort_shuffle_block(&data_path, 0)`. It returns bytes `10..30` instead 
of `0..10` followed by `10..20`.
   
   ### Expected behavior
   
   The response holds exactly the header bytes followed by the partition's 
bytes.
   
   ### Additional context
   
   It became visible while working on #2809, which makes `BlockDataStream` 
reject a stream that ends inside a message. With that check, every 
block-transport read of a sort-shuffle partition failed, and the stage was 
retried over and over.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to