andygrove opened a new pull request, #2871:
URL: https://github.com/apache/datafusion-ballista/pull/2871

   # Which issue does this PR close?
   
   Closes #2870.
   
   # Rationale for this change
   
   `stream_sort_shuffle_block` reads the schema header and the partition 
through a file handle and its `try_clone()`. A cloned handle shares its file 
cursor, so seeking to the partition also moves the header read. Every 
`IO_BLOCK_TRANSPORT` fetch of a sort-shuffle partition came back as the 
partition followed by `header_end` stray bytes from after it, instead of the 
header followed by the partition.
   
   Decoders happened to tolerate this, because each partition stream carries 
its own schema and `StreamDecoder` drops a trailing partial message when the 
input ends. A stricter reader rejects the response, which is how this surfaced 
in #2809. The bug dates from #1615.
   
   # What changes are included in this PR?
   
   - `stream_sort_shuffle_block` opens the data file twice, so the header and 
the partition each have their own cursor.
   - A test writes a small data file and index and checks the exact bytes 
served for each partition. It fails on `main`: partition 0 comes back as bytes 
`10..30` instead of `0..10` followed by `10..20`.
   
   # Are there any user-facing changes?
   
   No. The bytes served for a sort-shuffle partition over block transport are 
now exactly what the index describes.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to