andygrove commented on issue #2599: URL: https://github.com/apache/datafusion-ballista/issues/2599#issuecomment-6417006956
> 3\. I'd like to pick this up. My plan is two `pub(crate)` helpers in `shuffle_writer.rs`. `take_handoff(name, &mut initialized, &mut handoffs, partition, spawn_coordinator)` does the first-call channel setup, hands the K senders to the closure, and returns the receiver for `partition` with the existing "called twice or out of range" error. `handoff_stream(name, rx, work_dir, job_id, stage_id)` wraps that receiver and `summaries_to_batch` in the `RecordBatchStreamAdapter`. > They take the two fields rather than `WriterState`, since `SortShuffleWriterExec` has its own state with `column_null_counts`. Each `execute` ends up as the lock plus those two calls, and the error strings still name the exec. > I'd keep the two `run_coordinator`s and the stats/`ShuffleWriter` impls out of this PR, because the sort coordinator also does the JoinSet fan-out and null-count folding. Any objections to that scope? Sounds good. Thanks @bharadwaj-pendyala. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
