viirya commented on PR #58097: URL: https://github.com/apache/spark/pull/58097#issuecomment-5361867213
Thanks, @dongjoon-hyun. Both fixed in a new commit. The writer now checks `killTaskIfInterrupted` each cycle, same as the reader `takeItem`. And the rendezvous is now keyed by a per-run epoch (the jobId), so a re-run of the same shuffleId uses physically separate queues -- a straggler writer from an aborted run cannot cross into the next run, and a re-run never drains leftover batches or counts leftover end-of-stream markers. That let me delete the between-run mark reset (and the `onPipelinedProducerStageSubmit` hook) entirely, since there is nothing to clear once runs are isolated. Replies inline. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
