[GitHub] [spark] otterc commented on pull request #39725: [SPARK-33573][FOLLOW-UP] Increment ignoredBlockBytes when shuffle push blocks are late or colliding

otterc commented on PR #39725: URL: https://github.com/apache/spark/pull/39725#issuecomment-1404061436 If we don't want to include failed writes, then I think your suggestion requires much more refactoring of the current code. I thought it was slightly riskier to make those changes just bec

[GitHub] [spark] otterc commented on pull request #39725: [SPARK-33573][FOLLOW-UP] Increment ignoredBlockBytes when shuffle push blocks are late or colliding

otterc commented on PR #39725: URL: https://github.com/apache/spark/pull/39725#issuecomment-1404040435 > Sum up the total number of bytes received in `onData` - does not matter if it was written, deferred, etc. > In onComplete/onFailure, add it to ignored bytes if we are deciding not to