Ngone51 commented on a change in pull request #32287: URL: https://github.com/apache/spark/pull/32287#discussion_r620777695
########## File path: core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala ########## @@ -683,7 +694,28 @@ final class ShuffleBlockFetcherIterator( } } - case FailureFetchResult(blockId, mapIndex, address, e) => + // Catching OOM and do something based on it is only a workaround for handling the + // Netty OOM issue, which is not the best way towards memory management. We can + // get rid of it when we find a way to manage Netty's memory precisely. + case FailureFetchResult(blockId, mapIndex, address, size, isNetworkReqDone, e) + if e.isInstanceOf[OutOfDirectMemoryError] || e.isInstanceOf[NettyOutOfMemoryError] => + assert(address != blockManager.blockManagerId && + !hostLocalBlocks.contains(blockId -> mapIndex), + "Netty OOM error should only happen on remote fetch requests") + logWarning(s"Failed to fetch block $blockId due to Netty OOM, will retry", e) + NettyUtils.isNettyOOMOnShuffle = true + numBlocksInFlightPerAddress(address) = numBlocksInFlightPerAddress(address) - 1 + bytesInFlight -= size + if (isNetworkReqDone) { + reqsInFlight -= 1 + logDebug("Number of requests in flight " + reqsInFlight) + } + val defReqQueue = + deferredFetchRequests.getOrElseUpdate(address, new Queue[FetchRequest]()) + defReqQueue.enqueue(FetchRequest(address, Array(FetchBlockInfo(blockId, size, mapIndex)))) Review comment: > When it is not a NettyManagedBuffer then can there still be a OutOfDirectMemoryError? It's possible. In general, for a remote request, it must rely on Netty to receive the data sent from the server, which of course consumes Netty Memory. So it doesn't matter whatever the final stored type is, e.g., `NettyManagedBuffer` or `FileSegmentManagedBuffer`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org