holdenk commented on a change in pull request #28708:
URL: https://github.com/apache/spark/pull/28708#discussion_r456062326
##########
File path:
core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala
##########
@@ -168,7 +168,10 @@ private[spark] class NettyBlockTransferService(
// Everything else is encoded using our binary protocol.
val metadata =
JavaUtils.bufferToArray(serializer.newInstance().serialize((level, classTag)))
- val asStream = blockData.size() >
conf.get(config.MAX_REMOTE_BLOCK_SIZE_FETCH_TO_MEM)
+ // We always transfer shuffle blocks as a stream for simplicity with the
receiving code since
+ // they are always written to disk. Otherwise we check the block size.
+ val asStream = (blockData.size() >
conf.get(config.MAX_REMOTE_BLOCK_SIZE_FETCH_TO_MEM) ||
Review comment:
I think it's easier to read as is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
