Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/22105#discussion_r210842394
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/protocol/MessageWithHeader.java
---
@@ -140,8 +140,24 @@ private int copyByteBuf(ByteBuf buf,
WritableByteChannel target) throws IOExcept
// SPARK-24578: cap the sub-region's size of returned nio buffer to
improve the performance
// for the case that the passed-in buffer has too many components.
int length = Math.min(buf.readableBytes(), NIO_BUFFER_LIMIT);
--- End diff --
```
IIRC socket buffers are 32k by default on Linux, so it seems unlikely you'd
be able to write 256k in one call (ignoring what IOUtil does internally). But
maybe in practice it works ok.
```
After reading the context in #12083 and this discussion, I want to provide
a possibility about 256k in one call can work in practice. As in our scenario,
user will change `/proc/sys/net/core/wmem_default` based on their online
behavior, generally we'll set this value larger than `wmem_default`.
So maybe 256k of NIO_BUFFER_LIMIT is ok here? We just need add more
annotation to remind what params related with this value.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
