RexXiong commented on code in PR #3479:
URL: https://github.com/apache/celeborn/pull/3479#discussion_r2357378252
##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/PushDataHandler.scala:
##########
@@ -1514,6 +1515,25 @@ class PushDataHandler(val workerSource: WorkerSource)
extends BaseMessageHandler
hardSplitIndexes: Array[Int] = Array.empty[Int]): Unit = {
val length = fileWriters.length
val result = new Array[StatusCode](length)
+
+ var finalBody: ByteBuf = body
Review Comment:
Copying and quickly releasing buffers can indeed reduce direct memory usage.
However, I prefer to perform this in the PartitionWriter before generating the
FlushTask, because after the FlushTask is completed, the chunk memory for that
part can be completely released. With the current PR, the memory cannot be
released until all related tasks are finished. In addition, I recommend copying
the buffer only after the direct memory exceeds a certain threshold, so that
when the system is not under heavy pressure, we can reduce unnecessary CPU
consumption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
