marin-ma commented on code in PR #11083:
URL:
https://github.com/apache/incubator-gluten/pull/11083#discussion_r2576006246
##########
backends-velox/src-celeborn/main/scala/org/apache/spark/shuffle/VeloxCelebornColumnarShuffleWriter.scala:
##########
@@ -77,7 +77,7 @@ class VeloxCelebornColumnarShuffleWriter[K, V](
val columnarBatchHandle =
ColumnarBatches.getNativeHandle(BackendsApiManager.getBackendName,
cb)
val startTime = System.nanoTime()
- shuffleWriterJniWrapper.write(
+ val bytesWritten = shuffleWriterJniWrapper.write(
Review Comment:
I think `VeloxUniffleColumnarShuffleWriter` also needs to be updated. It's
using the same native partition writer as celeborn.
##########
backends-velox/src/main/scala/org/apache/spark/shuffle/ColumnarShuffleWriter.scala:
##########
@@ -223,6 +223,7 @@ class ColumnarShuffleWriter[K, V](
dep.metrics("numInputRows").add(rows)
dep.metrics("inputBatches").add(1)
// This metric is important, AQE use it to decide if EliminateLimit
+ writeMetrics.incBytesWritten(bytesWritten)
Review Comment:
Please move the comment above this line to below.
##########
cpp/core/shuffle/ShuffleWriter.cc:
##########
@@ -61,8 +61,8 @@ int32_t ShuffleWriter::numPartitions() const {
return numPartitions_;
}
-int64_t ShuffleWriter::totalBytesWritten() const {
- return metrics_.totalBytesWritten;
+int64_t ShuffleWriter::bytesWritten() const {
Review Comment:
Can we simply modify the signature of`ShuffleWriter::write` and
`ShuffleWriter::stop` to return the bytes evicted as `Result<int64_t>`? And
seems like this value is not updated and used for local shuffle write, please
add some explanation that only rss shuffle write returns the evicted bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
