LuciferYang commented on code in PR #13548:
URL: https://github.com/apache/gluten/pull/13548#discussion_r4536919942
##########
cpp/velox/shuffle/VeloxHashShuffleWriter.cc:
##########
@@ -229,12 +229,9 @@ arrow::Result<std::shared_ptr<arrow::Buffer>>
VeloxHashShuffleWriter::generateCo
const facebook::velox::IndexRange allRows{0, vector->size()};
serializer->append(vector, folly::Range(&allRows, 1));
auto serializedSize = serializer->maxSerializedSize();
- auto flushBuffer = complexTypeFlushBuffer_[0];
Review Comment:
Thanks @marin-ma. I tried `auto& flushBuffer` at both sites, and it corrupts
shuffle data. I ran a query with an `array` and a `map` column through 12
shuffle setups (single partition, hash, and hash with
`spark.gluten.sql.columnar.shuffle.partitionBufferEvictThreshold=1`, each with
and without a codec and with 50- and 1024-row batches). 6 of them fail in the
shuffle reader with Presto deserialization errors such as `Received corrupted
serialized page`, and one run crashed the JVM. The current code and this PR
pass all 12.
The failing cases are the evicts with `reuseBuffers=false`: the
single-partition path through `generateComplexTypeBuffers`, and the evict after
split. There `LocalPartitionWriter` caches the payload without copying it
unless it compresses it, which needs a codec and at least
`spark.gluten.sql.columnar.shuffle.compression.threshold` rows (100 by
default). The cached `BlockPayload` still points into the flush buffer, which
the next flush overwrites or reallocates.
Caching the buffer only when `reuseBuffers` is true, where the payload is
copied or compressed before `hashEvict` returns, also passes all 12. It saves
one allocation per flush, but each partition then holds its flush buffer
between evicts, and `reclaimFixedSize` only frees it if that partition still
has unflushed complex rows. So I lean towards removing it. If you'd rather keep
the reuse, I can switch this PR to the conditional version and make spill
release the buffers. I'd like to hear what you and @jinchengchenghh think
before changing it.
##########
cpp/velox/shuffle/VeloxHashShuffleWriter.cc:
##########
@@ -1172,13 +1161,10 @@
arrow::Result<std::vector<std::shared_ptr<arrow::Buffer>>> VeloxHashShuffleWrite
}
}
if (hasComplexType_ && complexTypeData_[partitionId] != nullptr) {
- auto flushBuffer = complexTypeFlushBuffer_[partitionId];
Review Comment:
Same as in
https://github.com/apache/gluten/pull/13548#discussion_r4527330782. This is the
site the hash shuffle cases go through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]