LuciferYang commented on code in PR #13548:
URL: https://github.com/apache/gluten/pull/13548#discussion_r4536919942


##########
cpp/velox/shuffle/VeloxHashShuffleWriter.cc:
##########
@@ -229,12 +229,9 @@ arrow::Result<std::shared_ptr<arrow::Buffer>> 
VeloxHashShuffleWriter::generateCo
   const facebook::velox::IndexRange allRows{0, vector->size()};
   serializer->append(vector, folly::Range(&allRows, 1));
   auto serializedSize = serializer->maxSerializedSize();
-  auto flushBuffer = complexTypeFlushBuffer_[0];

Review Comment:
   Thanks @marin-ma. I tried `auto& flushBuffer` at both sites, and it corrupts 
shuffle data. I ran a query with an `array` and a `map` column through 12 
shuffle setups (single partition, hash, and hash with 
`spark.gluten.sql.columnar.shuffle.partitionBufferEvictThreshold=1`, each with 
and without a codec and with 50- and 1024-row batches). 6 of them fail in the 
shuffle reader with Presto deserialization errors such as `Received corrupted 
serialized page`, and one run crashed the JVM. The current code and this PR 
pass all 12.
   
   The failing cases are the evicts with `reuseBuffers=false`: the 
single-partition path through `generateComplexTypeBuffers`, and the evict after 
split. There `LocalPartitionWriter` caches the payload without copying it 
unless it compresses it, which needs a codec and at least 
`spark.gluten.sql.columnar.shuffle.compression.threshold` rows (100 by 
default). The cached `BlockPayload` still points into the flush buffer, which 
the next flush overwrites or reallocates.
   
   Caching the buffer only when `reuseBuffers` is true, where the payload is 
copied or compressed before `hashEvict` returns, also passes all 12. It saves 
one allocation per flush, but each partition then holds its flush buffer 
between evicts, and `reclaimFixedSize` only frees it if that partition still 
has unflushed complex rows. So I lean towards removing it. If you'd rather keep 
the reuse, I can switch this PR to the conditional version and make spill 
release the buffers. I'd like to hear what you and @jinchengchenghh think 
before changing it.
   



##########
cpp/velox/shuffle/VeloxHashShuffleWriter.cc:
##########
@@ -1172,13 +1161,10 @@ 
arrow::Result<std::vector<std::shared_ptr<arrow::Buffer>>> VeloxHashShuffleWrite
     }
   }
   if (hasComplexType_ && complexTypeData_[partitionId] != nullptr) {
-    auto flushBuffer = complexTypeFlushBuffer_[partitionId];

Review Comment:
   Same as in 
https://github.com/apache/gluten/pull/13548#discussion_r4527330782. This is the 
site the hash shuffle cases go through.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to