Github user Ngone51 commented on a diff in the pull request:
https://github.com/apache/spark/pull/22163#discussion_r212167438
--- Diff:
core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java ---
@@ -206,14 +211,21 @@ private void writeSortedFile(boolean isLastFile) {
long recordReadPosition = recordOffsetInPage + uaoSize; // skip over
record length
while (dataRemaining > 0) {
final int toTransfer = Math.min(diskWriteBufferSize,
dataRemaining);
- Platform.copyMemory(
- recordPage, recordReadPosition, writeBuffer,
Platform.BYTE_ARRAY_OFFSET, toTransfer);
- writer.write(writeBuffer, 0, toTransfer);
+ if (bufferOffset > 0 && bufferOffset + toTransfer >
DISK_WRITE_BUFFER_SIZE) {
--- End diff --
> The numRecordsWritten in DiskBlockObjectWriter is still correct during
the process after this PR
The number is correct, but it is not consistent with what real happen
compare to current behaviour. But as you said, we will get correct result at
the end. So, it may not be a big deal.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
