zhli1142015 opened a new pull request, #11101: URL: https://github.com/apache/incubator-gluten/pull/11101
## What changes are proposed in this pull request? Fix overflow of pageNumber in VeloxSortShuffleWriter. In our testing of sort shuffle, we found that when the page number exceeded 8191, the compactRowId became corrupted, which caused a crash during data writing. This PR fixes this issue. ``` --------------- T H R E A D --------------- Stack: [0x00007e51b5400000,0x00007e51b5500000], sp=0x00007e51b54f8ef0, free space=995k Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libgluten.so+0x254a63d] LZ4_read_ARCH+0xc C [libgluten.so+0x257df2e] LZ4_compress_fast_continue+0x101d9 C [libgluten.so+0x2530f27] LZ4F_compressBlock_continue+0x5b C [libgluten.so+0x2530d5f] LZ4F_makeBlock+0x65 C [libgluten.so+0x25313b7] LZ4F_compressUpdateImpl+0x2f7 C [libgluten.so+0x25315ec] LZ4F_compressUpdate+0x4c C [libgluten.so+0x20fb4f3] arrow::util::internal::(anonymous namespace)::LZ4Compressor::Compress(long, unsigned char const*, long, unsigned char*)+0x1eb C [libgluten.so+0x17c7c35] gluten::ShuffleCompressedOutputStream::Write(void const*, long)+0x15f C [libgluten.so+0x17e4c18] gluten::InMemoryPayload::serialize(arrow::io::OutputStream*)+0xaa C [libgluten.so+0x17c989c] gluten::LocalPartitionWriter::LocalSpiller::spill(unsigned int, std::unique_ptr<gluten::InMemoryPayload, std::default_delete<gluten::InMemoryPayload> >)+0x18c C [libgluten.so+0x17c67ef] gluten::LocalPartitionWriter::sortEvict(unsigned int, std::unique_ptr<gluten::InMemoryPayload, std::default_delete<gluten::InMemoryPayload> >, bool)+0x41b C [libvelox.so+0x81a94ec] gluten::VeloxSortShuffleWriter::evictPartitionInternal(unsigned int, unsigned char*, long)+0x1e8 C [libvelox.so+0x81a909e] gluten::VeloxSortShuffleWriter::evictPartition(unsigned int, unsigned long, unsigned long)+0x3f4 C [libvelox.so+0x81a8970] gluten::VeloxSortShuffleWriter::evictAllPartitions()+0x2e2 C [libvelox.so+0x81a6b7a] gluten::VeloxSortShuffleWriter::stop()+0x74 C [libgluten.so+0x1795fd1] Java_org_apache_gluten_vectorized_ShuffleWriterJniWrapper_stop+0x12d ``` ## How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
