wankunde opened a new pull request, #11944: URL: https://github.com/apache/gluten/pull/11944
## What changes are proposed in this pull request? Bug fix for VeloxHashShuffleWriter. Description: * VeloxHashShuffleWriter::write() method will try to convert the input velox flat vectors into arrow buffer. * For StringView vector, VeloxHashShuffleWriter will calculate the total size of all input StringView, and initialize the arrow buffer. * But for null values, we don't really need to copy the dirty values to arrow buffers, and can save the execution memory. Test logs in our productions: ```log I20260415 17:50:28.520978 5474 VeloxHashShuffleWriter.cc:156] collectFlatVectorBufferStringView for VARCHAR I20260415 17:50:28.521070 5474 VeloxHashShuffleWriter.cc:110] DEBUG, lengthBufferSize = 40, flatVector->size() = 10 I20260415 17:50:28.521078 5474 VeloxHashShuffleWriter.cc:114] DEBUG flatVector = [FLAT VARCHAR: 10 elements, 9 nulls] data: 0: null 1: null 2: null 3: 114535255 4: null 5: null 6: null 7: null 8: null I20260415 17:50:28.521092 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 0, isNull = true, rawValues[i].size() = 1957213616, length = 0, offset = 0 I20260415 17:50:28.521100 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 1, isNull = true, rawValues[i].size() = 2588080704, length = 0, offset = 0 I20260415 17:50:28.521106 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 2, isNull = true, rawValues[i].size() = 2706762760, length = 0, offset = 0 I20260415 17:50:28.521111 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 3, isNull = false, rawValues[i].size() = 9, length = 9, offset = 9 I20260415 17:50:28.521117 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 4, isNull = true, rawValues[i].size() = 0, length = 0, offset = 9 I20260415 17:50:28.521122 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 5, isNull = true, rawValues[i].size() = 1, length = 0, offset = 9 I20260415 17:50:28.521127 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 6, isNull = true, rawValues[i].size() = 2832791872, length = 0, offset = 9 I20260415 17:50:28.521132 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 7, isNull = true, rawValues[i].size() = 2588222376, length = 0, offset = 9 I20260415 17:50:28.521138 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 8, isNull = true, rawValues[i].size() = 0, length = 0, offset = 9 I20260415 17:50:28.521143 5474 VeloxHashShuffleWriter.cc:120] DEBUG flatVector dump, i = 9, isNull = true, rawValues[i].size() = 0, length = 0, offset = 9 ``` ## How was this patch tested? Tested in out production environment. ## Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
