Smallfu666 commented on issue #11215: URL: https://github.com/apache/gluten/issues/11215#issuecomment-5278814957
Heads-up for whoever implements this: the sort / RssSort shuffle writers silently lose rows on a 0-column schema, so enabling empty-schema columnar shuffle has to route around them. Reproducer, on main with the 0-column path opened up (Velox `supportEmptySchemaColumnarShuffle()` returning true and the matching `outputAttributes.isEmpty` / `child.output.isEmpty` rejections in `VeloxValidatorApi` removed): ``` spark.gluten.sql.columnar.shuffle.sort.columns.threshold=0 spark.range(0, 10000).select().repartition(4).collect().length // 0, expected 10000 ``` Zero rows back, no exception. Same with `spark.gluten.sql.columnar.shuffle.sort.partitions.threshold=1` and 30000 rows over 9 partitions: 0 of 30000. The plan shows `shuffle_writer_type=sort`; forcing the hash writer when `output.isEmpty` restores correct counts in both cases. Worth noting `getShuffleWriterType` tests `output.size >= columnarShuffleSortColumnsThreshold`, so `0 >= 0` picks the sort writer even with the partitions threshold at its 4000 default — one user-settable config reaches it once the schema check is lifted. We have not established the cause. Changing the `bytesRead_ > 0` EOS check in `VeloxSortShuffleReaderDeserializer::next()` to a cached-rows check and rebuilding still returned 0 rows, so the reader predicate is not it. This is a reproducible symptom, not a diagnosis. Nothing is at risk on main today — the validator rejects 0-column shuffle first. Flagging it because it lands the moment this issue is implemented. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
