HyukjinKwon commented on code in PR #37140: URL: https://github.com/apache/spark/pull/37140#discussion_r917222344
########## sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala: ########## @@ -3029,7 +3029,10 @@ object SQLConf { " in memory, otherwise do a global sort which spills to disk if necessary.") .version("2.4.0") .intConf - .createWithDefault(ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH) + .checkValue(_ <= ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH, + "The top-K sort fallback threshold should be less than or equal to " + + s"${ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH}.") + .createWithDefault(800000) Review Comment: I would pick this number 1024 * N e.g., `655360` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org