Re: [PR] [SPARK-48644][SQL] Do a length check and throw COLLECTION_SIZE_LIMIT_EXCEEDED error in Hex.hex [spark]

[via GitHub]

yaooqinn commented on code in PR #47001:
URL: https://github.com/apache/spark/pull/47001#discussion_r1643683853


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala:
##########
@@ -1032,9 +1032,19 @@ object Hex {
     array
   }
 
+  private def maxArraySize(length: Long): Int = {
+    if (length > Int.MaxValue) {
+      throw QueryExecutionErrors.tooManyArrayElementsError(length, 
Int.MaxValue)

Review Comment:
   
   Hi @dongjoon-hyun 
   ```scala
    org.apache.spark.sql.catalyst.expressions.Hex.hex((" " * (Int.MaxValue / 2 
+ 1)).getBytes)
   org.apache.spark.SparkIllegalArgumentException: 
[COLLECTION_SIZE_LIMIT_EXCEEDED.INITIALIZE] Can't create array with 2147483648 
elements which exceeding the array size limit 2147483647, cannot initialize an 
array with specified parameters. SQLSTATE: 54000
     at 
org.apache.spark.sql.errors.QueryExecutionErrors$.tooManyArrayElementsError(QueryExecutionErrors.scala:2517)
     at 
org.apache.spark.sql.catalyst.expressions.Hex$.hex(mathExpressions.scala:1042)
     ... 42 elided
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org