[GitHub] [spark] goutam-git commented on a diff in pull request #37065: [SPARK-38699][SQL] Use error classes in the execution errors of dictionary encoding

[GitBox]

goutam-git commented on code in PR #37065:
URL: https://github.com/apache/spark/pull/37065#discussion_r922831090


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/compression/compressionSchemes.scala:
##########
@@ -421,7 +421,7 @@ private[columnar] case object DictionaryEncoding extends 
CompressionScheme {
 
     override def compress(from: ByteBuffer, to: ByteBuffer): ByteBuffer = {
       if (overflow) {

Review Comment:
   @MaxGekk
   
       I used the DictionaryEncodingSuite and found that in case of big enough 
Dictionary the users face java.lang.IllegalArgumentException  from 
ByteBuffer.allocate in build method in CompressibleColumnBuilder as a result of 
which they never reach encoder.compress  which throws  
useDictionaryEncodingWhenDictionaryOverflowError
   Please advice next steps



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org