Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22705#discussion_r224977082
  
    --- Diff: 
core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala ---
    @@ -195,7 +196,11 @@ object ChunkedByteBuffer {
         val is = new FileInputStream(file)
         ByteStreams.skipFully(is, offset)
         val in = new LimitedInputStream(is, length)
    -    val chunkSize = math.min(maxChunkSize, length).toInt
    +    // Though in theory you should be able to index into an array of size 
Int.MaxValue, in practice
    +    // jvms don't let you go up to limit.  It seems you may only need - 2, 
but we leave a little
    +    // extra room.
    +    val maxArraySize = Int.MaxValue - 512
    --- End diff --
    
    We already had seen the similar problem in other places. Can we use [this 
value](https://github.com/apache/spark/blob/master/common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java#L52)
 here, too?


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to