Re: Error:java.lang.RuntimeException: java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE

See slides starting with slide #25 of http://www.slideshare.net/cloudera/top-5-mistakes-to-avoid-when-writing-apache-spark-applications FYI On Wed, Feb 24, 2016 at 7:25 PM, xiazhuchang wrote: > When cache data to memory, the code DiskStore$getBytes will be called. If > there is

Error:java.lang.RuntimeException: java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE

When cache data to memory, the code DiskStore$getBytes will be called. If there is a big data, the code "channel.map(MapMode.READ_ONLY, offset, length)" will be called, and the "map" function's parameter "length" has a type of "long", but the valid range is "Integer". This results in the error: