"only option is to split you problem further by increasing parallelism" My understanding is by increasing the number of partitions, is that right? That didn't seem to help because it is seem the partitions are not uniformly sized. My observation is when I increase the number of partitions, it creates many "empty block partitions" and may larger partition is not broken down into smaller size. Any hints, on how I can get uniform partitions. I noticed many threads, but was not able to do any thing effective from Java api. I will appreciate any help/insight you can provide.
-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-OutOfMemoryError-Requested-array-size-exceeds-VM-limit-tp16809p19097.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
