Hi I am getting the above error in Spark SQL . I have increase (using 5000 ) number of partitions but still getting the same error .
My data most probably is skew. org.apache.spark.shuffle.FetchFailedException: Too large frame: 4247124829 at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:419) at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:349)