Hi Spark users, We are running Spark on Yarn and often query table partitions as big as 100~200 GB from hdfs. Hdfs is co-located on the same cluster on which Spark and Yarn run. I've noticed a much higher I/O read rates when I increase the number of executors cores from 2 to 8( Most tasks run in RACK_LOCAL and few in NODE_LOCAL) while keeping the #executors constant. The ram on my executor is around 24G. But the problem is that any subsequent shuffle stage starts failing if I do that. It runs fine if i leave the number of executors to 2 but then the read is much slower. The errors which I get from Yarn is
org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output > location for shuffle 1 > at > org.apache.spark.MapOutputTracker$$anonfun$org$apache$spark$MapOutputTracker$$convertMapStatuses$2.apply(MapOutputTracker.scala:460) It seem to indicate Yarn is killing executors for using too much memory but I can't be sure. I tried increasing the spark sql shuffle partitions as well but that didn't help much. Is there a way we can run more partition read tasks per executor but keep the #shuffle tasks still constant. Thanks
