Setting spark.driver.memory has no effect. It's still hanging trying to compute result.count when I'm sampling greater than 35% regardless of what value of spark.driver.memory I'm setting.
Here's my settings: export SPARK_JAVA_OPTS="-Xms5g -Xmx10g -XX:MaxPermSize=10g" export SPARK_MEM=10g in conf/spark-defaults: spark.driver.memory 1500 spark.serializer org.apache.spark.serializer.KryoSerializer spark.kryoserializer.buffer.mb 500 spark.executor.memory 58315m spark.executor.extraLibraryPath /root/ephemeral-hdfs/lib/native/ spark.executor.extraClassPath /root/ephemeral-hdfs/conf -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-akka-frameSize-stalls-job-in-1-1-0-tp7865p7877.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
