Thanks for the pointers! I did tried but didn't seem to help... In my latest try, I am doing spark-submit local
But see the same message in spark App ui (4040) localhost CANNOT FIND ADDRESS In the logs, I see a lot of in-memory map to disk. I don't understand why that is the case. There should be over 35 gb ram available for input that is not significantly large. It may be link to the performance issues that I am seeing. I have another post for seeing advice on that. It seems, I am not able to tune spark sufficiently to execute my process successfully. 14/10/31 13:45:12 INFO ExternalAppendOnlyMap: Thread 206 spilling in-memory map of 1777 MB to disk (2 times so far) 14/10/31 13:45:12 INFO ExternalAppendOnlyMap: Thread 228 spilling in-memory map of 393 MB to disk (1 time so f ar) 14/10/31 13:45:12 INFO ExternalAppendOnlyMap: Thread 259 spilling in-memory map of 392 MB to disk (2 times so far) 14/10/31 13:45:14 INctorsBySecId();^ZFO ExternalAppendOnlyMap: Thread 230 spilling in-memory map of 554 MB to disk (2 times so far) 14/10/31 13:45:15 INFO ExternalAppendOnlyMap: Thread 235 spilling in-memory map of 3990 MB to disk (1 time so far) 14/10/31 13:45:15 INFO ExternalAppendOnlyMap: Thread 236 spilling in-memory map of 2667 MB to disk (1 time so far) 14/10/31 13:45:17 INFO ExternalAppendOnlyMap: Thread 259 spilling in-memory map of 825 MB to disk (3 times so far) 14/10/31 13:45:24 INFO ExternalAppendOnlyMap: Thread 228 spilling in-memory map of 4618 MB to disk (2 times so far) 14/10/31 13:45:26 INFO ExternalAppendOnlyMap: Thread 233 spilling in-memory map of 15869 MB to disk (1 time so far) 14/10/31 13:45:37 INFO ExternalAppendOnlyMap: Thread 206 spilling in-memory map of 3026 MB to disk (3 times so far) 14/10/31 13:45:48 INFO ExternalAppendOnlyMap: Thread 228 spilling in-memory map of 401 MB to disk (3 times so far) 14/10/31 13:45:48 INFO ExternalAppendOnlyMap: Thread 259 spilling in-memory map of 392 MB to disk (4 times so My spark properties are: Name Value spark.akka.frameSize 50 spark.akka.timeout 900 spark.app.name Simple File Merge Application spark.core.connection.ack.wait.timeout 900 spark.default.parallelism 10 spark.driver.host spark-single.c.fi-mdd-poc.internal spark.driver.memory 35g spark.driver.port 40255 spark.eventLog.enabled true spark.fileserver.uri http://10.240.106.135:59255 spark.jars file:/home/ami_khandeshi_gmail_com/SparkExample-1.0.jar spark.master local[16] spark.scheduler.mode FIFO spark.shuffle.consolidateFiles true spark.storage.memoryFraction 0.3 spark.tachyonStore.baseDir /tmp spark.tachyonStore.folderName spark-21ad0fd2-2177-48ce-9242-8dbb33f2a1f1 spark.tachyonStore.url tachyon://mdd:19998 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/CANNOT-FIND-ADDRESS-tp17637p17824.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
