Tr this spark.storage.memoryFraction 0.9 On 31 Oct 2014 20:27, "akhandeshi" <ami.khande...@gmail.com> wrote:
> Thanks for the pointers! I did tried but didn't seem to help... > > In my latest try, I am doing spark-submit local > > But see the same message in spark App ui (4040) > localhost CANNOT FIND ADDRESS > > In the logs, I see a lot of in-memory map to disk. I don't understand why > that is the case. There should be over 35 gb ram available for input that > is not significantly large. It may be link to the performance issues that > I > am seeing. I have another post for seeing advice on that. It seems, I am > not able to tune spark sufficiently to execute my process successfully. > > 14/10/31 13:45:12 INFO ExternalAppendOnlyMap: Thread 206 spilling in-memory > map of 1777 MB to disk (2 times so > far) > 14/10/31 13:45:12 INFO ExternalAppendOnlyMap: Thread 228 spilling in-memory > map of 393 MB to disk (1 time so f > ar) > 14/10/31 13:45:12 INFO ExternalAppendOnlyMap: Thread 259 spilling in-memory > map of 392 MB to disk (2 times so > far) > 14/10/31 13:45:14 INctorsBySecId();^ZFO ExternalAppendOnlyMap: Thread 230 > spilling in-memory map of 554 MB to > disk (2 times so far) > 14/10/31 13:45:15 INFO ExternalAppendOnlyMap: Thread 235 spilling in-memory > map of 3990 MB to disk (1 time so > far) > 14/10/31 13:45:15 INFO ExternalAppendOnlyMap: Thread 236 spilling in-memory > map of 2667 MB to disk (1 time so > far) > 14/10/31 13:45:17 INFO ExternalAppendOnlyMap: Thread 259 spilling in-memory > map of 825 MB to disk (3 times so > far) > 14/10/31 13:45:24 INFO ExternalAppendOnlyMap: Thread 228 spilling in-memory > map of 4618 MB to disk (2 times so > far) > 14/10/31 13:45:26 INFO ExternalAppendOnlyMap: Thread 233 spilling in-memory > map of 15869 MB to disk (1 time so > far) > 14/10/31 13:45:37 INFO ExternalAppendOnlyMap: Thread 206 spilling in-memory > map of 3026 MB to disk (3 times so > far) > 14/10/31 13:45:48 INFO ExternalAppendOnlyMap: Thread 228 spilling in-memory > map of 401 MB to disk (3 times so > far) > 14/10/31 13:45:48 INFO ExternalAppendOnlyMap: Thread 259 spilling in-memory > map of 392 MB to disk (4 times so > > My spark properties are: > > Name Value > spark.akka.frameSize 50 > spark.akka.timeout 900 > spark.app.name Simple File Merge Application > spark.core.connection.ack.wait.timeout 900 > spark.default.parallelism 10 > spark.driver.host spark-single.c.fi-mdd-poc.internal > spark.driver.memory 35g > spark.driver.port 40255 > spark.eventLog.enabled true > spark.fileserver.uri http://10.240.106.135:59255 > spark.jars file:/home/ami_khandeshi_gmail_com/SparkExample-1.0.jar > spark.master local[16] > spark.scheduler.mode FIFO > spark.shuffle.consolidateFiles true > spark.storage.memoryFraction 0.3 > spark.tachyonStore.baseDir /tmp > spark.tachyonStore.folderName spark-21ad0fd2-2177-48ce-9242-8dbb33f2a1f1 > spark.tachyonStore.url tachyon://mdd:19998 > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/CANNOT-FIND-ADDRESS-tp17637p17824.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
