hey matei, most tasks have GC times of 200ms or less, and then a few tasks take many seconds. example GC activity for a slow one:
[GC [PSYoungGen: 1051814K->262624K(1398144K)] 3789259K->3524429K(5592448K), 0.0986800 secs] [Times: user=1.53 sys=0.01, real=0.10 secs] [GC [PSYoungGen: 786935K->524512K(1398144K)] 4048741K->4048762K(5592448K), 0.1132490 secs] [Times: user=1.70 sys=0.01, real=0.11 secs] [Full GC [PSYoungGen: 524512K->0K(1398144K)] [PSOldGen: 3524250K->2207344K(4194304K)] 4048762K->2207344K(5592448K) [PSPermGen: 56545K->54639K(83968K)], 7.7059350 secs] [Times:\ user=7.71 sys=0.00, real=7.70 secs] so looks like i am hit by stop-the-world gc? On Mon, Mar 10, 2014 at 7:00 PM, Koert Kuipers <ko...@tresata.com> wrote: > hey matei, > it happens repeatedly. > > we are currently runnning on java 6 with spark 0.9. > > i will add -XX:+PrintGCDetails and collect details, and also look into > java 7 G1. thanks > > > > > > > On Mon, Mar 10, 2014 at 6:27 PM, Matei Zaharia <matei.zaha...@gmail.com>wrote: > >> Does this happen repeatedly if you keep running the computation, or just >> the first time? It may take time to move these Java objects to the old >> generation the first time you run queries, which could lead to a GC pause >> that also slows down the small queries. >> >> If you can run with -XX:+PrintGCDetails in your Java options, it would >> also be good to see what percent of each GC generation is used. >> >> The concurrent mark-and-sweep GC -XX:+UseConcMarkSweepGC or the G1 GC in >> Java 7 (-XX:+UseG1GC) might also avoid these pauses by GCing concurrently >> with your application threads. >> >> Matei >> >> On Mar 10, 2014, at 3:18 PM, Koert Kuipers <ko...@tresata.com> wrote: >> >> hello all, >> i am observing a strange result. i have a computation that i run on a >> cached RDD in spark-standalone. it typically takes about 4 seconds. >> >> but when other RDDs that are not relevant to the computation at hand are >> cached in memory (in same spark context), the computation takes 40 seconds >> or more. >> >> the problem seems to be GC time, which goes from milliseconds to tens of >> seconds. >> >> note that my issue is not that memory is full. i have cached about 14G in >> RDDs with 66G available across workers for the application. also my >> computation did not push any cached RDD out of memory. >> >> any ideas? >> >> >> >
