I found my task takes so long time for YoungGen GC, I set the young gen size to about 1.5G, I wonder why it takes so long time? not all the tasks take such long time, only about 1% tasks so long...
180.426: [GC [PSYoungGen: 9916105K->1676785K(14256640K)] 26201020K->18690057K(53403648K), 17.3581500 secs] [Times: user=104.48 sys=152.56, real=17.36 secs] 198.986: [GC [PSYoungGen: 10724337K->1664837K(14881280K)] 27737609K->20205003K(54028288K), 55.0331460 secs] [Times: user=47.67 sys=776.67, real=55.02 secs] 255.339: [GC [PSYoungGen: 11605317K->1550792K(14632960K)] 30145483K->21580959K(53779968K), 187.7893060 secs] [Times: user=110.10 sys=2704.33, real=187.76 secs] 444.569: [GC [PSYoungGen: 11491272K->988109K(15197696K)] 31521439K->22370228K(54344704K), 366.2677820 secs] [Times: user=78.76 sys=5193.95, real=366.21 secs] 812.450: [GC [PSYoungGen: 11704781K->1087351K(15092736K)] 33086900K->23295623K(54239744K), 163.0328770 secs] [Times: user=97.71 sys=2134.11, real=163.01 secs] 977.207: [GC [PSYoungGen: 11804023K->1058228K(15470592K)] 34012295K->24294801K(54617600K), 176.5011980 secs] [Times: user=106.06 sys=1700.11, real=176.47 secs] 1155.439: [GC [PSYoungGen: 12288948K->1248832K(15333888K)] 35525521K->25495802K(54480896K), 123.5496760 secs] [Times: user=59.09 sys=940.54, real=123.53 secs] 1280.796: [GC [PSYoungGen: 12479552K->1097796K(15785472K)] 36726522K->26527988K(54932480K), 157.1727550 secs] [Times: user=11.14 sys=1376.93, real=157.15 secs] 1439.923: [GC [PSYoungGen: 12953668K->1178863K(15644160K)] 38383860K->27637944K(54791168K), 119.3690200 secs] [Times: user=107.54 sys=937.40, real=119.35 secs] 1561.236: [GC [PSYoungGen: 13034735K->1190849K(16093696K)] 39493816K->28787207K(55240704K), 159.1091770 secs] [Times: user=237.16 sys=1653.59, real=159.08 secs] 1722.406: [GC [PSYoungGen: 13651905K->1245193K(15941120K)] 41248263K->29995801K(55088128K), 168.0231330 secs] [Times: user=81.20 sys=2203.72, real=168.00 secs] 1892.731: [GC [PSYoungGen: 13706249K->1161132K(16381952K)] 42456857K->31117445K(55528960K), 162.0598100 secs] [Times: user=126.67 sys=1689.32, real=162.04 secs] -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/why-my-YoungGen-GC-takes-so-long-time-tp21941.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org