Increase the number of executors, :-)
At 2015-08-26 16:57:48, "Ted Yu" <yuzhih...@gmail.com> wrote: Mind sharing how you fixed the issue ? Cheers On Aug 26, 2015, at 1:50 AM, Todd <bit1...@163.com> wrote: Sorry for the noise, It's my bad...I have worked it out now. At 2015-08-26 13:20:57, "Todd" <bit1...@163.com> wrote: I think the answer is No. I only see such message on the console..and #2 is the thread stack trace。 I am thinking is that in Spark SQL Perf forks many dsdgen process to generate data when the scalafactor is increased which at last exhaust the JVM When thread exception is thrown on the console and I leave it there for some while(15min about),then eventually I will see OutOfMemory occur Can you guys try to run it if you have the environment ? I think you may reproduce it. Thanks! At 2015-08-26 13:01:34, "Ted Yu" <yuzhih...@gmail.com> wrote: The error in #1 below was not informative. Are you able to get more detailed error message ? Thanks On Aug 25, 2015, at 6:57 PM, Todd <bit1...@163.com> wrote: Thanks Ted Yu. Following are the error message: 1. The exception that is shown on the UI is : Exception in thread "Thread-113" Exception in thread "Thread-126" Exception in thread "Thread-64" Exception in thread "Thread-90" Exception in thread "Thread-117" Exception in thread "Thread-80" Exception in thread "Thread-115" Exception in thread "ResponseProcessor for block BP-1564562096-172.18.149.132-1435294011279:blk_1073846767_105984" Exception in thread "qtp1270119920-57" Exception in thread "Thread-77" Exception in thread "Thread-132" Exception in thread "Thread-68" Exception in thread "Thread-61" Exception in thread "Thread-70" Exception in thread "qtp1270119920-52" Exception in thread "Thread-88" Exception in thread "qtp318933312-47" Exception in thread "qtp1270119920-56" 2. jstack the process, I see bunch of following message: Thread 31258: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=503 (Interpreted frame) - java.lang.UNIXProcess.waitFor() @bci=8, line=263 (Interpreted frame) - scala.sys.process.ProcessImpl$SimpleProcess.exitValue() @bci=4, line=218 (Interpreted frame) - scala.sys.process.ProcessBuilderImpl$AbstractBuilder$$anonfun$lines$1.apply$mcV$sp() @bci=11, line=142 (Interpreted frame) - scala.sys.process.ProcessImpl$Spawn$$anon$1.run() @bci=4, line=22 (Interpreted frame) Thread 31257: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=503 (Interpreted frame) - java.lang.UNIXProcess.waitFor() @bci=8, line=263 (Interpreted frame) - scala.sys.process.ProcessImpl$SimpleProcess.exitValue() @bci=4, line=218 (Interpreted frame) - scala.sys.process.ProcessBuilderImpl$AbstractBuilder$$anonfun$lines$1.apply$mcV$sp() @bci=11, line=142 (Interpreted frame) - scala.sys.process.ProcessImpl$Spawn$$anon$1.run() @bci=4, line=22 (Interpreted frame) At 2015-08-25 19:32:56, "Ted Yu" <yuzhih...@gmail.com> wrote: Looks like you were attaching images to your email which didn't go through. Consider using third party site for images - or paste error in text. Cheers On Tue, Aug 25, 2015 at 4:22 AM, Todd <bit1...@163.com> wrote: Hi， The spark sql perf itself contains benchmark data generation. I am using spark shell to run the spark sql perf to generate the data with 10G memory for both driver and executor. When I increase the scalefactor to be 30,and run the job, Then I got the following error: When I jstack it to see the status of the thread. I see the following: looks it is waiting for the process that the spark job kicks off.
