Mind sharing how you fixed the issue ? Cheers
> On Aug 26, 2015, at 1:50 AM, Todd <bit1...@163.com> wrote: > > > Sorry for the noise, It's my bad...I have worked it out now. > > At 2015-08-26 13:20:57, "Todd" <bit1...@163.com> wrote: > > I think the answer is No. I only see such message on the console..and #2 is > the thread stack trace。 > I am thinking is that in Spark SQL Perf forks many dsdgen process to generate > data when the scalafactor is increased which at last exhaust the JVM > When thread exception is thrown on the console and I leave it there for some > while(15min about),then eventually I will see OutOfMemory occur > > Can you guys try to run it if you have the environment ? I think you may > reproduce it. Thanks! > > > > > > At 2015-08-26 13:01:34, "Ted Yu" <yuzhih...@gmail.com> wrote: > The error in #1 below was not informative. > > Are you able to get more detailed error message ? > > Thanks > > > >> On Aug 25, 2015, at 6:57 PM, Todd <bit1...@163.com> wrote: >> >> >> Thanks Ted Yu. >> >> Following are the error message: >> 1. The exception that is shown on the UI is : >> Exception in thread "Thread-113" Exception in thread "Thread-126" Exception >> in thread "Thread-64" Exception in thread "Thread-90" Exception in thread >> "Thread-117" Exception in thread "Thread-80" Exception in thread >> "Thread-115" Exception in thread "ResponseProcessor for block >> BP-1564562096-172.18.149.132-1435294011279:blk_1073846767_105984" Exception >> in thread "qtp1270119920-57" Exception in thread "Thread-77" Exception in >> thread "Thread-132" Exception in thread "Thread-68" Exception in thread >> "Thread-61" Exception in thread "Thread-70" Exception in thread >> "qtp1270119920-52" Exception in thread "Thread-88" Exception in thread >> "qtp318933312-47" Exception in thread "qtp1270119920-56" >> >> 2. jstack the process, I see bunch of following message: >> >> Thread 31258: (state = BLOCKED) >> - java.lang.Object.wait(long) @bci=0 (Interpreted frame) >> - java.lang.Object.wait() @bci=2, line=503 (Interpreted frame) >> - java.lang.UNIXProcess.waitFor() @bci=8, line=263 (Interpreted frame) >> - scala.sys.process.ProcessImpl$SimpleProcess.exitValue() @bci=4, line=218 >> (Interpreted frame) >> - >> scala.sys.process.ProcessBuilderImpl$AbstractBuilder$$anonfun$lines$1.apply$mcV$sp() >> @bci=11, line=142 (Interpreted frame) >> - scala.sys.process.ProcessImpl$Spawn$$anon$1.run() @bci=4, line=22 >> (Interpreted frame) >> >> >> Thread 31257: (state = BLOCKED) >> - java.lang.Object.wait(long) @bci=0 (Interpreted frame) >> - java.lang.Object.wait() @bci=2, line=503 (Interpreted frame) >> - java.lang.UNIXProcess.waitFor() @bci=8, line=263 (Interpreted frame) >> - scala.sys.process.ProcessImpl$SimpleProcess.exitValue() @bci=4, line=218 >> (Interpreted frame) >> - >> scala.sys.process.ProcessBuilderImpl$AbstractBuilder$$anonfun$lines$1.apply$mcV$sp() >> @bci=11, line=142 (Interpreted frame) >> - scala.sys.process.ProcessImpl$Spawn$$anon$1.run() @bci=4, line=22 >> (Interpreted frame) >> >> >> >> >> >> At 2015-08-25 19:32:56, "Ted Yu" <yuzhih...@gmail.com> wrote: >> Looks like you were attaching images to your email which didn't go through. >> >> Consider using third party site for images - or paste error in text. >> >> Cheers >> >>> On Tue, Aug 25, 2015 at 4:22 AM, Todd <bit1...@163.com> wrote: >>> Hi， >>> The spark sql perf itself contains benchmark data generation. I am using >>> spark shell to run the spark sql perf to generate the data with 10G memory >>> for both driver and executor. >>> When I increase the scalefactor to be 30,and run the job, Then I got the >>> following error: >>> >>> >>> >>> When I jstack it to see the status of the thread. I see the following: >>> looks it is waiting for the process that the spark job kicks off. >>> >>> >>> >>> >>
