Re: Can not set spark dynamic resource allocation

2016-05-27 Thread Cui, Weifeng
gt; > >Cheers! >David Newberger > >-Original Message- >From: Cui, Weifeng [mailto:weife...@a9.com] >Sent: Friday, May 20, 2016 4:26 PM >To: Marcelo Vanzin >Cc: Ted Yu; Rodrick Brown; user; Zhao, Jun; Aulakh, Sahib; Song, Yiwei >Subject: Re

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Cui, Weifeng
log. You get the >latter using the "yarn logs" command. > >On Fri, May 20, 2016 at 1:14 PM, Cui, Weifeng <weife...@a9.com> wrote: >> Here is the application log for this spark job. >> >> http://pastebin.com/2UJS9L4e >> >> >> >> Thanks,

Re: Can not set spark dynamic resource allocation

2016-05-20 Thread Cui, Weifeng
Here is the application log for this spark job. http://pastebin.com/2UJS9L4e Thanks, Weifeng From: "Aulakh, Sahib" <aula...@a9.com> Date: Friday, May 20, 2016 at 12:43 PM To: Ted Yu <yuzhih...@gmail.com> Cc: Rodrick Brown <rodr...@orchardplatform.com>, Cui Weifeng &l

Can not set spark dynamic resource allocation

2016-05-20 Thread Cui, Weifeng
Hi guys, Our team has a hadoop 2.6.0 cluster with Spark 1.6.1. We want to set dynamic resource allocation for spark and we followed the following link. After the changes, all spark jobs failed. https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation This test was