Thanks. Would that distribution work for hdp 2.2?
On Fri, Apr 17, 2015 at 2:19 PM, Zhan Zhang zzh...@hortonworks.com wrote:
You don’t need to put any yarn assembly in hdfs. The spark assembly jar
will include everything. It looks like your package does not include yarn
module, although I
Hi Udit,
By the way, do you mind to share the whole log trace?
Thanks.
Zhan Zhang
On Apr 17, 2015, at 2:26 PM, Udit Mehta
ume...@groupon.commailto:ume...@groupon.com wrote:
I am just trying to launch a spark shell and not do anything fancy. I got the
binary distribution from apache and put
You don’t need to put any yarn assembly in hdfs. The spark assembly jar will
include everything. It looks like your package does not include yarn module,
although I didn’t find anything wrong in your mvn command. Can you check
whether the ExecutorLauncher class is in your jar file or not?
BTW:
You probably want to first try the basic configuration to see whether it works,
instead of setting SPARK_JAR pointing to the hdfs location. This error is
caused by not finding ExecutorLauncher in class path, and not HDP specific, I
think.
Thanks.
Zhan Zhang
On Apr 17, 2015, at 2:26 PM, Udit
Hi,
This is the log trace:
https://gist.github.com/uditmehta27/511eac0b76e6d61f8b47
On the yarn RM UI, I see :
Error: Could not find or load main class
org.apache.spark.deploy.yarn.ExecutorLauncher
The command I run is: bin/spark-shell --master yarn-client
The spark defaults I use is:
I followed the steps described above and I still get this error:
Error: Could not find or load main class
org.apache.spark.deploy.yarn.ExecutorLauncher
I am trying to build spark 1.3 on hdp 2.2.
I built spark from source using:
build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -Phive
Besides the hdp.version in spark-defaults.conf, I think you probably forget to
put the file java-opts under $SPARK_HOME/conf with following contents.
[root@c6402 conf]# pwd
/usr/hdp/current/spark-client/conf
[root@c6402 conf]# ls
fairscheduler.xml.template java-opts
Thanks Zhang, that solved the error. This is probably not documented
anywhere so I missed it.
Thanks again,
Udit
On Fri, Apr 17, 2015 at 3:24 PM, Zhan Zhang zzh...@hortonworks.com wrote:
Besides the hdp.version in spark-defaults.conf, I think you probably
forget to put the file* java-opts*
The “best” solution to spark-shell’s problem is creating a file
$SPARK_HOME/conf/java-opts
with “-Dhdp.version=2.2.0.0-2014”
Cheers,
Doug
On Mar 28, 2015, at 1:25 PM, Michael Stone mst...@mathom.us wrote:
I've also been having trouble running 1.3.0 on HDP. The
Hi Folks,
Just to summarize it to run SPARK on HDP distribution.
1. The spark version has to be 1.3.0 and above if you are using upstream
distribution. This configuration is mainly for HDP rolling upgrade purpose,
and the patch only went into spark upstream from 1.3.0.
2. In
I've also been having trouble running 1.3.0 on HDP. The
spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.0.0-2041
configuration directive seems to work with pyspark, but not propagate
when using spark-shell. (That is, everything works find with pyspark,
and spark-shell fails with the bad
I found the problem.
In mapped-site.xml, mapreduce.application.classpath has references to
“${hdp.version}” which is not getting replaced
when launch_container.sh is created. The executor fails with a substitution
error at line 27 in launch_container.sh because bash
can’t deal with
Hi Doug,
I did try setting that config parameter to a larger number (several
minutes), but still wasn't able to retrieve additional context logs. Let us
know if you have any success with it.
Thanks,
Bharath
On Fri, Mar 20, 2015 at 3:21 AM, Doug Balog doug.sparku...@dugos.com
wrote:
I’m seeing
I’m seeing the same problem.
I’ve set logging to DEBUG, and I think some hints are in the “Yarn AM launch
context” that is printed out
before Yarn runs java.
My next step is to talk to the admins and get them to set
yarn.nodemanager.delete.debug-delay-sec
in the config, as recommended in
Thanks for clarifying Todd. This may then be an issue specific to the HDP
version we're using. Will continue to debug and post back if there's any
resolution.
On Thu, Mar 19, 2015 at 3:40 AM, Todd Nist tsind...@gmail.com wrote:
Yes I believe you are correct.
For the build you may need to
Yes I believe you are correct.
For the build you may need to specify the specific HDP version of hadoop to
use with the -Dhadoop.version=. I went with the default 2.6.0, but
Horton may have a vendor specific version that needs to go here. I know I
saw a similar post today where the solution
Hi Todd,
Yes, those entries were present in the conf under the same SPARK_HOME that
was used to run spark-submit. On a related note, I'm assuming that the
additional spark yarn options (like spark.yarn.jar) need to be set in the
same properties file that is passed to spark-submit. That apart, I
Hi Bharath,
Do you have these entries in your $SPARK_HOME/conf/spark-defaults.conf file?
spark.driver.extraJavaOptions -Dhdp.version=2.2.0.0-2041
spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.0.0-2041
On Tue, Mar 17, 2015 at 1:04 AM, Bharath Ravi Kumar reachb...@gmail.com
wrote:
Still
Hi,
Trying to run spark ( 1.2.1 built for hdp 2.2) against a yarn cluster
results in the AM failing to start with following error on stderr:
Error: Could not find or load main class
org.apache.spark.deploy.yarn.ExecutorLauncher
An application id was assigned to the job, but there were no logs.
Hi Todd,
Thanks for the help. I'll try again after building a distribution with the
1.3 sources. However, I wanted to confirm what I mentioned earlier: is it
sufficient to copy the distribution only to the client host from where
spark-submit is invoked(with spark.yarn.jar set), or is there a
Hi Bharath,
I ran into the same issue a few days ago, here is a link to a post on
Horton's fourm. http://hortonworks.com/community/forums/search/spark+1.2.1/
Incase anyone else needs to perform this these are the steps I took to get
it to work with Spark 1.2.1 as well as Spark 1.3.0-RC3:
1.
Still no luck running purpose-built 1.3 against HDP 2.2 after following all
the instructions. Anyone else faced this issue?
On Mon, Mar 16, 2015 at 8:53 PM, Bharath Ravi Kumar reachb...@gmail.com
wrote:
Hi Todd,
Thanks for the help. I'll try again after building a distribution with the
1.3
22 matches
Mail list logo
