I found the problem.
In mapped-site.xml, mapreduce.application.classpath has references to
“${hdp.version}” which is not getting replaced
when launch_container.sh is created. The executor fails with a substitution
error at line 27 in launch_container.sh because bash
can’t deal with “${hdp.version}."
I have hdp.version defined in my spark-defaults.conf via
spark.{driver,yarn.am}.extraJavaOptions -Dhdp.version=2.2.0-2041,
so something is not doing the substitution.
To work around this problem, I replaced "${hdp.version}” with “current” in
mapred-site.xml.
I found a similar bug, https://issues.apache.org/jira/browse/AMBARI-8028, and
the fix was exactly what I did to work around it.
Not sure if this is an AMBARI bug (not doing variable substitution when writing
mapred-site.xml) or YARN bug (its not doing the variable substitution when
writing launch_container.sh)
Anybody have an opinion ?
Doug
> On Mar 19, 2015, at 5:51 PM, Doug Balog <doug.sparku...@dugos.com> wrote:
>
> I’m seeing the same problem.
> I’ve set logging to DEBUG, and I think some hints are in the “Yarn AM launch
> context” that is printed out
> before Yarn runs java.
>
> My next step is to talk to the admins and get them to set
> yarn.nodemanager.delete.debug-delay-sec
> in the config, as recommended in
> http://spark.apache.org/docs/latest/running-on-yarn.html
> Then I can see exactly whats in the directory.
>
> Doug
>
> ps Sorry for the dup message Bharath and Todd, used wrong email address.
>
>
>> On Mar 19, 2015, at 1:19 AM, Bharath Ravi Kumar <reachb...@gmail.com> wrote:
>>
>> Thanks for clarifying Todd. This may then be an issue specific to the HDP
>> version we're using. Will continue to debug and post back if there's any
>> resolution.
>>
>> On Thu, Mar 19, 2015 at 3:40 AM, Todd Nist <tsind...@gmail.com> wrote:
>> Yes I believe you are correct.
>>
>> For the build you may need to specify the specific HDP version of hadoop to
>> use with the -Dhadoop.version=????. I went with the default 2.6.0, but
>> Horton may have a vendor specific version that needs to go here. I know I
>> saw a similar post today where the solution was to use
>> -Dhadoop.version=2.5.0-cdh5.3.2 but that was for a cloudera installation. I
>> am not sure what the HDP version would be to put here.
>>
>> -Todd
>>
>> On Wed, Mar 18, 2015 at 12:49 AM, Bharath Ravi Kumar <reachb...@gmail.com>
>> wrote:
>> Hi Todd,
>>
>> Yes, those entries were present in the conf under the same SPARK_HOME that
>> was used to run spark-submit. On a related note, I'm assuming that the
>> additional spark yarn options (like spark.yarn.jar) need to be set in the
>> same properties file that is passed to spark-submit. That apart, I assume
>> that no other host on the cluster should require a "deployment of" the spark
>> distribution or any other config change to support a spark job. Isn't that
>> correct?
>>
>> On Tue, Mar 17, 2015 at 6:19 PM, Todd Nist <tsind...@gmail.com> wrote:
>> Hi Bharath,
>>
>> Do you have these entries in your $SPARK_HOME/conf/spark-defaults.conf file?
>>
>> spark.driver.extraJavaOptions -Dhdp.version=2.2.0.0-2041
>> spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.0.0-2041
>>
>>
>>
>>
>> On Tue, Mar 17, 2015 at 1:04 AM, Bharath Ravi Kumar <reachb...@gmail.com>
>> wrote:
>> Still no luck running purpose-built 1.3 against HDP 2.2 after following all
>> the instructions. Anyone else faced this issue?
>>
>> On Mon, Mar 16, 2015 at 8:53 PM, Bharath Ravi Kumar <reachb...@gmail.com>
>> wrote:
>> Hi Todd,
>>
>> Thanks for the help. I'll try again after building a distribution with the
>> 1.3 sources. However, I wanted to confirm what I mentioned earlier: is it
>> sufficient to copy the distribution only to the client host from where
>> spark-submit is invoked(with spark.yarn.jar set), or is there a need to
>> ensure that the entire distribution is deployed made available pre-deployed
>> on every host in the yarn cluster? I'd assume that the latter shouldn't be
>> necessary.
>>
>> On Mon, Mar 16, 2015 at 8:38 PM, Todd Nist <tsind...@gmail.com> wrote:
>> Hi Bharath,
>>
>> I ran into the same issue a few days ago, here is a link to a post on
>> Horton's fourm. http://hortonworks.com/community/forums/search/spark+1.2.1/
>> Incase anyone else needs to perform this these are the steps I took to get
>> it to work with Spark 1.2.1 as well as Spark 1.3.0-RC3:
>>
>> 1. Pull 1.2.1 Source
>> 2. Apply the following patches
>> a. Address jackson version, https://github.com/apache/spark/pull/3938
>> b. Address the propagation of the hdp.version set in the spark-default.conf,
>> https://github.com/apache/spark/pull/3409
>> 3. build with $SPARK_HOME./make-distribution.sh –name hadoop2.6 –tgz -Pyarn
>> -Phadoop-2.4 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -DskipTests
>> package
>>
>> Then deploy the resulting artifact => spark-1.2.1-bin-hadoop2.6.tgz
>> following instructions in the HDP Spark preview
>> http://hortonworks.com/hadoop-tutorial/using-apache-spark-hdp/
>>
>> FWIW spark-1.3.0 appears to be working fine with HDP as well and steps 2a
>> and 2b are not required.
>>
>> HTH
>>
>> -Todd
>>
>>
>> On Mon, Mar 16, 2015 at 10:13 AM, Bharath Ravi Kumar <reachb...@gmail.com>
>> wrote:
>> Hi,
>>
>> Trying to run spark ( 1.2.1 built for hdp 2.2) against a yarn cluster
>> results in the AM failing to start with following error on stderr:
>> Error: Could not find or load main class
>> org.apache.spark.deploy.yarn.ExecutorLauncher
>> An application id was assigned to the job, but there were no logs. Note that
>> the spark distribution has not been "installed" on every host in the cluster
>> and the aforementioned spark build was copied to one of the hadoop client
>> hosts in the cluster to launch the
>> job. Spark-submit was run with --master yarn-client and spark.yarn.jar was
>> set to the assembly jar from the above distribution. Switching the spark
>> distribution to the HDP recommended version
>> and following the instructions on this page did not fix the problem either.
>> Any idea what may have caused this error ?
>>
>> Thanks,
>> Bharath
>>
>>
>>
>>
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
