Hi Sasai,
Thanks for your time. I have followed your inputs and downloaded
"spark-1.5.1-bin-hadoop2.6" on one of the node say node1. And when I did a
pie test everything seems to be working fine, except that the spark-history
-server running on this node1 has gone down. It was complaining about
missing class:
15/10/21 16:41:28 INFO HistoryServer: Registered signal handlers for [TERM,
HUP, INT]
15/10/21 16:41:28 WARN SparkConf: The configuration key
'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark
1.3 and and may be removed in the future. Please use the new key
'spark.yarn.am.waitTime' instead.
15/10/21 16:41:29 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/10/21 16:41:29 INFO SecurityManager: Changing view acls to: root
15/10/21 16:41:29 INFO SecurityManager: Changing modify acls to: root
15/10/21 16:41:29 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root); users
with modify permissions: Set(root)
Exception in thread "main" java.lang.ClassNotFoundException:
org.apache.spark.deploy.yarn.history.YarnHistoryProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
at
org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:231)
at
org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
I went to the lib folder and noticed that
"spark-assembly-1.5.1-hadoop2.6.0.jar" is missing that class. I was able to
get the spark history server started with 1.3.1 but not 1.5.1. Any inputs
on this?
Really appreciate your help. Thanks
Regards,
Ajay
On Wednesday, October 21, 2015, Saisai Shao <sai.sai.s...@gmail.com
<javascript:_e(%7B%7D,'cvml','sai.sai.s...@gmail.com');>> wrote:
> Hi Ajay,
>
> You don't need to copy tarball to all the nodes, only one node you want to
> run spark application is enough (mostly the master node), Yarn will help to
> distribute the Spark dependencies. The link I mentioned before is the one
> you could follow, please read my previous mail.
>
> Thanks
> Saisai
>
>
>
> On Thu, Oct 22, 2015 at 1:56 AM, Ajay Chander <itsche...@gmail.com> wrote:
>
>> Thanks for your kind inputs. Right now I am running spark-1.3.1 on YARN(4
>> node cluster) on a HortonWorks distribution. Now I want to upgrade
>> spark-1.3.1 to
>> spark-1.5.1. So at this point of time, do I have to manually go and copy
>> spark-1.5.1 tarbal to all the nodes or is there any alternative so that I
>> can get it upgraded through Ambari UI ? If possible can anyone point me to
>> a documentation online? Thank you.
>>
>> Regards,
>> Ajay
>>
>>
>> On Wednesday, October 21, 2015, Saisai Shao <sai.sai.s...@gmail.com>
>> wrote:
>>
>>> Hi Frans,
>>>
>>> You could download Spark 1.5.1-hadoop 2.6 pre-built tarball and copy
>>> into HDP 2.3 sandbox or master node. Then copy all the conf files from
>>> /usr/hdp/current/spark-client/ to your <spark-1.5.1>/conf, or you could
>>> refer to this tech preview (
>>> http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/
>>> ), in "installing chapter", step 4 ~ 8 is what you need to do.
>>>
>>> Thanks
>>> Saisai
>>>
>>> On Wed, Oct 21, 2015 at 1:27 PM, Frans Thamura <fr...@meruvian.org>
>>> wrote:
>>>
>>>> Doug
>>>>
>>>> is it possible to put in HDP 2.3?
>>>>
>>>> esp in Sandbox
>>>>
>>>> can share how do you install it?
>>>>
>>>>
>>>> F
>>>> --
>>>> Frans Thamura (曽志胜)
>>>> Java Champion
>>>> Shadow Master and Lead Investor
>>>> Meruvian.
>>>> Integrated Hypermedia Java Solution Provider.
>>>>
>>>> Mobile: +628557888699
>>>> Blog: http://blogs.mervpolis.com/roller/flatburger (id)
>>>>
>>>> FB: http://www.facebook.com/meruvian
>>>> TW: http://www.twitter.com/meruvian / @meruvian
>>>> Website: http://www.meruvian.org
>>>>
>>>> "We grow because we share the same belief."
>>>>
>>>>
>>>> On Wed, Oct 21, 2015 at 12:24 PM, Doug Balog <doug.sparku...@dugos.com>
>>>> wrote:
>>>> > I have been running 1.5.1 with Hive in secure mode on HDP 2.2.4
>>>> without any problems.
>>>> >
>>>> > Doug
>>>> >
>>>> >> On Oct 21, 2015, at 12:05 AM, Ajay Chander <itsche...@gmail.com>
>>>> wrote:
>>>> >>
>>>> >> Hi Everyone,
>>>> >>
>>>> >> Any one has any idea if spark-1.5.1 is available as a service on
>>>> HortonWorks ? I have spark-1.3.1 installed on the Cluster and it is a
>>>> HortonWorks distribution. Now I want upgrade it to spark-1.5.1. Anyone here
>>>> have any idea about it? Thank you in advance.
>>>> >>
>>>> >> Regards,
>>>> >> Ajay
>>>> >
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>>> > For additional commands, e-mail: user-h...@spark.apache.org
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>>
>>>>
>>>
>
