RE: Spark_1.5.1_on_HortonWorks

[Sun, Rui]

Frans,

SparkR runs with R 3.1+. If possible, latest verison of R is recommended.

From: Saisai Shao [mailto:sai.sai.s...@gmail.com]
Sent: Thursday, October 22, 2015 11:17 AM
To: Frans Thamura
Cc: Ajay Chander; Doug Balog; user spark mailing list
Subject: Re: Spark_1.5.1_on_HortonWorks

SparkR is shipped with Hortonworks version of Spark 1.4.1, there's no 
difference compared to community version, you could refer to the docs of Apache 
Spark. It would be better to ask HDP related questions in ( 
http://hortonworks.com/community/forums/forum/spark/ ). Sorry for not so 
familiar with SparkR related things.

Thanks
Saisai

On Thu, Oct 22, 2015 at 11:02 AM, Frans Thamura 
<fr...@meruvian.org<mailto:fr...@meruvian.org>> wrote:
talking about spark in hdp

Is there reference about Spark-R, and what version should we install in R?
--
Frans Thamura (曽志胜)
Java Champion
Shadow Master and Lead Investor
Meruvian.
Integrated Hypermedia Java Solution Provider.

Mobile: +628557888699<tel:%2B628557888699>
Blog: http://blogs.mervpolis.com/roller/flatburger (id)

FB: http://www.facebook.com/meruvian
TW: http://www.twitter.com/meruvian / @meruvian
Website: http://www.meruvian.org

"We grow because we share the same belief."

On Thu, Oct 22, 2015 at 8:56 AM, Saisai Shao 
<sai.sai.s...@gmail.com<mailto:sai.sai.s...@gmail.com>> wrote:
> How you start history server, do you still use the history server of 1.3.1,
> or you started the history server in 1.5.1?
>
> The Spark tarball you used is the community version, so Application
> TimelineServer based history provider is not supported, you could comment
> this configuration "spark.history.provider", so it will use default
> FsHistoryProvider, or you could configure "spark.history.provider" to
> "org.apache.spark.deploy.history.FsHistoryProvider".
>
> If you still want to use this ATS based history server, you have to wait for
> the technical preview release of Hortonworks.
>
> Thanks
> Saisai
>
>
> On Thu, Oct 22, 2015 at 9:47 AM, Ajay Chander 
> <itsche...@gmail.com<mailto:itsche...@gmail.com>> wrote:
>>
>> Hi Sasai,
>>
>> Thanks for your time. I have followed your inputs and downloaded
>> "spark-1.5.1-bin-hadoop2.6" on one of the node say node1. And when I did a
>> pie test everything seems to be working fine, except that the spark-history
>> -server running on this node1 has gone down. It was complaining about
>> missing class:
>>
>> 15/10/21 16:41:28 INFO HistoryServer: Registered signal handlers for
>> [TERM, HUP, INT]
>> 15/10/21 16:41:28 WARN SparkConf: The configuration key
>> 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3
>> and and may be removed in the future. Please use the new key
>> 'spark.yarn.am.waitTime' instead.
>> 15/10/21 16:41:29 WARN NativeCodeLoader: Unable to load native-hadoop
>> library for your platform... using builtin-java classes where applicable
>> 15/10/21 16:41:29 INFO SecurityManager: Changing view acls to: root
>> 15/10/21 16:41:29 INFO SecurityManager: Changing modify acls to: root
>> 15/10/21 16:41:29 INFO SecurityManager: SecurityManager: authentication
>> disabled; ui acls disabled; users with view permissions: Set(root); users
>> with modify permissions: Set(root)
>> Exception in thread "main" java.lang.ClassNotFoundException:
>> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:348)
>> at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
>> at
>> org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:231)
>> at
>> org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
>>
>>
>> I went to the lib folder and noticed that
>> "spark-assembly-1.5.1-hadoop2.6.0.jar" is missing that class. I was able to
>> get the spark history server started with 1.3.1 but not 1.5.1. Any inputs on
>> this?
>>
>> Really appreciate your help. Thanks
>>
>> Regards,
>> Ajay
>>
>>
>>
>> On Wednesday, October 21, 2015, Saisai Shao 
>> <sai.sai.s...@gmail.com<mailto:sai.sai.s...@gmail.com>>
>> wrote:
>>>
>>> Hi Ajay,
>>>
>>> You don't need to copy tarball to all the nod

Re: Spark_1.5.1_on_HortonWorks

[Steve Loughran]

On 22 Oct 2015, at 02:47, Ajay Chander 
> wrote:

Thanks for your time. I have followed your inputs and downloaded 
"spark-1.5.1-bin-hadoop2.6" on one of the node say node1. And when I did a pie 
test everything seems to be working fine, except that the spark-history -server 
running on this node1 has gone down. It was complaining about  missing class:

15/10/21 16:41:28 INFO HistoryServer: Registered signal handlers for [TERM, 
HUP, INT]
15/10/21 16:41:28 WARN SparkConf: The configuration key 
'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 
and and may be removed in the future. Please use the new key 
'spark.yarn.am.waitTime' instead.
15/10/21 16:41:29 WARN NativeCodeLoader: Unable to load native-hadoop library 
for your platform... using builtin-java classes where applicable
15/10/21 16:41:29 INFO SecurityManager: Changing view acls to: root
15/10/21 16:41:29 INFO SecurityManager: Changing modify acls to: root
15/10/21 16:41:29 INFO SecurityManager: SecurityManager: authentication 
disabled; ui acls disabled; users with view permissions: Set(root); users with 
modify permissions: Set(root)
Exception in thread "main" java.lang.ClassNotFoundException: 
org.apache.spark.deploy.yarn.history.YarnHistoryProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
at 
org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:231)
at 
org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)


OK, that's happening because the version of spark we ship comes with 
integration with the yarn history provider, and that's not on your classpath.


find the history server config and either unset spark.history.provider or set 
it back to the filesystem one:

spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider

The 1.5.1-based binaries we'll push out will be set up for this and for the 
spark apps to be able to publish to the timeline server.

note that irrespective of how histories are saved, a 1.3 or 1.4 history server 
cannot view a 1.5 one. So you will need to run a 1.5.x history server to view 
the most recent histories. But you can run a 1.4 history server talking to the 
timeline service alongside a 1.5 history server working through HDFS.

see: SPARK-1537 for the details of what's happening

Re: Spark_1.5.1_on_HortonWorks

[Ajay Chander]

Thanks for your kind inputs. Right now I am running spark-1.3.1 on YARN(4
node cluster) on a HortonWorks distribution. Now I want to upgrade
spark-1.3.1 to
spark-1.5.1. So at this point of time, do I have to manually go and copy
spark-1.5.1 tarbal to all the nodes or is there any alternative so that I
can get it upgraded through Ambari UI ? If possible can anyone point me to
a documentation online? Thank you.

Regards,
Ajay

On Wednesday, October 21, 2015, Saisai Shao  wrote:

> Hi Frans,
>
> You could download Spark 1.5.1-hadoop 2.6 pre-built tarball and copy into
> HDP 2.3 sandbox or master node. Then copy all the conf files from
> /usr/hdp/current/spark-client/ to your /conf, or you could
> refer to this tech preview (
> http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/
> ), in "installing chapter", step 4 ~ 8 is what you need to do.
>
> Thanks
> Saisai
>
> On Wed, Oct 21, 2015 at 1:27 PM, Frans Thamura  > wrote:
>
>> Doug
>>
>> is it possible to put in HDP 2.3?
>>
>> esp in Sandbox
>>
>> can share how do you install it?
>>
>>
>> F
>> --
>> Frans Thamura (曽志胜)
>> Java Champion
>> Shadow Master and Lead Investor
>> Meruvian.
>> Integrated Hypermedia Java Solution Provider.
>>
>> Mobile: +628557888699
>> Blog: http://blogs.mervpolis.com/roller/flatburger (id)
>>
>> FB: http://www.facebook.com/meruvian
>> TW: http://www.twitter.com/meruvian / @meruvian
>> Website: http://www.meruvian.org
>>
>> "We grow because we share the same belief."
>>
>>
>> On Wed, Oct 21, 2015 at 12:24 PM, Doug Balog > > wrote:
>> > I have been running 1.5.1 with Hive in secure mode on HDP 2.2.4 without
>> any problems.
>> >
>> > Doug
>> >
>> >> On Oct 21, 2015, at 12:05 AM, Ajay Chander > > wrote:
>> >>
>> >> Hi Everyone,
>> >>
>> >> Any one has any idea if spark-1.5.1 is available as a service on
>> HortonWorks ? I have spark-1.3.1 installed on the Cluster and it is a
>> HortonWorks distribution. Now I want upgrade it to spark-1.5.1. Anyone here
>> have any idea about it? Thank you in advance.
>> >>
>> >> Regards,
>> >> Ajay
>> >
>> >
>> > -
>> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> 
>> > For additional commands, e-mail: user-h...@spark.apache.org
>> 
>> >
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> 
>> For additional commands, e-mail: user-h...@spark.apache.org
>> 
>>
>>
>

Re: Spark_1.5.1_on_HortonWorks

[Artem Ervits]

You can use these steps
http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/

1.5.1 is not officially supported yet but should be coming in a month or so.
On Oct 21, 2015 1:56 PM, "Ajay Chander"  wrote:

> Thanks for your kind inputs. Right now I am running spark-1.3.1 on YARN(4
> node cluster) on a HortonWorks distribution. Now I want to upgrade 
> spark-1.3.1 to
> spark-1.5.1. So at this point of time, do I have to manually go and copy
> spark-1.5.1 tarbal to all the nodes or is there any alternative so that I
> can get it upgraded through Ambari UI ? If possible can anyone point me to
> a documentation online? Thank you.
>
> Regards,
> Ajay
>
> On Wednesday, October 21, 2015, Saisai Shao 
> wrote:
>
>> Hi Frans,
>>
>> You could download Spark 1.5.1-hadoop 2.6 pre-built tarball and copy into
>> HDP 2.3 sandbox or master node. Then copy all the conf files from
>> /usr/hdp/current/spark-client/ to your /conf, or you could
>> refer to this tech preview (
>> http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/
>> ), in "installing chapter", step 4 ~ 8 is what you need to do.
>>
>> Thanks
>> Saisai
>>
>> On Wed, Oct 21, 2015 at 1:27 PM, Frans Thamura 
>> wrote:
>>
>>> Doug
>>>
>>> is it possible to put in HDP 2.3?
>>>
>>> esp in Sandbox
>>>
>>> can share how do you install it?
>>>
>>>
>>> F
>>> --
>>> Frans Thamura (曽志胜)
>>> Java Champion
>>> Shadow Master and Lead Investor
>>> Meruvian.
>>> Integrated Hypermedia Java Solution Provider.
>>>
>>> Mobile: +628557888699
>>> Blog: http://blogs.mervpolis.com/roller/flatburger (id)
>>>
>>> FB: http://www.facebook.com/meruvian
>>> TW: http://www.twitter.com/meruvian / @meruvian
>>> Website: http://www.meruvian.org
>>>
>>> "We grow because we share the same belief."
>>>
>>>
>>> On Wed, Oct 21, 2015 at 12:24 PM, Doug Balog 
>>> wrote:
>>> > I have been running 1.5.1 with Hive in secure mode on HDP 2.2.4
>>> without any problems.
>>> >
>>> > Doug
>>> >
>>> >> On Oct 21, 2015, at 12:05 AM, Ajay Chander 
>>> wrote:
>>> >>
>>> >> Hi Everyone,
>>> >>
>>> >> Any one has any idea if spark-1.5.1 is available as a service on
>>> HortonWorks ? I have spark-1.3.1 installed on the Cluster and it is a
>>> HortonWorks distribution. Now I want upgrade it to spark-1.5.1. Anyone here
>>> have any idea about it? Thank you in advance.
>>> >>
>>> >> Regards,
>>> >> Ajay
>>> >
>>> >
>>> > -
>>> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> > For additional commands, e-mail: user-h...@spark.apache.org
>>> >
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>
>>>
>>

Re: Spark_1.5.1_on_HortonWorks

[Saisai Shao]

How you start history server, do you still use the history server of 1.3.1,
or you started the history server in 1.5.1?

The Spark tarball you used is the community version, so Application
TimelineServer based history provider is not supported, you could comment
this configuration "spark.history.provider", so it will use default
FsHistoryProvider, or you could configure "spark.history.provider" to
"org.apache.spark.deploy.history.FsHistoryProvider".

If you still want to use this ATS based history server, you have to wait
for the technical preview release of Hortonworks.

Thanks
Saisai


On Thu, Oct 22, 2015 at 9:47 AM, Ajay Chander  wrote:

> Hi Sasai,
>
> Thanks for your time. I have followed your inputs and downloaded
> "spark-1.5.1-bin-hadoop2.6" on one of the node say node1. And when I did a
> pie test everything seems to be working fine, except that the spark-history
> -server running on this node1 has gone down. It was complaining about
>  missing class:
>
> 15/10/21 16:41:28 INFO HistoryServer: Registered signal handlers for
> [TERM, HUP, INT]
> 15/10/21 16:41:28 WARN SparkConf: The configuration key
> 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark
> 1.3 and and may be removed in the future. Please use the new key
> 'spark.yarn.am.waitTime' instead.
> 15/10/21 16:41:29 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 15/10/21 16:41:29 INFO SecurityManager: Changing view acls to: root
> 15/10/21 16:41:29 INFO SecurityManager: Changing modify acls to: root
> 15/10/21 16:41:29 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(root); users
> with modify permissions: Set(root)
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
> at
> org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:231)
> at
> org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
>
>
> I went to the lib folder and noticed that
> "spark-assembly-1.5.1-hadoop2.6.0.jar" is missing that class. I was able to
> get the spark history server started with 1.3.1 but not 1.5.1. Any inputs
> on this?
>
> Really appreciate your help. Thanks
>
> Regards,
> Ajay
>
>
>
> On Wednesday, October 21, 2015, Saisai Shao 
> wrote:
>
>> Hi Ajay,
>>
>> You don't need to copy tarball to all the nodes, only one node you want
>> to run spark application is enough (mostly the master node), Yarn will help
>> to distribute the Spark dependencies. The link I mentioned before is the
>> one you could follow, please read my previous mail.
>>
>> Thanks
>> Saisai
>>
>>
>>
>> On Thu, Oct 22, 2015 at 1:56 AM, Ajay Chander 
>> wrote:
>>
>>> Thanks for your kind inputs. Right now I am running spark-1.3.1 on
>>> YARN(4 node cluster) on a HortonWorks distribution. Now I want to upgrade
>>> spark-1.3.1 to spark-1.5.1. So at this point of time, do I have to
>>> manually go and copy spark-1.5.1 tarbal to all the nodes or is there any
>>> alternative so that I can get it upgraded through Ambari UI ? If possible
>>> can anyone point me to a documentation online? Thank you.
>>>
>>> Regards,
>>> Ajay
>>>
>>>
>>> On Wednesday, October 21, 2015, Saisai Shao 
>>> wrote:
>>>
 Hi Frans,

 You could download Spark 1.5.1-hadoop 2.6 pre-built tarball and copy
 into HDP 2.3 sandbox or master node. Then copy all the conf files from
 /usr/hdp/current/spark-client/ to your /conf, or you could
 refer to this tech preview (
 http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/
 ), in "installing chapter", step 4 ~ 8 is what you need to do.

 Thanks
 Saisai

 On Wed, Oct 21, 2015 at 1:27 PM, Frans Thamura 
 wrote:

> Doug
>
> is it possible to put in HDP 2.3?
>
> esp in Sandbox
>
> can share how do you install it?
>
>
> F
> --
> Frans Thamura (曽志胜)
> Java Champion
> Shadow Master and Lead Investor
> Meruvian.
> Integrated Hypermedia Java Solution Provider.
>
> Mobile: +628557888699
> Blog: http://blogs.mervpolis.com/roller/flatburger (id)
>
> FB: http://www.facebook.com/meruvian
> TW: http://www.twitter.com/meruvian / @meruvian
> Website: http://www.meruvian.org

Re: Spark_1.5.1_on_HortonWorks

[Saisai Shao]

Hi Ajay,

You don't need to copy tarball to all the nodes, only one node you want to
run spark application is enough (mostly the master node), Yarn will help to
distribute the Spark dependencies. The link I mentioned before is the one
you could follow, please read my previous mail.

Thanks
Saisai



On Thu, Oct 22, 2015 at 1:56 AM, Ajay Chander  wrote:

> Thanks for your kind inputs. Right now I am running spark-1.3.1 on YARN(4
> node cluster) on a HortonWorks distribution. Now I want to upgrade 
> spark-1.3.1 to
> spark-1.5.1. So at this point of time, do I have to manually go and copy
> spark-1.5.1 tarbal to all the nodes or is there any alternative so that I
> can get it upgraded through Ambari UI ? If possible can anyone point me to
> a documentation online? Thank you.
>
> Regards,
> Ajay
>
>
> On Wednesday, October 21, 2015, Saisai Shao 
> wrote:
>
>> Hi Frans,
>>
>> You could download Spark 1.5.1-hadoop 2.6 pre-built tarball and copy into
>> HDP 2.3 sandbox or master node. Then copy all the conf files from
>> /usr/hdp/current/spark-client/ to your /conf, or you could
>> refer to this tech preview (
>> http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/
>> ), in "installing chapter", step 4 ~ 8 is what you need to do.
>>
>> Thanks
>> Saisai
>>
>> On Wed, Oct 21, 2015 at 1:27 PM, Frans Thamura 
>> wrote:
>>
>>> Doug
>>>
>>> is it possible to put in HDP 2.3?
>>>
>>> esp in Sandbox
>>>
>>> can share how do you install it?
>>>
>>>
>>> F
>>> --
>>> Frans Thamura (曽志胜)
>>> Java Champion
>>> Shadow Master and Lead Investor
>>> Meruvian.
>>> Integrated Hypermedia Java Solution Provider.
>>>
>>> Mobile: +628557888699
>>> Blog: http://blogs.mervpolis.com/roller/flatburger (id)
>>>
>>> FB: http://www.facebook.com/meruvian
>>> TW: http://www.twitter.com/meruvian / @meruvian
>>> Website: http://www.meruvian.org
>>>
>>> "We grow because we share the same belief."
>>>
>>>
>>> On Wed, Oct 21, 2015 at 12:24 PM, Doug Balog 
>>> wrote:
>>> > I have been running 1.5.1 with Hive in secure mode on HDP 2.2.4
>>> without any problems.
>>> >
>>> > Doug
>>> >
>>> >> On Oct 21, 2015, at 12:05 AM, Ajay Chander 
>>> wrote:
>>> >>
>>> >> Hi Everyone,
>>> >>
>>> >> Any one has any idea if spark-1.5.1 is available as a service on
>>> HortonWorks ? I have spark-1.3.1 installed on the Cluster and it is a
>>> HortonWorks distribution. Now I want upgrade it to spark-1.5.1. Anyone here
>>> have any idea about it? Thank you in advance.
>>> >>
>>> >> Regards,
>>> >> Ajay
>>> >
>>> >
>>> > -
>>> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> > For additional commands, e-mail: user-h...@spark.apache.org
>>> >
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>
>>>
>>

Re: Spark_1.5.1_on_HortonWorks

[Frans Thamura]

talking about spark in hdp

Is there reference about Spark-R, and what version should we install in R?
--
Frans Thamura (曽志胜)
Java Champion
Shadow Master and Lead Investor
Meruvian.
Integrated Hypermedia Java Solution Provider.

Mobile: +628557888699
Blog: http://blogs.mervpolis.com/roller/flatburger (id)

FB: http://www.facebook.com/meruvian
TW: http://www.twitter.com/meruvian / @meruvian
Website: http://www.meruvian.org

"We grow because we share the same belief."


On Thu, Oct 22, 2015 at 8:56 AM, Saisai Shao  wrote:
> How you start history server, do you still use the history server of 1.3.1,
> or you started the history server in 1.5.1?
>
> The Spark tarball you used is the community version, so Application
> TimelineServer based history provider is not supported, you could comment
> this configuration "spark.history.provider", so it will use default
> FsHistoryProvider, or you could configure "spark.history.provider" to
> "org.apache.spark.deploy.history.FsHistoryProvider".
>
> If you still want to use this ATS based history server, you have to wait for
> the technical preview release of Hortonworks.
>
> Thanks
> Saisai
>
>
> On Thu, Oct 22, 2015 at 9:47 AM, Ajay Chander  wrote:
>>
>> Hi Sasai,
>>
>> Thanks for your time. I have followed your inputs and downloaded
>> "spark-1.5.1-bin-hadoop2.6" on one of the node say node1. And when I did a
>> pie test everything seems to be working fine, except that the spark-history
>> -server running on this node1 has gone down. It was complaining about
>> missing class:
>>
>> 15/10/21 16:41:28 INFO HistoryServer: Registered signal handlers for
>> [TERM, HUP, INT]
>> 15/10/21 16:41:28 WARN SparkConf: The configuration key
>> 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3
>> and and may be removed in the future. Please use the new key
>> 'spark.yarn.am.waitTime' instead.
>> 15/10/21 16:41:29 WARN NativeCodeLoader: Unable to load native-hadoop
>> library for your platform... using builtin-java classes where applicable
>> 15/10/21 16:41:29 INFO SecurityManager: Changing view acls to: root
>> 15/10/21 16:41:29 INFO SecurityManager: Changing modify acls to: root
>> 15/10/21 16:41:29 INFO SecurityManager: SecurityManager: authentication
>> disabled; ui acls disabled; users with view permissions: Set(root); users
>> with modify permissions: Set(root)
>> Exception in thread "main" java.lang.ClassNotFoundException:
>> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:348)
>> at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
>> at
>> org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:231)
>> at
>> org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
>>
>>
>> I went to the lib folder and noticed that
>> "spark-assembly-1.5.1-hadoop2.6.0.jar" is missing that class. I was able to
>> get the spark history server started with 1.3.1 but not 1.5.1. Any inputs on
>> this?
>>
>> Really appreciate your help. Thanks
>>
>> Regards,
>> Ajay
>>
>>
>>
>> On Wednesday, October 21, 2015, Saisai Shao 
>> wrote:
>>>
>>> Hi Ajay,
>>>
>>> You don't need to copy tarball to all the nodes, only one node you want
>>> to run spark application is enough (mostly the master node), Yarn will help
>>> to distribute the Spark dependencies. The link I mentioned before is the one
>>> you could follow, please read my previous mail.
>>>
>>> Thanks
>>> Saisai
>>>
>>>
>>>
>>> On Thu, Oct 22, 2015 at 1:56 AM, Ajay Chander 
>>> wrote:

 Thanks for your kind inputs. Right now I am running spark-1.3.1 on
 YARN(4 node cluster) on a HortonWorks distribution. Now I want to upgrade
 spark-1.3.1 to spark-1.5.1. So at this point of time, do I have to manually
 go and copy spark-1.5.1 tarbal to all the nodes or is there any alternative
 so that I can get it upgraded through Ambari UI ? If possible can anyone
 point me to a documentation online? Thank you.

 Regards,
 Ajay


 On Wednesday, October 21, 2015, Saisai Shao 
 wrote:
>
> Hi Frans,
>
> You could download Spark 1.5.1-hadoop 2.6 pre-built tarball and copy
> into HDP 2.3 sandbox or master node. Then copy all the conf files from
> /usr/hdp/current/spark-client/ to your /conf, or you could
> refer to this tech preview (
> http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/
> ), in "installing chapter", step 4 ~ 8 is what you need to do.

Re: Spark_1.5.1_on_HortonWorks

[Saisai Shao]

SparkR is shipped with Hortonworks version of Spark 1.4.1, there's no
difference compared to community version, you could refer to the docs of
Apache Spark. It would be better to ask HDP related questions in (
http://hortonworks.com/community/forums/forum/spark/ ). Sorry for not so
familiar with SparkR related things.

Thanks
Saisai

On Thu, Oct 22, 2015 at 11:02 AM, Frans Thamura  wrote:

> talking about spark in hdp
>
> Is there reference about Spark-R, and what version should we install in R?
> --
> Frans Thamura (曽志胜)
> Java Champion
> Shadow Master and Lead Investor
> Meruvian.
> Integrated Hypermedia Java Solution Provider.
>
> Mobile: +628557888699
> Blog: http://blogs.mervpolis.com/roller/flatburger (id)
>
> FB: http://www.facebook.com/meruvian
> TW: http://www.twitter.com/meruvian / @meruvian
> Website: http://www.meruvian.org
>
> "We grow because we share the same belief."
>
>
> On Thu, Oct 22, 2015 at 8:56 AM, Saisai Shao 
> wrote:
> > How you start history server, do you still use the history server of
> 1.3.1,
> > or you started the history server in 1.5.1?
> >
> > The Spark tarball you used is the community version, so Application
> > TimelineServer based history provider is not supported, you could comment
> > this configuration "spark.history.provider", so it will use default
> > FsHistoryProvider, or you could configure "spark.history.provider" to
> > "org.apache.spark.deploy.history.FsHistoryProvider".
> >
> > If you still want to use this ATS based history server, you have to wait
> for
> > the technical preview release of Hortonworks.
> >
> > Thanks
> > Saisai
> >
> >
> > On Thu, Oct 22, 2015 at 9:47 AM, Ajay Chander 
> wrote:
> >>
> >> Hi Sasai,
> >>
> >> Thanks for your time. I have followed your inputs and downloaded
> >> "spark-1.5.1-bin-hadoop2.6" on one of the node say node1. And when I
> did a
> >> pie test everything seems to be working fine, except that the
> spark-history
> >> -server running on this node1 has gone down. It was complaining about
> >> missing class:
> >>
> >> 15/10/21 16:41:28 INFO HistoryServer: Registered signal handlers for
> >> [TERM, HUP, INT]
> >> 15/10/21 16:41:28 WARN SparkConf: The configuration key
> >> 'spark.yarn.applicationMaster.waitTries' has been deprecated as of
> Spark 1.3
> >> and and may be removed in the future. Please use the new key
> >> 'spark.yarn.am.waitTime' instead.
> >> 15/10/21 16:41:29 WARN NativeCodeLoader: Unable to load native-hadoop
> >> library for your platform... using builtin-java classes where applicable
> >> 15/10/21 16:41:29 INFO SecurityManager: Changing view acls to: root
> >> 15/10/21 16:41:29 INFO SecurityManager: Changing modify acls to: root
> >> 15/10/21 16:41:29 INFO SecurityManager: SecurityManager: authentication
> >> disabled; ui acls disabled; users with view permissions: Set(root);
> users
> >> with modify permissions: Set(root)
> >> Exception in thread "main" java.lang.ClassNotFoundException:
> >> org.apache.spark.deploy.yarn.history.YarnHistoryProvider
> >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> >> at java.lang.Class.forName0(Native Method)
> >> at java.lang.Class.forName(Class.java:348)
> >> at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
> >> at
> >>
> org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:231)
> >> at
> >> org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
> >>
> >>
> >> I went to the lib folder and noticed that
> >> "spark-assembly-1.5.1-hadoop2.6.0.jar" is missing that class. I was
> able to
> >> get the spark history server started with 1.3.1 but not 1.5.1. Any
> inputs on
> >> this?
> >>
> >> Really appreciate your help. Thanks
> >>
> >> Regards,
> >> Ajay
> >>
> >>
> >>
> >> On Wednesday, October 21, 2015, Saisai Shao 
> >> wrote:
> >>>
> >>> Hi Ajay,
> >>>
> >>> You don't need to copy tarball to all the nodes, only one node you want
> >>> to run spark application is enough (mostly the master node), Yarn will
> help
> >>> to distribute the Spark dependencies. The link I mentioned before is
> the one
> >>> you could follow, please read my previous mail.
> >>>
> >>> Thanks
> >>> Saisai
> >>>
> >>>
> >>>
> >>> On Thu, Oct 22, 2015 at 1:56 AM, Ajay Chander 
> >>> wrote:
> 
>  Thanks for your kind inputs. Right now I am running spark-1.3.1 on
>  YARN(4 node cluster) on a HortonWorks distribution. Now I want to
> upgrade
>  spark-1.3.1 to spark-1.5.1. So at this point of time, do I have to
> manually
>  go and copy spark-1.5.1 tarbal to all the nodes or is there any
> alternative
>  so that I can get it upgraded 

Re: Spark_1.5.1_on_HortonWorks

[Frans Thamura]

Doug

is it possible to put in HDP 2.3?

esp in Sandbox

can share how do you install it?


F
--
Frans Thamura (曽志胜)
Java Champion
Shadow Master and Lead Investor
Meruvian.
Integrated Hypermedia Java Solution Provider.

Mobile: +628557888699
Blog: http://blogs.mervpolis.com/roller/flatburger (id)

FB: http://www.facebook.com/meruvian
TW: http://www.twitter.com/meruvian / @meruvian
Website: http://www.meruvian.org

"We grow because we share the same belief."


On Wed, Oct 21, 2015 at 12:24 PM, Doug Balog  wrote:
> I have been running 1.5.1 with Hive in secure mode on HDP 2.2.4 without any 
> problems.
>
> Doug
>
>> On Oct 21, 2015, at 12:05 AM, Ajay Chander  wrote:
>>
>> Hi Everyone,
>>
>> Any one has any idea if spark-1.5.1 is available as a service on HortonWorks 
>> ? I have spark-1.3.1 installed on the Cluster and it is a HortonWorks 
>> distribution. Now I want upgrade it to spark-1.5.1. Anyone here have any 
>> idea about it? Thank you in advance.
>>
>> Regards,
>> Ajay
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Spark_1.5.1_on_HortonWorks

[Doug Balog]

I have been running 1.5.1 with Hive in secure mode on HDP 2.2.4 without any 
problems.

Doug

> On Oct 21, 2015, at 12:05 AM, Ajay Chander  wrote:
> 
> Hi Everyone,
> 
> Any one has any idea if spark-1.5.1 is available as a service on HortonWorks 
> ? I have spark-1.3.1 installed on the Cluster and it is a HortonWorks 
> distribution. Now I want upgrade it to spark-1.5.1. Anyone here have any idea 
> about it? Thank you in advance.
> 
> Regards,
> Ajay


-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: Spark_1.5.1_on_HortonWorks

[Saisai Shao]

Hi Frans,

You could download Spark 1.5.1-hadoop 2.6 pre-built tarball and copy into
HDP 2.3 sandbox or master node. Then copy all the conf files from
/usr/hdp/current/spark-client/ to your /conf, or you could
refer to this tech preview (
http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/
), in "installing chapter", step 4 ~ 8 is what you need to do.

Thanks
Saisai

On Wed, Oct 21, 2015 at 1:27 PM, Frans Thamura  wrote:

> Doug
>
> is it possible to put in HDP 2.3?
>
> esp in Sandbox
>
> can share how do you install it?
>
>
> F
> --
> Frans Thamura (曽志胜)
> Java Champion
> Shadow Master and Lead Investor
> Meruvian.
> Integrated Hypermedia Java Solution Provider.
>
> Mobile: +628557888699
> Blog: http://blogs.mervpolis.com/roller/flatburger (id)
>
> FB: http://www.facebook.com/meruvian
> TW: http://www.twitter.com/meruvian / @meruvian
> Website: http://www.meruvian.org
>
> "We grow because we share the same belief."
>
>
> On Wed, Oct 21, 2015 at 12:24 PM, Doug Balog 
> wrote:
> > I have been running 1.5.1 with Hive in secure mode on HDP 2.2.4 without
> any problems.
> >
> > Doug
> >
> >> On Oct 21, 2015, at 12:05 AM, Ajay Chander  wrote:
> >>
> >> Hi Everyone,
> >>
> >> Any one has any idea if spark-1.5.1 is available as a service on
> HortonWorks ? I have spark-1.3.1 installed on the Cluster and it is a
> HortonWorks distribution. Now I want upgrade it to spark-1.5.1. Anyone here
> have any idea about it? Thank you in advance.
> >>
> >> Regards,
> >> Ajay
> >
> >
> > -
> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> > For additional commands, e-mail: user-h...@spark.apache.org
> >
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>