RE: Spark_1.5.1_on_HortonWorks

Frans, SparkR runs with R 3.1+. If possible, latest verison of R is recommended. From: Saisai Shao [mailto:sai.sai.s...@gmail.com] Sent: Thursday, October 22, 2015 11:17 AM To: Frans Thamura Cc: Ajay Chander; Doug Balog; user spark mailing list Subject: Re: Spark_1.5.1_on_HortonWorks SparkR

Re: Spark_1.5.1_on_HortonWorks

On 22 Oct 2015, at 02:47, Ajay Chander > wrote: Thanks for your time. I have followed your inputs and downloaded "spark-1.5.1-bin-hadoop2.6" on one of the node say node1. And when I did a pie test everything seems to be working fine, except that

Re: Spark_1.5.1_on_HortonWorks

Thanks for your kind inputs. Right now I am running spark-1.3.1 on YARN(4 node cluster) on a HortonWorks distribution. Now I want to upgrade spark-1.3.1 to spark-1.5.1. So at this point of time, do I have to manually go and copy spark-1.5.1 tarbal to all the nodes or is there any alternative so

Re: Spark_1.5.1_on_HortonWorks

You can use these steps http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/ 1.5.1 is not officially supported yet but should be coming in a month or so. On Oct 21, 2015 1:56 PM, "Ajay Chander" wrote: > Thanks for your kind inputs. Right now

Spark_1.5.1_on_HortonWorks

Hi Sasai, Thanks for your time. I have followed your inputs and downloaded "spark-1.5.1-bin-hadoop2.6" on one of the node say node1. And when I did a pie test everything seems to be working fine, except that the spark-history -server running on this node1 has gone down. It was complaining about

Re: Spark_1.5.1_on_HortonWorks

How you start history server, do you still use the history server of 1.3.1, or you started the history server in 1.5.1? The Spark tarball you used is the community version, so Application TimelineServer based history provider is not supported, you could comment this configuration

Re: Spark_1.5.1_on_HortonWorks

Hi Ajay, You don't need to copy tarball to all the nodes, only one node you want to run spark application is enough (mostly the master node), Yarn will help to distribute the Spark dependencies. The link I mentioned before is the one you could follow, please read my previous mail. Thanks Saisai

Re: Spark_1.5.1_on_HortonWorks

talking about spark in hdp Is there reference about Spark-R, and what version should we install in R? -- Frans Thamura (曽志胜) Java Champion Shadow Master and Lead Investor Meruvian. Integrated Hypermedia Java Solution Provider. Mobile: +628557888699 Blog:

Re: Spark_1.5.1_on_HortonWorks

SparkR is shipped with Hortonworks version of Spark 1.4.1, there's no difference compared to community version, you could refer to the docs of Apache Spark. It would be better to ask HDP related questions in ( http://hortonworks.com/community/forums/forum/spark/ ). Sorry for not so familiar with

Spark_1.5.1_on_HortonWorks

Hi Everyone, Any one has any idea if spark-1.5.1 is available as a service on HortonWorks ? I have spark-1.3.1 installed on the Cluster and it is a HortonWorks distribution. Now I want upgrade it to spark-1.5.1. Anyone here have any idea about it? Thank you in advance. Regards, Ajay

Re: Spark_1.5.1_on_HortonWorks

Doug is it possible to put in HDP 2.3? esp in Sandbox can share how do you install it? F -- Frans Thamura (曽志胜) Java Champion Shadow Master and Lead Investor Meruvian. Integrated Hypermedia Java Solution Provider. Mobile: +628557888699 Blog: http://blogs.mervpolis.com/roller/flatburger (id)

Re: Spark_1.5.1_on_HortonWorks

I have been running 1.5.1 with Hive in secure mode on HDP 2.2.4 without any problems. Doug > On Oct 21, 2015, at 12:05 AM, Ajay Chander wrote: > > Hi Everyone, > > Any one has any idea if spark-1.5.1 is available as a service on HortonWorks > ? I have spark-1.3.1

Re: Spark_1.5.1_on_HortonWorks

Hi Frans, You could download Spark 1.5.1-hadoop 2.6 pre-built tarball and copy into HDP 2.3 sandbox or master node. Then copy all the conf files from /usr/hdp/current/spark-client/ to your /conf, or you could refer to this tech preview (