talking about spark in hdp Is there reference about Spark-R, and what version should we install in R? -- Frans Thamura (曽志胜) Java Champion Shadow Master and Lead Investor Meruvian. Integrated Hypermedia Java Solution Provider.
Mobile: +628557888699 Blog: http://blogs.mervpolis.com/roller/flatburger (id) FB: http://www.facebook.com/meruvian TW: http://www.twitter.com/meruvian / @meruvian Website: http://www.meruvian.org "We grow because we share the same belief." On Thu, Oct 22, 2015 at 8:56 AM, Saisai Shao <sai.sai.s...@gmail.com> wrote: > How you start history server, do you still use the history server of 1.3.1, > or you started the history server in 1.5.1? > > The Spark tarball you used is the community version, so Application > TimelineServer based history provider is not supported, you could comment > this configuration "spark.history.provider", so it will use default > FsHistoryProvider, or you could configure "spark.history.provider" to > "org.apache.spark.deploy.history.FsHistoryProvider". > > If you still want to use this ATS based history server, you have to wait for > the technical preview release of Hortonworks. > > Thanks > Saisai > > > On Thu, Oct 22, 2015 at 9:47 AM, Ajay Chander <itsche...@gmail.com> wrote: >> >> Hi Sasai, >> >> Thanks for your time. I have followed your inputs and downloaded >> "spark-1.5.1-bin-hadoop2.6" on one of the node say node1. And when I did a >> pie test everything seems to be working fine, except that the spark-history >> -server running on this node1 has gone down. It was complaining about >> missing class: >> >> 15/10/21 16:41:28 INFO HistoryServer: Registered signal handlers for >> [TERM, HUP, INT] >> 15/10/21 16:41:28 WARN SparkConf: The configuration key >> 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 >> and and may be removed in the future. Please use the new key >> 'spark.yarn.am.waitTime' instead. >> 15/10/21 16:41:29 WARN NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> 15/10/21 16:41:29 INFO SecurityManager: Changing view acls to: root >> 15/10/21 16:41:29 INFO SecurityManager: Changing modify acls to: root >> 15/10/21 16:41:29 INFO SecurityManager: SecurityManager: authentication >> disabled; ui acls disabled; users with view permissions: Set(root); users >> with modify permissions: Set(root) >> Exception in thread "main" java.lang.ClassNotFoundException: >> org.apache.spark.deploy.yarn.history.YarnHistoryProvider >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:348) >> at org.apache.spark.util.Utils$.classForName(Utils.scala:173) >> at >> org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:231) >> at >> org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala) >> >> >> I went to the lib folder and noticed that >> "spark-assembly-1.5.1-hadoop2.6.0.jar" is missing that class. I was able to >> get the spark history server started with 1.3.1 but not 1.5.1. Any inputs on >> this? >> >> Really appreciate your help. Thanks >> >> Regards, >> Ajay >> >> >> >> On Wednesday, October 21, 2015, Saisai Shao <sai.sai.s...@gmail.com> >> wrote: >>> >>> Hi Ajay, >>> >>> You don't need to copy tarball to all the nodes, only one node you want >>> to run spark application is enough (mostly the master node), Yarn will help >>> to distribute the Spark dependencies. The link I mentioned before is the one >>> you could follow, please read my previous mail. >>> >>> Thanks >>> Saisai >>> >>> >>> >>> On Thu, Oct 22, 2015 at 1:56 AM, Ajay Chander <itsche...@gmail.com> >>> wrote: >>>> >>>> Thanks for your kind inputs. Right now I am running spark-1.3.1 on >>>> YARN(4 node cluster) on a HortonWorks distribution. Now I want to upgrade >>>> spark-1.3.1 to spark-1.5.1. So at this point of time, do I have to manually >>>> go and copy spark-1.5.1 tarbal to all the nodes or is there any alternative >>>> so that I can get it upgraded through Ambari UI ? If possible can anyone >>>> point me to a documentation online? Thank you. >>>> >>>> Regards, >>>> Ajay >>>> >>>> >>>> On Wednesday, October 21, 2015, Saisai Shao <sai.sai.s...@gmail.com> >>>> wrote: >>>>> >>>>> Hi Frans, >>>>> >>>>> You could download Spark 1.5.1-hadoop 2.6 pre-built tarball and copy >>>>> into HDP 2.3 sandbox or master node. Then copy all the conf files from >>>>> /usr/hdp/current/spark-client/ to your <spark-1.5.1>/conf, or you could >>>>> refer to this tech preview ( >>>>> http://hortonworks.com/hadoop-tutorial/apache-spark-1-4-1-technical-preview-with-hdp/ >>>>> ), in "installing chapter", step 4 ~ 8 is what you need to do. >>>>> >>>>> Thanks >>>>> Saisai >>>>> >>>>> On Wed, Oct 21, 2015 at 1:27 PM, Frans Thamura <fr...@meruvian.org> >>>>> wrote: >>>>>> >>>>>> Doug >>>>>> >>>>>> is it possible to put in HDP 2.3? >>>>>> >>>>>> esp in Sandbox >>>>>> >>>>>> can share how do you install it? >>>>>> >>>>>> >>>>>> F >>>>>> -- >>>>>> Frans Thamura (曽志胜) >>>>>> Java Champion >>>>>> Shadow Master and Lead Investor >>>>>> Meruvian. >>>>>> Integrated Hypermedia Java Solution Provider. >>>>>> >>>>>> Mobile: +628557888699 >>>>>> Blog: http://blogs.mervpolis.com/roller/flatburger (id) >>>>>> >>>>>> FB: http://www.facebook.com/meruvian >>>>>> TW: http://www.twitter.com/meruvian / @meruvian >>>>>> Website: http://www.meruvian.org >>>>>> >>>>>> "We grow because we share the same belief." >>>>>> >>>>>> >>>>>> On Wed, Oct 21, 2015 at 12:24 PM, Doug Balog >>>>>> <doug.sparku...@dugos.com> wrote: >>>>>> > I have been running 1.5.1 with Hive in secure mode on HDP 2.2.4 >>>>>> > without any problems. >>>>>> > >>>>>> > Doug >>>>>> > >>>>>> >> On Oct 21, 2015, at 12:05 AM, Ajay Chander <itsche...@gmail.com> >>>>>> >> wrote: >>>>>> >> >>>>>> >> Hi Everyone, >>>>>> >> >>>>>> >> Any one has any idea if spark-1.5.1 is available as a service on >>>>>> >> HortonWorks ? I have spark-1.3.1 installed on the Cluster and it is a >>>>>> >> HortonWorks distribution. Now I want upgrade it to spark-1.5.1. >>>>>> >> Anyone here >>>>>> >> have any idea about it? Thank you in advance. >>>>>> >> >>>>>> >> Regards, >>>>>> >> Ajay >>>>>> > >>>>>> > >>>>>> > >>>>>> > --------------------------------------------------------------------- >>>>>> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>>> > For additional commands, e-mail: user-h...@spark.apache.org >>>>>> > >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>>>>> For additional commands, e-mail: user-h...@spark.apache.org >>>>>> >>>>> >>> > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
