Re: Is it possible to change the default port number 7077 for spark?
Many thanks for your response. Regards, Ashish -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Is-it-possible-to-change-the-default-port-number-7077-for-spark-tp23774p23797.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: SparkR Error in sparkR.init(master=“local”) in RStudio
Hi, Try this Sys.setenv(SPARK_HOME=C:\\spark-1.4.0) # The path to your spark installation .libPaths(c(file.path(Sys.getenv(SPARK_HOME), R, lib), .libPaths())) library(SparkR, lib.loc=C:\\spark-1.4.0\\lib) # The path to the lib folder in the spark location library(SparkR) sc=sparkR.init(master=local) Launching java with spark-submit command C:\spark-1.4.0/bin/spark-submit.cmd sparkr-shell C:\Users\Ashish\AppData\Local\Temp\RtmpWqFsOB\backend_portbdc329477c6 Cheers, Ashish -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-Error-in-sparkR-init-master-local-in-RStudio-tp23768p23800.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: sparkR
Please can you explain how did you set this second step in windows environment? .libPaths(c(file.path(Sys.getenv(SPARK_HOME), R, lib), .libPaths())) I mean to ask where do I type this command, at R prompt or in command prompt? Thanks for your time. Regards, Ashish -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sparkR-tp23412p23798.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: sparkR
I had been facing this problem for a long time and this practically forced me to move to pyspark. This is what I tried after reading the posts here Sys.setenv(SPARK_HOME=C:\\spark-1.4.0) .libPaths(c(file.path(Sys.getenv(SPARK_HOME), R, lib), .libPaths())) library(SparkR, lib.loc=C:\\spark-1.4.0\\lib) library(SparkR) sc=sparkR.init(master=local) OMFG It works, Bravo... Thanks a ton. Cheers Ashish -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sparkR-tp23412p23799.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Upgrade Spark-1.3.0 to Spark-1.4.0 in CDH5.4
Hello all, The configuration of my cluster is as follows; # 4 noded cluster running on Centos OS 6.4 # spark-1.3.0 installed on all I would like to use SparkR shipped with spark-1.4.0. I checked Cloudera and find that the latest release CDH5.4 still does not have the spark-1.4.0. Forums like StackExchange and Google groups suggest to do the process manually by building the binaries and also caution that if spark-1.4.0 will be used in CDH5.4 then it will not be supported by Cloudera as of now see here http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/can-i-upgrade-spark-from-1-3-0-to-1-3-1-in-CDH-5-4-0/td-p/27137 . (this link refers to an older version though) I dont mind if it will be supported by Cloudera or not because eventually it will but dont know when. Meanwhile, production is at stake and I am not sure how will i go about building the binaries (coming from Windows environment) Therefore, can you suggest how can I upgrade spark-1.3.0 to spark-1.4.0 on Centos 6.4 Final or do i wait for Cloudera to roll out the upgrade? Thanks for your help. Cheers, Ashish -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Upgrade-Spark-1-3-0-to-Spark-1-4-0-in-CDH5-4-tp23814.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Is it possible to change the default port number 7077 for spark?
Hello all, In my lab a colleague installed and configured spark 1.3.0 on a 4 noded cluster on CDH5.4 environment. The default port number for our spark configuration is 7456. I have been trying to SSH to spark-master from using this port number but it fails every time giving error JVM is timed out. After reading the documentation , given by Cloudera http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_ports_cdh5.html it says that the default port number for spark configuration should be 7077 and that is what i see in all the posts here and elsewhere on search results in Google. So now I have three question please if anyone can help me answer all or any of them Q1) Will the spark configuration work only with port number 7077? If yes, then how can I change the port number? Q2) Do i need to install spark on all the machines in the cluster? Q3) Do run any spark job do I always have to SSH into the spark-master machine ? or is it possible to connect my laptop to the spark-master and invoke commands from my laptop to spark-master and worker machines? Thank you for your time. Ashish -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Is-it-possible-to-change-the-default-port-number-7077-for-spark-tp23774.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: DLL load failed: %1 is not a valid win32 application on invoking pyspark
Not really a clean solution but I solved the problem by reinstalling Anaconda -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/DLL-load-failed-1-is-not-a-valid-win32-application-on-invoking-pyspark-tp23733p23743.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
DLL load failed: %1 is not a valid win32 application on invoking pyspark
Hi, I get the error, DLL load failed: %1 is not a valid win32 application whenever I invoke pyspark. Attached is the screenshot of the same. Is there any way I can get rid of it. Still being new to PySpark and have had, a not so pleasant experience so far most probably because I am on a windows environment. Therefore, I am afraid that this error might cause me trouble as I continue my journey exploring pyspark. I have already check SO and this user list but there are no posts for the same. My environment is python version 2.7, OS- Windows 7, Spark- ver 1.3.0 Appreciate your help. Sincerely, Ashish Dutt error.PNG http://apache-spark-user-list.1001560.n3.nabble.com/file/n23733/error.PNG -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/DLL-load-failed-1-is-not-a-valid-win32-application-on-invoking-pyspark-tp23733.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
JVM is not ready after 10 seconds
Hi, I am trying to connect a worker to the master. The spark master is on cloudera manager and I know the master IP address and port number. I downloaded the spark binary for CDH4 on the worker machine and then when I try to invoke the command sc = sparkR.init(master=ip address:port number) I get the following error. sc=sparkR.init(master=spark://10.229.200.250:7377) Launching java with spark-submit command C:\spark-1.4.0\bin/bin/spark-submit.cmd sparkr-shell C:\Users\ASHISH~1\AppData\Local\Temp\Rtmp82kCxH\backend_port4281739d85 Error in sparkR.init(master = spark://10.229.200.250:7377) : JVM is not ready after 10 seconds In addition: Warning message: running command 'C:\spark-1.4.0\bin/bin/spark-submit.cmd sparkr-shell C:\Users\ASHISH~1\AppData\Local\Temp\Rtmp82kCxH\backend_port4281739d85' had status 127 I am using windows 7 as the OS on the worker machine and I am invoking the sparkR.init() from RStudio Any help in this reference will be appreciated Thank you, Ashish Dutt -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/JVM-is-not-ready-after-10-seconds-tp23658.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
