Re: Run a self-contained Spark app on a Spark standalone cluster

[kevllino]

Update: 
- I managed to login to the cluster 
- I want to use copy-dir to deploy my Python files to all nodes, I read I
need to copy them to /ephemeral/hdfs. But I don't know how to move them from
local to the cluster in HDFS?

Thanks in advance,
Kevin 



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Run-a-self-contained-Spark-app-on-a-Spark-standalone-cluster-tp26753p26761.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Run a self-contained Spark app on a Spark standalone cluster

[kevllino]

Hi, 

I need to know how to run a self-contained Spark app  (3 python files) in a
Spark standalone cluster. Can I move the .py files to the cluster, or should
I store them locally, on HDFS or S3? I tried the following locally and on S3
with a zip of my .py files as suggested  here
  : 

./bin/spark-submit --master
spark://ec2-54-51-23-172.eu-west-1.compute.amazonaws.com:5080--py-files
s3n://AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY@mubucket//weather_predict.zip

But get: “Error: Must specify a primary resource (JAR or Python file)”

Best, 
Kevin 



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Run-a-self-contained-Spark-app-on-a-Spark-standalone-cluster-tp26753.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org