Hi, Thanks for your emails, I tried running your command but it returned: "No such file or directory". So I definitely need to move my local .py files to the cluster, I tried login but (before sshing) but could not find the master: ./spark-ec2 -k key -i key.pem login weather-cluster - then sshing, the copy-dir is located in the spark-ec2 but to, replicate my files across all nodes I need to get them into the root folder in the spark EC2 cluster: ./spark-ec2/copy-dir /root/spark/myfiles
I used that: http://spark.apache.org/docs/latest/ec2-scripts.html. Do you have any suggestions about how to move those files from local to the cluster? Thanks in advance, Kevin On 12 April 2016 at 12:19, Sun, Rui <rui....@intel.com> wrote: > Which py file is your main file (primary py file)? Zip the other two py > files. Leave the main py file alone. Don't copy them to S3 because it seems > that only local primary and additional py files are supported. > > ./bin/spark-submit --master spark://... --py-files <zip file> <main py > file> > > -----Original Message----- > From: kevllino [mailto:kevin.e...@mail.dcu.ie] > Sent: Tuesday, April 12, 2016 5:07 PM > To: user@spark.apache.org > Subject: Run a self-contained Spark app on a Spark standalone cluster > > Hi, > > I need to know how to run a self-contained Spark app (3 python files) in > a Spark standalone cluster. Can I move the .py files to the cluster, or > should I store them locally, on HDFS or S3? I tried the following locally > and on S3 with a zip of my .py files as suggested here < > http://spark.apache.org/docs/latest/submitting-applications.html> : > > ./bin/spark-submit --master > spark://ec2-54-51-23-172.eu-west-1.compute.amazonaws.com:5080 > --py-files > s3n://AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY@mubucket > //weather_predict.zip > > But get: “Error: Must specify a primary resource (JAR or Python file)” > > Best, > Kevin > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Run-a-self-contained-Spark-app-on-a-Spark-standalone-cluster-tp26753.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Kevin EID M.Sc. in Computing, Data Analytics <https://fr.linkedin.com/pub/kevin-eid/85/689/b01>
