read about *--total-executor-cores* not sure why you specify port 6066 in master...usually it's 7077 verify in master ui(usually port 8080) how many cores are there(depends on other configs, but usually workers connect to master with all their cores)
On 7 January 2016 at 23:46, Michael Pisula <michael.pis...@tngtech.com> wrote: > Hi, > > I start the cluster using the spark-ec2 scripts, so the cluster is in > stand-alone mode. > Here is how I submit my job: > spark/bin/spark-submit --class demo.spark.StaticDataAnalysis --master > spark://<host>:6066 --deploy-mode cluster demo/Demo-1.0-SNAPSHOT-all.jar > > Cheers, > Michael > > > On 07.01.2016 22:41, Igor Berman wrote: > > share how you submit your job > what cluster(yarn, standalone) > > On 7 January 2016 at 23:24, Michael Pisula <michael.pis...@tngtech.com> > wrote: > >> Hi there, >> >> I ran a simple Batch Application on a Spark Cluster on EC2. Despite >> having 3 >> Worker Nodes, I could not get the application processed on more than one >> node, regardless if I submitted the Application in Cluster or Client mode. >> I also tried manually increasing the number of partitions in the code, no >> effect. I also pass the master into the application. >> I verified on the nodes themselves that only one node was active while the >> job was running. >> I pass enough data to make the job take 6 minutes to process. >> The job is simple enough, reading data from two S3 files, joining records >> on >> a shared field, filtering out some records and writing the result back to >> S3. >> >> Tried all kinds of stuff, but could not make it work. I did find similar >> questions, but had already tried the solutions that worked in those cases. >> Would be really happy about any pointers. >> >> Cheers, >> Michael >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-job-uses-only-one-Worker-tp25909.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > > -- > Michael Pisula * michael.pis...@tngtech.com * +49-174-3180084 > TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring > Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke > Sitz: Unterföhring * Amtsgericht München * HRB 135082 > >
