[jira] [Commented] (SPARK-4360) task only execute on one node when spark on yarn

2015-02-10 Thread seekerak (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14313898#comment-14313898
 ] 

seekerak commented on SPARK-4360:
-

i have resolved this issue by configure yarn scheduler like this:

property
nameyarn.resourcemanager.scheduler.class/name

valueorg.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler/value
 
/property

or 
property
nameyarn.resourcemanager.scheduler.class/name

valueorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler/value
 
/property
   and
   property
nameyarn.scheduler.capacity.resource-calculator/name

valueorg.apache.hadoop.yarn.util.resource.DominantResourceCalculator/value
   /property
  
so the real reason is yarn's resource scheduler, if one node can provide all 
resource that tasks required， all task maybe run on one node only.



 task only execute on one node when spark on yarn
 

 Key: SPARK-4360
 URL: https://issues.apache.org/jira/browse/SPARK-4360
 Project: Spark
  Issue Type: Bug
  Components: YARN
Affects Versions: 1.0.2
Reporter: seekerak

 hadoop version: hadoop 2.0.3-alpha
 spark version: 1.0.2
 when i run spark jobs on yarn, i found all the task only run on one node, my 
 cluster has 4 nodes, executors has 3, but only one has task, the others 
 hasn't, my command like this :
 /opt/hadoopcluster/spark-1.0.2-bin-hadoop2/bin/spark-submit --class 
 org.sr.scala.Spark_LineCount_G0 --executor-memory 2G --num-executors 12 
 --master yarn-cluster /home/Spark_G0.jar /data /output/ou_1
 is there any one knows why?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-4360) task only execute on one node when spark on yarn

2014-11-12 Thread Sean Owen (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14207894#comment-14207894
 ] 

Sean Owen commented on SPARK-4360:
--

I don't think there's enough info here; this maybe should have been a question 
on the list first.

Is there more than 1 partition in the input? did more than 1 executor actually 
allocate? are you definitely observing tasks running and not some 
single-threaded process on the driver?

 task only execute on one node when spark on yarn
 

 Key: SPARK-4360
 URL: https://issues.apache.org/jira/browse/SPARK-4360
 Project: Spark
  Issue Type: Bug
Affects Versions: 1.0.2
Reporter: seekerak

 hadoop version: hadoop 2.0.3-alpha
 spark version: 1.0.2
 when i run spark jobs on yarn, i found all the task only run on one node, my 
 cluster has 4 nodes, executors has 3, but only one has task, the others 
 hasn't, my command like this :
 /opt/hadoopcluster/spark-1.0.2-bin-hadoop2/bin/spark-submit --class 
 org.sr.scala.Spark_LineCount_G0 --executor-memory 2G --num-executors 12 
 --master yarn-cluster /home/Spark_G0.jar /data /output/ou_1
 is there any one knows why?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org