Yuechen Chen created SPARK-19894: ------------------------------------ Summary: Tasks entirely assigned to one executor on Yarn-cluster mode for default-rack Key: SPARK-19894 URL: https://issues.apache.org/jira/browse/SPARK-19894 Project: Spark Issue Type: Bug Components: Scheduler, YARN Affects Versions: 2.1.0 Environment: Yarn-cluster Reporter: Yuechen Chen
In YARN-cluster mode, if driver has no rack information on two different hosts, these two hosts would both be recoginized as "/default-rack", which may cause some bugs. For example, if hosts of one executor and one external datasource are unknown by driver, this two hosts would be recoginized as the same rack "/default-rack", and then all tasks would be assigned to the executor. This bug would be avoided, if getRackForHost("unknown host") in YarnScheduler returns None, not Some("/default-rack"). -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org