[ https://issues.apache.org/jira/browse/SPARK-20560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved SPARK-20560. ------------------------------------ Resolution: Invalid "localhost" is filtered, been done in {{HadoopRDD.getPreferredLocations()}} since commit #06aac8a. > Review Spark's handling of filesystems returning "localhost" in > getFileBlockLocations > ------------------------------------------------------------------------------------- > > Key: SPARK-20560 > URL: https://issues.apache.org/jira/browse/SPARK-20560 > Project: Spark > Issue Type: Bug > Components: Scheduler > Affects Versions: 2.1.0 > Reporter: Steve Loughran > Priority: Minor > > Some filesystems (S3a, Azure WASB) return "localhost" as the response to > {{FileSystem.getFileBlockLocations(path)}}. If this is then used as the > preferred host when scheduling work, there's a risk that work will be queued > on one host, rather than spread across the cluster. > HIVE-14060 and TEZ-3291 have both seen it in their schedulers. > I don't know if Spark does it, someone needs to look at the code, maybe write > some tests -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org