[ https://issues.apache.org/jira/browse/SPARK-24764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-24764. ---------------------------------- Resolution: Incomplete > Add ServiceLoader implementation for SparkHadoopUtil > ---------------------------------------------------- > > Key: SPARK-24764 > URL: https://issues.apache.org/jira/browse/SPARK-24764 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.3.0, 2.3.1 > Reporter: Shruti Gumma > Assignee: Shruti Gumma > Priority: Major > Labels: bulk-closed > > Currently SparkHadoopUtil creation is static and cannot be changed. This > proposal is to move the creation of SparkHadoopUtil to a ServiceLoader, so > that external cluster managers can create their own SparkHadoopUtil. > In the case of Yarn, a specific YarnSparkHadoopUtil is used in the Yarn > packages whereas in other places, SparkHadoopUtil is used. SparkHadoopUtil > has been changed in v2.3.0 to work with Yarn, leaving external cluster > managers with no configurable way of doing the same. > The util classes such as SparkHadoopUtil should be made configurable, similar > to how the ExternalClusterManager was made pluggable through a ServiceLoader. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org