[
https://issues.apache.org/jira/browse/SPARK-30689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated SPARK-30689:
----------------------------------
Summary: Allow custom resource scheduling to work with YARN versions that
don't support custom resource scheduling (was: Allow custom resource
scheduling to work with Hadoop versions that don't support it)
> Allow custom resource scheduling to work with YARN versions that don't
> support custom resource scheduling
> ---------------------------------------------------------------------------------------------------------
>
> Key: SPARK-30689
> URL: https://issues.apache.org/jira/browse/SPARK-30689
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 3.0.0
> Reporter: Thomas Graves
> Priority: Major
>
> Many people/companies will not be moving to Hadoop 3.0 that it supports
> custom resource scheduling for things like GPUs soon and have requested
> support for it in older hadoop 2.x versions. This also means that they may
> not have isolation enabled which is what the default behavior relies on.
> right now the option is to write a custom discovery script to handle on their
> own. This is ok but has some limitation because the script runs as a separate
> process. It also just a shell script.
> I think we can make this a lot more flexible by making the entire resource
> discovery class pluggable. The default one would stay as is and call the
> discovery script, but if an advanced user wanted to replace the entire thing
> they could implement a pluggable class which they could write custom code on
> how to discovery resource addresses.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
