[
https://issues.apache.org/jira/browse/SPARK-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164019#comment-14164019
]
Patrick Wendell commented on SPARK-3561:
----------------------------------------
I wanted to wait for a more complete design - but I can short-circuit this a
bit and say that in general, I don’t think it makes much sense to open up Spark
internals to be extended like this. The goal here as I understand it is to
improve the multi-tenancy of Spark-on-YARN. The proposed solution is to add a
Tez dependency through this pluggable execution model. For a few reasons I
don’t think that approach makes much sense:
1. This opens up Spark internals and adds very invasive new unstable API’s. As
of Spark 1.0 we’ve stabilized the API’s and made the extension points of Spark
fairly clear. Things like runJob are internal hooks into the scheduler we do
not want developers to override. An actual implementation would likely need to
extend/access a bunch of other internals like Mridul mentioned.
2. There is existing work to make Spark scale elastically on YARN (see
SPARK-3174). The current work will apply to all modes and not expose new public
API’s. The proposal here is a different architecture, but as I understand, the
main motivation is to make Spark function better in large multi-tenant YARN
clusters.
3. For users, having multiple execution engines would lead to fragmentation and
discrepancies. I'd prefer community efforts around YARN focus on the existing
extension points of Spark and cases where we can make changes that benefit all
deployment modes. This is especially true given that Tez requires YARN and yet
> 50% of Spark users do not use YARN (Mesos, Standalone, etc).
4. In the short term, if someone really wants to prototype this they can extend
SparkContext, use reflection, etc and provide Tez integration. Therefore, I
don’t think we need to have changes in upstream Spark to allow the community to
prototype this. Merging this API upstream would IMO signify the blessing of the
maintainers of Spark core for this approach.
tl;dr: Spark can't depend on Tez in order to function well on YARN. From day
one the design of YARN was to allow multiple applications to run without having
to all depend on each other. My feeling is we should focus instead on improving
Spark-on-YARN via the existing points of extension.
> Allow for pluggable execution contexts in Spark
> -----------------------------------------------
>
> Key: SPARK-3561
> URL: https://issues.apache.org/jira/browse/SPARK-3561
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
> Affects Versions: 1.1.0
> Reporter: Oleg Zhurakousky
> Labels: features
> Fix For: 1.2.0
>
> Attachments: SPARK-3561.pdf
>
>
> Currently Spark provides integration with external resource-managers such as
> Apache Hadoop YARN, Mesos etc. Specifically in the context of YARN, the
> current architecture of Spark-on-YARN can be enhanced to provide
> significantly better utilization of cluster resources for large scale, batch
> and/or ETL applications when run alongside other applications (Spark and
> others) and services in YARN.
> Proposal:
> The proposed approach would introduce a pluggable JobExecutionContext (trait)
> - a gateway and a delegate to Hadoop execution environment - as a non-public
> api (@DeveloperAPI) not exposed to end users of Spark.
> The trait will define 4 only operations:
> * hadoopFile
> * newAPIHadoopFile
> * broadcast
> * runJob
> Each method directly maps to the corresponding methods in current version of
> SparkContext. JobExecutionContext implementation will be accessed by
> SparkContext via master URL as
> "execution-context:foo.bar.MyJobExecutionContext" with default implementation
> containing the existing code from SparkContext, thus allowing current
> (corresponding) methods of SparkContext to delegate to such implementation.
> An integrator will now have an option to provide custom implementation of
> DefaultExecutionContext by either implementing it from scratch or extending
> form DefaultExecutionContext.
> Please see the attached design doc for more details.
> Pull Request will be posted shortly as well
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
