[ https://issues.apache.org/jira/browse/SPARK-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14162351#comment-14162351 ]
Patrick Wendell commented on SPARK-3797: ---------------------------------------- For the dependencies issue - the plan is to create a separate build module that only contains the jar for the shuffle service so we can produce a jar with only this service and not the rest of Spark's dependency graph. This won't have any dependencies except for netty which is already a dependency of YARN and we are using the same version, and potentially the scala library jar (though we've even discussed writing this particular component in Java). I think that fully solves the issues Sandy has mentioned. BTW in general I don't think we are going to require this to run Spark-on-YARN in the future - it will just be a mode that people can run in if they want to have better elasticity. > Run the shuffle service inside the YARN NodeManager as an AuxiliaryService > -------------------------------------------------------------------------- > > Key: SPARK-3797 > URL: https://issues.apache.org/jira/browse/SPARK-3797 > Project: Spark > Issue Type: Sub-task > Components: YARN > Reporter: Patrick Wendell > Assignee: Andrew Or > > It's also worth considering running the shuffle service in a YARN container > beside the executor(s) on each node. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org