[ https://issues.apache.org/jira/browse/SPARK-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161703#comment-14161703 ]
Reynold Xin commented on SPARK-3797: ------------------------------------ What if we support using AuxiliaryService to run the shuffle service, but also a mode that runs the shuffle service directly in each executor? That should provide a good compromise. It shouldn't be too hard to support both. > Run the shuffle service inside the YARN NodeManager as an AuxiliaryService > -------------------------------------------------------------------------- > > Key: SPARK-3797 > URL: https://issues.apache.org/jira/browse/SPARK-3797 > Project: Spark > Issue Type: Sub-task > Components: YARN > Reporter: Patrick Wendell > Assignee: Andrew Or > > It's also worth considering running the shuffle service in a YARN container > beside the executor(s) on each node. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org