On 19 Mar 2016, at 02:25, Koert Kuipers <ko...@tresata.com<mailto:ko...@tresata.com>> wrote:
spark on yarn is nice because i can bring my own spark. i am worried that the shuffle service forces me to use some "sanctioned" spark version that is officially "installed" on the cluster. so... can i safely install the spark 1.3 shuffle service on yarn and use it with other 1.x versions of spark? That's an interesting question. I don't know the answer there. Normally I'd just say "use the latest, hope they got compatibility right". Spark 1.6.1 fixed some classpath issues related to shuffle JAR and hadoop namenode dependencies. There's a YARN JIRA on isolating nodemanager plugins, first with classpath, ideally with forked processes (for failure resilience and isolation of native libs). I guess a goal there should be "supports different versions of the same app"; that may need app changes doo
