Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20669 That's one scenario where an init-container /might/ help. But be aware that your YARN example only works in a single one scenario - YARN cluster mode, and maybe even then may need some config customization. And it may not even work in some cases (e.g. YARN data directory mounted with `noexec`). YARN client, nor any other supported cluster manager, is able to do what you're saying. Personally I feel it's perfectly ok to require a custom docker image in these cases, since they're so uncommon (I've never seen one of our users use the yarn-cluster feature for this purpose). People can have a "main" Spark image and a "debug" one that can be easily chosen from when submitting the app. During this discussion I think someone mentioned that it might be possible to side-load init containers into Spark without this. I'm not that familiar with kubernetes, but if that's possible, it's another way you could achieve this without Spark having its own init container.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org