Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/539#issuecomment-41703171
This case you describe wasn't originally supported on yarn. Perhaps it
should have been fixed up when sparkConf was added but it looks like it wasn't
so I would consider this more of an improvement rather then a bug.. minor nit.
This patch only makes it work for the yarn-client mode, I would like to see it
work in both if we are changing it. Would you be able to look into
yarn-cluster mode also?
The thing that I noticed while looking at this that I do consider a bug
(but not what you reported in this pr) is that if I specify --name with
spark-submit script with yarn-client mode it doesn't properly set the name on
the RM or in spark. the yarn-cluster mode does work fine though. I will file a
separate jira for that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
