[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/29 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36375413 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12931/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36375411 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36372790 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36372766 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36372765 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36372574 Uploaded a rebased patch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36368483 @sryza it looks like this is no longer mergeable due to other check ins. Can you please update it to the latest. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36367636 +1. Looks Good. Thanks Sandy! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36313360 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36313361 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12924/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36311620 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36311619 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36311628 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36311392 Oops, actually pushed it this time. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36293609 No problem. I attached a new patch that does this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36286358 I believe secrets are mostly for users adding secrets for other services. secrets are also used by the MR2 framework. secret keys are supported in MR1 via Credentials, I believe they just added the convenience methods to UserGroupInformation in mr2 for dealing with them. It looks like MR1 handled credentials through the JobConf. So I guess if you don't mind, overriding it for Yarn at least to pass them along might be best. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36275981 I'm not familiar with what the secrets are used for as opposed to the tokens. Do you know the reason they're not needed in the MR1 code? Either way, having YarnSparkHadoopUtils override runAsUser with something that calls addCredentials doesn't sound too difficult. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36253998 Thats really unfortunately that hadoop 1.x doesn't support it, as I would prefer to use the addCredentials since it also handles secrets. Our only option right now would be to add a separate function in SparkHadoopUtils and YarnSparkHadoopUtils. This would allow spark on yarn to use the addCredentials at least. Maybe its not critical right now. thoughts? Otherwise changes look good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36226281 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/12914/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36226279 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36223994 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36223980 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/29#issuecomment-36223981 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1051. On YARN, executors don't doAs subm...
GitHub user sryza opened a pull request: https://github.com/apache/spark/pull/29 SPARK-1051. On YARN, executors don't doAs submitting user This reopens https://github.com/apache/incubator-spark/pull/538 against the new repo You can merge this pull request into a Git repository by running: $ git pull https://github.com/sryza/spark sandy-spark-1051 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/29.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #29 commit f2a3c24ebbc6f89be5f88c4a3f34a3c05ff8a5c2 Author: Sandy Ryza Date: 2014-02-06T01:43:33Z SPARK-1051 second way --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---