[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77941 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77941/testReport)** for PR 17723 at commit [`c684d88`](https://github.com/apache/spark/commit/c684d8810824d18f9c9bacdddc69dad4c6fe325b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17723 Thanks! Will review it tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77822/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77822 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77822/testReport)** for PR 17723 at commit [`563b80a`](https://github.com/apache/spark/commit/563b80ab7da0e0f5ac0c615bca46a2a9eae58122). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77820/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77820 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77820/testReport)** for PR 17723 at commit [`7e2f90d`](https://github.com/apache/spark/commit/7e2f90d083c9bb3a9ab1ebc2edede92e1eb6259f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77822 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77822/testReport)** for PR 17723 at commit [`563b80a`](https://github.com/apache/spark/commit/563b80ab7da0e0f5ac0c615bca46a2a9eae58122). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77820 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77820/testReport)** for PR 17723 at commit [`7e2f90d`](https://github.com/apache/spark/commit/7e2f90d083c9bb3a9ab1ebc2edede92e1eb6259f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 Thanks. I updated the description. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 You still haven't updated the PR description. It still references things that you have removed from the PR. I tested the latest version with our kerberos tests and so far it looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/9/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #9 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/9/testReport)** for PR 17723 at commit [`4d57f7b`](https://github.com/apache/spark/commit/4d57f7bb76e424b48e71b18250cb018e09e2d173). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #9 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/9/testReport)** for PR 17723 at commit [`4d57f7b`](https://github.com/apache/spark/commit/4d57f7bb76e424b48e71b18250cb018e09e2d173). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 @vanzin tests are passing. ready for re-review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77761/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77761 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77761/testReport)** for PR 17723 at commit [`1479c60`](https://github.com/apache/spark/commit/1479c60b3059e17a29e23a309f1b38e364bb2451). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77761 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77761/testReport)** for PR 17723 at commit [`1479c60`](https://github.com/apache/spark/commit/1479c60b3059e17a29e23a309f1b38e364bb2451). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77688/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77688 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77688/testReport)** for PR 17723 at commit [`7796e14`](https://github.com/apache/spark/commit/7796e14791f73f6bc97d9b7d4c891cbf48add8f0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77688 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77688/testReport)** for PR 17723 at commit [`7796e14`](https://github.com/apache/spark/commit/7796e14791f73f6bc97d9b7d4c891cbf48add8f0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77686 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77686/testReport)** for PR 17723 at commit [`0ffe8f0`](https://github.com/apache/spark/commit/0ffe8f077867ebf4eb84c0e27f98e475e2b0af9d). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77686/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77686 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77686/testReport)** for PR 17723 at commit [`0ffe8f0`](https://github.com/apache/spark/commit/0ffe8f077867ebf4eb84c0e27f98e475e2b0af9d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77683/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77683 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77683/testReport)** for PR 17723 at commit [`376dba0`](https://github.com/apache/spark/commit/376dba09a490afc68285d26025f45af8b9da5510). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77683 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77683/testReport)** for PR 17723 at commit [`376dba0`](https://github.com/apache/spark/commit/376dba09a490afc68285d26025f45af8b9da5510). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 @vanzin All comments addressed. Ready for re-review (after tests pass). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 Thanks @mridulm. We'll move forward without you on this one. Have a great vacation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17723 Hi @mgummelt, I am quite tied up and will be back from vacation July mid - which is when I will have bandwidth for reviews (which is, IMO, too long to wait for this PR !). I see that Wenchen Fan is already taking a look along with @vanzin, I will try to follow the discussions, but wont be able to participate unfortunately. Thanks for the great work ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 Perfect, thanks. @mridulm Any thoughts? He might still be out on vacation... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 I'm super busy with things as usual. But I hope to be able to take a look at this sometime this week. I'm also sort of waiting for Mridul to chime in since he's the one that's super concerned about the API here... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 Hey guys, this has been open for > 1 month now and I'd love to get it over the line. Are there any other concerns or questions I can answer? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 @gatorsmile @vanzin @mridulm Any thoughts here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 @gatorsmile All comments addressed. Ready for re-review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77259/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77259 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77259/testReport)** for PR 17723 at commit [`e820b09`](https://github.com/apache/spark/commit/e820b093d4d9c8ba18b3e5d71d04de1791426264). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77259/testReport)** for PR 17723 at commit [`e820b09`](https://github.com/apache/spark/commit/e820b093d4d9c8ba18b3e5d71d04de1791426264). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77232/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77232 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77232/testReport)** for PR 17723 at commit [`e820b09`](https://github.com/apache/spark/commit/e820b093d4d9c8ba18b3e5d71d04de1791426264). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77232 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77232/testReport)** for PR 17723 at commit [`e820b09`](https://github.com/apache/spark/commit/e820b093d4d9c8ba18b3e5d71d04de1791426264). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77212 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77212/testReport)** for PR 17723 at commit [`bf758e6`](https://github.com/apache/spark/commit/bf758e64f699f3211e15767d0f1854cd47cecb29). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77212/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77212 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77212/testReport)** for PR 17723 at commit [`bf758e6`](https://github.com/apache/spark/commit/bf758e64f699f3211e15767d0f1854cd47cecb29). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77210 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77210/testReport)** for PR 17723 at commit [`cd58b6c`](https://github.com/apache/spark/commit/cd58b6c3e6d498a756a8e61d2c25d96b352537c1). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77210/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17723 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77210 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77210/testReport)** for PR 17723 at commit [`cd58b6c`](https://github.com/apache/spark/commit/cd58b6c3e6d498a756a8e61d2c25d96b352537c1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77065 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77065/testReport)** for PR 17723 at commit [`92ac3f0`](https://github.com/apache/spark/commit/92ac3f0790647f2159f02766a8cb20077ee6c9e5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 I have: * Made ServiceCredentialProvider private * Made ConfigurableCredentialManager load a hardcoded list of (HDFS, Hive, HBase) providers, rather than service loading * Renamed ServiceCredentialProvider to HadoopDelegationTokenProvider, a more specific name * Addressed all style comments Ready for review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17723 **[Test build #77064 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77064/testReport)** for PR 17723 at commit [`38adaae`](https://github.com/apache/spark/commit/38adaaec80cd172f7b5b92e5b85e78f5879577eb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 I'm fine with that if you're ok with Mesos being restricted to the built-in providers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 @jerryshao Are those providers different than the Hive and HBase providers already in the Spark codebase? Regardless, with what I'm proposing, the `yarn.ServiceCredentialProvider` would remain, so you would still retain the ability to plugin those providers for the YARN scheduler to use. It's just the Mesos scheduler (or any other scheduler that uses core) that wouldn't support pluggable providers. I'm not worried about this, because I haven't seen any demand from Mesos users for any provider other than HDFS, Hive, and HBase. We can expose it once, if ever, this demand arises, but by keeping it private for now, we gain the option to redesign the interface if a new provider type emerges, which is one of the things @vanzin and I were worried about. And we still wouldn't be exposing Hadoop types publicly, which is what rxin and @mridulm were worried about. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17723 @mgummelt We have in house delegation provider for HiveServer2, multi HBase cluster. I think this is useful in Hadoop world. So better to keep it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 I'm working on this now, and am definitely willing to execute the plan we've agreed on, but the more I think about it, the more I think it makes sense to make `ServiceCredentialProvider` private and not pluggable. The benefit is that we don't have to design a possibly incorrect delegation token interface until we actually have another implementation to design against. The cost is that Mesos users would not be able to provide their own delegation token implementations. However, YARN users still would, since the `yarn.ServiceCredentialProvider` would still exist. I would remove the deprecation. @vanzin @mridulm Does this work for both of you? We briefly considered this at the beginning of this PR conversation, and I think both of you suggested it's acceptable. Out of curiousity, what was the benefit of making delegation token provision pluggable in the first place? What other Hadoop delegation token providers other than HDFS, HBase, and Hive are there? What was the be --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 BTW, if @mgummelt is ok with spending time working on this super generic interface, then it's up to him. I find it unnecessary and a waste of resources at this point in time, but it's his time. :-) At best it will prove useful, most probably it will forever live with a single Hadoop-specific implementation, and at worst we'll have to change it to accommodate a system we did not predict, which is basically the same risk we have with just exposing the Hadoop one in the first place. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 > On the contrary, this is handled if you read the proposal - ServiceTokenManager.acquireTokens will do the necessary setup Yes, that's approach (b) which I said still assumes that the interface you created is enough for other security systems. Which is a big if, since we don't know any of them. > Given rxin's comment above, I am actually more convinced that taking a hadoop-security only route is detrimental Reynold mentioned that a lot of users don't need Hadoop security, which is true. But he didn't mention any other security system that actually needs anything like this, so that's not an argument for a super generic interface. Which is what I've been saying all along. You don't know of any other system that needs anything like this, so you don't know if the interface you're creating will be useful to them or cover all their needs. If there the least bit of difference in how this hypothetical system works, your abstraction goes down the drain. Instead, we have actual use cases for Hadoop security. They don't benefit from the abstraction you're trying to create at all, since internally they still need to deal with UGI and all the rest of the Hadoop security API. So why force the abstraction right now? I'm not saying that abstraction are bad. I'm just saying that at this point they're completely unnecessary and avoidable, and I haven't seen any use case for them here. Give me a *single* use case and I'll agree with you, but I haven't seen one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17723 @vanzin > But it's just a small piece of the puzzle. For example, for Hadoop token providers, obtainCredentials has to be called with a very specific environment set up depending on when it's called. On the contrary, this is handled if you read the proposal - ServiceTokenManager.acquireTokens will do the necessary setup : before acquiring tokens from BaseServiceTokenProvider's which match its service type. To elaborate: * configure - grab principal/keytab/configure itself based on the SparkConf provided. * acquireTokens - setup environment specific to the model. * Execute BaseServiceTokenProvider.obtainCredentials * In case of hadoop security, obtainCredentials will be invoked within ugi setup; in case of other models, it could be something else - that is an implementation detail which must not leak out. * serialize result to Array[Byte] and return. * applyTokens - at the executor's - for application of the obtained bytes. * Each ServiceTokenManager would have different deserialized object representing tokens, and different semantics on how tokens are applied to the executor environment. The whole point of generalizing is that existing functionality should continue to work, without needing to depend on specifics of hadoop-security. Given @rxin's comment above, I am actually more convinced that taking a hadoop-security only route is detrimental to future integration with other systems - I just did not have examples to elaborate my point. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 > b) Eliminate ConfigurableCredentialManager entirely, and have each ServiceTokenProvider load their specific sub-interface of BaseServiceTokenProvider; which gives better type safety. This may solve that particular problem, but why? What are you gaining from it? What if you never have to implement any different credential provider, because there's no need to? That's my main problem with your suggestion. You're trying to create something generic without any specific need for it other than wanting to create something generic. You're assuming that at some time in the future this generality will pay off, but you don't know. On the other hand, there's a lot of simplification that can be achieved by kicking the "let's create something generic" can down the road. > The life cycle is fairly generic IMO, But it's just a small piece of the puzzle. For example, for Hadoop token providers, `obtainCredentials` has to be called with a very specific environment set up depending on when it's called. For example, when creating new delegation tokens to work around the 7-day TTL, a new UGI has to be created by logging in with the user provided keytab, or no new delegation tokens will be created (because the currently active UGI still has valid tokens). So you've abstracted some aspects of how the mechanics of shipping tokens back and forth works, but you haven't abstracted enough that it's possible to write this feature without baking in logic about how Hadoop security works. What I'm trying to say is that it's a futile goal to try to create this generic interface now. You're going to end up with either of the following: - an overly generic interface, and make Hadoop token providers (which are the very thing we want to support!) way more complicated to code, since all the semantics that are now in the internal Spark code would be pushed to the providers. I really don't like this, because to be useful the interface would become more complicated, so that all things related to Hadoop are exposed in some way (e.g. "here's the kerberos keytab so that you can create your own UGI so that you can create new delegation tokens"). - a generic interface with a single Hadoop-specific implementation, which exposes its own interface so that Hadoop token providers can avoid having to re-implement all of the Hadoop-specific semantics. I assume this is what you mean with your (b) suggestion I quoted above. I really don't like the first option, and I think it's premature to invest time in the second option. So instead, what I'm proposing is to just expose the Hadoop-specific one. If it helps, change names to make it really clear that it's really specific to how Hadoop security works. Mask Hadoop names if desired, so that you have wrappers, and the API Spark exposes is not dependent on Hadoop (even though it is at runtime). And then, when we have a second use case for something like this, then think of how you can merge the two with a generic interface, if that would be even possible. You will have a ton more context about what's needed at that time, instead of trying to guess it up front. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17723
@vanzin:
>> // Modify ConfigurableCredentialManager to segregate by serviceType of
ServiceTokenProvider
> See, you're already making assumptions that these other systems use
credentials in a certain way.
serviceType defines which ServiceTokenManager to use for
configuring/managing the BaseServiceTokenProvider loaded - there is nothing
specific about how credentials are used.
Probably should have been more detailed in my proposal, apologies for the
confusion - detailing it below:
We can do it two ways :
a) I was initially envisioning ConfigurableCredentialManager will
essentially be doing only service loader lookup for BaseServiceTokenProvider
and partitioning the loaded BaseServiceTokenProvider's by serviceType.
All other existing methods (iirc token acquisition ?) will move into
YarnServiceTokenManager.
The only reason to do this would be for each ServiceTokenManager to not
need to do service load.
b) Eliminate ConfigurableCredentialManager entirely, and have each
ServiceTokenProvider load their specific sub-interface of
BaseServiceTokenProvider; which gives better type safety.
And perhaps also eliminate the confusion above ?
>> override def acquireTokens(manager: ConfigurableCredentialManager):
Array[Byte] = {
> Now you're exposing ConfigurableCredentialManager as a public API, which
it was never meant to be.
See above if we dont want to expose ConfigurableCredentialManager.
As I mentioned above, other than as a container for
BaseServiceTokenProvider, ConfigurableCredentialManager does not really do
anything else now.
The token acquisition is an implementation detail of the specific
ServiceTokenProvider and will move into that.
>> This ensures that future requirements are reasonably captured
> No, you just created an interface that is Hadoop security without Hadoop
types. That does not take into account any other service type. You're just
assuming that they'll behave like Hadoop services.
The life cycle is fairly generic IMO, and actually matches general SPI
patterns - not sure why you think it is specific to hadoop-security, probably I
am missing something ?
It currently does :
* configure/initialize
* invoke
* apply
Perhaps you refer to the specificity that configure and invoke happens in
driver and application in executor ?
> I really don't understand what the abstraction is buying anybody at this
point. If you don't know what other types of services you want to support, by
definition you can't create an abstraction.
The abstraction defines a general lifecycle of how spark expects
ServiceTokenManager to conform to - without tying it to any specific
implementation.
If the concern is that it requires more iteration/evolution, I am fine with
that.
Bottomline is, unless there is consensus that we want to adopt hadoop
security as the model for authenticating/authorizing not just existing - we
should not be dependent on it.
> How can we make progress? Does @mridulm get a -1 (veto) on this PR?
Veto is an extreme measure, and not my intention :-)
It is not like I dont understand where @vanzin is coming from, and I agree
with him for the most part - and he probably understands hadoop security way
better than I do.
As @vanzin mentioned, I am just more queasy about interface change, even if
marked Unstable :-)
> To complicate matters even further, I'll be on vacation next week. Feel
free to continue discussions, but I'll be absent until May 15.
Ah, looks like it is not just me who will be leaving for vacation soon !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 OK, I'll submit an updated version once I get back from vacation around May 15th. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 > I'm saying avoid exposing Hadoop APIs. Wrap them around something if possible. If that's all you'd like to see, it's not hard. But at the same time, it doesn't solve a whole lot of problems; as I mentioned, both the implementation of the provider and the Spark code calling into the provider have to use Hadoop APIs, so they still have to agree at some point. To try to give a more concrete example: let's say that we hide Hadoop types, and now some credential provider uses a UGI API that's only available in Hadoop 2.9, and someone tries to run it on Spark built Hadoop 2.6. Even though Spark hasn't exposed the Hadoop API, the dependency leaks, and things will still not work. > it looks like the best way to make progress is to generalize the interface. Are you OK with this? If masking the Hadoop types in the API make people happy, sure, do that. I don't see much benefit, since the implementation will still be pretty coupled to Hadoop one way or another, but it's not a big change from the current change anyway. I just don't want this to turn into "let's solve the problem of how to support all the security frameworks out there", when we have no idea of what other security frameworks we even might want to support. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 @vanzin I agree with you, but it looks like the best way to make progress is to generalize the interface. Are you OK with this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17723 I'm saying avoid exposing Hadoop APIs. Wrap them around something if possible. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 > I didn't read through the super long debate here, but I have a strong preference to not expose Hadoop APIs directly. I'm seeing more and more deployments out there that do not use Hadoop (e.g. connect directly to cloud storage, connect to some on-premise object store, connect to Redis, connect to some netapp appliance, connect directly to a message queue or just run Spark on a laptop). Are you saying exposing the Hadoop API in a public interface (like ServiceCredentialProvider) would make it more difficult to integrate with non-Hadoop data stores? How so? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17723 I didn't read through the super long debate here, but I have a strong preference to not expose Hadoop APIs directly. I'm seeing more and more deployments out there that do not use Hadoop (e.g. connect directly to cloud storage, connect to some on-premise object store, connect to Redis, connect to some netapp appliance, connect directly to a message queue or just run Spark on a laptop). Hadoop APIs were designed for a different world pre Spark. Serialization is painful (Configuration?) to deal with, API breaking changes are painful to deal with, size of the dependencies are painful to deal with (especially considering the single node use cases in which ideally we'd just want a super trimmed down jar). As you can see (although most of you that have chimed in here don't know much about the new components), the newer components (Spark SQL) does not expose Hadoop APIs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 Technically anyone can -1 a change, AFAIK. In practice I see that done very rarely. To clarify my position, I see it as counter-productive to try to create an abstraction now. We don't know what this abstraction would look like, so as Mridul's example proves, the abstraction is basically just Hadoop-security-without-Hadoop-types. We don't know if that's going to be useful to anyone else, which makes its standing as an abstraction questionable. Exposing Hadoop security directly at this point has a single drawback I can see: potentially exposing Hadoop types in the API. That's already done in a bunch of parts of core, so I don't see that as a strong argument. And hiding those from the API doesn't really hide them from the implementations; the Hadoop credential providers will still deal with UGI and Credentials internally, and since everybody is running under the same class loader, they'll be the same classes. So there's still tight coupling, just not explicit in the API. Also, exposing Hadoop security this way does not preclude adding an abstraction (like the one Mridul sketched out) later. Assuming that there exists a class of services that has a very similar requirement for how to create and distribute credentials, you just add the abstraction at that point, and have an implementation of the new interface that wraps the old Hadoop interface. It's pretty easy to do, and avoids having to figure out what this hypothetical interface would look like right now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 @vanzin @mridulm I think we've reached the crux of the disagreement. @mridulm thinks we should generalize ServiceCredentialProvider. @vanzin and I think this would do more harm than good since we don't have another delegation token provider to generalize against. You two might know the PR protocol better than I do. How can we make progress? Does @mridulm get a -1 (veto) on this PR? If so, I'm willing to just write the generalization in order to make progress, unless of course @vanzin is a -1 on that. Should I open this up to dev@? Maybe add a deadline for comments and see where we are after that? What if we're still split? To complicate matters even further, I'll be on vacation next week. Feel free to continue discussions, but I'll be absent until May 15. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17723 I see. Will try to catch the whole history. Thanks! @mgummelt @vanzin Also cc @jerryshao I just read his [design doc](https://docs.google.com/document/d/1Y8CY3XViViTYiIQO9ySoid0t9q3H163fmroCV1K3NTk/edit#). It sounds like he also expect moving `ServiceCredentialProvider` and `ConfigurableCredentialManager` from `Yarn` to `core` for supporting his proposed Credentials push command. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 @gatorsmile Yes. This PR originally contained the Mesos propagation code as well, but I split it out to simplify the review process (can't you see how simple it's become!) :) The old PR is here: https://github.com/apache/spark/pull/17665 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 @gatorsmile > Does that mean Mesos plans to use a different solution for delegation token management? There's still Mesos code to be added for this; if you trace back through the JIRA links, Michael submitted a PR earlier that had both this change + the Mesos code, I just asked him to separate the two. At the end, both YARN and Mesos code will call into these APIs to create delegation tokens; there might be some extra code needed in Mesos, since YARN already has built-in support for managing the initial token set (distribution and renewal). But creating and distributing new delegation tokens after the TTL of the initial set should be the same for both. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17723
> As I detailed above, the exposed spi contract is not just the interface,
but also how it is to be used by spark.
There's very little lee way on that front. Those semantics are mostly
already defined by Hadoop security; as I said, you can try to create some
abstraction on top of that, but it would be abstraction for the sake of
abstraction: the Hadoop implementation would basically be exactly this code,
and we don't have an example of any other types of services to see if the
abstraction actually makes sense for them.
> It would ensure that spark is not tied to hadoop security
There's nothing tying Spark to Hadoop security here. There's code that
allows Spark to use Hadoop services with security enabled, but that does not
preclude other types of services from being used. I've given many examples of
that already.
> Once interfaces are exposed, and have been exposed for a few releases, it
is exceedingly difficult for users if we break/modify them
Then we make an effort to not break them. Adding an abstraction that does
not abstract anything useful does not help with that. If anything, it makes
things worse.
> // Modify ConfigurableCredentialManager to segregate by serviceType of
ServiceTokenProvider
See, you're already making assumptions that these other systems use
credentials in a certain way.
> override def acquireTokens(manager: ConfigurableCredentialManager):
Array[Byte] = {
Now you're exposing `ConfigurableCredentialManager` as a public API, which
it was never meant to be.
> // Spark core itself becomes decoupled from hadoop-security with the
above.
It doesn't. You just created an interface that mimics the Hadoop security
interfaces, and if you change the semantics of those, you'll end up breaking
Hadoop security.
> This ensures that future requirements are reasonably captured
No, you just created an interface that is Hadoop security without Hadoop
types. That does not take into account any other service type. You're just
assuming that they'll behave like Hadoop services.
So, again, abstraction for the sake of abstraction. It is much simpler if
instead of trying to do this, you leave them as separate concerns. The
Hadoop-based security interface exposes Hadoop semantics. If later there's a
different type of service, it will probably need to expose a different public
interface for services to provide their credentials. You're just guessing that
they behave similarly enough to Hadoop that your proposed interface will
suffice.
Then, whether internally to Spark those two can be handled similarly (so
that code can be shared) is a different question that can only be answered when
those services are known.
> // Invoked in executor
> def applyTokens(data: Array[Byte])
This is maybe the only good idea I see in your proposal, but at the same
time, it's already internally handled by Spark with the current code and
doesn't need to be exposed to the credential provider implementations.
I really don't understand what the abstraction is buying anybody at this
point. If you don't know what other types of services you want to support, by
definition you can't create an abstraction. So, as I pointed out, your
abstraction looks exactly like the only type of service it does support -
Hadoop. Which makes it kinda pointless as an abstraction.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17723 Based on the PR title `Move kerberos delegation token code from yarn to core`, it sounds like it moves the whole delegation token logics into core. I found it does not move the logics for pushing and propagating delegation tokens. Does that mean Mesos plans to use a different solution for delegation token management? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mridulm commented on the issue:
https://github.com/apache/spark/pull/17723
> The only public interface being added in this change is
ServiceCredentialProvider. It's an interface that service-specific libraries
(e.g. a Solr connector, or a Kudu connector) would extend, and user
applications would never touch. It's also an existing interface, which was in
the YARN module but, really, has functionality that is independent of YARN.
As I detailed above, the exposed spi contract is not just the interface,
but also how it is to be used by spark.
It includes the environment `obtainCredentials` is invoked within, use of
`Credentials` as container for tokens and how `Credentials` are leveraged in
driver/executor.
This, in entirety, defines the public contract - not just the methods in
the interface : and this is dependent on how hadoop-security
authenticates/authorizes services.
> And, on top of that, I don't see a point in abstracting it further. It
would just be abstraction for the sake of abstraction.
It would ensure that spark is not tied to hadoop security, but can continue
to leverage it for managing long running applications/services.
Once interfaces are exposed, and have been exposed for a few releases, it
is exceedingly difficult for users if we break/modify them - irrespective of
the stability annotation's we add.
A straw man pseudocode proposal which elaborates is:
```
@Unstable
// rename
trait BaseServiceTokenProvider[C, T] {
// Must match a supported ServiceTokenManager.serviceType
def serviceType: String
def serviceName: String
def credentialsRequired(conf: C): Boolean
def obtainCredentials(conf: C, tokenContainer: T): Option[Long]
}
// Existing ServiceCredentialProvider becomes
@Unstable
trait ServiceCredentialProvider extends
BaseServiceTokenProvider[Configuration, Credentials] {
override def serviceType: String =
ServiceTokenManager.HADOOP_SECURITY_SERVICE_TYPE
}
// Modify ConfigurableCredentialManager to segregate by serviceType of
ServiceTokenProvider
// which will be queried by ServiceTokenManager. If unknown serviceType,
then log error.
@Unstable
trait ServiceTokenManager {
val HADOOP_SECURITY_SERVICE_TYPE = "hadoop-security"
def serviceType: String
// Return false if validation fails. Invoked in driver.
def configure(conf: org.apache.spark.SparkConf): Boolean
// Invoked in driver. If required, application of tokens to driver to be
done as part of
// acquireTokens
def acquireTokens(manager: ConfigurableCredentialManager): Array[Byte]
// Invoked in executor
def applyTokens(data: Array[Byte])
}
// Existing implementation will be encapsulated within this class.
private[spark] class YarnServiceTokenManager extends ServiceTokenManager {
override def serviceType: String = HADOOP_SECURITY_SERVICE_TYPE
// In driver
override def configure(conf: SparkConf): Boolean = {
// save principal, keytab; create freshHadoopConf
// configure
true
}
override def acquireTokens(manager: ConfigurableCredentialManager):
Array[Byte] = {
val ugi = UGI.loginUserFromKeytabAndReturnUGI(principal, keytab)
val tempCreds = ugi.getCredentials
ugi.doAs {
manager.getCredentialProviders(serviceType).flatMap { provider =>
// existing code in ConfigurableCredentialManager.obtainCredentials
}.foldLeft(Long.MaxValue)(math.min)
}
// apply it to driver
UGI.currentUser.addCredentials(tempCreds)
// serialize tempCreds to Array[Byte]
serialize(tempCreds)
}
override def applyTokens(data: Array[Byte]): Unit = {
// deserialize data into Credentials; apply it to UGI.currentUser
}
}
// Spark core itself becomes decoupled from hadoop-security with the above.
// It also allows us to add other models if applicable.
// Changes to driver to leverage above would essentially be:
// a) load ServiceTokenManager's via ServiceLoader.
// b) create CredentialManager - which loads BaseServiceTokenProvider via
service loader.
// b.1) Reject any token provider which has a service type not loaded in (a)
// c) configure (a)
managers.foreach(_.configure(sparkConf))
// token acquisition
val acquiredTokenMap = managers.map(manager => (manager.serviceType,
manager.acquireTokens(credentialManager)))
// serialize and send acquiredTokenMap to executors.
// At executors, application is:
acquiredTokenMap.map {
case (managerType, tokenData) =>
serviceTokenManagers(managerType).applyTokens(tokenData)
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 To ask a more direct question: The only public interface being added in this change is `ServiceCredentialProvider`. It's an interface that service-specific libraries (e.g. a Solr connector, or a Kudu connector) would extend, and user applications would never touch. It's also an existing interface, which was in the YARN module but, really, has functionality that is independent of YARN. So the question is: why is it a bad thing to expose this interface more widely? It serves a specific purpose (provide a way for services that use Hadoop-style security to integrate with Spark applications). It doesn't dictate that all Spark code must use Hadoop delegation tokens, it just allows Spark applications to better integrate with secure Hadoop services. And, on top of that, I don't see a point in abstracting it further. It would just be abstraction for the sake of abstraction. If a different class of secure services requires something like that at some point, we can evaluate then what to do. But right now, that class of services *does not exist*. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 > a) we have explicitly based our support on it What does this mean? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 > This was my point - we should not introduce system specific api's into spark core infrastructure api's/spi's Sorry, I still have no idea what your point is. How do you suggest we support Hadoop security without having code to do it? How can a Spark application running on Mesos talk to secure HDFS without this code? Please explain that, because so far, I'm really confused about exactly what you think this change is about. Adding support for Hadoop security does not preclude adding support for other systems. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17723 @vanzip wrote: > So, this is purely about handling Hadoop authentication for Hadoop services. This was my point - we should not introduce system specific api's into spark core infrastructure api's/spi's : unless a) we have explicitly based our support on it, or b) generalized it sufficiently that we can support others, or c) keep it an impl detail in core (but exposed in yarn for backward compatibility). IMO (a) or (b) require a dev@ discussion. Until now, this (hadoop security) support was restricted to yarn in spark (with a couple of minor other uses iirc). @mgummelt: > The only thing the new ServiceCredentialProvider interface enforces is that the credentials must be added to a Credentials object, which is a hadoop class. The spi makes assumptions about the environment within which the credential provider is invoked, how the tokens are updated at driver/executor as well in addition to use of Credentials - and these are driven by hadoop security DT design. > Do we need to generalize ServiceCredentialProvider to support non-hadoop delegation tokens? IMO that depends on what the answer to the design choice above is. If (a) or (c) - then no. If (b), then yes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 > Do we need to generalize ServiceCredentialProvider to support non-hadoop delegation tokens? I don't see a need for that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 > but I am unsure about kubernetes (perusing their documentation left me unsure) or any others That's kinda orthogonal. This PR isn't Mesos-specific. Mesos, AFAICT, does not support Kerberos (or so I've gathered from Michael's comments on JIRA and here). But that has no bearing on whether applications running on Mesos can talk to Hadoop services that use kerberos. So, this is purely about handling Hadoop authentication for Hadoop services. The authentication used to launch the application on Mesos has nothing to do with the auth used to talk to HDFS or other services. The same applies if you replace Mesos with any other cluster manager. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 I also have to admit that even after all this discussion, I'm still unclear what the concern is. Hadoop security classes, such as `UGI` are already exposed to users. For example, through `--proxy-user`. Also, even if `ServiceCredentialProvider`, which depends on hadoop, is a public interface, that doesn't mean we can't support non-hadoop security. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 > If I understand from @mgummelt, mesos does not require it Mesos has its own authn/authz system, but it has nothing to do with this PR: https://github.com/apache/spark/blob/master/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala#L82 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17723 @vanzin Thanks for the details and examples, that definitely clarifies things better for me ! I agree with you that opening it up, even if as unstable api, would be better than keeping it private[spark] (hence why I started down this tangent to begin with :-) ). My main concern, as I mentioned before, has been whether we would need to support auth models different from hadoop security or not. If I understand from @mgummelt, mesos does not require it - but I am unsure about kubernetes (perusing their documentation left me unsure) or any others. As I mentioned before, would be good to atleast solicit opinions on the dev@ list on this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 > Support for long running applications (which require token renewal, etc) was added much later in spark That's different and not what this change is about. Support for Hadoop security (i.e. delegation tokens) has existed at least since Spark 1.0 (I'm hazy before that since that's when I started playing with Spark). And it doesn't change the fact that it's the only custom security framework that people have ever tried to use with Spark, as far as I'm aware. Hadoop security is different and puts a lot of burden on clients to do things; there are good reasons for that, but it means that it's not as simple as just providing a password. I wish there was a library that made all this easier (wouldn't it be great if there was a single service to contact and ask for "delegation token for service X", like the Kerberos TGS?), but that's not the case. I can think of 3 other types of systems that Spark supports (directly or through extensions): - Those with no security. e.g. Kudu (as far as I know that's still in the roadmap), Kafka 0.8, etc - Those that are happy with just a simple secret stashed somewhere. e.g. S3, JDBC drivers, etc. Even though I've never seen it, I also count cert-based authentication here, since I'm pretty sure you can achieve that with existing features in Spark. - Systems that implemented Kerberos-based auth but not Hadoop delegation tokens. (Looking at you, Kafka 0.10.) That means it's really hard to use those services in a distributed environment where security is enabled. None of those require code in Spark to handle things specially (the third would, but then you'd run into the good reasons why delegation tokens exist in the first place, so they really should start using them instead). > If we are not exposing an api for spark core, while maintaining backward compatibility I guess I'm a little less queasy than you are about exposing an unstable API. That's what the "Unstable" annotation means to me. It's an API that is still being designed, and exposing it serves as both a way to let people write extensions that fit the model and also collect feedback about things that don't fit well. I have issues with the things you're suggesting for a few different reasons. - keeping the API private means people won't extend it, so we won't get feedback if there's any. - moving the API to a separate module is a distinction without a difference. It will still be a public Spark API, and still should follow the rules of backwards compatibility. It would just increase coupling since it's very unlikely that core wouldn't call into that module (since many people have asked for Hadoop auth support in standalone too - I have some issues with the security model there but it's a separate discussion). - trying to work on an abstract interface to rule them all is a wild goose chase. Unless you can point me to the contrary, we don't have an example of what a different system would look like, so whatever abstract interface we end up with will still be heavily modeled after the Hadoop system. Not exposing Hadoop types is not a great gain if the whole mechanism still works like Hadoop security. (It makes Spark's handling of backwards compatibility easier, probably, but here the model is more important than the types exposed in the API.) Yes, exposing an unstable interface risks more work in the future. We may have to change it, and then we have to decide whether to write code to keep compatibility, or break all users. It's a risk, but at the same time, we have a few years of only needing to support this model, so it doesn't seem like it's that big of a risk to me. In the case that this elusive other system shows up in the future, we'll probably have a lot more issues; we could try to merge both APIs or just handle it with a completely separate one. In either case, there will be work. So I'm really not seeing the benefit of going out of our way to mitigate future work. It's better to do that work when we have a better idea of what it even looks like. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17723 @vanzin: > @mridulm the main argument for just dealing with Hadoop security is that it's been sufficient since the inception of Spark. I have never seen anyone ask for integration with any other type of system. Support for long running applications (which require token renewal, etc) was added much later in spark - in 1.4 IIRC : https://issues.apache.org/jira/browse/SPARK-5342 > Would it make you more comfortable if the new API were kept private[spark]? It would limit extensibility in the case of Mesos (would be restricted to built-in providers), but would free us from this discussion and allow progress to happen. If we are not exposing an api for spark core, while maintaining backward compatibility - I am fine with the change. (Please see [1] below too) Either we move implementations from yarn to core - or a new module which yarn and mesos depends on (if that helps). @mgummelt: > @mridulm You keep mentioning hadoop-security as if it's a library. It's not. UserGroupInformation and Credentials, for example, are are security classes in hadoop-commons, which core already depends on. So this coupling already exists. Is your concern that we're increasing this coupling? When I refer to hadoop-security, I do not mean a maven package - but use of `org.apache.hadoop.security` for handling authentication/authorization. hadoop-common contains a collection of libraries & utilities, bundled together for convenience; `security` package implements classes in context of security design of hadoop. [1] If we want to base security in spark on hadoop-security, and think it is sufficient for our current & anticipated needs - we should be explicit about the (design) dependency. We should solicit opinion about it in dev@ and proceed based on feedback from from the list (this PR discussion might not be followed by many interested parties). As @vanzin mentioned above, there have not integration requests for other systems - so perhaps it is sufficient for our needs and I might be being overly paranoid (due to my past experiences with api design). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mgummelt commented on the issue: https://github.com/apache/spark/pull/17723 > My point is that spark core should not depend on hadoop-security; @mridulm You keep mentioning `hadoop-security` as if it's a library. It's not. `UserGroupInformation` and `Credentials`, for example, are are security classes in `hadoop-commons`, which `core` already depends on. So this coupling already exists. Is your concern that we're increasing this coupling? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 @mridulm the main argument for just dealing with Hadoop security is that it's been sufficient since the inception of Spark. I have never seen anyone ask for integration with any other type of system. Most of the non-Hadoop systems supported by Spark don't need this kind of functionality (e.g. S3 just uses secrets stored in configuration or a credential store - both of which are handled by the Hadoop libraries btw, not Spark). That being said, I think this whole discussion is distracting from the main goal of the change, which is to expose *Hadoop*-based security in Mesos-based applications. Would it make you more comfortable if the new API were kept `private[spark]`? It would limit extensibility in the case of Mesos (would be restricted to built-in providers), but would free us from this discussion and allow progress to happen. Really, personally, unless someone can given me a concrete example, I don't see a pressing need to have a lengthy discussion about non-Hadoop-based security frameworks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17723 Hi @mgummelt, hopefully I have clarified some of my thinking above. Responding to specific points below. > It seems the first point of contention is the distinction between Hadoop and YARN. This PR relies on Hadoop libraries in core, but it shouldn't rely on yarn. If it is, that's a mistake and I should fix that. My query was not regarding the actual credential provider implementations themselves (which will require their dependencies), but whether spark core needs to depend on the api. Put another way, suppose we moved credential provider implementations into a separate module - will spark core still need to depend on this or not ? @vanzin's made the point that since we depend on hadoop-client, which depends on hadoop-security - this does not matter anymore :-) > Then the discussion becomes whether we should rely on Hadoop in core. It looks like @mridulm acknowledges we're already using Hadoop in core, so I hope we agree that this PR doesn't create a new problem, but that it does increase the coupling. Hopefully I covered this in my earlier comments. I was not revisiting use of hadoop in core (that is pervasive in spark), but whether hadoop-security model is sufficient for what we are attempting. > And also as @vanzin points out, ultimately, there's no way to get around the requirement of using Hadoop security libraries such as UGI if our goal is to access Hadoop services. Hadoop services require Hadoop delegation tokens, rather than some more broadly applicable security standard. And I hope we agree we don't want to duplicate the UGI client code in both the yarn and mesos module (sharing that client code was the whole motivation of this PR). Definitely agree on not duplicating code ! Paraphrasing what you mentioned earlier and elaborating, I am trying to understand if: * We can assume hadoop-security is sufficient for our usecases. * In this case, we leverage existing implementations as-is (more or less) - and can expose hadoop-security in our interface definitions (`traits`, execution environment, how we use the credentials). * hadoop-security becomes one supported model (and currently only one). For example, model definition could be: * Defining pre-requisites (principal/keytab, external credential update, etc). * environment setup (`UGI. loginUserFromKeytabAndReturnUGI.doAs`, etc currently used). * Application of acquired credentials (`UGI.getCurrentUser.addCredentials`) at executors and driver. * credential provider's declaring which model they are for. * Perhaps some other solution (synthesis of the above ? new ?) > So the only alternative I see would be to create a separate hadoop module, place all this code there, and create new interfaces that Hadoop-specific code would implement. One obstacle to that is the massive amount of work. The other is that I'm not a huge fan of creating interfaces when we only have on implementation, since you often end up with the wrong interface, so you have to rewrite it anyway. I share your concerns here ! Creating incorrect interface we get stuck with, a single implementation causing our interfaces to be very specialized, potentially over-designing. I am trying to understand if what we have is sufficient or do we need to look more closely at our dependency on hadoop-security. I am not very familiar with mesos or kubernetes, but a cursory search indicated they have other forms of authentication ? If yes can you comment if, with the current model, will mesos be able to evolve to support others ? Since I do not have sufficient compelling examples to give, I am unable to convince @vanzin :-) > So my proposal is that we acknowledge that decoupling all Hadoop code into a hadoop module and placing UGI behind a common interface should be done at some point, but we wait to do it until we at least have some other security provider that would implement that interface. Since we are exposing credential providers as an api from core, we will need to support it. If it is possible to support other models in future without breaking our exposed interfaces - that is something which would be an excellent way forward too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17723 @vanzin Looking at it from view of specific API usage, while relevant for analyzing implementation, is not the whole picture. Is hadoop-security model sufficient for spark ? Will spark need to support others as well - and if it does, will our design allow for it ? [1]. A specific implementation of what we expose will be dependent on hadoop-security (existing credential providers). In order to make progress here, let me highlight a few areas where hadoop-security is used explicitly or implicitly in current code. * Pre-requisites - principal/keytab : they are specific to use of kerberos in hadoop-security. * Environment in which credential provider's are invoked. * Expected to be within ugi returned via `ugi.loginUserFromKeytabAndReturnUGI` * Current credential providers assume this as impl detail (for example hbase - perhaps others too). * Use of o.a.h.security.Credential in our interface. * The api is specific to hadoop-security - and assumes a model of secret keys and `TokenIdentifier`. * The latter might be non trivial to adapt for non hadoop-security implementations. * Application of acquired tokens at executors/driver. * For hadoop-security, this becomes `ugi.getCurrentUser.addCredentials(Credentials)` * For others, it could be something specific to its implementation. I am assuming token distribution should not be an issue - though we might have multiple implementations for it. We are also assuming a model of periodic renewal of credentials (based on expiry) - are there other models we would need to support ? [1] Having said this, I am not suggesting that we must not use hadoop-security as basis for introduction of this feature into spark core : if we find the model suitable for spark's needs (present and future), that is great. I did not see an analysis being done here on that regard - If I missed it (likely), please point me to relevant discussion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17723: [SPARK-20434][YARN][CORE] Move kerberos delegation token...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17723 It would be easier if instead you said what particular API that is being proposed here you have issues with. Is it the storing of credentials in UGI? Or what? If that's what you're saying. I don't see why it's a problem to store credentials, even non-Hadoop ones, in UGI. It's just a container. Just like HDFS looks at UGI and only fetches the credentials it needs, some app blah that wants to hook up to Spark would do the same and look for its credentials there. I don't see a ton of benefits in abstracting that away into yet another security abstraction layer. And if for some reason that doesn't work, well, how do we know? Unless you have a concrete example, it's hard to develop an interface to satisfy an unknown use case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
