[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/15119 LGTM. Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15119 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71173/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15119 **[Test build #71173 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71173/testReport)** for PR 15119 at commit [`532e744`](https://github.com/apache/spark/commit/532e744eff04ce377ceae903c3d916091f551216). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/15119 Thanks @vanzin , good points. I took another read through the doc and thought it was better to move things around a little. I also noticed a misplaced tag in another setting. Let me know if you see any other issues. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15119 **[Test build #71173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71173/testReport)** for PR 15119 at commit [`532e744`](https://github.com/apache/spark/commit/532e744eff04ce377ceae903c3d916091f551216). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/15119 Hi @vanzin, any more comments or think this could be merged? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15119 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71100/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15119 **[Test build #71100 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71100/testReport)** for PR 15119 at commit [`9b6d67f`](https://github.com/apache/spark/commit/9b6d67f9529d9b83aa5f7480461d3bbb691cb881). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15119 **[Test build #71100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71100/testReport)** for PR 15119 at commit [`9b6d67f`](https://github.com/apache/spark/commit/9b6d67f9529d9b83aa5f7480461d3bbb691cb881). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71001/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15119 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15119 **[Test build #71001 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71001/testReport)** for PR 15119 at commit [`7a7b6ba`](https://github.com/apache/spark/commit/7a7b6ba213e57a705642d42220037a7b9a18e3a6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/15119 @vanzin and @themodernlife , I have this fixed up to use the additional repositories when loading ivy settings from a file. Added a new test for loading a settings and fixed up the docs for `spark.jars.ivy` - I agree that the name is confusing, but hopefully clear in the docs. Thanks @themodernlife for helping out with the docs too. If you're able to try out this latest revision too, that would be great! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15119 **[Test build #71001 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71001/testReport)** for PR 15119 at commit [`7a7b6ba`](https://github.com/apache/spark/commit/7a7b6ba213e57a705642d42220037a7b9a18e3a6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user themodernlife commented on the issue: https://github.com/apache/spark/pull/15119 @BryanCutler - #1 I think you're right... the naming of that key is unfortuante, `spark.jars.ivyUserDir` or something would have been better... it affects `defaultIvyUserDir` property of the Ivy settings - #2 sounds right to me --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/15119 Not sure about (1), maybe @brkyvz knows since I think he wrote that code. (2) sounds ok to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/15119 Sorry for the delay @vanzin and @themodernlife , I was only able to start looking at this today. I have a couple questions for you guys 1. I think the docs might not be quite right about `spark.jars.ivy`. It's supposed to be a local path to Ivy and used to make the Ivy cache path and a path to download resolved jars, not for a remote repository, but that might work. For additional repositories, the user should specify with the command line option `--repositories`, which doesn't seem to have a conf property. Do you guys agree? If so I could fix up the docs here. 2. To make this PR work with additional repositories, it's a little more complicated due to some needed checks. I think the best strategy would be to create a new `ChainResolver` with a unique name (to not conflict with defined resolvers). Add the default resolver from the settings file to this (if exists), followed by the additional repositories, then make this the new default resolver. The last artifact resolved is the one used, so the additional repositories would have precedence over those in the settings file. In the same vein, if `spark.jars.ivy` is specified, it would override the Ivy user dir and cache path, if defined in the settings file. Sound ok? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/15119 Thanks for taking a look @vanzin, it'll be a few days before I can work on this but I'll ping you and @themodernlife when ready. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/15119 Hi @themodernlife, thanks for testing this out! Glad it worked for you, I'll try to ping some others and see if I can get more interest. ping @JoshRosen @brkyvz --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user themodernlife commented on the issue: https://github.com/apache/spark/pull/15119 FYI I tried this out in our environment - Firewall (no access to maven central) - Custom ivysettings.xml to point to our internal Artifactory Everything worked just as I'd expected. Hope to see this merged! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15119 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15119 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65506/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15119 **[Test build #65506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65506/consoleFull)** for PR 15119 at commit [`1f472ff`](https://github.com/apache/spark/commit/1f472ff9962c5b59eaa6e287afcb421f886fc83b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15119: [SPARK-17568][CORE][DEPLOY] Add spark-submit option to o...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15119 **[Test build #65506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65506/consoleFull)** for PR 15119 at commit [`1f472ff`](https://github.com/apache/spark/commit/1f472ff9962c5b59eaa6e287afcb421f886fc83b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
