[GitHub] spark pull request #14140: [SPARK-16426][MLlib] Fix bug that caused NaNs in ...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/14140#discussion_r70819198
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala
---
@@ -408,9 +409,11 @@ class IsotonicRegression private (private var
isotonic: Boolean) extends Seriali
*/
private def parallelPoolAdjacentViolators(
input: RDD[(Double, Double, Double)]): Array[(Double, Double,
Double)] = {
-val parallelStepResult = input
- .sortBy(x => (x._2, x._1))
- .glom()
+val keyedInput = input.keyBy(_._2)
+val parallelStepResult = keyedInput
+ .partitionBy(new RangePartitioner(keyedInput.getNumPartitions,
keyedInput))
+ .values
+ .mapPartitions( p => Iterator(p.toSeq.sortBy(x => (x._2,
x._1)).toArray))
--- End diff --
Ah I'm reading badly this morning. Of course right about `keyedInput` and
yes the `Iterator` wraps the array, right. But I think that your last change is
spot on, at least one conversion could be avoided. LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14202: [SPARK-16230] [CORE] CoarseGrainedExecutorBackend to sel...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14202 **[Test build #62324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62324/consoleFull)** for PR 14202 at commit [`0c71699`](https://github.com/apache/spark/commit/0c71699894d4b7920388056a1d05d2277a79cf38). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14202: [SPARK-16230] [CORE] CoarseGrainedExecutorBackend...
GitHub user tejasapatil opened a pull request: https://github.com/apache/spark/pull/14202 [SPARK-16230] [CORE] CoarseGrainedExecutorBackend to self kill if there is an exception while creating an Executor ## What changes were proposed in this pull request? With the fix from SPARK-13112, I see that `LaunchTask` is always processed after `RegisteredExecutor` is done and so it gets chance to do all retries to startup an executor. There is still a problem that if `Executor` creation itself fails and there is some exception, it gets unnoticed and the executor is killed when it tries to process the `LaunchTask` as `executor` is null : https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala#L88 So if one looks at the logs, it does not tell that there was problem during `Executor` creation and thats why it was killed. This PR explicitly catches exception in `Executor` creation, logs a proper message and then exits the JVM. Also, I have changed the `exitExecutor` method to accept `reason` so that backends can use that reason and do stuff like logging to a DB to get an aggregate of such exits at a cluster level ## How was this patch tested? I am relying on existing tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/tejasapatil/spark exit_executor_failure Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14202.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14202 commit 0c71699894d4b7920388056a1d05d2277a79cf38 Author: Tejas Patil Date: 2016-07-14T14:36:36Z CoarseGrainedExecutorBackend to self kill if there is an exception while creating an Executor --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14140: [SPARK-16426][MLlib] Fix bug that caused NaNs in Isotoni...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14140 **[Test build #62323 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62323/consoleFull)** for PR 14140 at commit [`6222452`](https://github.com/apache/spark/commit/62224529cc7ec2d215e8fd47453ca73785ffbf53). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14181: [SPARK-15382][SQL] Fix a rule to push down projects bene...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14181 **[Test build #62322 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62322/consoleFull)** for PR 14181 at commit [`a50d3dc`](https://github.com/apache/spark/commit/a50d3dc27ca9880bf8034dbf2994982a50fb3c43). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13051: [SPARK-15271] [MESOS] Allow force pulling executor docke...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13051 **[Test build #62317 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62317/consoleFull)** for PR 13051 at commit [`c2d4b7f`](https://github.com/apache/spark/commit/c2d4b7f3d6db53375eebd30d660a9508cfb76b3b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14162: [SPARK-16505][yarn] Optionally propagate error du...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14162 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14140: [SPARK-16426][MLlib] Fix bug that caused NaNs in ...
Github user neggert commented on a diff in the pull request:
https://github.com/apache/spark/pull/14140#discussion_r70818220
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala
---
@@ -408,9 +409,11 @@ class IsotonicRegression private (private var
isotonic: Boolean) extends Seriali
*/
private def parallelPoolAdjacentViolators(
input: RDD[(Double, Double, Double)]): Array[(Double, Double,
Double)] = {
-val parallelStepResult = input
- .sortBy(x => (x._2, x._1))
- .glom()
+val keyedInput = input.keyBy(_._2)
+val parallelStepResult = keyedInput
+ .partitionBy(new RangePartitioner(keyedInput.getNumPartitions,
keyedInput))
+ .values
+ .mapPartitions( p => Iterator(p.toSeq.sortBy(x => (x._2,
x._1)).toArray))
--- End diff --
Actually, the `Iterator` is necessary. Since we removed the `glom`, we need
to return an `RDD[Array[T]]`. So the function passed to `mapPartitions` needs
to be `Iterator[T] => Iterator[Array[T]]`. Removed the extraneous `toSeq`,
though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14140: [SPARK-16426][MLlib] Fix bug that caused NaNs in ...
Github user neggert commented on a diff in the pull request:
https://github.com/apache/spark/pull/14140#discussion_r70816925
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/regression/IsotonicRegression.scala
---
@@ -408,9 +409,11 @@ class IsotonicRegression private (private var
isotonic: Boolean) extends Seriali
*/
private def parallelPoolAdjacentViolators(
input: RDD[(Double, Double, Double)]): Array[(Double, Double,
Double)] = {
-val parallelStepResult = input
- .sortBy(x => (x._2, x._1))
- .glom()
+val keyedInput = input.keyBy(_._2)
+val parallelStepResult = keyedInput
+ .partitionBy(new RangePartitioner(keyedInput.getNumPartitions,
keyedInput))
+ .values
+ .mapPartitions( p => Iterator(p.toSeq.sortBy(x => (x._2,
x._1)).toArray))
--- End diff --
`keyedInput` is separated out because it needs to be used several times.
Will clean up the sorting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14162: [SPARK-16505][yarn] Optionally propagate error during sh...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/14162 +1, thanks for fixing up the docs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13051: [SPARK-15271] [MESOS] Allow force pulling executor docke...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13051 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13051: [SPARK-15271] [MESOS] Allow force pulling executor docke...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13051 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62317/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14150 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62316/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14150 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14150 **[Test build #62316 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62316/consoleFull)** for PR 14150 at commit [`0192f99`](https://github.com/apache/spark/commit/0192f99ecc8f8b980b78e6f5c3c872c6c85ea56b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9571: [SPARK-11373] [CORE] Add metrics to the History Server an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/9571 **[Test build #62321 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62321/consoleFull)** for PR 9571 at commit [`4b2046f`](https://github.com/apache/spark/commit/4b2046fd47d58bc5814f6271aa1d67a6cdd0918c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9571: [SPARK-11373] [CORE] Add metrics to the History Server an...
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/9571 This patch adds separate average values of the load times vs merge times per event; this shows ~2x difference in replay from load in the test case., These `.time` gauges are little lambda expressions evaluated whenever the gauge value is extracted; they divide the total load/merge durations by the event count. The Timer metrics don't provide enough data here, because they support various decaying reservoirs/windows for their time values, not whole-life-of-app durations. There's some base class metric support for registration (everything is prefixed) and testability. There's now tests in `HistoryServerSuite."incomplete apps get refreshed"` for metrics being in the dumped list, and for specific values, especially averages (that they increase, that they don't trigger division-by-0 exceptions before there have been any loads) This is what the metrics look like after the tests (from a log of the toString value). The `.last.attempted` values are `System.currentTimeMillis` timestamps of the operations. I'd considered a "time-since" gauge, but after some offline discussion with Allen Wittenauer, went for the absolute values; I'll leave to to the management tooling to work out elapsed times from absolute values if they want to use that for alerts or UIs. ``` 16/07/14 14:17:03.837 ScalaTest-main-running-HistoryServerSuite INFO HistoryServerSuite: Metrics: Metrics for history: Counters Gauges Metrics for history.fs: Counters history.provider.appui.event.count = 103 history.provider.appui.load.count = 6 history.provider.appui.load.duration = 38244482 history.provider.appui.load.failure.count = 0 history.provider.appui.load.not-found.count = 0 history.provider.history.merge.duration = 16138193 history.provider.history.merge.event.count = 83 history.provider.update.count = 5 history.provider.update.failure.count = 0 Gauges history.provider.appui.event.replay.time = 371305 history.provider.history.merge.event.time = 194436 history.provider.update.last.attempted = 1468502223726 history.provider.update.last.succeeded = 1468502223000 Metrics for application.cache: Counters history.cache.eviction.count = 3 history.cache.load.count = 4 history.cache.lookup.count = 58 history.cache.lookup.failure.count = 0 history.cache.update.probe.count = 57 history.cache.update.triggered.count = 3 Gauges ``` One thing to consider: gauges of the number of complete and incomplete applications? I know the REST UI gives this, but only indirectly (you call, you count the size of the lists). Doing it in the metrics provides something that could be monitored or probed in tests without making REST calls. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9571: [SPARK-11373] [CORE] Add metrics to the History Server an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/9571 **[Test build #62320 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62320/consoleFull)** for PR 9571 at commit [`0b8db8b`](https://github.com/apache/spark/commit/0b8db8b285bfcc73e861a92303206f670ea40de1). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9571: [SPARK-11373] [CORE] Add metrics to the History Server an...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/9571 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62320/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9571: [SPARK-11373] [CORE] Add metrics to the History Server an...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/9571 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9571: [SPARK-11373] [CORE] Add metrics to the History Server an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/9571 **[Test build #62320 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62320/consoleFull)** for PR 9571 at commit [`0b8db8b`](https://github.com/apache/spark/commit/0b8db8b285bfcc73e861a92303206f670ea40de1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9571: [SPARK-11373] [CORE] Add metrics to the History Server an...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/9571 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62319/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9571: [SPARK-11373] [CORE] Add metrics to the History Server an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/9571 **[Test build #62319 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62319/consoleFull)** for PR 9571 at commit [`3971241`](https://github.com/apache/spark/commit/39712419d463bface2027fb7888f2c40f26fa412). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9571: [SPARK-11373] [CORE] Add metrics to the History Server an...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/9571 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9571: [SPARK-11373] [CORE] Add metrics to the History Server an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/9571 **[Test build #62319 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62319/consoleFull)** for PR 9571 at commit [`3971241`](https://github.com/apache/spark/commit/39712419d463bface2027fb7888f2c40f26fa412). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9571: [SPARK-11373] [CORE] Add metrics to the History Server an...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/9571 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9571: [SPARK-11373] [CORE] Add metrics to the History Server an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/9571 **[Test build #62318 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62318/consoleFull)** for PR 9571 at commit [`6e7f466`](https://github.com/apache/spark/commit/6e7f466da0681af2c40a6842ed6c542ca44c7bd5). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9571: [SPARK-11373] [CORE] Add metrics to the History Server an...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/9571 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62318/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14197: [SPARK-14963][Minor][YARN] Fix typo in YarnShuffl...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/14197 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #9571: [SPARK-11373] [CORE] Add metrics to the History Server an...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/9571 **[Test build #62318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62318/consoleFull)** for PR 9571 at commit [`6e7f466`](https://github.com/apache/spark/commit/6e7f466da0681af2c40a6842ed6c542ca44c7bd5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14197: [SPARK-14963][Minor][YARN] Fix typo in YarnShuffleServic...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/14197 +1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14201: [SPARK-14702] Expose SparkLauncher's ProcessBuilder for ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14201 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14136 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62315/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14136 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14136 **[Test build #62315 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62315/consoleFull)** for PR 14136 at commit [`2194c9e`](https://github.com/apache/spark/commit/2194c9ea966276e242eba29eaa65cbc7b1a7bda9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14201: [SPARK-14702] Expose SparkLauncher's ProcessBuilder for ...
Github user andreweduffy commented on the issue: https://github.com/apache/spark/pull/14201 Replaces #14185, @vanzin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14201: [SPARK-14702] Expose SparkLauncher's ProcessBuild...
GitHub user andreweduffy opened a pull request: https://github.com/apache/spark/pull/14201 [SPARK-14702] Expose SparkLauncher's ProcessBuilder for user flexibility ## What changes were proposed in this pull request? Adds a few public methods to `SparkLauncher` to allow configuring some extra features of the `ProcessBuilder`, including the working directory, output and error stream redirection. ## How was this patch tested? Unit testing + simple Spark driver programs You can merge this pull request into a Git repository by running: $ git pull https://github.com/andreweduffy/spark feature/launcher Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14201.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14201 commit 5a35963bc4661193d8c169c7f98fffe24b181124 Author: Andrew Duffy Date: 2016-07-14T11:57:15Z Make environment of SparkLauncher launched process more configurable --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14191: [SPARK-16217][SQL] Support SELECT INTO statement
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/14191#discussion_r70801792
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -1376,4 +1376,62 @@ class SparkSqlAstBuilder(conf: SQLConf) extends
AstBuilder {
reader, writer,
schemaLess)
}
+
+ /**
+ * Reuse CTAS, convert select into to CTAS,
+ * returning [[CreateHiveTableAsSelectLogicalPlan]].
+ * The SELECT INTO statement selects data from one table
+ * and inserts it into a new table.It is commonly used to
+ * create a backup copy for table or selected records.
+ *
+ * Expected format:
+ * {{{
+ * SELECT column_name(s)
+ * INTO new_table
+ * FROM old_table
+ * ...
+ * }}}
+ */
+ override protected def withSelectInto(
--- End diff --
The code below is duplicates. Why are we not using the existing CTAS code
path?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14197: [SPARK-14963][Minor][YARN] Fix typo in YarnShuffleServic...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/14197 Ping @tgravescs , please help to review, this typo is introduce by SPARK-14963. Thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13911: [SPARK-16215][SQL] Reduce runtime overhead of a program ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13911 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62312/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13911: [SPARK-16215][SQL] Reduce runtime overhead of a program ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13911 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13911: [SPARK-16215][SQL] Reduce runtime overhead of a program ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13911 **[Test build #62312 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62312/consoleFull)** for PR 13911 at commit [`4b70df9`](https://github.com/apache/spark/commit/4b70df92cf163ef8680ec744236ee2825dae85ad). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13051: [SPARK-15271] [MESOS] Allow force pulling executor docke...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13051 **[Test build #62317 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62317/consoleFull)** for PR 13051 at commit [`c2d4b7f`](https://github.com/apache/spark/commit/c2d4b7f3d6db53375eebd30d660a9508cfb76b3b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13051: [SPARK-15271] [MESOS] Allow force pulling executo...
Github user philipphoffmann commented on a diff in the pull request:
https://github.com/apache/spark/pull/13051#discussion_r70797577
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -517,16 +517,10 @@ private[spark] class MesosClusterScheduler(
.addAllResources(memResourcesToUse.asJava)
offer.resources = finalResources.asJava
submission.schedulerProperties.get("spark.mesos.executor.docker.image").foreach
{ image =>
- val container = taskInfo.getContainerBuilder()
- val volumes = submission.schedulerProperties
-.get("spark.mesos.executor.docker.volumes")
-.map(MesosSchedulerBackendUtil.parseVolumesSpec)
- val portmaps = submission.schedulerProperties
-.get("spark.mesos.executor.docker.portmaps")
-.map(MesosSchedulerBackendUtil.parsePortMappingsSpec)
- MesosSchedulerBackendUtil.addDockerInfo(
-container, image, volumes = volumes, portmaps = portmaps)
- taskInfo.setContainer(container.build())
+ MesosSchedulerBackendUtil.setupContainerBuilderDockerInfo(
+image,
+submission.schedulerProperties,
+taskInfo.getContainerBuilder())
--- End diff --
I thought so too but apparently `taskInfo.getContainerBuilder()` creates a
new `ContainerInfo`, if not present, adds it to the `TaskInfo` and returns the
Builder.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13051: [SPARK-15271] [MESOS] Allow force pulling executo...
Github user philipphoffmann commented on a diff in the pull request:
https://github.com/apache/spark/pull/13051#discussion_r70797441
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackendUtil.scala
---
@@ -134,6 +137,32 @@ private[mesos] object MesosSchedulerBackendUtil
extends Logging {
addDockerInfo(
builder,
imageName,
+ conf.getBoolean("spark.mesos.executor.docker.forcePullImage",
defaultValue = false),
+ volumes = volumes,
+ portmaps = portmaps)
+logDebug("setupContainerDockerInfo: using docker image: " + imageName)
+ }
+
+ /**
+ * Setup a docker containerizer from MesosDriverDescription scheduler
properties
+ */
+ def setupContainerBuilderDockerInfo(
--- End diff --
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14150 **[Test build #62316 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62316/consoleFull)** for PR 14150 at commit [`0192f99`](https://github.com/apache/spark/commit/0192f99ecc8f8b980b78e6f5c3c872c6c85ea56b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14136 **[Test build #62315 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62315/consoleFull)** for PR 14136 at commit [`2194c9e`](https://github.com/apache/spark/commit/2194c9ea966276e242eba29eaa65cbc7b1a7bda9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14181: [SPARK-15382][SQL] Fix a rule to push down projects bene...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14181 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62309/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14181: [SPARK-15382][SQL] Fix a rule to push down projects bene...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14181 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14181: [SPARK-15382][SQL] Fix a rule to push down projects bene...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14181 **[Test build #62309 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62309/consoleFull)** for PR 14181 at commit [`ca23f4f`](https://github.com/apache/spark/commit/ca23f4f94d6ca8a9b29a9bf06d760a3d9798aa9d). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14200: [SPARK-16528][SQL] Fix NPE problem in HiveClientImpl
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14200 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14200: [SPARK-16528][SQL] Fix NPE problem in HiveClientImpl
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14200 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62311/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14200: [SPARK-16528][SQL] Fix NPE problem in HiveClientImpl
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14200 **[Test build #62311 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62311/consoleFull)** for PR 14200 at commit [`49c5451`](https://github.com/apache/spark/commit/49c5451a0247fc26f184c02416c6902157880e64). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14200: [SPARK-16528][SQL] Fix NPE problem in HiveClientImpl
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14200 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14200: [SPARK-16528][SQL] Fix NPE problem in HiveClientImpl
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14200 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62310/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14151: [SPARK-16496][SQL] Add wholetext as option for reading t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14151 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14151: [SPARK-16496][SQL] Add wholetext as option for reading t...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14151 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62308/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14200: [SPARK-16528][SQL] Fix NPE problem in HiveClientImpl
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14200 **[Test build #62310 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62310/consoleFull)** for PR 14200 at commit [`667abc0`](https://github.com/apache/spark/commit/667abc0d842266405ba412f468358e76b3d815c8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14151: [SPARK-16496][SQL] Add wholetext as option for reading t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14151 **[Test build #62308 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62308/consoleFull)** for PR 14151 at commit [`82952e7`](https://github.com/apache/spark/commit/82952e7a1b504f61ec09dc50a31105e5cb405587). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14169: [SPARK-16515][SQL]set default record reader and writer f...
Github user chenghao-intel commented on the issue: https://github.com/apache/spark/pull/14169 LGTM. cc @yhuai @liancheng This breaks the existed application which using the default delimiter, and we've already verified in TPCx-BB. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14136 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14136 **[Test build #62314 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62314/consoleFull)** for PR 14136 at commit [`bf6f539`](https://github.com/apache/spark/commit/bf6f539649b33f5bb5b5e71e3b5a7dbcc77abdc3). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14136 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62314/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14136 **[Test build #62314 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62314/consoleFull)** for PR 14136 at commit [`bf6f539`](https://github.com/apache/spark/commit/bf6f539649b33f5bb5b5e71e3b5a7dbcc77abdc3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14199: [SPARK-16543][SQL] Rename the columns of `SHOW PARTITION...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14199 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14199: [SPARK-16543][SQL] Rename the columns of `SHOW PARTITION...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14199 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62307/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14199: [SPARK-16543][SQL] Rename the columns of `SHOW PARTITION...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14199 **[Test build #62307 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62307/consoleFull)** for PR 14199 at commit [`1f6ba91`](https://github.com/apache/spark/commit/1f6ba91714292a9eab5a360ff382f97089c573d5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/14136 @hvanhovell I've fixed most of the problems mentioned above, and I also added basic tests and comments as you required. Please find some time to do a pass, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14136 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14136 **[Test build #62313 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62313/consoleFull)** for PR 14136 at commit [`4914174`](https://github.com/apache/spark/commit/491417408c51d0b5d06070d81061614e9ffcbab8). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14136 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62313/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14136: [SPARK-16282][SQL] Implement percentile SQL function.
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14136 **[Test build #62313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62313/consoleFull)** for PR 14136 at commit [`4914174`](https://github.com/apache/spark/commit/491417408c51d0b5d06070d81061614e9ffcbab8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13911: [SPARK-16215][SQL] Reduce runtime overhead of a program ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13911 **[Test build #62312 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62312/consoleFull)** for PR 13911 at commit [`4b70df9`](https://github.com/apache/spark/commit/4b70df92cf163ef8680ec744236ee2825dae85ad). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14167: [SPARK-16194] Mesos Driver env vars
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/14167#discussion_r70784310
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -399,20 +421,20 @@ private[spark] class MesosClusterScheduler(
// Sandbox points to the current directory by default with Mesos.
(cmdExecutable, ".")
}
-val primaryResource = new File(sandboxPath,
desc.jarUrl.split("/").last).toString()
val cmdOptions = generateCmdOption(desc, sandboxPath).mkString(" ")
+val primaryResource = new File(sandboxPath,
desc.jarUrl.split("/").last).toString()
val appArguments = desc.command.arguments.mkString(" ")
-builder.setValue(s"$executable $cmdOptions $primaryResource
$appArguments")
-builder.setEnvironment(envBuilder.build())
-conf.getOption("spark.mesos.uris").map { uris =>
- setupUris(uris, builder)
-}
-desc.schedulerProperties.get("spark.mesos.uris").map { uris =>
- setupUris(uris, builder)
-}
-desc.schedulerProperties.get("spark.submit.pyFiles").map { pyFiles =>
- setupUris(pyFiles, builder)
-}
+
+s"$executable $cmdOptions $primaryResource $appArguments"
+ }
+
+ private def buildDriverCommand(desc: MesosDriverDescription):
CommandInfo = {
+val builder = CommandInfo.newBuilder()
+
+builder.setValue(getDriverCommandValue(desc))
+builder.setEnvironment(getDriverEnvironment(desc))
+builder.addAllUris(getDriverUris(desc).asJava)
+
--- End diff --
We should be consistent i think with spaces. In other methods i see no
space between return argument and the rest of the method body ( vs
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala#L422).
The same applies for the spaces between variable declaration and statements
( vs
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala#L142)
.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14150 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14150 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62306/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14150: [SPARK-16494] [ML] Upgrade breeze version to 0.12
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14150 **[Test build #62306 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62306/consoleFull)** for PR 14150 at commit [`ed28c3b`](https://github.com/apache/spark/commit/ed28c3b405277fd583143bf6cacfb48feb012b70). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14200: [SPARK-16528][SQL] Fix NPE problem in HiveClientImpl
Github user srowen commented on the issue: https://github.com/apache/spark/pull/14200 OK by me --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14200: [SPARK-16528][SQL] Fix NPE problem in HiveClientImpl
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14200 **[Test build #62311 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62311/consoleFull)** for PR 14200 at commit [`49c5451`](https://github.com/apache/spark/commit/49c5451a0247fc26f184c02416c6902157880e64). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14200: [SPARK-16528][SQL]: Fix NPE problem in HiveClientImpl
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14200 **[Test build #62310 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62310/consoleFull)** for PR 14200 at commit [`667abc0`](https://github.com/apache/spark/commit/667abc0d842266405ba412f468358e76b3d815c8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14200: SPARK-16528: Fix NPE problem in HiveClientImpl
Github user jacek-lewandowski commented on the issue: https://github.com/apache/spark/pull/14200 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14200: SPARK-16528: Fix NPE problem in HiveClientImpl
GitHub user jacek-lewandowski opened a pull request: https://github.com/apache/spark/pull/14200 SPARK-16528: Fix NPE problem in HiveClientImpl ## What changes were proposed in this pull request? There are some calls to methods or fields (getParameters, properties) which are then passed to Java/Scala collection converters. Unfortunately those fields can be null in some cases and then the conversions throws NPE. We fix it by wrapping calls to those fields and methods with option and then do the conversion. ## How was this patch tested? Manually tested with a custom Hive metastore. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jacek-lewandowski/spark SPARK-16528 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/14200.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14200 commit 667abc0d842266405ba412f468358e76b3d815c8 Author: Jacek Lewandowski Date: 2016-07-14T10:14:36Z SPARK-16528: Fix NPE problem in HiveClientImpl There are some calls to methods or fields (getParameters, properties) which are then passed to Java/Scala collection converters. Unfortunately those fields can be null in some cases and then the conversions throws NPE. We fix it by wrapping calls to those fields and methods with option and then do the conversion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14167: [SPARK-16194] Mesos Driver env vars
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/14167#discussion_r70781247
--- Diff:
core/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala
---
@@ -35,6 +34,7 @@ import org.apache.spark.{LocalSparkContext,
SecurityManager, SparkConf, SparkCon
import org.apache.spark.network.shuffle.mesos.MesosExternalShuffleClient
import org.apache.spark.rpc.RpcEndpointRef
import org.apache.spark.scheduler.TaskSchedulerImpl
+import org.apache.spark.scheduler.cluster.mesos.Utils._
--- End diff --
Wild card imports should be avoided (unless you are importing more than 6
entities, or implicit methods) https://github.com/databricks/scala-style-guide.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14167: [SPARK-16194] Mesos Driver env vars
Github user skonto commented on the issue: https://github.com/apache/spark/pull/14167 The build fails with: [error] /home/jenkins/workspace/SparkPullRequestBuilder@4/core/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterSchedulerSuite.scala:34: object Utils is not a member of package org.apache.spark.scheduler.cluster.mesos [error] import org.apache.spark.scheduler.cluster.mesos.Utils._ https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62270/consoleFull I reproduce it locally you are missing the Utils object. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14181: [SPARK-15382][SQL] Fix a rule to push down projects bene...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14181 **[Test build #62309 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62309/consoleFull)** for PR 14181 at commit [`ca23f4f`](https://github.com/apache/spark/commit/ca23f4f94d6ca8a9b29a9bf06d760a3d9798aa9d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14181: [SPARK-15382][SQL] Fix a rule to push down projects bene...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14181 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62303/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14181: [SPARK-15382][SQL] Fix a rule to push down projects bene...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14181 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14151: [SPARK-16496][SQL] Add wholetext as option for reading t...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14151 **[Test build #62308 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62308/consoleFull)** for PR 14151 at commit [`82952e7`](https://github.com/apache/spark/commit/82952e7a1b504f61ec09dc50a31105e5cb405587). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14181: [SPARK-15382][SQL] Fix a rule to push down projects bene...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14181 **[Test build #62303 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62303/consoleFull)** for PR 14181 at commit [`3885f21`](https://github.com/apache/spark/commit/3885f21a68d9f16f129d4198e89d4c5af05b72ed). * This patch **fails SparkR unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14167: [SPARK-16194] Mesos Driver env vars
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/14167#discussion_r70779638
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
---
@@ -353,38 +353,60 @@ private[spark] class MesosClusterScheduler(
}
}
- private def buildDriverCommand(desc: MesosDriverDescription):
CommandInfo = {
-val appJar = CommandInfo.URI.newBuilder()
-
.setValue(desc.jarUrl.stripPrefix("file:").stripPrefix("local:")).build()
-val builder = CommandInfo.newBuilder().addUris(appJar)
-val entries = conf.getOption("spark.executor.extraLibraryPath")
- .map(path => Seq(path) ++ desc.command.libraryPathEntries)
- .getOrElse(desc.command.libraryPathEntries)
-
-val prefixEnv = if (!entries.isEmpty) {
- Utils.libraryPathEnvPrefix(entries)
-} else {
- ""
+ private def getDriverExecutorURI(desc: MesosDriverDescription) = {
+desc.schedulerProperties.get("spark.executor.uri")
+ .orElse(desc.command.environment.get("SPARK_EXECUTOR_URI"))
+ }
+
+ private def getDriverEnvironment(desc: MesosDriverDescription):
Environment = {
+val env = {
+ val executorOpts = desc.schedulerProperties.map { case (k, v) =>
s"-D$k=$v" }.mkString(" ")
+ val executorEnv = Map("SPARK_EXECUTOR_OPTS" -> executorOpts)
+
+ val prefix = "spark.mesos.env."
--- End diff --
Spark does not have a spark.driverEnv.[EnvironmentVariableName] similar to
spark.executorEnv.[EnvironmentVariableName]
http://spark.apache.org/docs/latest/configuration.html.
From a UX experience and name consistency view i would expect something
like that. The problem is that this pr only handles the mesos case so we cannot
rename it to that name (unless we explicitly defined it in docs), also in
client mode you do not need that so you will need to ignore it.
"spark.mesos.env." needs to be more specific like spark.mesos.driver.env
but as i said it only makes sense in cluster mode and spark.driverEnv seems
more appropriate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14196: [SPARK-16540][YARN][CORE] Avoid adding jars twice for Sp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14196 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14196: [SPARK-16540][YARN][CORE] Avoid adding jars twice for Sp...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14196 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62302/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14185: [SPARK-14702][SUBMIT] Expose SparkLauncher's ProcessBuil...
Github user andreweduffy commented on the issue: https://github.com/apache/spark/pull/14185 That sounds reasonable, I can take a shot at that today, I'll submit it as a second PR and then we can close this one out. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14196: [SPARK-16540][YARN][CORE] Avoid adding jars twice for Sp...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14196 **[Test build #62302 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62302/consoleFull)** for PR 14196 at commit [`86205fc`](https://github.com/apache/spark/commit/86205fcef29515ba72809fc2541e5d6aacfa76a7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14199: [SPARK-16543][SQL] Rename the columns of `SHOW PARTITION...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14199 **[Test build #62307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62307/consoleFull)** for PR 14199 at commit [`1f6ba91`](https://github.com/apache/spark/commit/1f6ba91714292a9eab5a360ff382f97089c573d5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14199: [SPARK-16543][SQL] Rename the columns of `SHOW PA...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/14199
[SPARK-16543][SQL] Rename the columns of `SHOW PARTITION/COLUMNS` commands
## What changes were proposed in this pull request?
This PR changes the name of columns returned by `SHOW PARTITION` and `SHOW
COLUMNS` commands. Currently, both commands uses `result` as a column name.
**Comparison: Column Name**
Command|Spark(Before)|Spark(After)|Hive
--|--||-
SHOW PARTITIONS|result|partition|partition
SHOW COLUMNS|result|col_name|field
Note that Spark/Hive show `col_name` in `DESC TABLES`. So, this PR chooses
`col_name` for consistency among Spark commands.
**Before**
```scala
scala> sql("show partitions p").show()
+--+
|result|
+--+
| b=2|
+--+
scala> sql("show columns in p").show()
+--+
|result|
+--+
| a|
| b|
+--+
```
**After**
```scala
scala> sql("show partitions p").show
+-+
|partition|
+-+
| b=2|
+-+
scala> sql("show columns in p").show
++
|col_name|
++
| a|
| b|
++
```
## How was this patch tested?
Manual.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-16543
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14199.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14199
commit 1f6ba91714292a9eab5a360ff382f97089c573d5
Author: Dongjoon Hyun
Date: 2016-07-14T09:39:10Z
[SPARK-16543][SQL] Rename the columns of `SHOW PARTITION/COLUMNS` commands
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14181: [SPARK-15382][SQL] Fix a rule to push down projects bene...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14181 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/62301/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14181: [SPARK-15382][SQL] Fix a rule to push down projects bene...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14181 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14181: [SPARK-15382][SQL] Fix a rule to push down projects bene...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14181 **[Test build #62301 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62301/consoleFull)** for PR 14181 at commit [`e426fc3`](https://github.com/apache/spark/commit/e426fc3217753eef7f2388b598f31ba48d87dd0f). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14198: [SPARK-16542] Fix bugs about types that result an array ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14198 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
