[GitHub] spark pull request: [SPARK-4498][core] Don't transition ExecutorIn...
Github user markhamstra commented on the pull request: https://github.com/apache/spark/pull/3550#issuecomment-65194915 The application won't be killed if an executor has been recognized by master as RUNNING (https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/master/Master.scala#L328). The buggy host will just keep trying and failing to launch executors. Detecting and blacklisting buggy hosts seems like a separable and complex issue. It would also be a new feature that maybe we don't want to add to 1.2 at the last minute. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver - m...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3548#issuecomment-65195159 [Test build #24035 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24035/consoleFull) for PR 3548 at commit [`9dce0d4`](https://github.com/apache/spark/commit/9dce0d4ad5fcb40a227d9a2182439623ca0adfe8). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2960] Support executing Spark from syml...
Github user roji commented on the pull request: https://github.com/apache/spark/pull/1875#issuecomment-65195595 Sorry for dropping out, was involved in other things. @mateiz, I'll take a look at your suggestions in the coming week. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Indent license header properly for interfaces....
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/3552 Indent license header properly for interfaces.scala. A very small nit update. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark license-header Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3552.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3552 commit df8d1a400ea435f41d098da0efc970b980fc5fb6 Author: Reynold Xin r...@databricks.com Date: 2014-12-02T08:14:21Z Indent license header properly for interfaces.scala. A very small nit update. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4672][GraphX]Non-transient PartitionsRD...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3544#issuecomment-65196022 cc @ankurdave --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4680] none - NoOpCompressionCodec
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3540#issuecomment-65196135 @roxchkplusony thanks for submitting this. Wouldn't you be able to achieve the same thing by setting the compress flag to off? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4397][Core] Cleanup 'import SparkContex...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3530#issuecomment-65196225 Thanks. Merging in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4397][Core] Cleanup 'import SparkContex...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3530 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4680] none - NoOpCompressionCodec
Github user roxchkplusony closed the pull request at: https://github.com/apache/spark/pull/3540 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4680] none - NoOpCompressionCodec
Github user roxchkplusony commented on the pull request: https://github.com/apache/spark/pull/3540#issuecomment-65196744 You're absolutely right. I was simply not familiar with spark.shuffle.compress. I saw spark.broadcast.compress easily enough, hilariously... This PR and Jira should be closed. I can do the honors :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Indent license header properly for interfaces....
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3552#issuecomment-65197472 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24036/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4680] none - NoOpCompressionCodec
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3540#issuecomment-65198096 Thanks for doing that! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Indent license header properly for interfaces....
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3552#issuecomment-65198169 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Indent license header properly for interfaces....
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3552#issuecomment-65198674 [Test build #24037 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24037/consoleFull) for PR 3552 at commit [`df8d1a4`](https://github.com/apache/spark/commit/df8d1a400ea435f41d098da0efc970b980fc5fb6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver - m...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3548#issuecomment-65199081 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24031/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver - m...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3548#issuecomment-65199074 [Test build #24031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24031/consoleFull) for PR 3548 at commit [`4132892`](https://github.com/apache/spark/commit/413289235942dc5eb7d6020fa795f3db5ecf3c04). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class AppClientHeartbeat(appId: String, hasRegisteredExecutors: Boolean)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4397][Core] Cleanup 'import SparkContex...
Github user zzcclp commented on the pull request: https://github.com/apache/spark/pull/3530#issuecomment-65199688 Whether this PR will be merged into the branch 1.2? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4397][Core] Cleanup 'import SparkContex...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3530#issuecomment-65200062 No. This is for SPARK-4397(#3262) and #3262 wasn't merged to branch 1.2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][core] Don't transition ExecutorIn...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3550#issuecomment-65200639 [Test build #24033 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24033/consoleFull) for PR 3550 at commit [`8f543b1`](https://github.com/apache/spark/commit/8f543b10222bf5006326d3fe0605d54474175501). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][core] Don't transition ExecutorIn...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3550#issuecomment-65200648 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24033/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4680] none - NoOpCompressionCodec
Github user roxchkplusony commented on the pull request: https://github.com/apache/spark/pull/3540#issuecomment-65200691 :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4669]Allow users to set arbitrary akka ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3551#issuecomment-65200730 [Test build #24039 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24039/consoleFull) for PR 3551 at commit [`b5324b7`](https://github.com/apache/spark/commit/b5324b7f10a6c2157d6f54c52ab88990b0465220). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4669]Allow users to set arbitrary akka ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3551#issuecomment-65200849 [Test build #24039 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24039/consoleFull) for PR 3551 at commit [`b5324b7`](https://github.com/apache/spark/commit/b5324b7f10a6c2157d6f54c52ab88990b0465220). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4669]Allow users to set arbitrary akka ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3551#issuecomment-65200852 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24039/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver - m...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3548#issuecomment-65201367 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24034/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver - m...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3548#issuecomment-65201358 [Test build #24034 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24034/consoleFull) for PR 3548 at commit [`6cc7cad`](https://github.com/apache/spark/commit/6cc7caddc14ea9582c93c619932560bae9fdf37e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class AppClientHeartbeat(appId: String, hasRegisteredExecutors: Boolean)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4669]Allow users to set arbitrary akka ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3551#issuecomment-65201444 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24038/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver - m...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3548#issuecomment-65202804 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24035/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver - m...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3548#issuecomment-65202792 [Test build #24035 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24035/consoleFull) for PR 3548 at commit [`9dce0d4`](https://github.com/apache/spark/commit/9dce0d4ad5fcb40a227d9a2182439623ca0adfe8). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class AppClientHeartbeat(appId: String, hasRegisteredExecutors: Boolean)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4644][Core] Implement skewed join
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/3505#discussion_r21147642
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
---
@@ -481,9 +481,203 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
* (k, v2) is in `other`. Uses the given Partitioner to partition the
output RDD.
*/
def join[W](other: RDD[(K, W)], partitioner: Partitioner): RDD[(K, (V,
W))] = {
--- End diff --
A better signature of `join` is `def join[W: ClassTag](other: RDD[(K, W)],
partitioner: Partitioner): RDD[(K, (V, W))]`. However, it's a breaking change,
so I don't change it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Indent license header properly for interfaces....
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3552#issuecomment-65206246 [Test build #24037 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24037/consoleFull) for PR 3552 at commit [`df8d1a4`](https://github.com/apache/spark/commit/df8d1a400ea435f41d098da0efc970b980fc5fb6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Indent license header properly for interfaces....
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3552#issuecomment-65206253 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24037/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [spark-4691][shuffle]code optimization for jud...
GitHub user maji2014 opened a pull request: https://github.com/apache/spark/pull/3553 [spark-4691][shuffle]code optimization for judgement In HashShuffleReader.scala and HashShuffleWriter.scala, no need to judge dep.aggregator.isEmpty again as this is judged by dep.aggregator.isDefined In SortShuffleWriter.scala, dep.aggregator.isEmpty is better than !dep.aggregator.isDefined ? You can merge this pull request into a Git repository by running: $ git pull https://github.com/maji2014/spark spark-4691 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3553.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3553 commit d8f52dc7dc34bc6c1c368d790b7bdfe30c4eb529 Author: maji2014 ma...@asiainfo.com Date: 2014-12-02T09:54:33Z code optimization for judgement --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [spark-4691][shuffle]code optimization for jud...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3553#issuecomment-65207158 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4685] Update JavaDoc settings to includ...
GitHub user Lewuathe opened a pull request: https://github.com/apache/spark/pull/3554 [SPARK-4685] Update JavaDoc settings to include spark.ml and all spark.m... ...llib subpackages in the right sections Create subcategory, MLlib and ML for correspond packages. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Lewuathe/spark modify-javadoc-setting Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3554.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3554 commit f7535e64966cfeae9c1d4d895d03af91f2d7b1c7 Author: lewuathe lewua...@me.com Date: 2014-12-02T10:27:13Z [SPARK-4685] Update JavaDoc settings to include spark.ml and all spark.mllib subpackages in the right sections --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4685] Update JavaDoc settings to includ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3554#issuecomment-65210755 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4669]Allow users to set arbitrary akka ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3551#issuecomment-65213405 [Test build #24040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24040/consoleFull) for PR 3551 at commit [`13f2957`](https://github.com/apache/spark/commit/13f295780d5217464c8589963cb2b2cc02b9dcc5). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4692] [SQL] Support ! boolean logic ope...
GitHub user YanTangZhai opened a pull request: https://github.com/apache/spark/pull/3555 [SPARK-4692] [SQL] Support ! boolean logic operator like NOT Support ! boolean logic operator like NOT in sql as follows select * from for_test where !(col1 col2) You can merge this pull request into a Git repository by running: $ git pull https://github.com/YanTangZhai/spark SPARK-4692 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3555.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3555 commit cdef539abc5d2d42d4661373939bdd52ca8ee8e6 Author: YanTangZhai hakeemz...@tencent.com Date: 2014-08-06T13:07:08Z Merge pull request #1 from apache/master update commit cbcba66ad77b96720e58f9d893e87ae5f13b2a95 Author: YanTangZhai hakeemz...@tencent.com Date: 2014-08-20T13:14:08Z Merge pull request #3 from apache/master Update commit 8a0010691b669495b4c327cf83124cabb7da1405 Author: YanTangZhai hakeemz...@tencent.com Date: 2014-09-12T06:54:58Z Merge pull request #6 from apache/master Update commit 03b62b043ab7fd39300677df61c3d93bb9beb9e3 Author: YanTangZhai hakeemz...@tencent.com Date: 2014-09-16T12:03:22Z Merge pull request #7 from apache/master Update commit 76d40277d51f709247df1d3734093bf2c047737d Author: YanTangZhai hakeemz...@tencent.com Date: 2014-10-20T12:52:22Z Merge pull request #8 from apache/master update commit d26d98248a1a4d0eb15336726b6f44e05dd7a05a Author: YanTangZhai hakeemz...@tencent.com Date: 2014-11-04T09:00:31Z Merge pull request #9 from apache/master Update commit e249846d9b7967ae52ec3df0fb09e42ffd911a8a Author: YanTangZhai hakeemz...@tencent.com Date: 2014-11-11T03:18:24Z Merge pull request #10 from apache/master Update commit 6e643f81555d75ec8ef3eb57bf5ecb6520485588 Author: YanTangZhai hakeemz...@tencent.com Date: 2014-12-01T11:23:56Z Merge pull request #11 from apache/master Update commit 92242c7c07d7d9f5aea2111b548a3355f3633a7d Author: YanTangZhai hakeemz...@tencent.com Date: 2014-12-02T10:57:59Z Update HiveQl.scala --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4692] [SQL] Support ! boolean logic ope...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3555#issuecomment-65214491 [Test build #24041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24041/consoleFull) for PR 3555 at commit [`92242c7`](https://github.com/apache/spark/commit/92242c7c07d7d9f5aea2111b548a3355f3633a7d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3883 SSL support for HttpServer and Akka
Github user jacek-lewandowski commented on the pull request: https://github.com/apache/spark/pull/2739#issuecomment-65216596 Back to working on this... I've rebased against branch-1.2 and then I'll rebase against master if you want. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4692] [SQL] Support ! boolean logic ope...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3555#issuecomment-65221949 [Test build #24041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24041/consoleFull) for PR 3555 at commit [`92242c7`](https://github.com/apache/spark/commit/92242c7c07d7d9f5aea2111b548a3355f3633a7d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4692] [SQL] Support ! boolean logic ope...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3555#issuecomment-65221953 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24041/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4669]Allow users to set arbitrary akka ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3551#issuecomment-65222661 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24040/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4669]Allow users to set arbitrary akka ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3551#issuecomment-65222656 [Test build #24040 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24040/consoleFull) for PR 3551 at commit [`13f2957`](https://github.com/apache/spark/commit/13f295780d5217464c8589963cb2b2cc02b9dcc5). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2199] [mllib] topic modeling
Github user akopich commented on the pull request: https://github.com/apache/spark/pull/1269#issuecomment-65223123 @jkbradley, thank you for you comments! It seems like we should discuss API for this set of models first. As far as I can understand, you are not about to provide a user with an ability to implement their own regularizers. Are you? In this case it's possible to implement a facade class and set all the other classes private. But there's still an issue. First, note that `RobustDocumentParameters` inherits from `DocumentParameters` and `RobustGlobalParameters` inherits from `GlobalParameters`. Now imagine the facade class. Let it have a parameter `robust : Boolean`. Now let it have a method like `def infer(documents : RDD[Seq[String]]) : (RDD[DocumentParameters], GlobalParameter)` If `robust == false`, everything is OK, but how am I supposed to return `RobustDocumentParameters` and `RobustGlobalParameters` if `robust == true`? It's definitely possible, but user will have to explicitly cast returned values. The only way I see to solve this problem is to define a pair of facades (robust and non-robust) inheriting one another. Probably, it's just because I'm not a Scala expert. If there's a better solution, please give me a hint. By the way. You said, `PLSA` and `RobustPLSA` are very similar and suffer from code duplication. Their codes are similar, though different (e.g. in robust version I have to pass background and noise matrices to almost every function). Now I've done my best to avoid code duplication via inheriting both of these classes from `AbstractPLSA`. If there is a way for further code generalization, please point it. You've also mentioned mllib/linalg/. Should I use DenseMatrix instead of Array[Array[Float]]? I don't like this idea because DenseMatrix contains Double values -- it seems to be a waste of memory and network resourses. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4669]Allow users to set arbitrary akka ...
Github user WangTaoTheTonic commented on the pull request: https://github.com/apache/spark/pull/3551#issuecomment-65223765 I will amend the comments and docs once we can make sure to merge this commit. @ScrapCodes I saw you [changed the way](https://github.com/apache/spark/commit/b3018811e106e6414816380a35c07a8564945d37) to set akka configuration, any comment about this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4693] [SQL] PruningPredicates may be wr...
GitHub user YanTangZhai opened a pull request: https://github.com/apache/spark/pull/3556 [SPARK-4693] [SQL] PruningPredicates may be wrong if predicates contains an empty AttributeSet() references The sql select * from spark_test::for_test where abs(20141202) is not null has predicates=List(IS NOT NULL HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFAbs(20141202)) and partitionKeyIds=AttributeSet(). PruningPredicates is List(IS NOT NULL HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFAbs(20141202)). Then the exception java.lang.IllegalArgumentException: requirement failed: Partition pruning predicates only supported for partitioned tables. is thrown. The sql select * from spark_test::for_test_partitioned_table where abs(20141202) is not null and type_id=11 and platform = 3 with partitioned key insert_date has predicates=List(IS NOT NULL HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFAbs(20141202), (type_id#12 = 11), (platform#8 = 3)) and partitionKeyIds=AttributeSet(insert_date#24). PruningPredicates is List(IS NOT NULL HiveSimpleUdf#org.apache.hadoop.hive.ql.udf.UDFAbs(20141202)). You can merge this pull request into a Git repository by running: $ git pull https://github.com/YanTangZhai/spark SPARK-4693 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3556.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3556 commit cdef539abc5d2d42d4661373939bdd52ca8ee8e6 Author: YanTangZhai hakeemz...@tencent.com Date: 2014-08-06T13:07:08Z Merge pull request #1 from apache/master update commit cbcba66ad77b96720e58f9d893e87ae5f13b2a95 Author: YanTangZhai hakeemz...@tencent.com Date: 2014-08-20T13:14:08Z Merge pull request #3 from apache/master Update commit 8a0010691b669495b4c327cf83124cabb7da1405 Author: YanTangZhai hakeemz...@tencent.com Date: 2014-09-12T06:54:58Z Merge pull request #6 from apache/master Update commit 03b62b043ab7fd39300677df61c3d93bb9beb9e3 Author: YanTangZhai hakeemz...@tencent.com Date: 2014-09-16T12:03:22Z Merge pull request #7 from apache/master Update commit 76d40277d51f709247df1d3734093bf2c047737d Author: YanTangZhai hakeemz...@tencent.com Date: 2014-10-20T12:52:22Z Merge pull request #8 from apache/master update commit d26d98248a1a4d0eb15336726b6f44e05dd7a05a Author: YanTangZhai hakeemz...@tencent.com Date: 2014-11-04T09:00:31Z Merge pull request #9 from apache/master Update commit e249846d9b7967ae52ec3df0fb09e42ffd911a8a Author: YanTangZhai hakeemz...@tencent.com Date: 2014-11-11T03:18:24Z Merge pull request #10 from apache/master Update commit 6e643f81555d75ec8ef3eb57bf5ecb6520485588 Author: YanTangZhai hakeemz...@tencent.com Date: 2014-12-01T11:23:56Z Merge pull request #11 from apache/master Update commit e572b9a754a71da1f5bdb53c283b936ab803def2 Author: YanTangZhai hakeemz...@tencent.com Date: 2014-12-02T12:27:14Z Update HiveStrategies.scala --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4693] [SQL] PruningPredicates may be wr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3556#issuecomment-65224550 [Test build #24042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24042/consoleFull) for PR 3556 at commit [`e572b9a`](https://github.com/apache/spark/commit/e572b9a754a71da1f5bdb53c283b936ab803def2). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1888] enhance MEMORY_AND_DISK mode by d...
Github user suyanNone commented on a diff in the pull request:
https://github.com/apache/spark/pull/791#discussion_r21156020
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -837,11 +837,11 @@ private[spark] class BlockManager(
* Drop a block from memory, possibly putting it on disk if applicable.
Called when the memory
* store reaches its limit and needs to free up space.
*
- * Return the block status if the given block has been updated, else
None.
+ * Return the block status and dropped memory size if the given block
has been updated, else None.
*/
def dropFromMemory(
blockId: BlockId,
- data: Either[Array[Any], ByteBuffer]): Option[BlockStatus] = {
+ data: Either[Array[Any], ByteBuffer]): Option[(BlockStatus, Long)] =
{
logInfo(sDropping block $blockId from memory)
val info = blockInfo.get(blockId).orNull
--- End diff --
Hi, as for now, this method is called by doPut 1) and doGetLocal to put
data from disk into memory 2), all this two are got related
blockinfo.synchronized, So will there need used info.synchronized again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Get result using executeCollect
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3547#issuecomment-65231830 [Test build #24043 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24043/consoleFull) for PR 3547 at commit [`a5ab68e`](https://github.com/apache/spark/commit/a5ab68ecf599328630aa18e6403a02e8b56f1dbe). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4693] [SQL] PruningPredicates may be wr...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3556#issuecomment-65232200 [Test build #24042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24042/consoleFull) for PR 3556 at commit [`e572b9a`](https://github.com/apache/spark/commit/e572b9a754a71da1f5bdb53c283b936ab803def2). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4693] [SQL] PruningPredicates may be wr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3556#issuecomment-65232207 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24042/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Get result using executeCollect
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3547#issuecomment-65240856 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24043/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Get result using executeCollect
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3547#issuecomment-65240848 [Test build #24043 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24043/consoleFull) for PR 3547 at commit [`a5ab68e`](https://github.com/apache/spark/commit/a5ab68ecf599328630aa18e6403a02e8b56f1dbe). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4461][YARN] pass extra java options to ...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3409#issuecomment-65241674 memory should not be specified through the java options (I filed a separate jira to support that - I would expect to have something like spark.yarn.am.memory for that). Like I mentioned in one of my original comments we should do a check like SparkConf has. I can see valid use cases for wanting to set the java options or classpath for an application master differently then the driver and I don't see any harm in allowing users to do that. Most users wouldn't have to set anything for these. The reason I brought up doing all 3 at once is I don't want to do one and then 2 weeks later someone ask for the other. I would rather have it consistent across executor/driver/am. But since there has been so much discussion back and forth perhaps just the java options for now and see if there is a need for the others. I will file a separate jira to look at making the driver.extra* apply similarly across the board also. I find it inconsistent that the driver.extraClassPath applies to AM but the other don't. @vanzin I see what you were getting at now. You were just wondering if we should do something specific to hadoop configs. The thing is you can do a lot of stuff in the configs and its deployment specific. Did you have something in mind? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4574][SQL] Adding support for defining ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3431#issuecomment-65242931 [Test build #24044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24044/consoleFull) for PR 3431 at commit [`12fab0a`](https://github.com/apache/spark/commit/12fab0adf97de693884a00286e1485654edfb6b9). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4697]System properties should override ...
GitHub user WangTaoTheTonic opened a pull request: https://github.com/apache/spark/pull/3557 [SPARK-4697]System properties should override environment variables I found some arguments in yarn module take environment variables before system properties while the latter override the former in core module. You can merge this pull request into a Git repository by running: $ git pull https://github.com/WangTaoTheTonic/spark SPARK4697 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3557.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3557 commit 5f43f453df19c1b281aa0a0e74d06b543a6e2d95 Author: WangTao barneystin...@aliyun.com Date: 2014-12-02T15:02:21Z System property can override environment variable --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4697]System properties should override ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3557#issuecomment-65245329 [Test build #24045 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24045/consoleFull) for PR 3557 at commit [`5f43f45`](https://github.com/apache/spark/commit/5f43f453df19c1b281aa0a0e74d06b543a6e2d95). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3739] [SQL] Update the split num base o...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/2589#issuecomment-65248499 Thank you @marmbrus , I've updated the code as you suggested. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4573] [SQL] Add SettableStructObjectIns...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3429#issuecomment-65248898 [Test build #24046 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24046/consoleFull) for PR 3429 at commit [`f5a40e8`](https://github.com/apache/spark/commit/f5a40e8beca52e0f9ab5651711d34d5315933802). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3739] [SQL] Update the split num base o...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2589#issuecomment-65248882 [Test build #24047 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24047/consoleFull) for PR 2589 at commit [`dff38e7`](https://github.com/apache/spark/commit/dff38e7709e3e1e01431beb3ef68dcc99db44889). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4618][SQL] Make foreign DDL commands op...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3470#issuecomment-65253492 [Test build #24048 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24048/consoleFull) for PR 3470 at commit [`3c132ef`](https://github.com/apache/spark/commit/3c132efa9747448b73605e83594d4ae161b7b86e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4639] [SQL] Pass maxIterations in as a ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3499#issuecomment-65254021 [Test build #24050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24050/consoleFull) for PR 3499 at commit [`4c1252c`](https://github.com/apache/spark/commit/4c1252c8bbe3c1d5ef24e16223ce0b74e60e0f96). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4574][SQL] Adding support for defining ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3431#issuecomment-65254247 [Test build #24044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24044/consoleFull) for PR 3431 at commit [`12fab0a`](https://github.com/apache/spark/commit/12fab0adf97de693884a00286e1485654edfb6b9). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ParquetRelation2(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4574][SQL] Adding support for defining ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3431#issuecomment-65254255 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24044/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4662] [SQL] Whitelist more unittest
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3522#issuecomment-65254437 [Test build #24049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24049/consoleFull) for PR 3522 at commit [`f54e4c7`](https://github.com/apache/spark/commit/f54e4c760660cf3b7f82a035369d224231339f80). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [YARN][SPARK-3293]Fix yarn's web show SUCCEED...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3508#issuecomment-65259690 Yeah the performance regression was difficult to reproduce according to @vanzin. I haven't been able to reproduce it myself but others have. Yes, this patch should only deal with the exception case since we can't safely and efficiently deal with exit codes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4697]System properties should override ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3557#issuecomment-65259859 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24045/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4697]System properties should override ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3557#issuecomment-65259844 [Test build #24045 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24045/consoleFull) for PR 3557 at commit [`5f43f45`](https://github.com/apache/spark/commit/5f43f453df19c1b281aa0a0e74d06b543a6e2d95). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4573] [SQL] Add SettableStructObjectIns...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3429#issuecomment-65260446 [Test build #24046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24046/consoleFull) for PR 3429 at commit [`f5a40e8`](https://github.com/apache/spark/commit/f5a40e8beca52e0f9ab5651711d34d5315933802). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4573] [SQL] Add SettableStructObjectIns...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3429#issuecomment-65260456 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24046/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4269][SQL] make wait time configurable ...
Github user jackylk commented on the pull request: https://github.com/apache/spark/pull/3133#issuecomment-65260781 had updated according to @marmbrus comment --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3739] [SQL] Update the split num base o...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2589#issuecomment-65261259 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24047/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3739] [SQL] Update the split num base o...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2589#issuecomment-65261249 [Test build #24047 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24047/consoleFull) for PR 2589 at commit [`dff38e7`](https://github.com/apache/spark/commit/dff38e7709e3e1e01431beb3ef68dcc99db44889). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][core] Don't transition ExecutorIn...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/3550#issuecomment-65264098 Yeah, this seems safe to me. Even if the Master doesn't know that the driver has exited for some reason (i.e. if the `finishApplication` was somehow not triggered from the `DisassociatedEvent` [here](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/master/Master.scala#L411)), this will still fail the application correctly because all existing executors will have exited and new executors will fail immediately because they can't connect to the driver. I agree that this is a better fix for 1.2. Eventually it would be good to get Josh's changes in too because it's easier to test things there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB] [spark-2352] Implementation of an 1-hi...
Github user avulanov commented on the pull request: https://github.com/apache/spark/pull/1290#issuecomment-65264353 Just in case, I've tested the ANN without the hidden layer and it seems to work as multinomial regression, though with (one-half) squared-error cost function and without softmax output. One needs to pass an empty array `Array[Int]()` as a hidden layer parameter. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4618][SQL] Make foreign DDL commands op...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3470#issuecomment-65265127 [Test build #24048 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24048/consoleFull) for PR 3470 at commit [`3c132ef`](https://github.com/apache/spark/commit/3c132efa9747448b73605e83594d4ae161b7b86e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class CaseInsensitiveMap(_baseMap: Map[String, String]) extends Map[String, String] ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4618][SQL] Make foreign DDL commands op...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3470#issuecomment-65265133 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24048/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4686] Link to allowed master URLs is br...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3542 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4639] [SQL] Pass maxIterations in as a ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3499#issuecomment-65265709 [Test build #24050 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24050/consoleFull) for PR 3499 at commit [`4c1252c`](https://github.com/apache/spark/commit/4c1252c8bbe3c1d5ef24e16223ce0b74e60e0f96). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class Analyzer(catalog: Catalog,` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4639] [SQL] Pass maxIterations in as a ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3499#issuecomment-65265713 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24050/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4662] [SQL] Whitelist more unittest
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3522#issuecomment-65267595 [Test build #24049 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24049/consoleFull) for PR 3522 at commit [`f54e4c7`](https://github.com/apache/spark/commit/f54e4c760660cf3b7f82a035369d224231339f80). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4662] [SQL] Whitelist more unittest
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3522#issuecomment-65267607 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24049/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][core] Don't transition ExecutorIn...
Github user markhamstra commented on the pull request: https://github.com/apache/spark/pull/3550#issuecomment-65269257 It's worth spending a little time checking that any executors that are RUNNING for an application definitely will transition to a Finished state and be removed from the master's accounting if the application dies. If we are certain that all the running executors will finish after application death and that repeatedly failing executors from a bad node while a running executor remains on master's books will not progressively consume resources, then I think this PR solves the problems. The only sort-of negative that I am seeing is that there can be an arbitrarily large number of failed executor launch attempts while at least one executor remains running, which will at least fill up error logs; but that is arguably not an all bad thing and is something whose proper resolution can be better handled (at least for now) by a system administrator than by an attempt to automate resolution. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4699][SQL] make caseSensitive configura...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3558#issuecomment-65270774 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4699][SQL] make caseSensitive configura...
GitHub user jackylk opened a pull request: https://github.com/apache/spark/pull/3558 [SPARK-4699][SQL] make caseSensitive configurable in Analyzer.scala You can merge this pull request into a Git repository by running: $ git pull https://github.com/jackylk/spark case Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3558.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3558 commit 578d167bfccdc2d1d5ce9ca06cab7b7b753bb3eb Author: Jacky Li jacky.li...@huawei.com Date: 2014-12-02T17:33:44Z make caseSensitive configurable --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/3541#issuecomment-65272551 Thx @kayousterhout for the ping ! We are fairly aggressively using blacklisting executors - not hosts. The assumption that a task failed on an executor in a host will imply it will fail on all executors on the host is incorrect - particularly in multi-tennet environment : where number of executors per host can be fairly high : which is exactly the reason why I added this hack to begin with. Instead of modifying the existing behavior of executor level blacklist, it would be better to add to it - add a way to configure host level blacklisting in addition to existing functionality. I am -1 on changing the existing behavior. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3883 SSL support for HttpServer and Akka
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2739#issuecomment-65278543 @jacek-lewandowski please work on top of master. We can work on backporting it to branch-1.2 if there's a strong desire for it, but new features should always be checked into master first. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4609] Blacklist hosts rather than execu...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/3541#issuecomment-65278813 @mridulm Thanks for the feedback. Could you provide some details about configurations (such as the number of executors on a host and the number of hosts)? And in which case the host level blacklist is worse than executor level? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4685] Update JavaDoc settings to includ...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/3554#issuecomment-65279181 Does this require any corresponding change in the Maven build? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3883 SSL support for HttpServer and Akka
Github user jacek-lewandowski commented on the pull request: https://github.com/apache/spark/pull/2739#issuecomment-65279317 @vanzin already did that. Now I'm running tests - is there a new procedure for testing? Or just sbt clean assembly test? I can see one test failure which I can reproduce on master as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3883 SSL support for HttpServer and Akka
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2739#issuecomment-65280103 I think `dev/run-tests` will run more comprehensive checks. Also, always a good idea to try such a thing yourself on real jobs, aside from the unit tests. :-) You'll probably need to open a new PR, I don't think Github allows you to change the target branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3883 SSL support for HttpServer and Akka
Github user jacek-lewandowski commented on the pull request: https://github.com/apache/spark/pull/2739#issuecomment-65280862 @vanzin yeah, thats right I tried running the real jobs before rebasing and they worked :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4699][SQL] make caseSensitive configura...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3558#issuecomment-65281087 Thanks for working on this! What about HiveContext? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4699][SQL] make caseSensitive configura...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3558#issuecomment-65281098 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4699][SQL] make caseSensitive configura...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3558#issuecomment-65281697 [Test build #24051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24051/consoleFull) for PR 3558 at commit [`578d167`](https://github.com/apache/spark/commit/578d167bfccdc2d1d5ce9ca06cab7b7b753bb3eb). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4672][Core]Checkpoint() should clear f ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/3545#issuecomment-65282261 cc @tdas can you take a look at this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [YARN][SPARK-3293]Fix yarn's web show SUCCEED...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3508#issuecomment-65283129 Just to reinforce, we shouldn't add a security manager without better understanding what impact it has. Feel free to file a bug if you'd like this investigated, but I like the policy of don't use System.exit() better. In any case, it seems your patch will fail apps that do a `System.exit(0)` without the security manager, so that's something to think about. Cluster mode takes the opposite approach (all apps that use `System.exit()` succeed), but probably for a reason that does not apply here (AM retries). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver - m...
Github user JoshRosen closed the pull request at: https://github.com/apache/spark/pull/3548 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4498][SPARK-2424] [WIP] Add driver - m...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3548#issuecomment-65285340 I'm going to close this for now in favor of Mark's patch (#3550). There are a couple of ideas here that might be useful for future improvements to this code (including factoring out the policy into a separate file for easier testing, which would be important if we added features like timeout-based host blacklisting), but I agree that this PR is more complex than we need for a narrow fix for this bug. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
