[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Github user hhbyyh commented on the pull request: https://github.com/apache/spark/pull/8551#issuecomment-147304609 @feynmanliang @jkbradley Sorry for the late update. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8551#issuecomment-147304573 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8551#issuecomment-147304561 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8551#issuecomment-147305874 [Test build #43561 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43561/consoleFull) for PR 8551 at commit [`ee4832d`](https://github.com/apache/spark/commit/ee4832d512fdeb57773f51846c9e75ecd8774c01). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10974][Streaming]Add progress bar for o...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9010#issuecomment-147305902 [Test build #43558 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43558/console) for PR 9010 at commit [`dc89b8b`](https://github.com/apache/spark/commit/dc89b8b48687302c7d2243fa99d1b8c63d27a41d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10974][Streaming]Add progress bar for o...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9010#issuecomment-147306039 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10974][Streaming]Add progress bar for o...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9010#issuecomment-147306040 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43558/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10973][ML][PYTHON] __gettitem__ method ...
Github user jkbradley commented on the pull request: https://github.com/apache/spark/pull/9009#issuecomment-147308172 Yes, please. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11029] [ML] Add computeCost to KMeansMo...
GitHub user hhbyyh opened a pull request: https://github.com/apache/spark/pull/9073 [SPARK-11029] [ML] Add computeCost to KMeansModel in spark.ml jira: https://issues.apache.org/jira/browse/SPARK-11029 We should add a method analogous to spark.mllib.clustering.KMeansModel.computeCost to spark.ml.clustering.KMeansModel. This will be a temp fix until we have proper evaluators defined for clustering. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hhbyyh/spark computeCost Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9073.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9073 commit 5440dc598cf973f83fec1e0356325ebd6d762430 Author: yuhaoyang Date: 2015-10-12T07:13:44Z add computeCost to ml.Kmeans --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11029] [ML] Add computeCost to KMeansMo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9073#issuecomment-147309298 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11029] [ML] Add computeCost to KMeansMo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9073#issuecomment-147309315 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8551#issuecomment-147310348 [Test build #43561 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43561/console) for PR 8551 at commit [`ee4832d`](https://github.com/apache/spark/commit/ee4832d512fdeb57773f51846c9e75ecd8774c01). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8551#issuecomment-147310378 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43561/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8551#issuecomment-147310376 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11029] [ML] Add computeCost to KMeansMo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9073#issuecomment-147310580 [Test build #43562 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43562/consoleFull) for PR 9073 at commit [`5440dc5`](https://github.com/apache/spark/commit/5440dc598cf973f83fec1e0356325ebd6d762430). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-8426] [scheduler] enhance blacklist mec...
Github user mwws commented on the pull request: https://github.com/apache/spark/pull/8760#issuecomment-147312498 @andrewor14 , could you also help review and comment. Thanks in advance. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10958] Use json4s 3.3.0. Formats is now...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/8992#issuecomment-147313013 Sounds good. IIRC it used more Jackson artifacts than Spark does, so excluding all of them might leave it lacking a library it needs. If so the solution would be to just manage down the version in `dependencyManagement` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11060][Streaming] Fix some potential NP...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9070#discussion_r41731525
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/ConstantInputDStream.scala
---
@@ -27,6 +28,9 @@ import scala.reflect.ClassTag
class ConstantInputDStream[T: ClassTag](ssc_ : StreamingContext, rdd:
RDD[T])
extends InputDStream[T](ssc_) {
+ require(rdd != null,
--- End diff --
(This could have been resolved with
https://github.com/apache/spark/pull/8881 for more consistency rather than a
different patch)
Is this inconsistent with SPARK-10772 and the other changes, which cause
`SparkException`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11060][Streaming] Fix some potential NP...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9070#discussion_r41731681
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/TransformedDStream.scala
---
@@ -39,7 +39,10 @@ class TransformedDStream[U: ClassTag] (
override def slideDuration: Duration = parents.head.slideDuration
override def compute(validTime: Time): Option[RDD[U]] = {
-val parentRDDs = parents.map(_.getOrCompute(validTime).orNull).toSeq
+val parentRDDs = parents.map { parent =>
parent.getOrCompute(validTime).getOrElse(
--- End diff --
You've probably thought through this, so I'm mostly asking: do we want to
assume the parent can be computed? in some places `null` or `None` does
correctly mean "can't be computed now". That may not be here. I'm making sure
we don't 'fix' the wrong part.
PS you can still retain the compact `_.getOrCompute` syntax here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11060][Streaming] Fix some potential NP...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9070#discussion_r41731988
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/TransformedDStream.scala
---
@@ -39,7 +39,10 @@ class TransformedDStream[U: ClassTag] (
override def slideDuration: Duration = parents.head.slideDuration
override def compute(validTime: Time): Option[RDD[U]] = {
-val parentRDDs = parents.map(_.getOrCompute(validTime).orNull).toSeq
+val parentRDDs = parents.map { parent =>
parent.getOrCompute(validTime).getOrElse(
--- End diff --
If "can't be computed now" and returns None, here `orNull` should not be
used, it will get null and pass by user, which will lead to NPE. For example:
```
inputStream.transform(_.union(rdd))
```
If inputStream returns `None` as what you mean by "can't be computed now",
this clause will throw NPE, so we should guard out this situation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11029] [ML] Add computeCost to KMeansMo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9073#issuecomment-147323578 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43562/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11029] [ML] Add computeCost to KMeansMo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9073#issuecomment-147323392 [Test build #43562 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43562/console) for PR 9073 at commit [`5440dc5`](https://github.com/apache/spark/commit/5440dc598cf973f83fec1e0356325ebd6d762430). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11029] [ML] Add computeCost to KMeansMo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9073#issuecomment-147323575 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10668][ML] Use WeightedLeastSquares in ...
Github user Lewuathe commented on the pull request: https://github.com/apache/spark/pull/8884#issuecomment-147323507 @dbtsai I rebased master and updated. Could you check it again? Thank you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11060][Streaming] Fix some potential NP...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9070#discussion_r41732138
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/TransformedDStream.scala
---
@@ -39,7 +39,10 @@ class TransformedDStream[U: ClassTag] (
override def slideDuration: Duration = parents.head.slideDuration
override def compute(validTime: Time): Option[RDD[U]] = {
-val parentRDDs = parents.map(_.getOrCompute(validTime).orNull).toSeq
+val parentRDDs = parents.map { parent =>
parent.getOrCompute(validTime).getOrElse(
--- End diff --
Also see here in
[UnionDStream](https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/dstream/UnionDStream.scala#L44),
also has such defensive assumption.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11055][SQL] Use mixing hash-based and s...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9067#issuecomment-147324890 [Test build #43559 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43559/console) for PR 9067 at commit [`787846b`](https://github.com/apache/spark/commit/787846bc5106a6a2372e109598fd78c0a2b38427). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11055][SQL] Use mixing hash-based and s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9067#issuecomment-147325034 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11055][SQL] Use mixing hash-based and s...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9067#issuecomment-147325035 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43559/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147326801 [Test build #43560 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43560/console) for PR 9072 at commit [`124aab0`](https://github.com/apache/spark/commit/124aab0da39ef9024ff37eb6459afcd2ef18ed82). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147326864 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43560/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147326863 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2251] fix concurrency issues in random ...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/1229#issuecomment-147326922 @pauloangelo sounds like your RDD is not immutable then, in which case many bets are off. RDDs are generally always the same whenever you compute them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11056] Improve documentation of SBT bui...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/9068#issuecomment-147327076 Seems fine to me, because of the pointer to more info. I don't know what the right breakdown is between info on this page and on the wiki; I suppose all in one place would be nice (i.e. just a link here) but I can't argue with this; the build/sbt bit is important. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11060][Streaming] Fix some potential NP...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9070#discussion_r41733627
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/TransformedDStream.scala
---
@@ -39,7 +39,10 @@ class TransformedDStream[U: ClassTag] (
override def slideDuration: Duration = parents.head.slideDuration
override def compute(validTime: Time): Option[RDD[U]] = {
-val parentRDDs = parents.map(_.getOrCompute(validTime).orNull).toSeq
+val parentRDDs = parents.map { parent =>
parent.getOrCompute(validTime).getOrElse(
--- End diff --
Sounds good, I see why this is consistent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9072#discussion_r41734082
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1521,8 +1521,15 @@ abstract class RDD[T: ClassTag](
}
checkpointData match {
- case Some(reliable: ReliableRDDCheckpointData[_]) => logWarning(
-"RDD was already marked for reliable checkpointing: overriding
with local checkpoint.")
+ case Some(reliable: ReliableRDDCheckpointData[_]) =>
--- End diff --
I suspect this is a good restriction, but cc @tdas regarding the semantics.
I feel like this code construct is getting hard to read. I don't feel
strongly, but is `if (checkpointData.isDefined && isCheckpointed)` not simpler?
then there are just two branches, one return point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11062] Thrift server does not support o...
GitHub user navis opened a pull request: https://github.com/apache/spark/pull/9074 [SPARK-11062] Thrift server does not support operationLog Currently, SparkExecuteStatementOperation is skipping beforeRun/afterRun method. You can merge this pull request into a Git repository by running: $ git pull https://github.com/navis/spark SPARK-11062 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9074.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9074 commit a56728c48a49a7bd2ff850ce0b36ba7b3ae60051 Author: navis.ryu Date: 2015-10-12T08:39:40Z [SPARK-11062] Thrift server does not support operationLog --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11062] Thrift server does not support o...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9074#issuecomment-147330040 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11062] Thrift server does not support o...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9074#issuecomment-147330070 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11062] Thrift server does not support o...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9074#issuecomment-147331430 [Test build #43563 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43563/consoleFull) for PR 9074 at commit [`a56728c`](https://github.com/apache/spark/commit/a56728c48a49a7bd2ff850ce0b36ba7b3ae60051). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11063][Streaming]Change preferredLocati...
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/9075 [SPARK-11063][Streaming]Change preferredLocations of Receiver's RDD to hosts rather than hostports The format of RDD's preferredLocations must be hostname but the format of Streaming Receiver's scheduling executors is hostport. So it doesn't work. This PR converts `schedulerExecutors` to `hosts` before creating Receiver's RDD. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zsxwing/spark SPARK-11063 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9075.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9075 commit 4706ec0ba2ee9a82b279c46abf7005894e593b3c Author: zsxwing Date: 2015-10-12T08:47:36Z Change preferredLocations of Receiver's RDD to hosts rather than hostports --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11063][Streaming]Change preferredLocati...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9075#issuecomment-147332022 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11063][Streaming]Change preferredLocati...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9075#issuecomment-147332103 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11063][Streaming]Change preferredLocati...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9075#issuecomment-147335370 [Test build #43564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43564/consoleFull) for PR 9075 at commit [`4706ec0`](https://github.com/apache/spark/commit/4706ec0ba2ee9a82b279c46abf7005894e593b3c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9072#discussion_r41736088
--- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala ---
@@ -1521,8 +1521,15 @@ abstract class RDD[T: ClassTag](
}
checkpointData match {
- case Some(reliable: ReliableRDDCheckpointData[_]) => logWarning(
-"RDD was already marked for reliable checkpointing: overriding
with local checkpoint.")
+ case Some(reliable: ReliableRDDCheckpointData[_]) =>
--- End diff --
In addition to `checkpointData.isDefined && isCheckpointed`, we still need
to check `isLocallyCheckpointed`, because when `checkpointData.isDefined` is
true, it is possibly a local checkpointing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147335942 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147335883 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147335906 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147338950 [Test build #43565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43565/consoleFull) for PR 9072 at commit [`124aab0`](https://github.com/apache/spark/commit/124aab0da39ef9024ff37eb6459afcd2ef18ed82). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11062] Thrift server does not support o...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9074#issuecomment-147341866 [Test build #43563 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43563/console) for PR 9074 at commit [`a56728c`](https://github.com/apache/spark/commit/a56728c48a49a7bd2ff850ce0b36ba7b3ae60051). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11062] Thrift server does not support o...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9074#issuecomment-147341954 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43563/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11062] Thrift server does not support o...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9074#issuecomment-147341952 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11063][Streaming]Change preferredLocati...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9075#issuecomment-147354342 [Test build #43564 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43564/console) for PR 9075 at commit [`4706ec0`](https://github.com/apache/spark/commit/4706ec0ba2ee9a82b279c46abf7005894e593b3c). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11063][Streaming]Change preferredLocati...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9075#issuecomment-147354464 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43564/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11063][Streaming]Change preferredLocati...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9075#issuecomment-147354463 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147364369 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43565/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147364329 [Test build #43565 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43565/console) for PR 9072 at commit [`124aab0`](https://github.com/apache/spark/commit/124aab0da39ef9024ff37eb6459afcd2ef18ed82). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147364367 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147364822 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147365440 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147365411 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147365490 [Test build #43566 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43566/consoleFull) for PR 9072 at commit [`124aab0`](https://github.com/apache/spark/commit/124aab0da39ef9024ff37eb6459afcd2ef18ed82). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10515] When killing executor, the pendi...
Github user KaiXinXiaoLei commented on a diff in the pull request:
https://github.com/apache/spark/pull/8945#discussion_r41752408
--- Diff:
core/src/test/scala/org/apache/spark/deploy/StandaloneDynamicAllocationSuite.scala
---
@@ -369,6 +369,38 @@ class StandaloneDynamicAllocationSuite
assert(apps.head.getExecutorLimit === 1)
}
+ test("the pending replacement executors should not be lost
(SPARK-10515)") {
+sc = new SparkContext(appConf)
+val appId = sc.applicationId
+eventually(timeout(10.seconds), interval(10.millis)) {
+ val apps = getApplications()
+ assert(apps.size === 1)
+ assert(apps.head.id === appId)
+ assert(apps.head.executors.size === 2)
+ assert(apps.head.getExecutorLimit === Int.MaxValue)
+}
+// sync executors between the Master and the driver, needed because
+// the driver refuses to kill executors it does not know about
+syncExecutors(sc)
+val executors = getExecutorIds(sc)
+assert(executors.size === 2)
+
+// kill executor 1, and replace it
+assert(sc.killAndReplaceExecutor(executors.head))
+var apps = getApplications()
+assert(apps.head.executors.size === 2)
--- End diff --
@vanzin
```
assert(sc.killAndReplaceExecutor(executors.head))
var apps = getApplications()
assert(apps.head.executors.size === 2)
```
I want to say, kill a executor and a new executor should replaces it. The
total number of executor should be not changed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147389080 [Test build #43566 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43566/console) for PR 9072 at commit [`124aab0`](https://github.com/apache/spark/commit/124aab0da39ef9024ff37eb6459afcd2ef18ed82). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147389311 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11051][Core] Do not allow local checkpo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9072#issuecomment-147389316 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43566/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10930] Adds max task duration to all st...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9051#issuecomment-147394955 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10930] Adds max task duration to all st...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/9051#issuecomment-147394925 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10930] Adds max task duration to all st...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9051#issuecomment-147394979 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10930] Adds max task duration to all st...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9051#issuecomment-147396129 [Test build #43567 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43567/consoleFull) for PR 9051 at commit [`5588157`](https://github.com/apache/spark/commit/5588157c2ef149bb8017ebe52d3fc695c8e4200e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10930] Adds max task duration to all st...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9051#issuecomment-147401311 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43567/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10930] Adds max task duration to all st...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9051#issuecomment-147401247 [Test build #43567 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43567/console) for PR 9051 at commit [`5588157`](https://github.com/apache/spark/commit/5588157c2ef149bb8017ebe52d3fc695c8e4200e). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10930] Adds max task duration to all st...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9051#issuecomment-147401309 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10930] Adds max task duration to all st...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/9051#issuecomment-147402862 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10930] Adds max task duration to all st...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9051#issuecomment-147403438 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10930] Adds max task duration to all st...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9051#issuecomment-147403468 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [spark-11026][yarn]spark.yarn.user.classpath.f...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/9045#issuecomment-147404161 sorry your description isn't clarifying it for me. Could you perhaps give an example? what do you mean by linkNames? If I use --jars /users/foo.jar and spark.yarn.user.classpath.first=true. foo.jar gets put into the system classpath (CLASSPATH=.:foo.jar:...:__spark__.jar --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10930] Adds max task duration to all st...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9051#issuecomment-147404975 [Test build #43568 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43568/consoleFull) for PR 9051 at commit [`5588157`](https://github.com/apache/spark/commit/5588157c2ef149bb8017ebe52d3fc695c8e4200e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10876 - Display total uptime for complet...
Github user jbonofre commented on the pull request: https://github.com/apache/spark/pull/9059#issuecomment-147411507 Updated PR with a minor cleanup in the total uptime computation. @srowen I double checked and I confirm that the history server view is updated (as the application details UI actually goes to history server view). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11052] Spaces in the build dir causes f...
Github user trystanleftwich commented on a diff in the pull request: https://github.com/apache/spark/pull/9065#discussion_r41761657 --- Diff: make-distribution.sh --- @@ -121,7 +121,7 @@ if [ $(command -v git) ]; then fi -if [ ! $(command -v "$MVN") ] ; then +if [ ! "$(command -v "$MVN")" ] ; then --- End diff -- It does, here is the output from that line when it has quotes: + '[' '!' '/tmp/test spaces/spark/build/mvn' ']' Before: + '[' '!' /tmp/test spaces/spark/build/mvn ']' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9708] [MESOS] Spark should create local...
Github user Zariel commented on the pull request: https://github.com/apache/spark/pull/8358#issuecomment-147414856 @tnachen is there anything my end which needs to be done to merge this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10876 - Display total uptime for complet...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9059#discussion_r41768301
--- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala ---
@@ -289,13 +290,14 @@ private[ui] class AllJobsPage(parent: JobsTab)
extends WebUIPage("") {
val summary: NodeSeq =
-{if (parent.sc.isDefined) {
- // Total duration is not meaningful unless the UI is live
-
-Total Uptime:
-{UIUtils.formatDuration(System.currentTimeMillis() -
startTime)}
-
+
+ Total Uptime:
+ {if (parent.sc.isDefined) {
--- End diff --
This seems OK, but can it be even simpler?
`{UIUtils.formatDuration((if (parent.sc.isDefined)
System.currentTimeMillis() else endTime) - startTime)}`
Or maybe pull out a `val` to simplify.
In any event LGTM in principle
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [spark-11026][yarn]spark.yarn.user.classpath.f...
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/9045#issuecomment-147439210 @tgravescs if use --jars hdfs://users/foo.jar, now i can not put foo.jar into the system classpath without this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10876 - Display total uptime for complet...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9059#issuecomment-147439152 [Test build #1875 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1875/consoleFull) for PR 9059 at commit [`35770d6`](https://github.com/apache/spark/commit/35770d60462b821ab44fd6817eb0f9b5a9a91edc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10930] Adds max task duration to all st...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9051#issuecomment-147439788 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11052] Spaces in the build dir causes f...
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/9065#discussion_r41768863 --- Diff: make-distribution.sh --- @@ -121,7 +121,7 @@ if [ $(command -v git) ]; then fi -if [ ! $(command -v "$MVN") ] ; then +if [ ! "$(command -v "$MVN")" ] ; then --- End diff -- Yeah I tried it locally too. Just making sure the first two double qutoes didn't somehow bind more closely than the parens. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10930] Adds max task duration to all st...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9051#issuecomment-147439789 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43568/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11052] Spaces in the build dir causes f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9065#issuecomment-147440268 [Test build #1876 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1876/consoleFull) for PR 9065 at commit [`76dcd50`](https://github.com/apache/spark/commit/76dcd506c03441f15ee81128e555dcebf74ea58d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10930] Adds max task duration to all st...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9051#issuecomment-147439696 [Test build #43568 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43568/console) for PR 9051 at commit [`5588157`](https://github.com/apache/spark/commit/5588157c2ef149bb8017ebe52d3fc695c8e4200e). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8880#issuecomment-147441714 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8880#issuecomment-147441746 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10771: Implement the shuffle encryption ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8880#issuecomment-147443941 [Test build #43569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43569/consoleFull) for PR 8880 at commit [`6116b2c`](https://github.com/apache/spark/commit/6116b2c7adc7f54aa152fbd89f7a914d24aedb9e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update DAGScheduler's "misbehaved ResultHandle...
GitHub user shellberg opened a pull request: https://github.com/apache/spark/pull/9076 Update DAGScheduler's "misbehaved ResultHandler" Restrict tasks (of job) to only 1 to ensure that the causing Exception asserted for job failure is the deliberately thrown DAGSchedulerSuiteDummyException intended, not an UnsupportedOperationException from any second/subsequent tasks that can propagate from a race condition during code execution. You can merge this pull request into a Git repository by running: $ git pull https://github.com/shellberg/spark shellberg-DAGSchedulerSuite-misbehavedResultHandlerTest-patch-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9076.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9076 commit 84dc8bf06a94ebf8fa7094486752d44ad0d08b77 Author: shellberg Date: 2015-10-12T11:48:52Z Update DAGScheduler's "misbehaved ResultHandler" Restrict tasks (of job) to only 1 to ensure that the causing Exception asserted for job failure is the deliberately thrown DAGSchedulerSuiteDummyException intended, not an UnsupportedOperationException from any second/subsequent tasks that can propagate from a race condition during code execution. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update DAGScheduler's "misbehaved ResultHandle...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9076#issuecomment-147445583 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-10876 - Display total uptime for complet...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/9059#issuecomment-147448778 I applied your patch and I noticed you didn't care the case we watch the status of incomplete apps through the history server. Sometimes, apps are marked as incomplete because of some reasons e.g. users forget to stop `SparkContext`, those apps are just running or etc. When we watch the status of those incomplete apps through the history server, `endTime` should still be `-1` and then, we can get invalid Total Uptime like this screen shot. https://cloud.githubusercontent.com/assets/4736016/10432636/66b40c7a-7147-11e5-8afc-3a725308312f.png";> --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-147449167 LGTM. @viirya Thank you for fixing this bug. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/9011#issuecomment-147449256 Merging to master and 1.5 branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/9011#discussion_r41773288
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -833,6 +833,33 @@ class SQLQuerySuite extends QueryTest with
SQLTestUtils with TestHiveSingleton {
).map(i => Row(i._1, i._2, i._3)))
}
+ test("window function: refer column in inner select block") {
+val data = Seq(
+ WindowData(1, "a", 5),
+ WindowData(2, "a", 6),
+ WindowData(3, "b", 7),
+ WindowData(4, "b", 8),
+ WindowData(5, "c", 9),
+ WindowData(6, "c", 10)
+)
+sparkContext.parallelize(data).toDF().registerTempTable("windowData")
--- End diff --
btw, you can use `withTempTable` in future to let our test infra
automatically drop the temp table created at here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9011 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10960][SQL] SQL with windowing function...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/9011#discussion_r41773979
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
---
@@ -833,6 +833,33 @@ class SQLQuerySuite extends QueryTest with
SQLTestUtils with TestHiveSingleton {
).map(i => Row(i._1, i._2, i._3)))
}
+ test("window function: refer column in inner select block") {
+val data = Seq(
+ WindowData(1, "a", 5),
+ WindowData(2, "a", 6),
+ WindowData(3, "b", 7),
+ WindowData(4, "b", 8),
+ WindowData(5, "c", 9),
+ WindowData(6, "c", 10)
+)
+sparkContext.parallelize(data).toDF().registerTempTable("windowData")
--- End diff --
Ah, thanks for suggesting it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [spark-11026][yarn]spark.yarn.user.classpath.f...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/9045#issuecomment-147451561 ah I understand now, thanks. I'll take a look --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
