[GitHub] spark pull request: [SPARK-12864][YARN] initialize executorIdCount...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10794#issuecomment-204276434 **[Test build #54685 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54685/consoleFull)** for PR 10794 at commit [`ebe3c7f`](https://github.com/apache/spark/commit/ebe3c7f290929588c822137b8bf27b18fe75393f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12864][YARN] initialize executorIdCount...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10794#issuecomment-204276708 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54685/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12864][YARN] initialize executorIdCount...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10794#issuecomment-204276705 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14295][SPARK-14274][SQL] Implements bui...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/12088#issuecomment-204276794 Hm, seems that this PR broke master build. I'm looking into it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14299][Examples] Remove duplications fo...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/12092#issuecomment-204276885 @yinxusen Some example code can be used as command-line apps, e.g., `LinearRegressionExample`. I think we should keep them since people might want to use them as templates to build their own. Could you make a pass and exclude them from this PR? Later we can call them `App` instead of `Example`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13432][SQL] add the source file name an...
Github user kiszk commented on the pull request: https://github.com/apache/spark/pull/11301#issuecomment-204277041 @sarutak , thanks, I will address them. Since I am making a business trip next week, its progress may be slow. I realized one big issue when I tried to resolve conflicts. In the latest code, ASTNode, whose constructor captures a stack trace, was dropped. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14299][Examples] Remove duplications fo...
Github user yinxusen commented on the pull request: https://github.com/apache/spark/pull/12092#issuecomment-204277789 Sure thing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13321][SQL] Add nested union test cases
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11361#issuecomment-204278001 cc @hvanhovell @rxin Because new ANTLR4 parser seems can support this syntax. I updated this to add test cases only. Please take a look. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12081#discussion_r58169289
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
---
@@ -116,8 +116,9 @@ class SessionCatalog(
currentDb = db
}
- def getDefaultDBPath(db: String): String = {
-System.getProperty("java.io.tmpdir") + File.separator + db + ".db"
+ def getDatabasePath(dbName: String, path: Option[String]): String = {
--- End diff --
Actually, I am also wondering if we should return an empty String in
non-Hive tables? Let me know if we need to remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13321][SQL] Add nested union test cases
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11361#issuecomment-204278368 **[Test build #54689 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54689/consoleFull)** for PR 11361 at commit [`6775020`](https://github.com/apache/spark/commit/6775020adf1f1066f411937571b3b9664b627280). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Minor][DOC] Add python examples for DCT,MinMa...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/12063#issuecomment-204278357 @zhengruifeng Could you make JIRAs for this PR and #11844? `[MINOR]` is only used for apparent typo fixes or trivial changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12081#issuecomment-204278364 **[Test build #54688 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54688/consoleFull)** for PR 12081 at commit [`16ac0b1`](https://github.com/apache/spark/commit/16ac0b1a548fedf7f602097ebb4aa1e7ed285515). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-7425] [ML] spark.ml Predictor should su...
Github user BenFradet commented on the pull request: https://github.com/apache/spark/pull/10355#issuecomment-204279426 Will fix that, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11517#issuecomment-204279636 **[Test build #54690 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54690/consoleFull)** for PR 11517 at commit [`6dfecf1`](https://github.com/apache/spark/commit/6dfecf1626a8981dd9921322e578fa9e81974781). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/12060#discussion_r58169664
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -750,23 +764,20 @@ class DAGScheduler(
submitStage(stage)
}
}
-submitWaitingStages()
}
/**
* Check for waiting stages which are now eligible for resubmission.
- * Ordinarily run on every iteration of the event loop.
+ * Ordinarily run after the parent stage completed successfully.
*/
- private def submitWaitingStages() {
-// TODO: We might want to run this less often, when we are sure that
something has become
-// runnable that wasn't before.
+ private def submitWaitingChildStages(parent: Stage) {
logTrace("Checking for newly runnable parent stages")
logTrace("running: " + runningStages)
logTrace("waiting: " + waitingStages)
logTrace("failed: " + failedStages)
-val waitingStagesCopy = waitingStages.toArray
-waitingStages.clear()
-for (stage <- waitingStagesCopy.sortBy(_.firstJobId)) {
+val childStages =
waitingStages.filter(_.parents.contains(parent)).toArray
+waitingStages --= childStages
+for (stage <- childStages.sortBy(_.firstJobId)) {
submitStage(stage)
--- End diff --
Seems `submitWaitingChildStages ` is called to submit child stages when
the given `parent` stage is available. From this observation, do we have to
re-check missing parents inside `submitStage`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/12060#discussion_r58170068 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1245,9 +1250,8 @@ class DAGScheduler( markMapStageJobAsFinished(job, stats) } } +submitWaitingChildStages(shuffleStage) --- End diff -- Do we need to call this `submitWaitingChildStages(shuffleStage)` when the given `shuffleStage` is not available? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11517#issuecomment-204283360 **[Test build #54686 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54686/consoleFull)** for PR 11517 at commit [`ef588db`](https://github.com/apache/spark/commit/ef588db4a8f8d9d742225c16dbad9d8cb17e2c71). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11517#issuecomment-204284197 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11517#issuecomment-204284201 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54686/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14299][Examples] Remove duplications fo...
Github user yinxusen commented on the pull request: https://github.com/apache/spark/pull/12092#issuecomment-204285909 test it please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12864][YARN] initialize executorIdCount...
Github user zhonghaihua commented on the pull request: https://github.com/apache/spark/pull/10794#issuecomment-204288682 @andrewor14 @tgravescs @vanzin The code and the comment is optimized. And the description of this PR and jira is also updated. Please review it again. Thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11517#discussion_r58170614
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/BenchmarkWholeStageCodegen.scala
---
@@ -85,6 +85,66 @@ class BenchmarkWholeStageCodegen extends SparkFunSuite {
*/
}
+ ignore("range/sample/sum") {
+val N = 500 << 20
+runBenchmark("range/sample/sum", N) {
+ sqlContext.range(N).sample(true, 0.8).groupBy().sum().collect()
+}
+/*
+Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
+range/sample/sum: Best/Avg Time(ms)Rate(M/s)
Per Row(ns) Relative
+
---
+range/sample/sum codegen=false 55656 / 56490 9.4
106.2 1.0X
+range/sample/sum codegen=true 35423 / 35758 14.8
67.6 1.6X
+*/
+
+runBenchmark("range/sample/sum", N) {
+ sqlContext.range(N).sample(false, 0.8).groupBy().sum().collect()
+}
+/*
+Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
+range/sample/sum: Best/Avg Time(ms)Rate(M/s)
Per Row(ns) Relative
+
---
+range/sample/sum codegen=false 16460 / 17161 31.9
31.4 1.0X
+range/sample/sum codegen=true4081 / 5390128.5
7.8 4.0X
+*/
+ }
+
+ ignore("sort merge join/sample") {
--- End diff --
We may do not want this
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user davies commented on a diff in the pull request:
https://github.com/apache/spark/pull/11517#discussion_r58170654
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/BenchmarkWholeStageCodegen.scala
---
@@ -85,6 +85,66 @@ class BenchmarkWholeStageCodegen extends SparkFunSuite {
*/
}
+ ignore("range/sample/sum") {
+val N = 500 << 20
+runBenchmark("range/sample/sum", N) {
+ sqlContext.range(N).sample(true, 0.8).groupBy().sum().collect()
--- End diff --
Should this be more selective?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14299][Examples] Remove duplications fo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12092#issuecomment-204289142 **[Test build #54691 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54691/consoleFull)** for PR 12092 at commit [`2424729`](https://github.com/apache/spark/commit/242472937b45f21ed5d49702a8efe16bdfd8ae6b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/11517#discussion_r58170829 --- Diff: core/src/main/scala/org/apache/spark/util/random/RandomSampler.scala --- @@ -186,7 +186,7 @@ class BernoulliSampler[T: ClassTag](fraction: Double) extends RandomSampler[T, T * @tparam T item type */ @DeveloperApi -class PoissonSampler[T: ClassTag]( +class PoissonSampler[T]( --- End diff -- @mengxr Is it OK to change this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13321][SQL] Add nested union test cases
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/11361#discussion_r58171387
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala
---
@@ -426,4 +428,68 @@ class PlanParserSuite extends PlanTest {
"Number of aliases must match the number of fields in an inline
table.")
intercept[ArrayIndexOutOfBoundsException](parsePlan("values (1, 'a'),
(2, 'b', 5Y)"))
}
+
+ test("nesting UNION") {
+val parsed = parsePlan(
+ """
+ |SELECT `u_1`.`id` FROM (((SELECT `t0`.`id` FROM `default`.`t0`)
+ |UNION ALL (SELECT `t0`.`id` FROM `default`.`t0`)) UNION ALL
+ |(SELECT `t0`.`id` FROM `default`.`t0`)) AS u_1
+ """.stripMargin)
+
+val expected = Project(
--- End diff --
Minor: @viirya could you update this use the DSL and `assertEqual` equals?
It makes this a bit easier to read.
BTW this test is very similar to the following test case:
https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala#L384-L392
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13321][SQL] Add nested union test cases
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/11361#discussion_r58171562
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
---
@@ -1114,6 +1114,91 @@ class HiveQuerySuite extends HiveComparisonTest with
BeforeAndAfter {
}
}
+ test("nested union") {
+sql(
+ """
+| EXPLAIN
--- End diff --
What is the point of this SQL statement? See if it compiles?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14299][Examples] Remove duplications fo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12092#issuecomment-204293051 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54691/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14299][Examples] Remove duplications fo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12092#issuecomment-204293046 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14299][Examples] Remove duplications fo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12092#issuecomment-204292831 **[Test build #54691 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54691/consoleFull)** for PR 12092 at commit [`2424729`](https://github.com/apache/spark/commit/242472937b45f21ed5d49702a8efe16bdfd8ae6b). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class Params(` * ` case class Params(` * ` case class Params(` * ` case class Params(` * ` case class Params(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13321][SQL] Add nested union test cases
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/11361#discussion_r58171624
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
---
@@ -1114,6 +1114,91 @@ class HiveQuerySuite extends HiveComparisonTest with
BeforeAndAfter {
}
}
+ test("nested union") {
+sql(
+ """
+| EXPLAIN
+| SELECT count(1) FROM (
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+|
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+|
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+|
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+|
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src) src
+ """.stripMargin)
+
+val countForSrc = sql("SELECT count(1) FROM src").first()
+
+val countForUnion25Src = sql(
+ """
+| SELECT count(1) FROM (
--- End diff --
It seems a bit redundant to write the allmost the same query as above...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3724][ML] RandomForest: More options fo...
Github user MLnick commented on the pull request: https://github.com/apache/spark/pull/11989#issuecomment-204296175 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/12060#discussion_r58172201
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -750,23 +764,20 @@ class DAGScheduler(
submitStage(stage)
}
}
-submitWaitingStages()
}
/**
* Check for waiting stages which are now eligible for resubmission.
- * Ordinarily run on every iteration of the event loop.
+ * Ordinarily run after the parent stage completed successfully.
*/
- private def submitWaitingStages() {
-// TODO: We might want to run this less often, when we are sure that
something has become
-// runnable that wasn't before.
+ private def submitWaitingChildStages(parent: Stage) {
logTrace("Checking for newly runnable parent stages")
logTrace("running: " + runningStages)
logTrace("waiting: " + waitingStages)
logTrace("failed: " + failedStages)
-val waitingStagesCopy = waitingStages.toArray
-waitingStages.clear()
-for (stage <- waitingStagesCopy.sortBy(_.firstJobId)) {
+val childStages =
waitingStages.filter(_.parents.contains(parent)).toArray
+waitingStages --= childStages
+for (stage <- childStages.sortBy(_.firstJobId)) {
submitStage(stage)
--- End diff --
Yes, and the re-check is done in the `submitStage()`.
If there are some missing parent stages, the child will go to
`waitingStages` again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3724][ML] RandomForest: More options fo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11989#issuecomment-204297156 **[Test build #54692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54692/consoleFull)** for PR 11989 at commit [`b9416d6`](https://github.com/apache/spark/commit/b9416d61506b40ebbb52c591ebf001295744e316). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-14321. [SQL] Reduce date format cost and...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12105#issuecomment-204296880 **[Test build #2721 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2721/consoleFull)** for PR 12105 at commit [`6fd07db`](https://github.com/apache/spark/commit/6fd07db11b5c9eed795dde11177f1c245a6fef16). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14299][Examples] Remove duplications fo...
Github user yinxusen commented on the pull request: https://github.com/apache/spark/pull/12092#issuecomment-204297783 @mengxr Excluded those examples. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...
Github user ueshin commented on a diff in the pull request: https://github.com/apache/spark/pull/12060#discussion_r58173090 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1245,9 +1250,8 @@ class DAGScheduler( markMapStageJobAsFinished(job, stats) } } +submitWaitingChildStages(shuffleStage) --- End diff -- I don't think we need. The `shuffleStage` is re-submitted if it is not available, so the `submitWaitingChildStages(shuffleStage)` for the stage will call after it is processed again and becomes available. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-14113. Consider marking JobConf closure-...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11978#issuecomment-204300804 **[Test build #54687 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54687/consoleFull)** for PR 11978 at commit [`0c53ed2`](https://github.com/apache/spark/commit/0c53ed23e3fb24cc5d882272ddca629843005629). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-14113. Consider marking JobConf closure-...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11978#issuecomment-204301039 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54687/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-14113. Consider marking JobConf closure-...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11978#issuecomment-204301037 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/11517#discussion_r58174636
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/BenchmarkWholeStageCodegen.scala
---
@@ -85,6 +85,66 @@ class BenchmarkWholeStageCodegen extends SparkFunSuite {
*/
}
+ ignore("range/sample/sum") {
+val N = 500 << 20
+runBenchmark("range/sample/sum", N) {
+ sqlContext.range(N).sample(true, 0.8).groupBy().sum().collect()
--- End diff --
0.2?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/11517#discussion_r58174661
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/BenchmarkWholeStageCodegen.scala
---
@@ -85,6 +85,66 @@ class BenchmarkWholeStageCodegen extends SparkFunSuite {
*/
}
+ ignore("range/sample/sum") {
+val N = 500 << 20
+runBenchmark("range/sample/sum", N) {
+ sqlContext.range(N).sample(true, 0.8).groupBy().sum().collect()
+}
+/*
+Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
+range/sample/sum: Best/Avg Time(ms)Rate(M/s)
Per Row(ns) Relative
+
---
+range/sample/sum codegen=false 55656 / 56490 9.4
106.2 1.0X
+range/sample/sum codegen=true 35423 / 35758 14.8
67.6 1.6X
+*/
+
+runBenchmark("range/sample/sum", N) {
+ sqlContext.range(N).sample(false, 0.8).groupBy().sum().collect()
+}
+/*
+Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
+range/sample/sum: Best/Avg Time(ms)Rate(M/s)
Per Row(ns) Relative
+
---
+range/sample/sum codegen=false 16460 / 17161 31.9
31.4 1.0X
+range/sample/sum codegen=true4081 / 5390128.5
7.8 4.0X
+*/
+ }
+
+ ignore("sort merge join/sample") {
--- End diff --
ok. Let me remove it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/11517#discussion_r58175716
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/BenchmarkWholeStageCodegen.scala
---
@@ -85,6 +85,66 @@ class BenchmarkWholeStageCodegen extends SparkFunSuite {
*/
}
+ ignore("range/sample/sum") {
+val N = 500 << 20
+runBenchmark("range/sample/sum", N) {
+ sqlContext.range(N).sample(true, 0.8).groupBy().sum().collect()
--- End diff --
I remember that I have posted benchmark for `p=0.01` in #11578 as required
by @mengxr. Let me update this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/12060#discussion_r58175906
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -750,23 +764,20 @@ class DAGScheduler(
submitStage(stage)
}
}
-submitWaitingStages()
}
/**
* Check for waiting stages which are now eligible for resubmission.
- * Ordinarily run on every iteration of the event loop.
+ * Ordinarily run after the parent stage completed successfully.
*/
- private def submitWaitingStages() {
-// TODO: We might want to run this less often, when we are sure that
something has become
-// runnable that wasn't before.
+ private def submitWaitingChildStages(parent: Stage) {
logTrace("Checking for newly runnable parent stages")
logTrace("running: " + runningStages)
logTrace("waiting: " + waitingStages)
logTrace("failed: " + failedStages)
-val waitingStagesCopy = waitingStages.toArray
-waitingStages.clear()
-for (stage <- waitingStagesCopy.sortBy(_.firstJobId)) {
+val childStages =
waitingStages.filter(_.parents.contains(parent)).toArray
+waitingStages --= childStages
+for (stage <- childStages.sortBy(_.firstJobId)) {
submitStage(stage)
--- End diff --
Ahah, I see.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3724][ML] RandomForest: More options fo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11989#issuecomment-204306558 **[Test build #54692 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54692/consoleFull)** for PR 11989 at commit [`b9416d6`](https://github.com/apache/spark/commit/b9416d61506b40ebbb52c591ebf001295744e316). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3724][ML] RandomForest: More options fo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11989#issuecomment-204306689 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3724][ML] RandomForest: More options fo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11989#issuecomment-204306695 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54692/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-14113. Consider marking JobConf closure-...
Github user rajeshbalamohan commented on the pull request:
https://github.com/apache/spark/pull/11978#issuecomment-204307805
@andrewor14 - Not sure if I understood your last comment. Currently no
direct invocation to HadoopRDD (with initLocalJobConfFuncOpt) is made in
Spark. Later point in time, if change is needed to invoke HadoopRDD (with
initLocalJobConfFuncOpt) via SparkContext, following method could be added
which cleans up the function.
```
def hadoopRDD[K, V](
broadcastedConf: Broadcast[SerializableConfiguration],
initLocalJobConfFuncOpt: Option[JobConf => Unit],
inputFormatClass: Class[_ <: InputFormat[K, V]],
keyClass: Class[K],
valueClass: Class[V],
minPartitions: Int = defaultMinPartitions): RDD[(K, V)] = withScope {
assertNotStopped()
clean(initLocalJobConfFuncOpt)
new HadoopRDD(this, broadcastedConf, initLocalJobConfFuncOpt,
inputFormatClass, keyClass, valueClass, minPartitions)
}
```
But, I am not sure whether we need to clean sc.hadoopRDD in this patch.
Please let me know.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11517#issuecomment-204309253 **[Test build #54693 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54693/consoleFull)** for PR 11517 at commit [`12e1b37`](https://github.com/apache/spark/commit/12e1b37e076ca7161bae66ecec99938f1cd6e813). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14295][MLLIB][HOTFIX] Fixes Scala 2.10 ...
GitHub user liancheng opened a pull request: https://github.com/apache/spark/pull/12107 [SPARK-14295][MLLIB][HOTFIX] Fixes Scala 2.10 compilation failure ## What changes were proposed in this pull request? Fixes a compilation failure introduced in PR #12088 under Scala 2.10. ## How was this patch tested? Compilation. You can merge this pull request into a Git repository by running: $ git pull https://github.com/liancheng/spark spark-14295-hotfix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12107.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12107 commit a4e6c1bef0bd97302884f650de8941c33ca4a693 Author: Cheng Lian Date: 2016-04-01T08:30:14Z Fixes Scala 2.10 compilation failure --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13063] [YARN] Make the SPARK YARN STAGI...
Github user devaraj-kavali commented on the pull request: https://github.com/apache/spark/pull/12082#issuecomment-204309628 Thanks @tgravescs for looking into the patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13063] [YARN] Make the SPARK YARN STAGI...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12082#issuecomment-204310428 **[Test build #54695 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54695/consoleFull)** for PR 12082 at commit [`42848ad`](https://github.com/apache/spark/commit/42848adb25a8f19439ed439e1425441a4975c2f9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14295][MLLIB][HOTFIX] Fixes Scala 2.10 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12107#issuecomment-204310414 **[Test build #54694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54694/consoleFull)** for PR 12107 at commit [`a4e6c1b`](https://github.com/apache/spark/commit/a4e6c1bef0bd97302884f650de8941c33ca4a693). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13321][SQL] Add nested union test cases
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/11361#discussion_r58177517
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
---
@@ -1114,6 +1114,91 @@ class HiveQuerySuite extends HiveComparisonTest with
BeforeAndAfter {
}
}
+ test("nested union") {
+sql(
+ """
+| EXPLAIN
+| SELECT count(1) FROM (
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+|
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+|
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+|
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+|
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src UNION ALL
+| SELECT key, value FROM src) src
+ """.stripMargin)
+
+val countForSrc = sql("SELECT count(1) FROM src").first()
+
+val countForUnion25Src = sql(
+ """
+| SELECT count(1) FROM (
--- End diff --
ha. sure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13321][SQL] Add nested union test cases
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/11361#discussion_r58177573
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
---
@@ -1114,6 +1114,91 @@ class HiveQuerySuite extends HiveComparisonTest with
BeforeAndAfter {
}
}
+ test("nested union") {
+sql(
+ """
+| EXPLAIN
--- End diff --
Previous solution to the ANTLR3 parser will hang on this query.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13063] [YARN] Make the SPARK YARN STAGI...
Github user devaraj-kavali commented on a diff in the pull request:
https://github.com/apache/spark/pull/12082#discussion_r58177914
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -1444,4 +1444,19 @@ object Client extends Logging {
uri.startsWith(s"$LOCAL_SCHEME:")
}
+ /**
+ * Returns the app staging dir.
+ */
+ private def getAppStagingDirPath(
+ conf: SparkConf,
+ fs: FileSystem,
+ appStagingDir: String): Path = {
+val stagingRootDir = conf.get(STAGING_DIR).orNull
--- End diff --
`conf.get(STAGING_DIR).orElse(fs.getHomeDirectory)` gives type mismatch
compilation error since the `fs.getHomeDirectory` return type is Path and it
expects to be String as per `conf.get(STAGING_DIR)`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13321][SQL] Add nested union test cases
Github user viirya closed the pull request at: https://github.com/apache/spark/pull/11361 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13321][SQL] Add nested union test cases
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/11361#discussion_r58178060
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala
---
@@ -426,4 +428,68 @@ class PlanParserSuite extends PlanTest {
"Number of aliases must match the number of fields in an inline
table.")
intercept[ArrayIndexOutOfBoundsException](parsePlan("values (1, 'a'),
(2, 'b', 5Y)"))
}
+
+ test("nesting UNION") {
+val parsed = parsePlan(
+ """
+ |SELECT `u_1`.`id` FROM (((SELECT `t0`.`id` FROM `default`.`t0`)
+ |UNION ALL (SELECT `t0`.`id` FROM `default`.`t0`)) UNION ALL
+ |(SELECT `t0`.`id` FROM `default`.`t0`)) AS u_1
+ """.stripMargin)
+
+val expected = Project(
--- End diff --
hmm. indeed. If so, I think I can close this pr now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13321][SQL] Add nested union test cases
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/11361#discussion_r58178278
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala
---
@@ -426,4 +428,68 @@ class PlanParserSuite extends PlanTest {
"Number of aliases must match the number of fields in an inline
table.")
intercept[ArrayIndexOutOfBoundsException](parsePlan("values (1, 'a'),
(2, 'b', 5Y)"))
}
+
+ test("nesting UNION") {
+val parsed = parsePlan(
+ """
+ |SELECT `u_1`.`id` FROM (((SELECT `t0`.`id` FROM `default`.`t0`)
+ |UNION ALL (SELECT `t0`.`id` FROM `default`.`t0`)) UNION ALL
+ |(SELECT `t0`.`id` FROM `default`.`t0`)) AS u_1
+ """.stripMargin)
+
+val expected = Project(
--- End diff --
@hvanhovell Is new ANTLR4 parser natively to solve this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11517#issuecomment-204311884 **[Test build #54693 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54693/consoleFull)** for PR 11517 at commit [`12e1b37`](https://github.com/apache/spark/commit/12e1b37e076ca7161bae66ecec99938f1cd6e813). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11517#issuecomment-204311911 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54693/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11517#issuecomment-204311910 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13902][SPARK-14269][SCHEDULER] Eliminat...
Github user maropu commented on the pull request: https://github.com/apache/spark/pull/12060#issuecomment-204312096 This pr seems great in terms of spark-core performance, so could you assign qualified guys to review this? cc: @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/11517#issuecomment-204312598 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10530] [CORE] Kill other task attempts ...
Github user devaraj-kavali commented on the pull request: https://github.com/apache/spark/pull/11996#issuecomment-204313542 Thanks @tgravescs for checking this, I will add test for these changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12081#issuecomment-204313429 **[Test build #54688 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54688/consoleFull)** for PR 12081 at commit [`16ac0b1`](https://github.com/apache/spark/commit/16ac0b1a548fedf7f602097ebb4aa1e7ed285515). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12081#issuecomment-204314841 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54688/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14124] [SQL] [FOLLOWUP] Implement Datab...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12081#issuecomment-204314831 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14244][SQL] Don't use SizeBasedWindowFu...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/12040#discussion_r58179063
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/Window.scala ---
@@ -885,11 +886,21 @@ private[execution] object AggregateProcessor {
val evaluateExpressions =
mutable.Buffer.fill[Expression](ordinal)(NoOp)
val imperatives = mutable.Buffer.empty[ImperativeAggregate]
+// SPARK-14244: `SizeBasedWindowFunction`s are firstly created on
driver side and then
+// serialized to executor side. These functions all reference a global
singleton window
+// partition size attribute reference, i.e.,
`SizeBasedWindowFunction.n`. Here we must collect
+// the singleton instance created on driver side instead of using
executor side
+// `SizeBasedWindowFunction.n` to avoid binding failure caused by
mismatching expression ID.
+val partitionSize = {
--- End diff --
Yeah, saw discussion between you and Yin above. This improves readability.
TBH, I was also quite confused while updating this part of code. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11517#issuecomment-204316401 **[Test build #54696 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54696/consoleFull)** for PR 11517 at commit [`12e1b37`](https://github.com/apache/spark/commit/12e1b37e076ca7161bae66ecec99938f1cd6e813). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13321][SQL] Add nested union test cases
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11361#issuecomment-204317907 **[Test build #54689 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54689/consoleFull)** for PR 11361 at commit [`6775020`](https://github.com/apache/spark/commit/6775020adf1f1066f411937571b3b9664b627280). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14295][MLLIB][HOTFIX] Fixes Scala 2.10 ...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/12107#issuecomment-204317960 Jenkins compilation passed. I'm merging this to bring back Jenkins master build. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13321][SQL] Add nested union test cases
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11361#issuecomment-204318561 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54689/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13321][SQL] Add nested union test cases
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11361#issuecomment-204318557 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13063] [YARN] Make the SPARK YARN STAGI...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12082#issuecomment-204319140 **[Test build #54695 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54695/consoleFull)** for PR 12082 at commit [`42848ad`](https://github.com/apache/spark/commit/42848adb25a8f19439ed439e1425441a4975c2f9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13063] [YARN] Make the SPARK YARN STAGI...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12082#issuecomment-204319528 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13063] [YARN] Make the SPARK YARN STAGI...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12082#issuecomment-204319532 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54695/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14065]Increase probability of using cac...
Github user viper-kun commented on the pull request: https://github.com/apache/spark/pull/11886#issuecomment-204319818 @srowen The logic is the same to me.If necessary, I will put serializeMapStatus into match...None. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14295][MLLIB][HOTFIX] Fixes Scala 2.10 ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/12107 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13112]CoarsedExecutorBackend register t...
Github user viper-kun commented on the pull request: https://github.com/apache/spark/pull/12078#issuecomment-204322877 I don't think so. It is a wrong order "after registed to drive first, it begin new Executor. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/12076#issuecomment-204324891 1. Hive uses Kryo "the guava of serialization" internally; I don't know the specifics, but its not insignificant. 1. they moved ahead of spark's version to fix some bugs; that's the kryo which repackaged things. 1. Kryo types were used across methods called from Spark; the unshading was needed to get things to link, same with the switch from 2.22 to 2.21 (and a change to some files). It was less traumatic to push Hive back slightly than to try to force an update into unknown territory for chill & dependencies, even if it were possible. Upgrading to hive 2 would finally get everything in sync: ideally eliminate the need to have a custom hive JAR at all. Essentially all spark needs is a version of Hive with parameters that can be exchanged across all needed methods, with the conflict packages shaded, and the non-conflict packages generally omitted If you want to find out where things don't link, edit the spark pom to pull in org.apache.hive/hive and see what breaks. (of course, the other thing that would be nice would be for Hive to make it possible to subclass their thrift service cleanly; I've got a PoC of that somewhere ... it's not hard to add) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11416][BUILD] Update to Chill 0.8.0 & K...
Github user steveloughran commented on the pull request: https://github.com/apache/spark/pull/12076#issuecomment-204325755 ...Thinking about this; it might be possible to go to hive with a shaded kryo, with the invocation of those methods which exchange kryo types referring to the shaded values. That doesn't do anything for marshalling stuff between kryo versions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14295][MLLIB][HOTFIX] Fixes Scala 2.10 ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12107#issuecomment-204326073 **[Test build #54694 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54694/consoleFull)** for PR 12107 at commit [`a4e6c1b`](https://github.com/apache/spark/commit/a4e6c1bef0bd97302884f650de8941c33ca4a693). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14295][MLLIB][HOTFIX] Fixes Scala 2.10 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12107#issuecomment-204326442 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14138][SQL] Fix generated SpecificColum...
GitHub user kiszk opened a pull request: https://github.com/apache/spark/pull/12108 [Spark-14138][SQL] Fix generated SpecificColumnarIterator code can exceed JVM size limit for cached DataFrames ## What changes were proposed in this pull request? This PR reduces Java byte code size of method in ```SpecificColumnarIterator``` by using a approach to make a group for lot of ```ColumnAccessor``` instantiations or method calls (more than 200) into a method ## How was this patch tested? Added a new unit test, which includes large instantiations and method calls, to ```InMemoryColumnarQuerySuite``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/kiszk/spark SPARK-14138-master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12108.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12108 commit 9ce2ad637ceef472f7df405b3203159e60574837 Author: Kazuaki Ishizaki Date: 2016-04-01T09:19:07Z group calls of accessors or extractors into methods if their numbers exceeds a threshold commit 0a1b8818838c0149f73bab7a7638c09f351548a7 Author: Kazuaki Ishizaki Date: 2016-04-01T09:19:32Z add test suite --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14295][MLLIB][HOTFIX] Fixes Scala 2.10 ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12107#issuecomment-204326446 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54694/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13321][SQL] Add nested union test cases
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/11361#discussion_r58181875
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala
---
@@ -426,4 +428,68 @@ class PlanParserSuite extends PlanTest {
"Number of aliases must match the number of fields in an inline
table.")
intercept[ArrayIndexOutOfBoundsException](parsePlan("values (1, 'a'),
(2, 'b', 5Y)"))
}
+
+ test("nesting UNION") {
+val parsed = parsePlan(
+ """
+ |SELECT `u_1`.`id` FROM (((SELECT `t0`.`id` FROM `default`.`t0`)
+ |UNION ALL (SELECT `t0`.`id` FROM `default`.`t0`)) UNION ALL
+ |(SELECT `t0`.`id` FROM `default`.`t0`)) AS u_1
+ """.stripMargin)
+
+val expected = Project(
--- End diff --
@viirya the new parser handles nested queries a lot better. This is mainly
due to ANTLR4's better parsing algorithms.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14138][SQL][master] Fix generated Speci...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12108#issuecomment-204327067 **[Test build #54697 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54697/consoleFull)** for PR 12108 at commit [`0a1b881`](https://github.com/apache/spark/commit/0a1b8818838c0149f73bab7a7638c09f351548a7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11517#issuecomment-204327322 **[Test build #54690 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54690/consoleFull)** for PR 11517 at commit [`6dfecf1`](https://github.com/apache/spark/commit/6dfecf1626a8981dd9921322e578fa9e81974781). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class BatchPythonEvaluation(udfs: Seq[PythonUDF], output: Seq[Attribute], child: SparkPlan)` * ` // enable memo iff we serialize the row with schema (schema and class should be memorized)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11517#issuecomment-204327572 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13674][SQL] Add wholestage codegen supp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11517#issuecomment-204327574 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54690/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13112]CoarsedExecutorBackend register t...
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/12078#issuecomment-204329728
I see your point, just defer the registering of executors until fully
created. But here Spark do take care of this issue by such code, though as I
said not so elegant to handle this race condition.
```scala
if (executor == null) {
logError("Received LaunchTask command but executor was null")
System.exit(1)
} else {
```
Looking at the description of this JIRA, a more deeper problem is that
driver scheduler is not aware of this bad machine and repeatedly assign tasks
on this node, and finally make the job failure. So in short term maybe this pr
can solve this race condition problem, but this race condition will only be
happened on some slow machines (that's why I haven't met this problem before),
so maybe a more generic solution is that scheduler should be aware of bad
executor/node. Just my two cents, not so relevant to this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MINOR][SQL] Fix comments style and corrects s...
GitHub user HyukjinKwon opened a pull request: https://github.com/apache/spark/pull/12109 [MINOR][SQL] Fix comments style and corrects several styles in CSV data source ## What changes were proposed in this pull request? While trying to create a PR (which was not an issue at the end), I just corrected some style nits. So, I removed the changes except for some coding style corrections. - According to the [scala-style-guide#documentation-style](https://github.com/databricks/scala-style-guide#documentation-style), Scala style comments are discouraged. >```scala >/** This is a correct one-liner, short description. */ > >/** > * This is correct multi-line JavaDoc comment. And > * this is my second line, and if I keep typing, this would be > * my third line. > */ > >/** In Spark, we don't use the ScalaDoc style so this > * is not correct. > */ >``` - According to [scala-style-guide#blank-lines-vertical-whitespace](https://github.com/databricks/scala-style-guide#blank-lines-vertical-whitespace), single newline appears when >Between consecutive members (or initializers) of a class: fields, constructors, methods, nested classes, static initializers, instance initializers. So, this PR also removes the line. - Remove uesless parentheses in tests - Use `mapPartitions` instead of `mapPartitionsWithIndex()`. ## How was this patch tested? Unit tests were used and `dev/run_tests` for style tests. You can merge this pull request into a Git repository by running: $ git pull https://github.com/HyukjinKwon/spark SPARK-14271 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/12109.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12109 commit c8f1745689d82b6b0cc714ac326ce8cf65888f61 Author: hyukjinkwon Date: 2016-04-01T08:58:29Z Support for reading with rowSeparator option commit b8f09347cfe4aa7f371d4964a0452ed711d82bfb Author: hyukjinkwon Date: 2016-04-01T09:46:26Z Reverts changes commit 784d8f7ba3120c9873517d3137284e44ebe5ccca Author: hyukjinkwon Date: 2016-04-01T09:48:45Z Java style comments commit 878ae8911480ca48e0dd5ed7d10fff2670d97514 Author: hyukjinkwon Date: 2016-04-01T09:50:25Z Revert other changes as well and correct style commit f56e748ed81711a98d70427b791a790f26da471d Author: hyukjinkwon Date: 2016-04-01T09:51:34Z Revert toString() to toString --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MINOR][SQL] Fix comments style and corrects s...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12109#issuecomment-204337478 **[Test build #54698 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54698/consoleFull)** for PR 12109 at commit [`f56e748`](https://github.com/apache/spark/commit/f56e748ed81711a98d70427b791a790f26da471d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14138][SQL][master] Fix generated Speci...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12108#issuecomment-204346020 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14138][SQL][master] Fix generated Speci...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12108#issuecomment-204345904 **[Test build #54697 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54697/consoleFull)** for PR 12108 at commit [`0a1b881`](https://github.com/apache/spark/commit/0a1b8818838c0149f73bab7a7638c09f351548a7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark-14138][SQL][master] Fix generated Speci...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/12108#issuecomment-204346023 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54697/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13108][SQL] Support for ascii compatibl...
Github user HyukjinKwon closed the pull request at: https://github.com/apache/spark/pull/11016 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13108][SQL] Support for ascii compatibl...
Github user HyukjinKwon commented on the pull request: https://github.com/apache/spark/pull/11016#issuecomment-204346308 Sorry, this will not work for Windows (`\r\n`). I am closing this. If you intend to just block non-ascii compatible encodings, then I will create a new PR or reopen this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-14244][SQL] Don't use SizeBasedWindowFu...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/12040#issuecomment-204347119 **[Test build #54699 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54699/consoleFull)** for PR 12040 at commit [`27c22f2`](https://github.com/apache/spark/commit/27c22f23836cade81ce7e21eb9137630da7efb6d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
