[GitHub] spark issue #19332: [SPARK-22093][TESTS] Fixes `assume` in `UtilsSuite` and ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19332 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82117/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18924: [SPARK-14371] [MLLIB] OnlineLDAOptimizer should not coll...
Github user akopich commented on the issue: https://github.com/apache/spark/pull/18924 @WeichenXu123, thanks for creating Jira. Yes, sure I will work on it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19329: [SPARK-22110][SQL][Documentation] Add usage and improve ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19329 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19329: [SPARK-22110][SQL][Documentation] Add usage and improve ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19329 Thanks! Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19329: [SPARK-22110][SQL][Documentation] Add usage and i...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19329 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadi...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19286#discussion_r140638128
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
---
@@ -749,6 +749,34 @@ class JDBCSuite extends SparkFunSuite
assert(agg.isCascadingTruncateTable() === Some(true))
}
+ test("Aggregated dialects: isCascadingTruncateTable") {
+def genDialect(cascadingTruncateTable: Option[Boolean]): JdbcDialect =
new JdbcDialect {
+ override def canHandle(url: String): Boolean = true
+ override def getCatalystType(
+sqlType: Int,
+typeName: String,
+size: Int,
+md: MetadataBuilder): Option[DataType] = None
+ override def isCascadingTruncateTable(): Option[Boolean] =
cascadingTruncateTable
+}
+
+val dialectCombination = Seq(
+ List(genDialect(Some(true)), genDialect(Some(false)),
genDialect(None)),
+ List(genDialect(Some(true)), genDialect(Some(true)),
genDialect(None)),
+ List(genDialect(Some(false)), genDialect(Some(false)),
genDialect(None)),
+ List(genDialect(Some(true)), genDialect(Some(true))),
+ List(genDialect(Some(false)), genDialect(Some(false))),
+ List(genDialect(None), genDialect(None))
+)
+
+val expectedCascading = Seq(Some(true), Some(true), None, Some(true),
Some(false), None)
+
+dialectCombination.zip(expectedCascading).foreach { case (dialects,
cascading) =>
--- End diff --
Could we combine `dialectCombination` and `expectedCascading` together? Or
we can create a separate helper function?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadi...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19286#discussion_r140638141
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/AggregatedDialect.scala ---
@@ -43,6 +43,17 @@ private class AggregatedDialect(dialects:
List[JdbcDialect]) extends JdbcDialect
}
override def isCascadingTruncateTable(): Option[Boolean] = {
-dialects.flatMap(_.isCascadingTruncateTable()).reduceOption(_ || _)
+// If any dialect claims cascading truncate, this dialect is also
cascading truncate.
+// Otherwise, if any dialect has unknown cascading truncate, this
dialect is also unknown.
+val cascading =
dialects.flatMap(_.isCascadingTruncateTable()).reduceOption(_ || _)
+if (cascading.getOrElse(false)) {
+ cascading
+} else {
+ if (dialects.exists(_.isCascadingTruncateTable().isEmpty)) {
--- End diff --
combine line 51 and 52?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadi...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19286#discussion_r140638171
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/AggregatedDialect.scala ---
@@ -43,6 +43,17 @@ private class AggregatedDialect(dialects:
List[JdbcDialect]) extends JdbcDialect
}
override def isCascadingTruncateTable(): Option[Boolean] = {
-dialects.flatMap(_.isCascadingTruncateTable()).reduceOption(_ || _)
+// If any dialect claims cascading truncate, this dialect is also
cascading truncate.
+// Otherwise, if any dialect has unknown cascading truncate, this
dialect is also unknown.
+val cascading =
dialects.flatMap(_.isCascadingTruncateTable()).reduceOption(_ || _)
+if (cascading.getOrElse(false)) {
--- End diff --
Use case-match?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19330: Orderable MapType
Github user tejasapatil commented on the issue: https://github.com/apache/spark/pull/19330 @hvanhovell : based on [your comment over the jira](https://issues.apache.org/jira/browse/SPARK-18134?focusedCommentId=15693519&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15693519), it seemed that the approach to be used is yet to be finalised. Are we moving ahead with this approach ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19333: [SPARK-22109][SQL][BRANCH-2.2] Resolves type conflicts b...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19333 **[Test build #82119 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82119/testReport)** for PR 19333 at commit [`42fa83c`](https://github.com/apache/spark/commit/42fa83caab5914a089d282e0c5009f7d0db42c46). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19333: [SPARK-22109][SQL][BRANCH-2.2] Resolves type conflicts b...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19333 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82119/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19333: [SPARK-22109][SQL][BRANCH-2.2] Resolves type conflicts b...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19333 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19277: [SPARK-22058][CORE]the BufferedInputStream will not be c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19277 **[Test build #3933 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3933/testReport)** for PR 19277 at commit [`2e5f21a`](https://github.com/apache/spark/commit/2e5f21a1a89bbf69431f918d3043ff6b58420dd8). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19333: [SPARK-22109][SQL][BRANCH-2.2] Resolves type conflicts b...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/19333 Thanks! merging to branch-2.2. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19333: [SPARK-22109][SQL][BRANCH-2.2] Resolves type conflicts b...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/19333 @HyukjinKwon Could you close this PR? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18747: [WIP][SPARK-20822][SQL] Generate code to directly get va...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18747 **[Test build #82121 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82121/testReport)** for PR 18747 at commit [`c77b129`](https://github.com/apache/spark/commit/c77b12919a928bfef65f653fad6948484ef8ff3d). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19295: [SPARK-22080][SQL] Adds support for allowing user...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19295#discussion_r140638641
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/ExperimentalMethods.scala ---
@@ -44,11 +44,14 @@ class ExperimentalMethods private[sql]() {
*/
@volatile var extraStrategies: Seq[Strategy] = Nil
+ @volatile var extraPreOptimizations: Seq[Rule[LogicalPlan]] = Nil
+
@volatile var extraOptimizations: Seq[Rule[LogicalPlan]] = Nil
--- End diff --
This is an API change. We can't do it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19295: [SPARK-22080][SQL] Adds support for allowing user to add...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19295 I do not think we should do it. The extra pre-optimizer rules can easily break our existing optimizer rules. Adding post optimizer rules should be enough for 99% cases. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19222 **[Test build #82118 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82118/testReport)** for PR 19222 at commit [`8ec08ba`](https://github.com/apache/spark/commit/8ec08ba822bca62db0d917b2908dfc509e08e96b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19222 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82118/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19222 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19321: [SPARK-22100] [SQL] Make percentile_approx support numer...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/19321 That's a good point, thanks --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest commit sh...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19290 **[Test build #82120 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82120/testReport)** for PR 19290 at commit [`7e6c2c5`](https://github.com/apache/spark/commit/7e6c2c546d86946ee4cd110321ae24226cb50917). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest commit sh...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19290 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82120/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest commit sh...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19290 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19326: [SPARK-22107] Change as to alias in python quickstart
Github user jgoleary commented on the issue: https://github.com/apache/spark/pull/19326 @HyukjinKwon looks like this does the same thing as `name()` is an alias for `alias()`. Happy to close but suggest merging one of these to help future noobs. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18747: [WIP][SPARK-20822][SQL] Generate code to directly get va...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18747 **[Test build #82121 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82121/testReport)** for PR 18747 at commit [`c77b129`](https://github.com/apache/spark/commit/c77b12919a928bfef65f653fad6948484ef8ff3d). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18747: [WIP][SPARK-20822][SQL] Generate code to directly get va...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18747 Build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18747: [WIP][SPARK-20822][SQL] Generate code to directly get va...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18747 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82121/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19333: [SPARK-22109][SQL][BRANCH-2.2] Resolves type conf...
Github user HyukjinKwon closed the pull request at: https://github.com/apache/spark/pull/19333 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19294: [SPARK-21549][CORE] Respect OutputFormats with no output...
Github user szhem commented on the issue: https://github.com/apache/spark/pull/19294 @mridulm > incorporating a test for the sql part will also help in this matter. What should be the expected behaviour in case of sql? I'm asking because [the sql part seems to fail even before setupJob the on committer is called](https://github.com/apache/spark/blob/3f958a99921d149fb9fdf7ba7e78957afdad1405/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala#L118). FileOutputFormat.setOutputPath(job, new Path(outputSpec.outputPath)) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadingTrunc...
Github user dilipbiswal commented on the issue:
https://github.com/apache/spark/pull/19286
@viirya Hey simon, thanks for catching this. Will it be little easier to
follow if we wrote like this ?
```
override def isCascadingTruncateTable(): Option[Boolean] = {
def compute(left: Option[Boolean], right: Option[Boolean]):
Option[Boolean] = {
(left, right) match {
case (_, Some(true)) => Some(true)
case (Some(true), _) => Some(true)
case (Some(false), Some(false)) => Some(false)
case (_, _) => None
}
}
// If any dialect claims cascading truncate, this dialect is also
cascading truncate.
// Otherwise, if any dialect has unknown cascading truncate, this
dialect is also unknown.
dialects.map(_.isCascadingTruncateTable()).reduce(compute(_, _))
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest co...
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/19290#discussion_r140643692
--- Diff: R/pkg/R/DataFrame.R ---
@@ -2650,8 +2650,9 @@ setMethod("merge",
#' @param suffix a suffix for the column name
#' @return list of columns
#'
-#' @note generateAliasesForIntersectedCols since 1.6.0
-generateAliasesForIntersectedCols <- function(x, intersectedColNames,
suffix) { # nolint
+#' @note genAliasesForIntersectedCols since 1.6.0
--- End diff --
nit: I'd remove this `@note` too
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19290: [WIP][SPARK-22063][R] Upgrades lintr to latest commit sh...
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/19290 oh, remember to remove WIP and update this line https://github.com/apache/spark/pull/19290/files#diff-74ca2b618d236bbd6faa23e13bff1403R30 before merging in? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18704: [SPARK-20783][SQL] Create ColumnVector to abstract exist...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18704 **[Test build #82122 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82122/testReport)** for PR 18704 at commit [`1607bd1`](https://github.com/apache/spark/commit/1607bd152c64bf7900e489eb2cbef086f44e0861). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18704: [SPARK-20783][SQL] Create ColumnVector to abstract exist...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18704 **[Test build #82122 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82122/testReport)** for PR 18704 at commit [`1607bd1`](https://github.com/apache/spark/commit/1607bd152c64bf7900e489eb2cbef086f44e0861). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18704: [SPARK-20783][SQL] Create ColumnVector to abstract exist...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18704 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82122/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18704: [SPARK-20783][SQL] Create ColumnVector to abstract exist...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18704 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadi...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19286#discussion_r140644507
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/AggregatedDialect.scala ---
@@ -43,6 +43,17 @@ private class AggregatedDialect(dialects:
List[JdbcDialect]) extends JdbcDialect
}
override def isCascadingTruncateTable(): Option[Boolean] = {
-dialects.flatMap(_.isCascadingTruncateTable()).reduceOption(_ || _)
+// If any dialect claims cascading truncate, this dialect is also
cascading truncate.
+// Otherwise, if any dialect has unknown cascading truncate, this
dialect is also unknown.
+val cascading =
dialects.flatMap(_.isCascadingTruncateTable()).reduceOption(_ || _)
+if (cascading.getOrElse(false)) {
--- End diff --
Done.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadingTrunc...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19286 @dilipbiswal Thanks for the suggestion. However, it looks more complicated, IMO. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadingTrunc...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19286 **[Test build #82123 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82123/testReport)** for PR 19286 at commit [`7e5a57c`](https://github.com/apache/spark/commit/7e5a57c3e4d9550d2ddd8a971293ace3984b5447). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18704: [SPARK-20783][SQL] Create ColumnVector to abstract exist...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18704 **[Test build #82124 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82124/testReport)** for PR 18704 at commit [`b8d5dec`](https://github.com/apache/spark/commit/b8d5decfa32a8d8c1eba331a976eb2e341c40b53). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18704: [SPARK-20783][SQL] Create ColumnVector to abstract exist...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18704 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18704: [SPARK-20783][SQL] Create ColumnVector to abstract exist...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18704 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82124/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18704: [SPARK-20783][SQL] Create ColumnVector to abstract exist...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18704 **[Test build #82124 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82124/testReport)** for PR 18704 at commit [`b8d5dec`](https://github.com/apache/spark/commit/b8d5decfa32a8d8c1eba331a976eb2e341c40b53). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18704: [SPARK-20783][SQL] Create ColumnVector to abstract exist...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18704 **[Test build #82125 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82125/testReport)** for PR 18704 at commit [`549b10f`](https://github.com/apache/spark/commit/549b10fac2e3b7a8cfd9d289ab4c152e7f764a17). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadingTrunc...
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/19286 @viirya No problem. The newer version you have looks clean as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadingTrunc...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19286 **[Test build #82123 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82123/testReport)** for PR 19286 at commit [`7e5a57c`](https://github.com/apache/spark/commit/7e5a57c3e4d9550d2ddd8a971293ace3984b5447). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadingTrunc...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19286 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82123/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadingTrunc...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19286 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18704: [SPARK-20783][SQL] Create ColumnVector to abstract exist...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18704 **[Test build #82125 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82125/testReport)** for PR 18704 at commit [`549b10f`](https://github.com/apache/spark/commit/549b10fac2e3b7a8cfd9d289ab4c152e7f764a17). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18704: [SPARK-20783][SQL] Create ColumnVector to abstract exist...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18704 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82125/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18704: [SPARK-20783][SQL] Create ColumnVector to abstract exist...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18704 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19330: Orderable MapType
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/19330 @hvanhovell Thanks a lot for comment. I got you point. I will refine soon. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadingTrunc...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19286 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadingTrunc...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19286 Thanks! Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19286: [SPARK-21338][SQL][FOLLOW-UP] Implement isCascadi...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/19286 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19317: [SPARK-22098][CORE] Add new method aggregateByKeyLocally...
Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/19317 Does not `treeAggregate` will introduce another `Shuffle`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19321: [SPARK-22100] [SQL] Make percentile_approx suppor...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19321#discussion_r140647683
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
---
@@ -134,7 +141,18 @@ case class ApproximatePercentile(
}
override def eval(buffer: PercentileDigest): Any = {
-val result = buffer.getPercentiles(percentages)
+val doubleResult = buffer.getPercentiles(percentages)
+val result = child.dataType match {
+ case DateType => doubleResult.map(_.toInt)
+ case TimestampType => doubleResult.map(_.toLong)
+ case ByteType => doubleResult.map(_.toByte)
+ case ShortType => doubleResult.map(_.toShort)
+ case IntegerType => doubleResult.map(_.toInt)
+ case LongType => doubleResult.map(_.toLong)
+ case FloatType => doubleResult.map(_.toFloat)
+ case DoubleType => doubleResult
+ case _: DecimalType => doubleResult.map(Decimal(_))
--- End diff --
Add
```Scala
case other: DataType =>
throw new UnsupportedOperationException(s"Unexpected data type
$other")
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #19321: [SPARK-22100] [SQL] Make percentile_approx suppor...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19321#discussion_r140647928
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala
---
@@ -123,7 +124,13 @@ case class ApproximatePercentile(
val value = child.eval(inputRow)
// Ignore empty rows, for example: percentile_approx(null)
if (value != null) {
- buffer.add(value.asInstanceOf[Double])
+ // Convert the value to a double value
+ val doubleValue = child.dataType match {
+case DateType => value.asInstanceOf[Int].toDouble
+case TimestampType => value.asInstanceOf[Long].toDouble
+case n: NumericType =>
n.numeric.toDouble(value.asInstanceOf[n.InternalType])
--- End diff --
The same here.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19321: [SPARK-22100] [SQL] Make percentile_approx support numer...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19321 Could you document the change in the output type of `percentile_approx ` in the following section? https://spark.apache.org/docs/latest/sql-programming-guide.html#migration-guide --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19293: [SPARK-22079][SQL] Serializer in HiveOutputWriter miss l...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19293 @LantaoJin Please add a test case. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19293: [SPARK-22079][SQL] Serializer in HiveOutputWriter miss l...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/19293 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19293: [SPARK-22079][SQL] Serializer in HiveOutputWriter miss l...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19293 **[Test build #82126 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82126/testReport)** for PR 19293 at commit [`45477fb`](https://github.com/apache/spark/commit/45477fbf00558066e3733a34e1d59ce22c192ee2). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
