[GitHub] spark issue #13491: [SPARK-15748][SQL] Replace inefficient foldLeft() call w...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13491
  
**[Test build #59921 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59921/consoleFull)**
 for PR 13491 at commit 
[`50a6270`](https://github.com/apache/spark/commit/50a62704362d8a0963b47cf4086e594a8f2c2f21).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13463: [SPARK-14959][SQL] handle partitioned table direc...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13463


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13463: [SPARK-14959][SQL] handle partitioned table directories ...

[liancheng]

Github user liancheng commented on the issue:

https://github.com/apache/spark/pull/13463
  
LGTM, merging to master and branch-2.0. Thanks for fixing this!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13491: [SPARK-15748][SQL] Replace inefficient foldLeft()...

[JoshRosen]

GitHub user JoshRosen opened a pull request:

https://github.com/apache/spark/pull/13491

[SPARK-15748][SQL] Replace inefficient foldLeft() call with flatMap() in 
PartitionStatistics

`PartitionStatistics` uses `foldLeft` and list concatenation (`++`) to 
flatten an iterator of lists, but this is extremely inefficient compared to 
simply doing `flatMap`/`flatten` because it performs many unnecessary object 
allocations. Simply replacing this `foldLeft` by a `flatMap` results in decent 
performance gains when constructing PartitionStatistics instances for tables 
with many columns.

This patch fixes this and also makes two similar changes in MLlib and 
streaming to try to fix all known occurrences of this pattern.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JoshRosen/spark foldleft-to-flatmap

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/13491.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #13491


commit 50a62704362d8a0963b47cf4086e594a8f2c2f21
Author: Josh Rosen 
Date:   2016-06-03T05:41:12Z

Replace inefficient foldLeft calls by flatMap().




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13491: [SPARK-15748][SQL] Replace inefficient foldLeft() call w...

[JoshRosen]

Github user JoshRosen commented on the issue:

https://github.com/apache/spark/pull/13491
  
@ericl also observed this same perf. bottleneck in his profiling.

I'll update the benchmark numbers tomorrrow.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13470: [SPARK-15733][SQL] Makes the explain output less ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13470


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13470: [SPARK-15733][SQL] Makes the explain output less verbose...

[liancheng]

Github user liancheng commented on the issue:

https://github.com/apache/spark/pull/13470
  
Merging to master and branch-2.0.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13269: [SPARK-15494][SQL] encoder code cleanup

[liancheng]

Github user liancheng commented on the issue:

https://github.com/apache/spark/pull/13269
  
Just rebased this branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13269: [SPARK-15494][SQL] encoder code cleanup

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13269
  
**[Test build #59920 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59920/consoleFull)**
 for PR 13269 at commit 
[`efe0cd5`](https://github.com/apache/spark/commit/efe0cd57056673fcd3406ffaa468b431fa71eac9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13446: [SPARK-15704] [SQL] fix a crash in TungstenAggregate whi...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13446
  
**[Test build #59919 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59919/consoleFull)**
 for PR 13446 at commit 
[`65a1534`](https://github.com/apache/spark/commit/65a1534c101f1bc0dd711af88d08dedc42df6c08).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13476: [SPARK-15684][SparkR]Not mask startsWith and ends...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/13476#discussion_r65657624
  
--- Diff: R/pkg/R/generics.R ---
@@ -695,10 +695,6 @@ setGeneric("desc", function(x) { 
standardGeneric("desc") })
 
 #' @rdname column
 #' @export
-setGeneric("endsWith", function(x, ...) { standardGeneric("endsWith") })
--- End diff --

to be clear, try something like
```
setGeneric("endsWith", function(x, suffix) { standardGeneric("endsWith") })
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13489: [SPARK-15745][SQL] Use classloader's getResource() for r...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13489
  
**[Test build #59918 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59918/consoleFull)**
 for PR 13489 at commit 
[`165ed51`](https://github.com/apache/spark/commit/165ed519b9d9910adb4bce6ed9a36b5456c8f02d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13476: [SPARK-15684][SparkR]Not mask startsWith and ends...

[wangmiao1981]

Github user wangmiao1981 commented on a diff in the pull request:

https://github.com/apache/spark/pull/13476#discussion_r65657352
  
--- Diff: R/pkg/R/generics.R ---
@@ -695,10 +695,6 @@ setGeneric("desc", function(x) { 
standardGeneric("desc") })
 
 #' @rdname column
 #' @export
-setGeneric("endsWith", function(x, ...) { standardGeneric("endsWith") })
--- End diff --

I kept this definition before. But sparkR complains no signature found. The 
tests only work if I remove it in generic.R. Let me try again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13483: [SPARK-15688][SQL] RelationalGroupedDataset.toDF ...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/13483#discussion_r65657150
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala ---
@@ -224,6 +224,26 @@ class DatasetAggregatorSuite extends QueryTest with 
SharedSQLContext {
 checkAnswer(df.groupBy($"j").agg(RowAgg.toColumn), Row("a", 1) :: 
Row("b", 5) :: Nil)
   }
 
+  test("SPARK-15688: Remove redundant expressions from aggregate") {
+val df = Seq(1 -> "a", 2 -> "b", 3 -> "b").toDF("col1", "col2")
--- End diff --

@gatorsmile Ah.. good idea. I will make the change. Thanks :-)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13176: [SPARK-15100][DOC] Modified user guide and examples for ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13176
  
**[Test build #59915 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59915/consoleFull)**
 for PR 13176 at commit 
[`5a53051`](https://github.com/apache/spark/commit/5a530519d77f036a90b71a654d377e333e99a7f9).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13176: [SPARK-15100][DOC] Modified user guide and examples for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13176
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13176: [SPARK-15100][DOC] Modified user guide and examples for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13176
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59915/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13394: [SPARK-15490][R][DOC] SparkR 2.0 QA: New R APIs a...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/13394#discussion_r65656790
  
--- Diff: R/pkg/R/DataFrame.R ---
@@ -1069,6 +1079,8 @@ setMethod("first",
 #'
 #' @param x A SparkDataFrame
 #'
+#' @family SparkDataFrame functions
+#' @rdname tordd
--- End diff --

let's revert these 2 lines as well? thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13061: [SPARK-14279] [Build] Pick the spark version from pom

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13061
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59914/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13061: [SPARK-14279] [Build] Pick the spark version from pom

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13061
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13483: [SPARK-15688][SQL] RelationalGroupedDataset.toDF should ...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/13483
  
LGTM except a few comments in test cases. : )

@cloud-fan @yhuai Could you please review this PR? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13061: [SPARK-14279] [Build] Pick the spark version from pom

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13061
  
**[Test build #59914 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59914/consoleFull)**
 for PR 13061 at commit 
[`031e5b0`](https://github.com/apache/spark/commit/031e5b030f24e4e0ef93ce22d3c2fdb8deefb53a).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13488: [MINOR][R][DOC] Fix R documentation generation instructi...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/13488
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13483: [SPARK-15688][SQL] RelationalGroupedDataset.toDF ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/13483#discussion_r65656657
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala ---
@@ -224,6 +224,26 @@ class DatasetAggregatorSuite extends QueryTest with 
SharedSQLContext {
 checkAnswer(df.groupBy($"j").agg(RowAgg.toColumn), Row("a", 1) :: 
Row("b", 5) :: Nil)
   }
 
+  test("SPARK-15688: Remove redundant expressions from aggregate") {
+val df = Seq(1 -> "a", 2 -> "b", 3 -> "b").toDF("col1", "col2")
--- End diff --

Could you use `WithSQLConf` here? This is to ensure this test case will not 
be affected by the others or the future change in the default value of 
`SQLConf.DATAFRAME_RETAIN_GROUP_COLUMNS.key`

Please also fix the related one in `DataFrameAggregateSuite` by adding 
`WithSQLConf`:  
https://github.com/apache/spark/blob/bb1362eb3b36b553dca246b95f59ba7fd8adcc8a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala#L210-L215


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13476: [SPARK-15684][SparkR]Not mask startsWith and ends...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/13476#discussion_r65656594
  
--- Diff: R/pkg/R/generics.R ---
@@ -695,10 +695,6 @@ setGeneric("desc", function(x) { 
standardGeneric("desc") })
 
 #' @rdname column
 #' @export
-setGeneric("endsWith", function(x, ...) { standardGeneric("endsWith") })
--- End diff --

you shouldn't need to remove it - I think if they are updated to match the 
signature of the existing generic (in newer R 3.3.0) it should work both in the 
case of R 3.3.0 and R 3.2.0.
With regards to the comment on R version check - it should not be necessary 
also.
@shivaram 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13483: [SPARK-15688][SQL] RelationalGroupedDataset.toDF ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/13483#discussion_r65656488
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala ---
@@ -224,6 +224,26 @@ class DatasetAggregatorSuite extends QueryTest with 
SharedSQLContext {
 checkAnswer(df.groupBy($"j").agg(RowAgg.toColumn), Row("a", 1) :: 
Row("b", 5) :: Nil)
   }
 
+  test("SPARK-15688: Remove redundant expressions from aggregate") {
+val df = Seq(1 -> "a", 2 -> "b", 3 -> "b").toDF("col1", "col2")
+// Aggregate expression contains the group by column and should appear 
once in the output.
+val df1 = df.groupBy("col1").agg($"col1", count("*"))
+assert(df1.schema.map(_.name) === Seq("col1", "count(1)"))
+// Only the column that is missing in aggregate expression should be 
added to aggregate expr.
+val df2 = df.groupBy("col1", "col2").agg($"col1", count("*"))
+assert(df2.schema.map(_.name) === Seq("col2", "col1", "count(1)"))
+// Complex expression in group by and aggregate clause.
+val df3 = df.groupBy(expr("col1 + 2")).agg(expr("col1 + 2"), 
count("*"))
+assert(df3.schema.map(_.name) === Seq("(col1 + 2)", "count(1)"))
+// Group by column should be added as its not part of aggregate 
expression.
--- End diff --

`Group by column` -> `Group-by expressions`. `its` -> `they are`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13476: [SPARK-15684][SparkR]Not mask startsWith and endsWith in...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/13476
  
please remove the workaround in test_context.R


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13476: [SPARK-15684][SparkR]Not mask startsWith and ends...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/13476#discussion_r65656453
  
--- Diff: R/pkg/R/column.R ---
@@ -151,6 +151,40 @@ setMethod("substr", signature(x = "Column"),
 column(jc)
   })
 
+#' startsWith
+#'
+#' Determines if entries of x start with string (entries of) prefix 
respectively,
+#' where strings are recycled to common lengths.
+#'
+#' @rdname startsWith
+#' @name startsWith
+#' @family colum_func
+#'
+#' @param vector of character string whose “starts” are considered
+#' @param character vector (often of length one)
+setMethod("startsWith", signature(x = "Column"),
+  function(x, prefix) {
+jc <- callJMethod(x@jc, "startsWith", as.vector(prefix))
+column(jc)
+  })
+
+#' endsWith
+#'
+#' Determines if entries of x end with string (entries of) suffix 
respectively,
+#' where strings are recycled to common lengths.
+#'
+#' @rdname endsWith
+#' @name endsWith
+#' @family colum_func
+#'
+#' @param vector of character string whose “ends” are considered
--- End diff --

ditto here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13476: [SPARK-15684][SparkR]Not mask startsWith and ends...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/13476#discussion_r65656447
  
--- Diff: R/pkg/R/column.R ---
@@ -151,6 +151,40 @@ setMethod("substr", signature(x = "Column"),
 column(jc)
   })
 
+#' startsWith
+#'
+#' Determines if entries of x start with string (entries of) prefix 
respectively,
+#' where strings are recycled to common lengths.
+#'
+#' @rdname startsWith
+#' @name startsWith
+#' @family colum_func
+#'
+#' @param vector of character string whose “starts” are considered
--- End diff --

`@param` should be once per parameter. I think we should leave out `@param` 
on the next line since it is describing the same parameter.
Also we should name the parameter first at the beginning of the text right 
after thr `@param` tag, see L146 for an example.
Finally, add `@param` and description for `x`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13415: [SPARK-15676] [SQL] Disallow Column Names as Partition C...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13415
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59912/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13483: [SPARK-15688][SQL] RelationalGroupedDataset.toDF should ...

[dilipbiswal]

Github user dilipbiswal commented on the issue:

https://github.com/apache/spark/pull/13483
  
@gatorsmile added comments and also updated the PR description.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13415: [SPARK-15676] [SQL] Disallow Column Names as Partition C...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13415
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13415: [SPARK-15676] [SQL] Disallow Column Names as Partition C...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13415
  
**[Test build #59912 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59912/consoleFull)**
 for PR 13415 at commit 
[`f4207e3`](https://github.com/apache/spark/commit/f4207e3c185a13a7f2866b0f12fcde5b28b8d948).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13483: [SPARK-15688][SQL] RelationalGroupedDataset.toDF should ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13483
  
**[Test build #59917 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59917/consoleFull)**
 for PR 13483 at commit 
[`6e59f9d`](https://github.com/apache/spark/commit/6e59f9d77d47b0994e104b9a378376e8d6d17924).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13176: [SPARK-15100][DOC] Modified user guide and examples for ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13176
  
**[Test build #59915 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59915/consoleFull)**
 for PR 13176 at commit 
[`5a53051`](https://github.com/apache/spark/commit/5a530519d77f036a90b71a654d377e333e99a7f9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13070: [SPARK-15286] [SQL] Make the output readable for EXPLAIN...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13070
  
**[Test build #59916 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59916/consoleFull)**
 for PR 13070 at commit 
[`73c0da2`](https://github.com/apache/spark/commit/73c0da2f1c087f97ff5b71a62f4c2cdc5894b78e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13481: [SPARK-15738][PYSPARK][ML] Adding Pyspark ml RFormula __...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13481
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59911/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13481: [SPARK-15738][PYSPARK][ML] Adding Pyspark ml RFormula __...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13481
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13481: [SPARK-15738][PYSPARK][ML] Adding Pyspark ml RFormula __...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13481
  
**[Test build #59911 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59911/consoleFull)**
 for PR 13481 at commit 
[`043e9bf`](https://github.com/apache/spark/commit/043e9bf5e29558334ea539f84ec84098a0f02786).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13070: [SPARK-15286] [SQL] Make the output readable for ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/13070#discussion_r65655636
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
 ---
@@ -140,6 +170,32 @@ case class CatalogTable(
   locationUri, inputFormat, outputFormat, serde, compressed, 
serdeProperties))
   }
 
+  override def toString: String = {
+val tableProperties = properties.map(p => p._1 + "=" + 
p._2).mkString("[", ", ", "]")
+val partitionColumns = partitionColumnNames.map("`" + _ + 
"`").mkString("[", ", ", "]")
+val sortColumns = sortColumnNames.map("`" + _ + "`").mkString("[", ", 
", "]")
+val bucketColumns = bucketColumnNames.map("`" + _ + "`").mkString("[", 
", ", "]")
+
+val output =
+  Seq(s"Table:${identifier.quotedString}",
+if (owner.nonEmpty) s"Owner:$owner" else "",
+s"Created:${new Date(createTime).toString}",
+s"Last Access:${new Date(lastAccessTime).toString}",
+s"Type:${tableType.name}",
+if (schema.nonEmpty) s"Schema:${schema.mkString("[", ", ", "]")}" 
else "",
+if (partitionColumnNames.nonEmpty) s"Partition 
Columns:$partitionColumns" else "",
+if (numBuckets != -1) s"Num Buckets:$numBuckets" else "",
+if (bucketColumnNames.nonEmpty) s"Bucket Columns:$bucketColumns" 
else "",
+if (sortColumnNames.nonEmpty) s"Sort Columns:$sortColumns" else "",
+viewOriginalText.map("Original View:" + _).getOrElse(""),
+viewText.map("View:" + _).getOrElse(""),
+comment.map("Comment:" + _).getOrElse(""),
+if (properties.nonEmpty) s"Properties:$tableProperties" else "",
+s"$storage")
+
+output.filter(_.nonEmpty).mkString("CatalogTable(", ", ", ")")
--- End diff --

Since we are using `, ` as a separator, the output of EXPLAIN is like:
```
CreateTableCommand CatalogTable(Table:`extTable_with_partitions`, 
Created:Thu Jun 02 21:25:26 PDT 2016, Last Access:Wed Dec 31 15:59:59 PST 1969, 
Type:EXTERNAL, Schema:[`key` int, `value` string, `ds` string, `hr` string], 
Partition Columns:[`ds`, `hr`], 
Storage(Location:/private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-472564ea-a321-447b-8224-7790edf454ab,
 InputFormat:org.apache.hadoop.mapred.TextInputFormat, 
OutputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat)), false
```

If we change it to `\n`, it will be like
```
CreateTableCommand CatalogTable(Table:`extTable_with_partitions`
Created:Thu Jun 02 21:28:32 PDT 2016
Last Access:Wed Dec 31 15:59:59 PST 1969
Type:EXTERNAL
Schema:[`key` int, `value` string, `ds` string, `hr` string]
Partition Columns:[`ds`, `hr`]

Storage(Location:/private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-a8e78990-3d9c-4f83-9ec4-6121473c1d09,
 InputFormat:org.apache.hadoop.mapred.TextInputFormat, 
OutputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat)), false
```

Another option is to use `\n\t` in `mkString` (i.e., 
`mkString("CatalogTable(\n\t", "\n\t", ")")`). Then, it will become 
```
CreateTableCommand CatalogTable(
Table:`extTable_with_partitions`
Created:Thu Jun 02 21:30:54 PDT 2016
Last Access:Wed Dec 31 15:59:59 PST 1969
Type:EXTERNAL
Schema:[`key` int, `value` string, `ds` string, `hr` string]
Partition Columns:[`ds`, `hr`]

Storage(Location:/private/var/folders/4b/sgmfldk15js406vk7lw5llzwgn/T/spark-a06083b8-8e88-4d07-9ff0-d6bd8d943ad3,
 InputFormat:org.apache.hadoop.mapred.TextInputFormat, 
OutputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat)), false
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13070: [SPARK-15286] [SQL] Make the output readable for ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/13070#discussion_r65654836
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
 ---
@@ -48,7 +49,26 @@ case class CatalogStorageFormat(
 outputFormat: Option[String],
 serde: Option[String],
 compressed: Boolean,
-serdeProperties: Map[String, String])
+serdeProperties: Map[String, String]) {
+
+  override def toString: String = {
+val serdePropsToString =
+  if (serdeProperties.nonEmpty) {
+s"Properties:" + serdeProperties.map(p => p._1 + "=" + 
p._2).mkString("[", ", ", "]")
+  } else {
+""
+  }
+val output =
+  Seq(locationUri.map("Location:" + _).getOrElse(""),
+inputFormat.map("InputFormat:" + _).getOrElse(""),
+outputFormat.map("OutputFormat:" + _).getOrElse(""),
+if (compressed) "Compressed:Yes" else "Compressed:No",
--- End diff --

Yeah, will do it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13070: [SPARK-15286] [SQL] Make the output readable for ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/13070#discussion_r65654827
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
 ---
@@ -65,8 +85,18 @@ case class CatalogColumn(
 // as a string due to issues in converting Hive varchars to and from 
SparkSQL strings.
 @Nullable dataType: String,
 nullable: Boolean = true,
-comment: Option[String] = None)
+comment: Option[String] = None) {
+
+  override def toString: String = {
+val output =
+  Seq(s"`$name`",
+dataType,
+if (nullable) "NULL" else "NOT NULL",
--- End diff --

Do not show anything if it is NULLABLE. If it is not nullable, we can show 
`NOT NULL`. This is the tradition way when we create a table. How about this? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13484: [SPARK-15742][SQL] Reduce temp collections allocations i...

[JoshRosen]

Github user JoshRosen commented on the issue:

https://github.com/apache/spark/pull/13484
  
Want to do more benchmarking to investigate something so don't merge yet.
On Thu, Jun 2, 2016 at 7:27 PM UCB AMPLab  wrote:

> Test PASSed.
> Refer to this link for build results (access rights to CI server needed):
> https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59901/
> Test PASSed.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13061: [SPARK-14279] [Build] Pick the spark version from pom

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13061
  
**[Test build #59914 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59914/consoleFull)**
 for PR 13061 at commit 
[`031e5b0`](https://github.com/apache/spark/commit/031e5b030f24e4e0ef93ce22d3c2fdb8deefb53a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13490: [SPARK-15722][SQL] Disallow specifying schema in CTAS st...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13490
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13490: [SPARK-15722][SQL] Disallow specifying schema in CTAS st...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13490
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59910/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13490: [SPARK-15722][SQL] Disallow specifying schema in CTAS st...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13490
  
**[Test build #59910 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59910/consoleFull)**
 for PR 13490 at commit 
[`fc372f5`](https://github.com/apache/spark/commit/fc372f5d487109bf5cf7cc59f41f38004f2bc108).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13061: [SPARK-14279] [Build] Pick the spark version from pom

[vanzin]

Github user vanzin commented on the issue:

https://github.com/apache/spark/pull/13061
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13415: [SPARK-15676] [SQL] Disallow Column Names as Partition C...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/13415
  
@cloud-fan @andrewor14 In this scenario, we do not have the case 
sensitivity issues. All the catalog columns are converted to lower case by 
https://github.com/apache/spark/blob/d109a1beeef5bca1e683247e0a5db4ec841bf3ba/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala#L1230

I remember we gave up the case sensitivity support in this release. 

Let me know if you have any question regarding the current implementation. 
Thanks! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13176: [SPARK-15100][DOC] Modified user guide and examples for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13176
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59913/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13176: [SPARK-15100][DOC] Modified user guide and examples for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13176
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13176: [SPARK-15100][DOC] Modified user guide and examples for ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13176
  
**[Test build #59913 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59913/consoleFull)**
 for PR 13176 at commit 
[`5df77c3`](https://github.com/apache/spark/commit/5df77c39ec98a14530d27366cb32ff236a29917c).
 * This patch **fails Python style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13415: [SPARK-15676] [SQL] Disallow Column Names as Partition C...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13415
  
**[Test build #59912 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59912/consoleFull)**
 for PR 13415 at commit 
[`f4207e3`](https://github.com/apache/spark/commit/f4207e3c185a13a7f2866b0f12fcde5b28b8d948).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13176: [SPARK-15100][DOC] Modified user guide and examples for ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13176
  
**[Test build #59913 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59913/consoleFull)**
 for PR 13176 at commit 
[`5df77c3`](https://github.com/apache/spark/commit/5df77c39ec98a14530d27366cb32ff236a29917c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13483: [SPARK-15688][SQL] RelationalGroupedDataset.toDF ...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/13483#discussion_r65650867
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala ---
@@ -46,7 +46,18 @@ class RelationalGroupedDataset protected[sql](
 
   private[this] def toDF(aggExprs: Seq[Expression]): DataFrame = {
 val aggregates = if 
(df.sparkSession.sessionState.conf.dataFrameRetainGroupColumns) {
-  groupingExprs ++ aggExprs
+  val agg = Aggregate(groupingExprs, aggExprs.map(alias), 
df.logicalPlan)
+  val plan = df.sparkSession.sessionState.executePlan(agg).analyzed
+  val aggAnalyzed = 
plan.find(_.isInstanceOf[Aggregate]).getOrElse(agg).asInstanceOf[Aggregate]
--- End diff --

Ok.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13483: [SPARK-15688][SQL] RelationalGroupedDataset.toDF ...

[dilipbiswal]

Github user dilipbiswal commented on a diff in the pull request:

https://github.com/apache/spark/pull/13483#discussion_r65650856
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala ---
@@ -224,6 +224,21 @@ class DatasetAggregatorSuite extends QueryTest with 
SharedSQLContext {
 checkAnswer(df.groupBy($"j").agg(RowAgg.toColumn), Row("a", 1) :: 
Row("b", 5) :: Nil)
   }
 
+  test("SPARK-15688: Remove redundant expressions from aggregate") {
+val df = Seq(1 -> "a", 2 -> "b", 3 -> "b").toDF("col1", "col2")
+val df1 = df.groupBy("col1").agg($"col1", count("*"))
+assert(df1.schema.map(_.name) === Seq("col1", "count(1)"))
+val df2 = df.groupBy("col1", "col2").agg($"col1", count("*"))
+assert(df2.schema.map(_.name) === Seq("col2", "col1", "count(1)"))
+val df3 = df.groupBy(expr("col1 + 2")).agg(expr("col1 + 2"), 
count("*"))
+assert(df3.schema.map(_.name) === Seq("(col1 + 2)", "count(1)"))
+val df4 = df.groupBy("col1").agg(min("col1"), max("col1"))
--- End diff --

Sure. Will do.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13061: [SPARK-14279] [Build] Pick the spark version from pom

[dhruve]

Github user dhruve commented on the issue:

https://github.com/apache/spark/pull/13061
  
Cool. Thanks for reviewing the changes. Once this gets merged, I will file 
a JIRA for the refactoring. 

This one's been open for quite some time now. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13269: [SPARK-15494][SQL] encoder code cleanup

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13269
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13269: [SPARK-15494][SQL] encoder code cleanup

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13269
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59906/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13269: [SPARK-15494][SQL] encoder code cleanup

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13269
  
**[Test build #59906 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59906/consoleFull)**
 for PR 13269 at commit 
[`6c793a8`](https://github.com/apache/spark/commit/6c793a8ebd5c00680699ecd5b2f08dce2d421b27).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13456: [SPARK-15724] Add benchmarks for performance over...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13456


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13487: [SPARK-15744][SQL] Rename two TungstenAggregation*Suites...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13487
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13487: [SPARK-15744][SQL] Rename two TungstenAggregation*Suites...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13487
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59905/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13481: [SPARK-15738][PYSPARK][ML] Adding Pyspark ml RFormula __...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/13481
  
Thanks @MLnick , I made those changes.  The string representation for the 
RFormulaModel is really ugly - it has lots of `WrappedArray` types.  I took all 
that out of the doc test, so let me know if what is there is ok.  I'm not 
really sure how it is even useful, maybe for debugging I guess?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13487: [SPARK-15744][SQL] Rename two TungstenAggregation*Suites...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13487
  
**[Test build #59905 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59905/consoleFull)**
 for PR 13487 at commit 
[`345b191`](https://github.com/apache/spark/commit/345b1916d8a6dcfc05c2b4958aec71e21138e3e5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13481: [SPARK-15738][PYSPARK][ML] Adding Pyspark ml RFormula __...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13481
  
**[Test build #59911 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59911/consoleFull)**
 for PR 13481 at commit 
[`043e9bf`](https://github.com/apache/spark/commit/043e9bf5e29558334ea539f84ec84098a0f02786).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13456: [SPARK-15724] Add benchmarks for performance over wide s...

[JoshRosen]

Github user JoshRosen commented on the issue:

https://github.com/apache/spark/pull/13456
  
LGTM. I'm going to merge this in for now so that we can divide-and-conquer 
on perf. improvement work. Let's deal with any additional feedback in followups.

I'm also going to merge this into `branch-2.0` so that PRs which update the 
benchmark results don't merge conflict as easily when cherry-picking them for 
2.0.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13485: [SPARK-15732][SQL] better error message when use java re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13485
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59903/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13473: [SPARK-15736][CORE] Gracefully handle loss of Dis...

[tedyu]

Github user tedyu commented on a diff in the pull request:

https://github.com/apache/spark/pull/13473#discussion_r65649617
  
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala 
---
@@ -403,6 +403,17 @@ private[spark] class BlockManager(
   }
 
   /**
+   * Cleanup code run in response to a failed local read.
+   * Must be called while holding a read lock on the block.
+   */
+  private def handleLocalReadFailure(blockId: BlockId): Nothing = {
+releaseLock(blockId)
+// Remove the missing block so that its unavailability is reported to 
the driver
+removeBlock(blockId)
--- End diff --

Looking at BlockInfoManager#lockForWriting(), I think you're right.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13485: [SPARK-15732][SQL] better error message when use java re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13485
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13485: [SPARK-15732][SQL] better error message when use java re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13485
  
**[Test build #59903 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59903/consoleFull)**
 for PR 13485 at commit 
[`51d403d`](https://github.com/apache/spark/commit/51d403dd632e81d9ebf0f943c2fb6be45edf50f3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13484: [SPARK-15742][SQL] Reduce temp collections allocations i...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13484
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13484: [SPARK-15742][SQL] Reduce temp collections allocations i...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13484
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59901/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13484: [SPARK-15742][SQL] Reduce temp collections allocations i...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13484
  
**[Test build #59901 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59901/consoleFull)**
 for PR 13484 at commit 
[`da79ec6`](https://github.com/apache/spark/commit/da79ec6207d4b78d0ebc461cacd37e14d6e4d2e3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12105: [SPARK-14321][SQL] Reduce date format cost and string-to...

[zzcclp]

Github user zzcclp commented on the issue:

https://github.com/apache/spark/pull/12105
  
@rajeshbalamohan , Is there any progress yet? We meet the same performance 
issues when formatting date or operating string-to-date .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13486: [SPARK-15743][SQL] Prevent saving with all-column partit...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13486
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59902/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13486: [SPARK-15743][SQL] Prevent saving with all-column partit...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13486
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13486: [SPARK-15743][SQL] Prevent saving with all-column partit...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13486
  
**[Test build #59902 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59902/consoleFull)**
 for PR 13486 at commit 
[`bb97467`](https://github.com/apache/spark/commit/bb97467dba96604d26d45763f4115152640ff189).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13489: [SPARK-15745][SQL] Use classloader's getResource() for r...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13489
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59908/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13258: [SPARK-15472][SQL][Streaming] Add partitioned `csv`, `js...

[lw-lin]

Github user lw-lin commented on the issue:

https://github.com/apache/spark/pull/13258
  
@zsxwing , yeah let me update this in the next one day or so - I was 
waiting for https://github.com/apache/spark/pull/13431. Thanks for the reminder!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13473: [SPARK-15736][CORE] Gracefully handle loss of Dis...

[JoshRosen]

Github user JoshRosen commented on a diff in the pull request:

https://github.com/apache/spark/pull/13473#discussion_r65648664
  
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala 
---
@@ -403,6 +403,17 @@ private[spark] class BlockManager(
   }
 
   /**
+   * Cleanup code run in response to a failed local read.
+   * Must be called while holding a read lock on the block.
+   */
+  private def handleLocalReadFailure(blockId: BlockId): Nothing = {
+releaseLock(blockId)
+// Remove the missing block so that its unavailability is reported to 
the driver
+removeBlock(blockId)
--- End diff --

No, I don't think so: internally, `removeBlock` acquires a write lock on 
the block, so if we called it before the `releaseLock` call then we'd be 
calling it while holding a read lock which would cause us to deadlock.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13489: [SPARK-15745][SQL] Use classloader's getResource() for r...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13489
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13430: [HOT-FIX] Fix the test case failures in DDLSuite

[JackyPJB]

Github user JackyPJB commented on the issue:

https://github.com/apache/spark/pull/13430
  
您好，感谢您的岗位推荐。我对这个职位不是很有å…
´è¶£ï¼Œå¦‚果有互联网企业的大数据岗位，请再推荐给我。



发件人: Xiao Li 
发送时间: 2016年6月1日 2:28
收件人: apache/spark
主题: [apache/spark] [HOT-FIX] Fix the test case failures in DDLSuite 
(#13430)

What changes were proposed in this pull request?

Test cases failure in DDLSuite introduced in 
#13427

How was this patch tested?

Local testing.


You can view, comment on, or merge this pull request online at:

  https://github.com/apache/spark/pull/13430

Commit Summary

  *   hot fix.

File Changes

  *   M 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
 (6)
  *   M 
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
 (8)

Patch Links:

  *   https://github.com/apache/spark/pull/13430.patch
  *   https://github.com/apache/spark/pull/13430.diff

―
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on 
GitHub, or mute the 
thread.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13489: [SPARK-15745][SQL] Use classloader's getResource() for r...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13489
  
**[Test build #59908 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59908/consoleFull)**
 for PR 13489 at commit 
[`65fa0f6`](https://github.com/apache/spark/commit/65fa0f6b39a9558c290d3ab9e6a223e65515ea8a).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13485: [SPARK-15732][SQL] better error message when use java re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13485
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13485: [SPARK-15732][SQL] better error message when use java re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13485
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59900/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13485: [SPARK-15732][SQL] better error message when use java re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13485
  
**[Test build #59900 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59900/consoleFull)**
 for PR 13485 at commit 
[`4ad5b7f`](https://github.com/apache/spark/commit/4ad5b7f40f1393c9894c3d1dcf4ef1e0e918d4eb).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13473: [SPARK-15736][CORE] Gracefully handle loss of Dis...

[tedyu]

Github user tedyu commented on a diff in the pull request:

https://github.com/apache/spark/pull/13473#discussion_r65648116
  
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala 
---
@@ -403,6 +403,17 @@ private[spark] class BlockManager(
   }
 
   /**
+   * Cleanup code run in response to a failed local read.
+   * Must be called while holding a read lock on the block.
+   */
+  private def handleLocalReadFailure(blockId: BlockId): Nothing = {
+releaseLock(blockId)
+// Remove the missing block so that its unavailability is reported to 
the driver
+removeBlock(blockId)
--- End diff --

Should this be called before the releaseLock() call ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13473: [SPARK-15736][CORE] Gracefully handle loss of Dis...

[tedyu]

Github user tedyu commented on a diff in the pull request:

https://github.com/apache/spark/pull/13473#discussion_r65648040
  
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala 
---
@@ -403,6 +403,17 @@ private[spark] class BlockManager(
   }
 
   /**
+   * Cleanup code run in response to a failed local read.
+   * Must be called while holding a read lock on the block.
+   */
+  private def handleLocalReadFailure(blockId: BlockId): Nothing = {
+releaseLock(blockId)
+// Remove the missing block so that its unavailability is reported to 
the driver
+removeBlock(blockId)
--- End diff --

Should this be called before the releaseLock() call ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13472: [SPARK-15735] Allow specifying min time to run in microb...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13472
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13472: [SPARK-15735] Allow specifying min time to run in microb...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13472
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59898/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13472: [SPARK-15735] Allow specifying min time to run in microb...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13472
  
**[Test build #59898 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59898/consoleFull)**
 for PR 13472 at commit 
[`239ee6e`](https://github.com/apache/spark/commit/239ee6ead2d162978b24b61f4eed8b5809be824c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13470: [SPARK-15733][SQL] Makes the explain output less verbose...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13470
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13470: [SPARK-15733][SQL] Makes the explain output less verbose...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13470
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59899/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13470: [SPARK-15733][SQL] Makes the explain output less verbose...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/13470
  
**[Test build #59899 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59899/consoleFull)**
 for PR 13470 at commit 
[`c979b69`](https://github.com/apache/spark/commit/c979b6924b3c47f9dbbe8a5d3f5b41ee91820991).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13483: [SPARK-15688][SQL] RelationalGroupedDataset.toDF should ...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/13483
  
Please update the PR description and post the analyzed plan. Also added 
another example to show the root node is `Window`. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13483: [SPARK-15688][SQL] RelationalGroupedDataset.toDF ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/13483#discussion_r65647177
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala ---
@@ -46,7 +46,18 @@ class RelationalGroupedDataset protected[sql](
 
   private[this] def toDF(aggExprs: Seq[Expression]): DataFrame = {
 val aggregates = if 
(df.sparkSession.sessionState.conf.dataFrameRetainGroupColumns) {
-  groupingExprs ++ aggExprs
+  val agg = Aggregate(groupingExprs, aggExprs.map(alias), 
df.logicalPlan)
+  val plan = df.sparkSession.sessionState.executePlan(agg).analyzed
+  val aggAnalyzed = 
plan.find(_.isInstanceOf[Aggregate]).getOrElse(agg).asInstanceOf[Aggregate]
--- End diff --

The same here. Can you put a comment to explain the top level node might 
not be an `Aggregate`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13483: [SPARK-15688][SQL] RelationalGroupedDataset.toDF ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/13483#discussion_r65647111
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/DatasetAggregatorSuite.scala ---
@@ -224,6 +224,21 @@ class DatasetAggregatorSuite extends QueryTest with 
SharedSQLContext {
 checkAnswer(df.groupBy($"j").agg(RowAgg.toColumn), Row("a", 1) :: 
Row("b", 5) :: Nil)
   }
 
+  test("SPARK-15688: Remove redundant expressions from aggregate") {
+val df = Seq(1 -> "a", 2 -> "b", 3 -> "b").toDF("col1", "col2")
+val df1 = df.groupBy("col1").agg($"col1", count("*"))
+assert(df1.schema.map(_.name) === Seq("col1", "count(1)"))
+val df2 = df.groupBy("col1", "col2").agg($"col1", count("*"))
+assert(df2.schema.map(_.name) === Seq("col2", "col1", "count(1)"))
+val df3 = df.groupBy(expr("col1 + 2")).agg(expr("col1 + 2"), 
count("*"))
+assert(df3.schema.map(_.name) === Seq("(col1 + 2)", "count(1)"))
+val df4 = df.groupBy("col1").agg(min("col1"), max("col1"))
--- End diff --

Could you put some comments to show the goal of each test cases? The 
comments can help reviewers understand the coverage of test cases. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13394: [SPARK-15490][R][DOC] SparkR 2.0 QA: New R APIs and API ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13394
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59909/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13394: [SPARK-15490][R][DOC] SparkR 2.0 QA: New R APIs and API ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/13394
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org