[GitHub] spark pull request: [SPARK-15165][SQL] Codegen can break because t...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12939#issuecomment-218368915
  
**[Test build #58333 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58333/consoleFull)**
 for PR 12939 at commit 
[`7106f23`](https://github.com/apache/spark/commit/7106f234fd4f04ce8e922e643bb343ac635d09d2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15263][Core] Make shuffle service dir c...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13042#issuecomment-218368903
  
**[Test build #58332 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58332/consoleFull)**
 for PR 13042 at commit 
[`1877ad4`](https://github.com/apache/spark/commit/1877ad4ea0dc003c042728d36c65e1f74297b7a1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15037][SQL][MLLIB] Part2 Use SparkSessi...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13044#issuecomment-218368867
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15037][SQL][MLLIB] Part2 Use SparkSessi...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13044#issuecomment-218368852
  
**[Test build #58330 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58330/consoleFull)**
 for PR 13044 at commit 
[`d7361d8`](https://github.com/apache/spark/commit/d7361d88835baf9fda8df3e59d6848f9fb62a19f).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15037][SQL][MLLIB] Part2 Use SparkSessi...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13044#issuecomment-218368868
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58330/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15263][Core] Make shuffle service dir c...

[tejasapatil]

Github user tejasapatil commented on a diff in the pull request:

https://github.com/apache/spark/pull/13042#discussion_r62794873
  
--- Diff: 
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
 ---
@@ -79,14 +80,27 @@ public static String bytesToString(ByteBuffer b) {
 return Unpooled.wrappedBuffer(b).toString(StandardCharsets.UTF_8);
   }
 
-  /*
+  /**
* Delete a file or directory and its contents recursively.
* Don't follow directories if they are symlinks.
-   * Throws an exception if deletion is unsuccessful.
+   *
+   * @param file Input file / dir to be deleted
+   * @throws IOException if deletion is unsuccessful
*/
   public static void deleteRecursively(File file) throws IOException {
 if (file == null) { return; }
 
+// On Unix systems, use operating system command to run faster
+// If that does not work out, fallback to the Java native way
+if (SystemUtils.IS_OS_UNIX) {
+  try {
+deleteRecursivelyForUnix(file);
+return;
+  } catch (IOException e) {
+// ignore and fall back to the Java native way
+  }
+}
+
 if (file.isDirectory() && !isSymlink(file)) {
--- End diff --

done


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15263][Core] Make shuffle service dir c...

[tejasapatil]

Github user tejasapatil commented on a diff in the pull request:

https://github.com/apache/spark/pull/13042#discussion_r62794845
  
--- Diff: 
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
 ---
@@ -79,14 +80,27 @@ public static String bytesToString(ByteBuffer b) {
 return Unpooled.wrappedBuffer(b).toString(StandardCharsets.UTF_8);
   }
 
-  /*
+  /**
* Delete a file or directory and its contents recursively.
* Don't follow directories if they are symlinks.
-   * Throws an exception if deletion is unsuccessful.
+   *
+   * @param file Input file / dir to be deleted
+   * @throws IOException if deletion is unsuccessful
*/
   public static void deleteRecursively(File file) throws IOException {
 if (file == null) { return; }
 
+// On Unix systems, use operating system command to run faster
+// If that does not work out, fallback to the Java native way
+if (SystemUtils.IS_OS_UNIX) {
+  try {
+deleteRecursivelyForUnix(file);
+return;
+  } catch (IOException e) {
+// ignore and fall back to the Java native way
--- End diff --

Added logging


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12719#issuecomment-218368305
  
**[Test build #58331 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58331/consoleFull)**
 for PR 12719 at commit 
[`a6b1cbc`](https://github.com/apache/spark/commit/a6b1cbce72db9ab65e923aae8f108bd5566c1bac).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15085][Streaming][Kafka] Rename streami...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12946#issuecomment-218368063
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58323/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15085][Streaming][Kafka] Rename streami...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12946#issuecomment-218368062
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15085][Streaming][Kafka] Rename streami...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12946#issuecomment-218367940
  
**[Test build #58323 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58323/consoleFull)**
 for PR 12946 at commit 
[`ab2bd63`](https://github.com/apache/spark/commit/ab2bd63d02c4b00edd4d776b4d2ce780b443f1ae).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15037][SQL][MLLIB] Part2 Use SparkSessi...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13044#issuecomment-218367680
  
**[Test build #58330 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58330/consoleFull)**
 for PR 13044 at commit 
[`d7361d8`](https://github.com/apache/spark/commit/d7361d88835baf9fda8df3e59d6848f9fb62a19f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15037][SQL][MLLIB] Part2 Use SparkSessi...

[techaddict]

GitHub user techaddict opened a pull request:

https://github.com/apache/spark/pull/13044

[SPARK-15037][SQL][MLLIB] Part2 Use SparkSession instead of SQLContext in 
Python TestSuites

## What changes were proposed in this pull request?
Use SparkSession instead of SQLContext in Python TestSuites

## How was this patch tested?
Existing tests

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/techaddict/spark SPARK-15037-python

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/13044.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #13044


commit d7361d88835baf9fda8df3e59d6848f9fb62a19f
Author: Sandeep Singh 
Date:   2016-05-07T15:03:37Z

first pass




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15150][EXAMPLE][DOC] Update LDA example...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12927#issuecomment-218366438
  
**[Test build #58329 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58329/consoleFull)**
 for PR 12927 at commit 
[`eab9511`](https://github.com/apache/spark/commit/eab9511f04968bf766aaf06f2a0773e1a0ffdacf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12719#issuecomment-218366226
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58324/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12719#issuecomment-218366224
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12719#issuecomment-218366088
  
**[Test build #58324 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58324/consoleFull)**
 for PR 12719 at commit 
[`b8ddf0c`](https://github.com/apache/spark/commit/b8ddf0c382e7c2499974b8c5e6cde41d51806361).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15235][WEBUI] Corresponding row cannot ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13016


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15264][SQL] CSV Reader Error on Blank C...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/13041#discussion_r62793802
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/DefaultSource.scala
 ---
@@ -61,7 +61,9 @@ class DefaultSource extends FileFormat with 
DataSourceRegister {
 val firstRow = new LineCsvReader(csvOptions).parseLine(firstLine)
 
 val header = if (csvOptions.headerFlag) {
-  firstRow
+  firstRow.zipWithIndex.map { case (value, index) =>
+if (value == "" || value == null) s"C$index" else value
--- End diff --

I mean if the header is a empty string then, I think the field name should 
be a empty string since apparently it works with fields named empty strings. I 
tested this by manually giving a schema. If the header is used for schema, then 
I think the names should be as they are. We don't really change field names 
specified in ORC, Parquet or JSON.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15259] Sort time metric should not incl...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/13035#issuecomment-218365971
  
Why shouldn't we include the time it takes for record insertion?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15235][WEBUI] Corresponding row cannot ...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/13016#issuecomment-218365775
  
Thanks - merging in master/2.0.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15255] [SQL] limit the length of name f...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13033


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15246][Core] Fix code style and improve...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13020


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15246][Core] Fix code style and improve...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/13020#issuecomment-218365539
  
Merging in master/2.0. Thanks.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15265][SQL][MINOR] Fix Union query erro...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the pull request:

https://github.com/apache/spark/pull/13043#issuecomment-218365410
  
Thank you, @rxin .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15264][SQL] CSV Reader Error on Blank C...

[anabranch]

Github user anabranch commented on a diff in the pull request:

https://github.com/apache/spark/pull/13041#discussion_r62793571
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/DefaultSource.scala
 ---
@@ -61,7 +61,9 @@ class DefaultSource extends FileFormat with 
DataSourceRegister {
 val firstRow = new LineCsvReader(csvOptions).parseLine(firstLine)
 
 val header = if (csvOptions.headerFlag) {
-  firstRow
+  firstRow.zipWithIndex.map { case (value, index) =>
+if (value == "" || value == null) s"C$index" else value
--- End diff --

This code does rename it with the index and prefix, `C`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15265][SQL][MINOR] Fix Union query erro...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13043


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15255] [SQL] limit the length of name f...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/13033#issuecomment-218365388
  
Thanks - merging in master/2.0.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15265][SQL][MINOR] Fix Union query erro...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/13043#issuecomment-218365133
  
LGTM - merging in master/2.0.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the pull request:

https://github.com/apache/spark/pull/12719#issuecomment-218364963
  
Anyway, I made a long detour for this PR. Sorry for that, @cloud-fan , and 
thank you again. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15263][Core] Make shuffle service dir c...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/13042#discussion_r62793386
  
--- Diff: 
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
 ---
@@ -79,14 +80,27 @@ public static String bytesToString(ByteBuffer b) {
 return Unpooled.wrappedBuffer(b).toString(StandardCharsets.UTF_8);
   }
 
-  /*
+  /**
* Delete a file or directory and its contents recursively.
* Don't follow directories if they are symlinks.
-   * Throws an exception if deletion is unsuccessful.
+   *
+   * @param file Input file / dir to be deleted
+   * @throws IOException if deletion is unsuccessful
*/
   public static void deleteRecursively(File file) throws IOException {
 if (file == null) { return; }
 
+// On Unix systems, use operating system command to run faster
+// If that does not work out, fallback to the Java native way
+if (SystemUtils.IS_OS_UNIX) {
+  try {
+deleteRecursivelyForUnix(file);
+return;
+  } catch (IOException e) {
+// ignore and fall back to the Java native way
+  }
+}
+
 if (file.isDirectory() && !isSymlink(file)) {
--- End diff --

can we move this also into a private function named 
deleteRecursivelyUsingJavaIO


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15263][Core] Make shuffle service dir c...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/13042#discussion_r62793337
  
--- Diff: 
common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java
 ---
@@ -79,14 +80,27 @@ public static String bytesToString(ByteBuffer b) {
 return Unpooled.wrappedBuffer(b).toString(StandardCharsets.UTF_8);
   }
 
-  /*
+  /**
* Delete a file or directory and its contents recursively.
* Don't follow directories if they are symlinks.
-   * Throws an exception if deletion is unsuccessful.
+   *
+   * @param file Input file / dir to be deleted
+   * @throws IOException if deletion is unsuccessful
*/
   public static void deleteRecursively(File file) throws IOException {
 if (file == null) { return; }
 
+// On Unix systems, use operating system command to run faster
+// If that does not work out, fallback to the Java native way
+if (SystemUtils.IS_OS_UNIX) {
+  try {
+deleteRecursivelyForUnix(file);
+return;
+  } catch (IOException e) {
+// ignore and fall back to the Java native way
--- End diff --

hm silent fallback isn't always the best idea. we should maybe at least log 
something.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14495][SQL][1.6] fix resolution failure...

[xwu0226]

Github user xwu0226 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12974#discussion_r62793285
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala ---
@@ -44,10 +44,12 @@ class PlannerSuite extends SharedSQLContext {
 fail(s"Could query play aggregation query $query. Is it an 
aggregation query?"))
 val aggregations = planned.collect { case n if n.nodeName contains 
"Aggregate" => n }
 
-// For the new aggregation code path, there will be four aggregate 
operator for
-// distinct aggregations.
+// For the new aggregation code path, there will be three aggregate 
operator for
+// distinct aggregations. There used to be four aggregate operators 
because single
+// distinct aggregate used to trigger DistinctAggregationRewriter 
rewrite. Now the
+// the rewrite only happens when there are multiple distinct 
aggregations.
 assert(
-  aggregations.size == 2 || aggregations.size == 4,
+  aggregations.size == 2 || aggregations.size == 3,
--- End diff --

The rewrite I mentioned was by Analyzer. In this testcase, the analyzed 
logical plan, instead of the optimized one, is passed to generate physical 
plan.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15250][SQL] Remove deprecated json API ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13040


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14495][SQL][1.6] fix resolution failure...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12974#issuecomment-218364589
  
**[Test build #58327 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58327/consoleFull)**
 for PR 12974 at commit 
[`e0eeb7d`](https://github.com/apache/spark/commit/e0eeb7d92f397f94698208afa4081a083e835074).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12719#issuecomment-218364599
  
**[Test build #58328 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58328/consoleFull)**
 for PR 12719 at commit 
[`a73deaf`](https://github.com/apache/spark/commit/a73deafe591db0af6c8567da8439840ec3588ec2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15080][CORE] Break copyAndReset into co...

[techaddict]

Github user techaddict commented on a diff in the pull request:

https://github.com/apache/spark/pull/12936#discussion_r62793263
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala 
---
@@ -29,9 +29,18 @@ class SQLMetric(val metricType: String, initValue: Long 
= 0L) extends Accumulato
   // We may use -1 as initial value of the accumulator, if the accumulator 
is valid, we will
   // update it at the end of task and the value will be at least 0. Then 
we can filter out the -1
   // values before calculate max, min, etc.
-  private[this] var _value = initValue
+  private var _value = initValue
--- End diff --

Something like this ?
```
  private[this] var _value = initValue
  private var _zeroValue = initValue

  override def copy(): SQLMetric = {
val newAcc = new SQLMetric(metricType, _value)
newAcc._zeroValue = initValue
newAcc
  }

  override def reset(): Unit = _value = _zeroValue
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15250][SQL] Remove deprecated json API ...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/13040#issuecomment-218364416
  
LGTM - merging in master/2.0.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15250][SQL] Remove deprecated json API ...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/13040#issuecomment-218364439
  
Yes please submit another pr for those.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/12719#discussion_r62793156
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -650,15 +646,15 @@ object FoldablePropagation extends Rule[LogicalPlan] {
 
 if (foldableExprSet.isEmpty) {
   plan
-} else if (hasCommandOrUnion(plan)) {
-  replaceInSubplan(plan)
 } else {
   val foldableMap = 
AttributeMap(foldableExprSet.toSeq.map(_.asInstanceOf[Alias])
 .map(a => (a.toAttribute, a.child)))
 
-  CleanupAliases(plan.transformAllExpressions {
-case a : AttributeReference if foldableMap.contains(a) =>
-  Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()
+  CleanupAliases(plan.transformUp {
+case p: LogicalPlan if !hasCommandOrUnion(p) => 
p.transformAllExpressions {
+  case a: AttributeReference if foldableMap.contains(a) =>
+Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()
+}
--- End diff --

Right! That sounds great.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15094][SPARK-14803][SQL] Add ObjectProj...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12926#issuecomment-218364032
  
**[Test build #58326 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58326/consoleFull)**
 for PR 12926 at commit 
[`29a0c70`](https://github.com/apache/spark/commit/29a0c70488fc8d3f7157679d8f41f7ceb5af9bc4).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15080][CORE] Break copyAndReset into co...

[techaddict]

Github user techaddict commented on a diff in the pull request:

https://github.com/apache/spark/pull/12936#discussion_r62792987
  
--- Diff: core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala 
---
@@ -291,12 +291,23 @@ private[spark] object TaskMetrics extends Logging {
 
 private[spark] class BlockStatusesAccumulator
   extends AccumulatorV2[(BlockId, BlockStatus), Seq[(BlockId, 
BlockStatus)]] {
-  private[this] var _seq = ArrayBuffer.empty[(BlockId, BlockStatus)]
+  private var _seq = ArrayBuffer.empty[(BlockId, BlockStatus)]
 
   override def isZero(): Boolean = _seq.isEmpty
 
   override def copyAndReset(): BlockStatusesAccumulator = new 
BlockStatusesAccumulator
 
+  override def copy(): BlockStatusesAccumulator = {
+val newAcc = new BlockStatusesAccumulator
+newAcc._seq = _seq.clone()
--- End diff --

Not sure, but In general `clone`'s are faster.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15250][SQL] Remove deprecated json API ...

[HyukjinKwon]

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/13040#issuecomment-218363946
  
cc @rxin 

(I see doc tests and test in Python is still using `SqlContext`. Do you 
mind if I correct this in another PR?)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15080][CORE] Break copyAndReset into co...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/12936#discussion_r62792794
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala 
---
@@ -29,9 +29,18 @@ class SQLMetric(val metricType: String, initValue: Long 
= 0L) extends Accumulato
   // We may use -1 as initial value of the accumulator, if the accumulator 
is valid, we will
   // update it at the end of task and the value will be at least 0. Then 
we can filter out the -1
   // values before calculate max, min, etc.
-  private[this] var _value = initValue
+  private var _value = initValue
--- End diff --

`initValue` will be assigned to `_value` when creating accumulator, and 
`zeroValue` will be assigned to `_value` when calling `reset`, how about it?

Most of the time `initValue` is equal to `zeroValue`,  it's only used in 
`copy`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15264][SQL] CSV Reader Error on Blank C...

[HyukjinKwon]

Github user HyukjinKwon commented on the pull request:

https://github.com/apache/spark/pull/13041#issuecomment-218363294
  
@anabranch First of all, I think currently (at least for me) it is really 
confusing to deal with `null`, `""` and empty string in CSV. I am trying to 
make this clear here and there (eg.https://github.com/apache/spark/pull/12921). 
So, I hope we can hold off this at least that PR is merged.

Secondly, JSON data source would not support empty strings as fields. I 
think we should clarify what we want for empty strings. For example, JSON data 
sources simply ignores when it meets empty strings as far as I know but CSV 
data sources currently throws NPE. Also, I think we need to decide if we are 
going to support fields named empty strings or not for data sources.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/12719#discussion_r62792569
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -650,15 +646,15 @@ object FoldablePropagation extends Rule[LogicalPlan] {
 
 if (foldableExprSet.isEmpty) {
   plan
-} else if (hasCommandOrUnion(plan)) {
-  replaceInSubplan(plan)
 } else {
   val foldableMap = 
AttributeMap(foldableExprSet.toSeq.map(_.asInstanceOf[Alias])
 .map(a => (a.toAttribute, a.child)))
 
-  CleanupAliases(plan.transformAllExpressions {
-case a : AttributeReference if foldableMap.contains(a) =>
-  Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()
+  CleanupAliases(plan.transformUp {
+case p: LogicalPlan if !hasCommandOrUnion(p) => 
p.transformAllExpressions {
+  case a: AttributeReference if foldableMap.contains(a) =>
+Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()
+}
--- End diff --

And we can improve this later, in another way: make `Union` and `Command`(I 
think it's only `InsertIntoHadoopFsRelation`) work with it, i.e. do not hold 
child's attributes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-1301] [Web UI] Added anchor links to Ac...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13037#issuecomment-218362808
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58320/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-1301] [Web UI] Added anchor links to Ac...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13037#issuecomment-218362807
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/12719#discussion_r62792327
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -650,15 +646,15 @@ object FoldablePropagation extends Rule[LogicalPlan] {
 
 if (foldableExprSet.isEmpty) {
   plan
-} else if (hasCommandOrUnion(plan)) {
-  replaceInSubplan(plan)
 } else {
   val foldableMap = 
AttributeMap(foldableExprSet.toSeq.map(_.asInstanceOf[Alias])
 .map(a => (a.toAttribute, a.child)))
 
-  CleanupAliases(plan.transformAllExpressions {
-case a : AttributeReference if foldableMap.contains(a) =>
-  Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()
+  CleanupAliases(plan.transformUp {
+case p: LogicalPlan if !hasCommandOrUnion(p) => 
p.transformAllExpressions {
+  case a: AttributeReference if foldableMap.contains(a) =>
+Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()
+}
--- End diff --

Ur, we should preserve the name of AttributeReference. So, `foldableMap` 
has the child, not Alias until now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15080][CORE] Break copyAndReset into co...

[techaddict]

Github user techaddict commented on a diff in the pull request:

https://github.com/apache/spark/pull/12936#discussion_r62792294
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala 
---
@@ -29,9 +29,18 @@ class SQLMetric(val metricType: String, initValue: Long 
= 0L) extends Accumulato
   // We may use -1 as initial value of the accumulator, if the accumulator 
is valid, we will
   // update it at the end of task and the value will be at least 0. Then 
we can filter out the -1
   // values before calculate max, min, etc.
-  private[this] var _value = initValue
+  private var _value = initValue
--- End diff --

That won't be correct. If we create a new accumulator with `initValue` as 
`_value` and do `reset` it will reset to `_value` instead of `initValue`. Can't 
think of a way around this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-1301] [Web UI] Added anchor links to Ac...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13037#issuecomment-218362701
  
**[Test build #58320 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58320/consoleFull)**
 for PR 13037 at commit 
[`2b39686`](https://github.com/apache/spark/commit/2b3968623b6fb994ba58e9740072a34cba755f33).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/12719#discussion_r62792130
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -650,15 +646,15 @@ object FoldablePropagation extends Rule[LogicalPlan] {
 
 if (foldableExprSet.isEmpty) {
   plan
-} else if (hasCommandOrUnion(plan)) {
-  replaceInSubplan(plan)
 } else {
   val foldableMap = 
AttributeMap(foldableExprSet.toSeq.map(_.asInstanceOf[Alias])
 .map(a => (a.toAttribute, a.child)))
 
-  CleanupAliases(plan.transformAllExpressions {
-case a : AttributeReference if foldableMap.contains(a) =>
-  Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()
+  CleanupAliases(plan.transformUp {
+case p: LogicalPlan if !hasCommandOrUnion(p) => 
p.transformAllExpressions {
+  case a: AttributeReference if foldableMap.contains(a) =>
+Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()
+}
--- End diff --

I see. If you think so, I will update again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14495][SQL][1.6] fix resolution failure...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/12974#discussion_r62792092
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala ---
@@ -44,10 +44,12 @@ class PlannerSuite extends SharedSQLContext {
 fail(s"Could query play aggregation query $query. Is it an 
aggregation query?"))
 val aggregations = planned.collect { case n if n.nodeName contains 
"Aggregate" => n }
 
-// For the new aggregation code path, there will be four aggregate 
operator for
-// distinct aggregations.
+// For the new aggregation code path, there will be three aggregate 
operator for
+// distinct aggregations. There used to be four aggregate operators 
because single
+// distinct aggregate used to trigger DistinctAggregationRewriter 
rewrite. Now the
+// the rewrite only happens when there are multiple distinct 
aggregations.
 assert(
-  aggregations.size == 2 || aggregations.size == 4,
+  aggregations.size == 2 || aggregations.size == 3,
--- End diff --

Why is there no rewrite after your change? You just moved the rule but 
didn't change it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15259] Sort time metric should not incl...

[davies]

Github user davies commented on a diff in the pull request:

https://github.com/apache/spark/pull/13035#discussion_r62792091
  
--- Diff: 
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
 ---
@@ -248,6 +249,17 @@ public long getPeakMemoryUsedBytes() {
   }
 
   /**
+   * @return the total amount of time spent sorting data (in-memory only).
+   */
+  public long getSortTimeNanos() {
+UnsafeInMemorySorter sorter = inMemSorter;
+if (sorter != null) {
+  return inMemSorter.getSortTimeNanos();
--- End diff --

sorter ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15264][SQL] CSV Reader Error on Blank C...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/13041#discussion_r62792044
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/DefaultSource.scala
 ---
@@ -61,7 +61,9 @@ class DefaultSource extends FileFormat with 
DataSourceRegister {
 val firstRow = new LineCsvReader(csvOptions).parseLine(firstLine)
 
 val header = if (csvOptions.headerFlag) {
-  firstRow
+  firstRow.zipWithIndex.map { case (value, index) =>
+if (value == "" || value == null) s"C$index" else value
--- End diff --

I see Spark allows a empty string as a field. So, I wonder if we should 
rename this with the index and prefix, `C`. Also, I think `""` will throw an 
NPE whereas empty string without quotes will produce a correct field because 
the default of `nullValue` is `""`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15265][SQL][MINOR] Fix Union query erro...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13043#issuecomment-218361757
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15265][SQL][MINOR] Fix Union query erro...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13043#issuecomment-218361760
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58322/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15261][SQL] Remove experimental tag fro...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13038


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15265][SQL][MINOR] Fix Union query erro...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13043#issuecomment-218361663
  
**[Test build #58322 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58322/consoleFull)**
 for PR 13043 at commit 
[`c102d4d`](https://github.com/apache/spark/commit/c102d4d393c5ea6921a674df6a3433ccff959395).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15261][SQL] Remove experimental tag fro...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/13038#issuecomment-218361546
  
Merging in master/2.0.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/12719#discussion_r62791648
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -650,15 +646,15 @@ object FoldablePropagation extends Rule[LogicalPlan] {
 
 if (foldableExprSet.isEmpty) {
   plan
-} else if (hasCommandOrUnion(plan)) {
-  replaceInSubplan(plan)
 } else {
   val foldableMap = 
AttributeMap(foldableExprSet.toSeq.map(_.asInstanceOf[Alias])
 .map(a => (a.toAttribute, a.child)))
 
-  CleanupAliases(plan.transformAllExpressions {
-case a : AttributeReference if foldableMap.contains(a) =>
-  Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()
+  CleanupAliases(plan.transformUp {
+case p: LogicalPlan if !hasCommandOrUnion(p) => 
p.transformAllExpressions {
+  case a: AttributeReference if foldableMap.contains(a) =>
+Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()
+}
--- End diff --

I'd like to write:
```
var stop = false
plan.transformUp {
  case u: Union =>
stop = true
u
  case c: Command =>
stop = true
c
  case p if !stop => p.transformExpressions {
case a: AttributeReference if foldableMap.contains(a) =>
  foldableMap(a) // foldableMap's value shoud be Alias, not its child
  }
}
```

if earlier sibling is `Union`, this is suboptimal, but still safe(it's ok 
to not apply this optimization for some corner cases)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15080][CORE] Break copyAndReset into co...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/12936#discussion_r62791621
  
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -112,7 +112,18 @@ abstract class AccumulatorV2[IN, OUT] extends 
Serializable {
* Creates a new copy of this accumulator, which is zero value. i.e. 
call `isZero` on the copy
* must return true.
*/
-  def copyAndReset(): AccumulatorV2[IN, OUT]
+  def copyAndReset(): AccumulatorV2[IN, OUT] = copy().reset()
--- End diff --

mainly here for now


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14476][SQL] Improve the physical plan v...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/12947


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14476][SQL] Improve the physical plan v...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/12947#issuecomment-218361168
  
Thanks - merging in master/2.0.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14857] [SQL] Table/Database Name Valida...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12618#issuecomment-218360331
  
**[Test build #58325 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58325/consoleFull)**
 for PR 12618 at commit 
[`3788e86`](https://github.com/apache/spark/commit/3788e86830c0608c064eb421f34d565d110649f1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14800][SQL] Dealing with null as a valu...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12629#issuecomment-218360303
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58321/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14800][SQL] Dealing with null as a valu...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12629#issuecomment-218360301
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14800][SQL] Dealing with null as a valu...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12629#issuecomment-218360205
  
**[Test build #58321 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58321/consoleFull)**
 for PR 12629 at commit 
[`ef4e6be`](https://github.com/apache/spark/commit/ef4e6be908a08840884e77f0ad0620f79026b4ad).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-218359655
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58319/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-218359654
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15143][SPARK-15144][SQL] Add CSV tests ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12921#issuecomment-218359513
  
**[Test build #58319 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58319/consoleFull)**
 for PR 12921 at commit 
[`75f1cb8`](https://github.com/apache/spark/commit/75f1cb8d7c7e89103b1ac0607512837693f7f699).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14476][SQL] Improve the physical plan v...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12947#issuecomment-218359085
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14476][SQL] Improve the physical plan v...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12947#issuecomment-218359086
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58318/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14476][SQL] Improve the physical plan v...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12947#issuecomment-218358970
  
**[Test build #58318 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58318/consoleFull)**
 for PR 12947 at commit 
[`59f816f`](https://github.com/apache/spark/commit/59f816f4cf91979282d3b9385d746099b040fbc1).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15195][PySpark][DOCS] Update ml.tuning ...

[MLnick]

Github user MLnick commented on a diff in the pull request:

https://github.com/apache/spark/pull/12967#discussion_r62790418
  
--- Diff: python/pyspark/ml/tuning.py ---
@@ -33,6 +33,8 @@
 
 class ParamGridBuilder(object):
 r"""
+.. note:: Experimental
--- End diff --

I don't think it's required there? Anyway I left it as is
On Tue, 10 May 2016 at 23:21, Holden Karau  wrote:

> In python/pyspark/ml/tuning.py
> :
>
> > @@ -33,6 +33,8 @@
> >
> >  class ParamGridBuilder(object):
> >  r"""
> > +.. note:: Experimental
>
> r is for raw string literal (often used to avoid needing certain things)
>
> —
> You are receiving this because you commented.
> Reply to this email directly or view it on GitHub
> 

>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/12719#discussion_r62790052
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -650,15 +646,15 @@ object FoldablePropagation extends Rule[LogicalPlan] {
 
 if (foldableExprSet.isEmpty) {
   plan
-} else if (hasCommandOrUnion(plan)) {
-  replaceInSubplan(plan)
 } else {
   val foldableMap = 
AttributeMap(foldableExprSet.toSeq.map(_.asInstanceOf[Alias])
 .map(a => (a.toAttribute, a.child)))
 
-  CleanupAliases(plan.transformAllExpressions {
-case a : AttributeReference if foldableMap.contains(a) =>
-  Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()
+  CleanupAliases(plan.transformUp {
+case p: LogicalPlan if !hasCommandOrUnion(p) => 
p.transformAllExpressions {
+  case a: AttributeReference if foldableMap.contains(a) =>
+Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()
+}
--- End diff --

By the way, in the above code using `stop` flag, if earlier sibling has 
`Union`, the later sibling is disabled. So, it's not safe. It's just an example.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/12719#discussion_r62789853
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -650,15 +646,15 @@ object FoldablePropagation extends Rule[LogicalPlan] {
 
 if (foldableExprSet.isEmpty) {
   plan
-} else if (hasCommandOrUnion(plan)) {
-  replaceInSubplan(plan)
 } else {
   val foldableMap = 
AttributeMap(foldableExprSet.toSeq.map(_.asInstanceOf[Alias])
 .map(a => (a.toAttribute, a.child)))
 
-  CleanupAliases(plan.transformAllExpressions {
-case a : AttributeReference if foldableMap.contains(a) =>
-  Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()
+  CleanupAliases(plan.transformUp {
+case p: LogicalPlan if !hasCommandOrUnion(p) => 
p.transformAllExpressions {
+  case a: AttributeReference if foldableMap.contains(a) =>
+Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()
+}
--- End diff --

As you told me, for the `stop` flag, I can change the above code more like 
the following. IMHO, the above code is simpler. The query optimization speed is 
more important? If then, I'll update that too.
```
var stop = false
CleanupAliases(plan.transformUp {
  case p: LogicalPlan if !stop && hasCommandOrUnion(p) =>

stop = true
p
  case p: LogicalPlan if !stop && !hasCommandOrUnion(p) => 
p.transformAllExpressions {
case a: AttributeReference if foldableMap.contains(a) =>

  Alias(foldableMap.getOrElse(a.toAttribute, a), a.name)()

  }
})
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15250][SQL] Remove deprecated json API ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13040#issuecomment-218357373
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15250][SQL] Remove deprecated json API ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13040#issuecomment-218357375
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58313/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15264][SQL] CSV Reader Error on Blank C...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13041#issuecomment-218357313
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14495][SQL][1.6] fix resolution failure...

[xwu0226]

Github user xwu0226 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12974#discussion_r62789705
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala ---
@@ -44,10 +44,12 @@ class PlannerSuite extends SharedSQLContext {
 fail(s"Could query play aggregation query $query. Is it an 
aggregation query?"))
 val aggregations = planned.collect { case n if n.nodeName contains 
"Aggregate" => n }
 
-// For the new aggregation code path, there will be four aggregate 
operator for
-// distinct aggregations.
+// For the new aggregation code path, there will be three aggregate 
operator for
+// distinct aggregations. There used to be four aggregate operators 
because single
+// distinct aggregate used to trigger DistinctAggregationRewriter 
rewrite. Now the
+// the rewrite only happens when there are multiple distinct 
aggregations.
 assert(
-  aggregations.size == 2 || aggregations.size == 4,
+  aggregations.size == 2 || aggregations.size == 3,
--- End diff --

Before my change of moving the `DistinctAggregationRewriter` rule to 
Optimizer, Analyzer resolved the single distinct case into 2-layered 
non-distinct Aggregates and an EXPAND as following:
```
Aggregate [value#3], [value#3,(count(if ((gid#48 = 1)) key#49 else 
null),mode=Complete,isDistinct=false) AS count(key)#47L]
+- Aggregate [value#3,key#49,gid#48], [value#3,key#49,gid#48]
   +- Expand [ArrayBuffer(value#3, key#2, 1)], [value#3,key#49,gid#48]
  +- LogicalRDD [key#2,value#3], MapPartitionsRDD[3] at beforeAll at 
BeforeAndAfterAll.scala:187
```
Then, the resulted physical Plan in this testcase had 4 `TungstenAggregate` 
generated by 
[aggregate/utils.planAggregateWithoutDistinct](https://github.com/apache/spark/blob/branch-1.6/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/utils.scala#L65-L131).
 So this testcase checked for 4 aggregations. 

After my change, there is no rewrite for the single distinct aggregate 
anymore, the analyzed plan looks like this:
```
Aggregate [value#3], [value#3,(count(key#2),mode=Complete,isDistinct=true) 
AS count(key)#47L]
+- LogicalRDD [key#2,value#3], MapPartitionsRDD[3] at beforeAll at 
BeforeAndAfterAll.scala:187
```
Then, the physical plan are resulted in 3 `TungstenAggregate` by 
[aggregate/utils.planAggregateWithOneDistinct](https://github.com/apache/spark/blob/branch-1.6/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/utils.scala#L160-L288)

Therefore, I need to modify this testcase. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15264][SQL] CSV Reader Error on Blank C...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13041#issuecomment-218357314
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58315/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15264][SQL] CSV Reader Error on Blank C...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13041#issuecomment-218357216
  
**[Test build #58315 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58315/consoleFull)**
 for PR 13041 at commit 
[`85d0843`](https://github.com/apache/spark/commit/85d08434b620241d698edeaa6557e8a715816185).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15250][SQL] Remove deprecated json API ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13040#issuecomment-218357297
  
**[Test build #58313 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58313/consoleFull)**
 for PR 13040 at commit 
[`d28b24d`](https://github.com/apache/spark/commit/d28b24d7537435d06bca59b478b59a4c4e312305).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14939][SQL] Add FoldablePropagation opt...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12719#issuecomment-218357235
  
**[Test build #58324 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58324/consoleFull)**
 for PR 12719 at commit 
[`b8ddf0c`](https://github.com/apache/spark/commit/b8ddf0c382e7c2499974b8c5e6cde41d51806361).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15263][Core] Make shuffle service dir c...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13042#issuecomment-218356728
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58312/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15263][Core] Make shuffle service dir c...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13042#issuecomment-218356727
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15263][Core] Make shuffle service dir c...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13042#issuecomment-218356644
  
**[Test build #58312 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58312/consoleFull)**
 for PR 13042 at commit 
[`32cc1e6`](https://github.com/apache/spark/commit/32cc1e63fde168e71a6d392106f551e874889a22).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14427][SQL] Support persisting partitio...

[viirya]

Github user viirya commented on the pull request:

https://github.com/apache/spark/pull/12204#issuecomment-218355986
  
@yhuai Sure. No problem. Please ping me when you think it is appropriate 
time to revisit this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15085][Streaming][Kafka] Rename streami...

[koeninger]

Github user koeninger commented on the pull request:

https://github.com/apache/spark/pull/12946#issuecomment-218355852
  
@JoshRosen let me know if that fix to pass mima is ok with you.  I can add 
settings for the kafka subproject instead, but this seemed like the most likely 
place to get noticed and cleaned up once the version changes and it's no longer 
needed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15085][Streaming][Kafka] Rename streami...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12946#issuecomment-218355738
  
**[Test build #58323 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58323/consoleFull)**
 for PR 12946 at commit 
[`ab2bd63`](https://github.com/apache/spark/commit/ab2bd63d02c4b00edd4d776b4d2ce780b443f1ae).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE] Data Property accumulators...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11105#issuecomment-218355389
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58311/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE] Data Property accumulators...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/11105#issuecomment-218355386
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12469][CORE] Data Property accumulators...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/11105#issuecomment-218355286
  
**[Test build #58311 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58311/consoleFull)**
 for PR 11105 at commit 
[`e797227`](https://github.com/apache/spark/commit/e797227bce87e8f02b4c2fcfb67c9672d26d712a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15259] Sort time metric should not incl...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13035#issuecomment-218355176
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15259] Sort time metric should not incl...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13035#issuecomment-218355178
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58310/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15259] Sort time metric should not incl...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13035#issuecomment-218355071
  
**[Test build #58310 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/58310/consoleFull)**
 for PR 13035 at commit 
[`1c253fb`](https://github.com/apache/spark/commit/1c253fb6f7b7a294eb543dc01e262eeb06ece610).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-14495][SQL][1.6] fix resolution failure...

[xwu0226]

Github user xwu0226 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12974#discussion_r62788662
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/AggregationQuerySuite.scala
 ---
@@ -560,80 +560,73 @@ abstract class AggregationQuerySuite extends 
QueryTest with SQLTestUtils with Te
   }
 
   test("single distinct column set") {
-Seq(true, false).foreach { specializeSingleDistinctAgg =>
--- End diff --

yes. you are right. I will add it back. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15264][SQL] CSV Reader Error on Blank C...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/13041#issuecomment-218354784
  
sure, you don't have to ask for permission to review a patch


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15150][EXAMPLE][DOC] Update LDA example...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12927#issuecomment-218353892
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/58317/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org