[GitHub] spark issue #14915: [SPARK-17356][SQL] Fix out of memory issue when generati...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14915
  
**[Test build #64971 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64971/consoleFull)**
 for PR 14915 at commit 
[`20fa7e3`](https://github.com/apache/spark/commit/20fa7e37946267b7a126653ba81270a404497e45).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14915: [SPARK-17356][SQL] Fix out of memory issue when generati...

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/14915
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14966: Merge pull request #8 from apache/master

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14966
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64960/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14966: Merge pull request #8 from apache/master

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14966
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14966: Merge pull request #8 from apache/master

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14966
  
**[Test build #64960 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64960/consoleFull)**
 for PR 14966 at commit 
[`b6b0d0a`](https://github.com/apache/spark/commit/b6b0d0a41c1aa59bc97a0aa438619d903b78b108).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14969: [SPARK-17406][WEB-UI] limit timeline executor events

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14969
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64970/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14969: [SPARK-17406][WEB-UI] limit timeline executor events

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14969
  
**[Test build #64970 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64970/consoleFull)**
 for PR 14969 at commit 
[`c368f88`](https://github.com/apache/spark/commit/c368f885aa539da622f95093c51205af11c9d7a1).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14915: [SPARK-17356][SQL] Fix out of memory issue when generati...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14915
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64967/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14969: [SPARK-17406][WEB-UI] limit timeline executor events

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14969
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14915: [SPARK-17356][SQL] Fix out of memory issue when generati...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14915
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14921: [SPARK-17361][SQL] file-based external table without pat...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14921
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64961/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14915: [SPARK-17356][SQL] Fix out of memory issue when generati...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14915
  
**[Test build #64967 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64967/consoleFull)**
 for PR 14915 at commit 
[`20fa7e3`](https://github.com/apache/spark/commit/20fa7e37946267b7a126653ba81270a404497e45).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14921: [SPARK-17361][SQL] file-based external table without pat...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14921
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14921: [SPARK-17361][SQL] file-based external table without pat...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14921
  
**[Test build #64961 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64961/consoleFull)**
 for PR 14921 at commit 
[`43fb72e`](https://github.com/apache/spark/commit/43fb72ee54686b1cd8c6befcffe4f93a8ce8d0be).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14921: [SPARK-17361][SQL] file-based external table without pat...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14921
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14638: [SPARK-11374][SQL] Support `skip.header.line.count` opti...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14638
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14921: [SPARK-17361][SQL] file-based external table without pat...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14921
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64962/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14638: [SPARK-11374][SQL] Support `skip.header.line.count` opti...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14638
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64963/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14638: [SPARK-11374][SQL] Support `skip.header.line.count` opti...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14638
  
**[Test build #64963 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64963/consoleFull)**
 for PR 14638 at commit 
[`2b1c99b`](https://github.com/apache/spark/commit/2b1c99b4f2f573c91342a9ad58a50a73a358beba).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14921: [SPARK-17361][SQL] file-based external table without pat...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14921
  
**[Test build #64962 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64962/consoleFull)**
 for PR 14921 at commit 
[`4071bec`](https://github.com/apache/spark/commit/4071bec22849ea999c3c7e1b2e73f7cf2b8f9722).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14960: [WIP][SPARK-17339][SPARKR][CORE] Fix Windows path...

[sarutak]

Github user sarutak commented on a diff in the pull request:

https://github.com/apache/spark/pull/14960#discussion_r77575448
  
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1900,7 +1900,20 @@ private[spark] object Utils extends Logging {
*/
   def resolveURI(path: String): URI = {
 try {
-  val uri = new URI(path)
+  val osSafePath = if (Path.isWindowsAbsolutePath(path, false)) {
+// Make sure C:/ part becomes /C/.
+val windowsUri = new URI(path)
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
+  } else if (Path.isWindowsAbsolutePath(path, true)) {
+// Make sure /C:/ part becomes /C/.
+val windowsUri = new URI(path.substring(1))
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
--- End diff --

O.K, let's fix the other problem of `resolveURI` in another PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14968: [SPARK-17369][SQL][2.0] MetastoreRelation toJSON throws ...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/14968
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14967: [Minor] [ML] Correct weights doc of MultilayerPerceptron...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14967
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64968/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14967: [Minor] [ML] Correct weights doc of MultilayerPerceptron...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14967
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14967: [Minor] [ML] Correct weights doc of MultilayerPerceptron...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14967
  
**[Test build #64968 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64968/consoleFull)**
 for PR 14967 at commit 
[`cd6ad06`](https://github.com/apache/spark/commit/cd6ad06551c58972c4f4d14568e8ce10d4ecae49).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14623: [SPARK-17044][SQL] Make test files for window functions ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14623
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14623: [SPARK-17044][SQL] Make test files for window functions ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14623
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64964/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14527: [SPARK-16938][SQL] `drop/dropDuplicate` should handle th...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14527
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14623: [SPARK-17044][SQL] Make test files for window functions ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14623
  
**[Test build #64964 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64964/consoleFull)**
 for PR 14623 at commit 
[`9cb28c1`](https://github.com/apache/spark/commit/9cb28c1d702826c4b8dd944572c96662efd1e6cf).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14527: [SPARK-16938][SQL] `drop/dropDuplicate` should handle th...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14527
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64965/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14969: [SPARK-17406][WEB-UI] limit timeline executor events

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14969
  
**[Test build #64970 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64970/consoleFull)**
 for PR 14969 at commit 
[`c368f88`](https://github.com/apache/spark/commit/c368f885aa539da622f95093c51205af11c9d7a1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14527: [SPARK-16938][SQL] `drop/dropDuplicate` should handle th...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14527
  
**[Test build #64965 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64965/consoleFull)**
 for PR 14527 at commit 
[`a30cd6e`](https://github.com/apache/spark/commit/a30cd6e4c65663fe530135eb0f589af56f77a82b).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14426: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14426
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14426: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14426
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64966/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14426: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14426
  
**[Test build #64966 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64966/consoleFull)**
 for PR 14426 at commit 
[`42248a1`](https://github.com/apache/spark/commit/42248a10daeee9f8ac62e0713e82cee04cc23ab1).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class Hint(name: String, parameters: Seq[String], child: 
LogicalPlan) extends UnaryNode `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14969: [SPARK-17406][WEB-UI] limit timeline executor eve...

[cenyuhai]

GitHub user cenyuhai opened a pull request:

https://github.com/apache/spark/pull/14969

[SPARK-17406][WEB-UI] limit timeline executor events

## What changes were proposed in this pull request?
The job page will be too slow to open when there are thousands of executor 
events(added or removed). I found that in ExecutorsTab file, executorIdToData 
will not remove elements, it will increase all the time.Before this pr, it 
looks like 
[timeline1.png](https://issues.apache.org/jira/secure/attachment/12827112/timeline1.png).
 After this pr, it looks like 
[timeline2.png](https://issues.apache.org/jira/secure/attachment/12827113/timeline2.png)(we
 can set how many events will be displayed)



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cenyuhai/spark SPARK-17406

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14969.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14969


commit c368f885aa539da622f95093c51205af11c9d7a1
Author: cenyuhai 
Date:   2016-09-06T05:25:53Z

limit timeline executor events




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14960: [WIP][SPARK-17339][SPARKR][CORE] Fix Windows path...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/14960#discussion_r77574745
  
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1900,7 +1900,20 @@ private[spark] object Utils extends Logging {
*/
   def resolveURI(path: String): URI = {
 try {
-  val uri = new URI(path)
+  val osSafePath = if (Path.isWindowsAbsolutePath(path, false)) {
+// Make sure C:/ part becomes /C/.
+val windowsUri = new URI(path)
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
+  } else if (Path.isWindowsAbsolutePath(path, true)) {
+// Make sure /C:/ part becomes /C/.
+val windowsUri = new URI(path.substring(1))
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
--- End diff --

Let me please just use `new Path(...).toUri` directly to deal with this if 
it seems okay.

I just tried to fix `resolveURI` to use `new Path(path).toUri` instead of 
`new URI(path)` but I found it breaks existing tests for `resolveURI`. It seems 
parsing special characters differently, for example , `#`

```
"hdfs:/root/spark.jar[%23]app.jar" did not equal 
"hdfs:/root/spark.jar[#]app.jar"
```




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14915: [SPARK-17356][SQL] Fix out of memory issue when g...

[clockfly]

Github user clockfly commented on a diff in the pull request:

https://github.com/apache/spark/pull/14915#discussion_r77574630
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala 
---
@@ -617,7 +617,9 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] 
extends Product {
 case s: String => JString(s)
 case u: UUID => JString(u.toString)
 case dt: DataType => dt.jsonValue
-case m: Metadata => m.jsonValue
+// SPARK-17356: In usage of mllib, Metadata may store a huge vector of 
data, transforming
+// it to JSON may trigger OutOfMemoryError.
+case m: Metadata => Metadata.empty.jsonValue
--- End diff --

No, we should not. JNothing is to map `scala.Option`.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14960: [WIP][SPARK-17339][SPARKR][CORE] Fix Windows path...

[sarutak]

Github user sarutak commented on a diff in the pull request:

https://github.com/apache/spark/pull/14960#discussion_r77574419
  
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1900,7 +1900,20 @@ private[spark] object Utils extends Logging {
*/
   def resolveURI(path: String): URI = {
 try {
-  val uri = new URI(path)
+  val osSafePath = if (Path.isWindowsAbsolutePath(path, false)) {
+// Make sure C:/ part becomes /C/.
+val windowsUri = new URI(path)
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
+  } else if (Path.isWindowsAbsolutePath(path, true)) {
+// Make sure /C:/ part becomes /C/.
+val windowsUri = new URI(path.substring(1))
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
--- End diff --

The way @shivaram mentioned works well and doesn't affect  SPARK-11227.
@HyukjinKwon You can fix this problem with the way but if you will fix 
`resolveURI`, adding new test cases to `UtilsSuite` is desireble.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14966: Merge pull request #8 from apache/master

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14966
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14966: Merge pull request #8 from apache/master

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14966
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64959/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14966: Merge pull request #8 from apache/master

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14966
  
**[Test build #64959 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64959/consoleFull)**
 for PR 14966 at commit 
[`869eaaf`](https://github.com/apache/spark/commit/869eaaf23f79eefbc6a8ff7a7b9efbc4a9f8c6b7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14968: [SPARK-17369][SQL][2.0] MetastoreRelation toJSON throws ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14968
  
**[Test build #64969 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64969/consoleFull)**
 for PR 14968 at commit 
[`0101f7a`](https://github.com/apache/spark/commit/0101f7ac0ec6a88c4f0ed8f77ac48c7470a4b41e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14968: [SPARK-17369][SQL][2.0] MetastoreRelation toJSON throws ...

[clockfly]

Github user clockfly commented on the issue:

https://github.com/apache/spark/pull/14968
  
Target 2.0 branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14968: [SPARK-17369][SQL][2.0] MetastoreRelation toJSON ...

[clockfly]

GitHub user clockfly opened a pull request:

https://github.com/apache/spark/pull/14968

[SPARK-17369][SQL][2.0] MetastoreRelation toJSON throws AssertException due 
to missing otherCopyArgs

backport https://github.com/apache/spark/pull/14928 to 2.0

## What changes were proposed in this pull request?

`TreeNode.toJSON` requires a subclass to explicitly override otherCopyArgs 
to include currying construction arguments, otherwise it reports 
AssertException telling that the construction argument values' count doesn't 
match the construction argument names' count.

For class `MetastoreRelation`, it has a currying construction parameter 
`client: HiveClient`, but Spark forgets to add it to the list of otherCopyArgs.

## How was this patch tested?

Unit tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/clockfly/spark 
metastore_toJSON_fix_for_spark_2.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14968.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14968


commit 0101f7ac0ec6a88c4f0ed8f77ac48c7470a4b41e
Author: Sean Zhong 
Date:   2016-09-06T02:50:07Z

[SPARK-17369][SQL] MetastoreRelation toJSON throws AssertException due to 
missing otherCopyArgs

## What changes were proposed in this pull request?

`TreeNode.toJSON` requires a subclass to explicitly override otherCopyArgs 
to include currying construction arguments, otherwise it reports 
AssertException telling that the construction argument values' count doesn't 
match the construction argument names' count.

For class `MetastoreRelation`, it has a currying construction parameter 
`client: HiveClient`, but Spark forgets to add it to the list of otherCopyArgs.

## How was this patch tested?

Unit tests.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/14797
  
Just created a JIRA: https://issues.apache.org/jira/browse/SPARK-17409 

FYI, Spark 2.0.1 has been fixed by this PR, but Spark 2.1.0 has the issue 
due to the related codes have been changed. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14960: [WIP][SPARK-17339][SPARKR][CORE] Fix Windows path...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/14960#discussion_r77573054
  
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1900,7 +1900,20 @@ private[spark] object Utils extends Logging {
*/
   def resolveURI(path: String): URI = {
 try {
-  val uri = new URI(path)
+  val osSafePath = if (Path.isWindowsAbsolutePath(path, false)) {
+// Make sure C:/ part becomes /C/.
+val windowsUri = new URI(path)
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
+  } else if (Path.isWindowsAbsolutePath(path, true)) {
+// Make sure /C:/ part becomes /C/.
+val windowsUri = new URI(path.substring(1))
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
--- End diff --

Yeap, meanwhile, I will try to use that and run tests. Thanks for your 
quick feedback..


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14915: [SPARK-17356][SQL] Fix out of memory issue when g...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14915#discussion_r77573016
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala 
---
@@ -617,7 +617,9 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] 
extends Product {
 case s: String => JString(s)
 case u: UUID => JString(u.toString)
 case dt: DataType => dt.jsonValue
-case m: Metadata => m.jsonValue
+// SPARK-17356: In usage of mllib, Metadata may store a huge vector of 
data, transforming
+// it to JSON may trigger OutOfMemoryError.
+case m: Metadata => Metadata.empty.jsonValue
--- End diff --

shall we use `JNothing` instead of `Metadata.empty.jsonValue`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14960: [WIP][SPARK-17339][SPARKR][CORE] Fix Windows path...

[shivaram]

Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/14960#discussion_r77573002
  
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1900,7 +1900,20 @@ private[spark] object Utils extends Logging {
*/
   def resolveURI(path: String): URI = {
 try {
-  val uri = new URI(path)
+  val osSafePath = if (Path.isWindowsAbsolutePath(path, false)) {
+// Make sure C:/ part becomes /C/.
+val windowsUri = new URI(path)
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
+  } else if (Path.isWindowsAbsolutePath(path, true)) {
+// Make sure /C:/ part becomes /C/.
+val windowsUri = new URI(path.substring(1))
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
--- End diff --

The other option is to of course just use the Hadoop `Path` class and do 
something like `new Path(path).toURI` -- I think they handle `C:/` correctly. I 
don't know if this affects other functionality though (like SPARK-11227) and we 
should check with @sarutak 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14712: [SPARK-17072] [SQL] support table-level statistic...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/14712#discussion_r77572880
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala ---
@@ -168,6 +171,154 @@ class StatisticsSuite extends QueryTest with 
TestHiveSingleton with SQLTestUtils
   TableIdentifier("tempTable"), ignoreIfNotExists = true, purge = 
false)
   }
 
+  private def checkMetastoreRelationStats(
+  tableName: String,
+  expectedStats: Option[Statistics]): Unit = {
+val df = sql(s"SELECT * FROM $tableName")
+val relations = df.queryExecution.analyzed.collect { case rel: 
MetastoreRelation =>
+  expectedStats match {
+case Some(es) =>
+  assert(rel.catalogTable.stats.isDefined)
+  val stats = rel.catalogTable.stats.get
+  assert(stats.sizeInBytes === es.sizeInBytes)
+  assert(stats.rowCount === es.rowCount)
+case None =>
+  assert(rel.catalogTable.stats.isEmpty)
+  }
+  rel
+}
+assert(relations.size === 1)
+  }
+
+  test("test table-level statistics for hive tables created in 
HiveExternalCatalog") {
+val textTable = "textTable"
+withTable(textTable) {
+  // Currently Spark's statistics are self-contained, we don't have 
statistics until we use
+  // the `ANALYZE TABLE` command.
+  sql(s"CREATE TABLE $textTable (key STRING, value STRING) STORED AS 
TEXTFILE")
+  checkMetastoreRelationStats(textTable, expectedStats = None)
+  sql(s"INSERT INTO TABLE $textTable SELECT * FROM src")
+  checkMetastoreRelationStats(textTable, expectedStats = None)
+
+  // noscan won't count the number of rows
+  sql(s"ANALYZE TABLE $textTable COMPUTE STATISTICS noscan")
+  checkMetastoreRelationStats(textTable, expectedStats =
+Some(Statistics(sizeInBytes = 5812, rowCount = None)))
--- End diff --

sorry I missed this, we should avoid hardcode nondeterministic values(like 
file size) in test, for this case, we only need to make sure the first 
`sizeInBytes` is greater than 0, and the second `sizeInBytes` is equal to the 
first one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/14797
  
Thanks! Is there a jira?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/14797
  
The test failure in Spark 2.0 has been fixed in 
https://github.com/apache/spark/pull/14951. However, after this fix, we still 
optimize the query in CTAS twice, as reported in this PR. Thus, I plan to fix 
it soon. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14960: [WIP][SPARK-17339][SPARKR][CORE] Fix Windows path...

[shivaram]

Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/14960#discussion_r77572574
  
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1900,7 +1900,20 @@ private[spark] object Utils extends Logging {
*/
   def resolveURI(path: String): URI = {
 try {
-  val uri = new URI(path)
+  val osSafePath = if (Path.isWindowsAbsolutePath(path, false)) {
+// Make sure C:/ part becomes /C/.
+val windowsUri = new URI(path)
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
+  } else if (Path.isWindowsAbsolutePath(path, true)) {
+// Make sure /C:/ part becomes /C/.
+val windowsUri = new URI(path.substring(1))
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
--- End diff --

I see - Unless there is a reason not to handle such paths it will be good 
to make the change to convert `C:/../..` to valid URIs as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14797: [SPARK-17230] [SQL] Should not pass optimized query into...

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/14797
  
@gatorsmile want to put the regression tests at here? Or, you have already 
have a pr?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14967: [Minor] [ML] Correct weights doc of MultilayerPerceptron...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14967
  
**[Test build #64968 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64968/consoleFull)**
 for PR 14967 at commit 
[`cd6ad06`](https://github.com/apache/spark/commit/cd6ad06551c58972c4f4d14568e8ce10d4ecae49).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14967: [Minor] [ML] Correct weights doc of MultilayerPer...

[yanboliang]

GitHub user yanboliang opened a pull request:

https://github.com/apache/spark/pull/14967

[Minor] [ML] Correct weights doc of MultilayerPerceptronClassificationModel.

## What changes were proposed in this pull request?
```weights``` of ```MultilayerPerceptronClassificationModel``` should be 
the output weights of layers rather than initial weights, this PR correct it.


## How was this patch tested?
Doc change.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yanboliang/spark mlp-weights

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14967.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14967


commit cd6ad06551c58972c4f4d14568e8ce10d4ecae49
Author: Yanbo Liang 
Date:   2016-09-06T04:20:42Z

Correct weights doc of MultilayerPerceptronClassificationModel.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14915: [SPARK-17356][SQL] Fix out of memory issue when generati...

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/14915
  
LGTM. Pending jenkins. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/14712
  
I have created https://issues.apache.org/jira/browse/SPARK-17408. @wzhfy 
Can you take a look?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14915: [SPARK-17356][SQL] Fix out of memory issue when generati...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14915
  
**[Test build #64967 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64967/consoleFull)**
 for PR 14915 at commit 
[`20fa7e3`](https://github.com/apache/spark/commit/20fa7e37946267b7a126653ba81270a404497e45).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/14712
  
Can you take a look at the test at 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64956/testReport/junit/org.apache.spark.sql.hive/StatisticsSuite/test_statistics_of_LogicalRelation_converted_from_MetastoreRelation/?
 It is flaky.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14915: [SPARK-17356][SQL] Fix out of memory issue when generati...

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/14915
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14928: [SPARK-17369][SQL] MetastoreRelation toJSON throws Asser...

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/14928
  
I will revert it from branch 2.0.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14928: [SPARK-17369][SQL] MetastoreRelation toJSON throws Asser...

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/14928
  
Seems this breaks 2.0 build.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14964: [DO NOT MERGE] Test DefinedByConstructorParams

[yhuai]

Github user yhuai closed the pull request at:

https://github.com/apache/spark/pull/14964


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14527: [SPARK-16938][SQL] `drop/dropDuplicate` should handle th...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14527
  
**[Test build #64965 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64965/consoleFull)**
 for PR 14527 at commit 
[`a30cd6e`](https://github.com/apache/spark/commit/a30cd6e4c65663fe530135eb0f589af56f77a82b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14638: [SPARK-11374][SQL] Support `skip.header.line.count` opti...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14638
  
**[Test build #64963 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64963/consoleFull)**
 for PR 14638 at commit 
[`2b1c99b`](https://github.com/apache/spark/commit/2b1c99b4f2f573c91342a9ad58a50a73a358beba).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14426: [SPARK-16475][SQL] Broadcast Hint for SQL Queries

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14426
  
**[Test build #64966 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64966/consoleFull)**
 for PR 14426 at commit 
[`42248a1`](https://github.com/apache/spark/commit/42248a10daeee9f8ac62e0713e82cee04cc23ab1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14623: [SPARK-17044][SQL] Make test files for window functions ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14623
  
**[Test build #64964 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64964/consoleFull)**
 for PR 14623 at commit 
[`9cb28c1`](https://github.com/apache/spark/commit/9cb28c1d702826c4b8dd944572c96662efd1e6cf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14966: Merge pull request #8 from apache/master

[cenyuhai]

Github user cenyuhai closed the pull request at:

https://github.com/apache/spark/pull/14966


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14964: [DO NOT MERGE] Test DefinedByConstructorParams

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14964
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14964: [DO NOT MERGE] Test DefinedByConstructorParams

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14964
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64958/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14964: [DO NOT MERGE] Test DefinedByConstructorParams

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14964
  
**[Test build #64958 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64958/consoleFull)**
 for PR 14964 at commit 
[`546742a`](https://github.com/apache/spark/commit/546742a70e88751a82bf81ee1141c6761dfd2e0e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14913: [SPARK-17358][SQL] Cached table(parquet/orc) should be s...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/14913
  
thanks, merging to master and 2.0!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14913: [SPARK-17358][SQL] Cached table(parquet/orc) shou...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/14913


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14961: [SPARK-17379] [BUILD] Upgrade netty-all to 4.0.41 final ...

[jerryshao]

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/14961
  
Also many other downstream and upstream applications may also use different 
version of Netty jar, it would be better to keep stable for these fundamental 
dependences.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14960: [WIP][SPARK-17339][SPARKR][CORE] Fix Windows path...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/14960#discussion_r77568884
  
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1900,7 +1900,20 @@ private[spark] object Utils extends Logging {
*/
   def resolveURI(path: String): URI = {
 try {
-  val uri = new URI(path)
+  val osSafePath = if (Path.isWindowsAbsolutePath(path, false)) {
+// Make sure C:/ part becomes /C/.
+val windowsUri = new URI(path)
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
+  } else if (Path.isWindowsAbsolutePath(path, true)) {
+// Make sure /C:/ part becomes /C/.
+val windowsUri = new URI(path.substring(1))
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
--- End diff --

So, I am a bit confused I should still change R path.

```scala
new Path(path, "rMetadata").toString
```

is being used pretty much in those wrappers. 

Would this make sense if this case (handling `C:/../..`)  is treated within 
`Utils.resolveURI` cc - @sarutak ? If so, I will try to neat up the codes more.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14921: [SPARK-17361][SQL] file-based external table without pat...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14921
  
**[Test build #64962 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64962/consoleFull)**
 for PR 14921 at commit 
[`4071bec`](https://github.com/apache/spark/commit/4071bec22849ea999c3c7e1b2e73f7cf2b8f9722).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14961: [SPARK-17379] [BUILD] Upgrade netty-all to 4.0.41 final ...

[jerryshao]

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/14961
  
Upgrading Netty version to branch 1.6 may cause API version incompatible 
issue for yarn shuffle service, please see 
[SPARK-16018](https://issues.apache.org/jira/browse/SPARK-16018) and 
[SPARK-15178](https://issues.apache.org/jira/browse/SPARK-15178). We also need 
to shade netty if we want to backport to 1.6.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14921: [SPARK-17361][SQL] file-based external table without pat...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14921
  
**[Test build #64961 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64961/consoleFull)**
 for PR 14921 at commit 
[`43fb72e`](https://github.com/apache/spark/commit/43fb72ee54686b1cd8c6befcffe4f93a8ce8d0be).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14960: [WIP][SPARK-17339][SPARKR][CORE] Fix Windows path...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/14960#discussion_r77568657
  
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -1900,7 +1900,20 @@ private[spark] object Utils extends Logging {
*/
   def resolveURI(path: String): URI = {
 try {
-  val uri = new URI(path)
+  val osSafePath = if (Path.isWindowsAbsolutePath(path, false)) {
+// Make sure C:/ part becomes /C/.
+val windowsUri = new URI(path)
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
+  } else if (Path.isWindowsAbsolutePath(path, true)) {
+// Make sure /C:/ part becomes /C/.
+val windowsUri = new URI(path.substring(1))
+val driveLetter = windowsUri.getScheme
+s"/$driveLetter/${windowsUri.getSchemeSpecificPart()}"
--- End diff --

I see. Actually, it seems it has to be 
`C:\Users\appveyor\AppData\Local\Temp\1\RtmpkzTq6t\glm78c146111b1.tmp` for path 
without the `slash`.

It seems the problem is, in many classes such as `RWrappers`, there are the 
usages of `new Path(path, "rMetadata").toString`.

It seems this converts the path, for example,

from

```
C:\Users\appveyor\AppData\Local\Temp\1\RtmpkzTq6t\glm78c146111b1.tmp
```

to

```
C:/Users/appveyor/AppData/Local/Temp/1/RtmpkzTq6t/glm78c146111b1.tmp
```

So, for example, it goes through `textFile` -> `hadoopFile` -> 
`Utils.resolveURI`. Here the given path is 
`C:/Users/appveyor/AppData/Local/Temp/1/RtmpkzTq6t/glm78c146111b1.tmp` but it 
seems `Utils.resolveURI` does not handle this case.

So, this returns this as it is.

```scala
...
val uri = new URI(path)
if (uri.getScheme() != null) { // here it just passes becuase `C:/.../...` 
has the scheme `C`.
  return uri
}
...
```

and it ends up with an exception as below:

```scala
java.io.IOException: No FileSystem for scheme: C
at 
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2421)
at 
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2428)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
```




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14966: Merge pull request #8 from apache/master

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14966
  
**[Test build #64960 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64960/consoleFull)**
 for PR 14966 at commit 
[`b6b0d0a`](https://github.com/apache/spark/commit/b6b0d0a41c1aa59bc97a0aa438619d903b78b108).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14966: Merge pull request #8 from apache/master

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/14966
  
Please close it. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14966: Merge pull request #8 from apache/master

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14966
  
**[Test build #64959 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64959/consoleFull)**
 for PR 14966 at commit 
[`869eaaf`](https://github.com/apache/spark/commit/869eaaf23f79eefbc6a8ff7a7b9efbc4a9f8c6b7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14966: Merge pull request #8 from apache/master

[cenyuhai]

Github user cenyuhai commented on the issue:

https://github.com/apache/spark/pull/14966
  
Sorry, I make a mistake... I want to merge pull request to my fork.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14966: Merge pull request #8 from apache/master

[cenyuhai]

GitHub user cenyuhai opened a pull request:

https://github.com/apache/spark/pull/14966

Merge pull request #8 from apache/master

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)


## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)


(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)


merge latest code to my fork

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cenyuhai/spark master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14966.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14966


commit 869eaaf23f79eefbc6a8ff7a7b9efbc4a9f8c6b7
Author: 岑玉海 <261810...@qq.com>
Date:   2016-08-21T03:55:04Z

Merge pull request #8 from apache/master

merge latest code to my fork




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14928: [SPARK-17369][SQL] MetastoreRelation toJSON throws Asser...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/14928
  
LGTM, merging to master and 2.0!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14928: [SPARK-17369][SQL] MetastoreRelation toJSON throw...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/14928


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14219: [SPARK-16566][MLLib] sort sparseVector's indices before ...

[wilson-lauw]

Github user wilson-lauw commented on the issue:

https://github.com/apache/spark/pull/14219
  
Closing this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14946: [SPARK-17353] [SPARK-16943] [SPARK-16942] [BACKPORT-2.0]...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/14946
  
Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14219: [SPARK-16566][MLLib] sort sparseVector's indices ...

[wilson-lauw]

Github user wilson-lauw closed the pull request at:

https://github.com/apache/spark/pull/14219


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14946: [SPARK-17353] [SPARK-16943] [SPARK-16942] [BACKPO...

[gatorsmile]

Github user gatorsmile closed the pull request at:

https://github.com/apache/spark/pull/14946


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14946: [SPARK-17353] [SPARK-16943] [SPARK-16942] [BACKPORT-2.0]...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/14946
  
LGTM, merging to 2.0!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14850: [SPARK-17279][SQL] better error message for excep...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/14850


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14850: [SPARK-17279][SQL] better error message for exceptions d...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/14850
  
thanks for the review, merging to master!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14965: minor change to regex pattern

[yiwenzhang92]

Github user yiwenzhang92 closed the pull request at:

https://github.com/apache/spark/pull/14965


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14965: minor change to regex pattern

[yiwenzhang92]

GitHub user yiwenzhang92 opened a pull request:

https://github.com/apache/spark/pull/14965

minor change to regex pattern

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)


## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)


(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yiwenzhang92/spark spark-mod

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14965.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14965


commit 1cd86c96eabbaebaec4e7a9bd346032aff8fb93b
Author: yiwenzhang92 
Date:   2016-09-06T02:10:01Z

minor change to regex pattern




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14964: [DO NOT MERGE] Test DefinedByConstructorParams

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14964
  
**[Test build #64958 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64958/consoleFull)**
 for PR 14964 at commit 
[`546742a`](https://github.com/apache/spark/commit/546742a70e88751a82bf81ee1141c6761dfd2e0e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14964: [DO NOT MERGE] Test DefinedByConstructorParams

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/14964
  
test this please



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #14964: [DO NOT MERGE] Test DefinedByConstructorParams

[yhuai]

GitHub user yhuai reopened a pull request:

https://github.com/apache/spark/pull/14964

[DO NOT MERGE] Test DefinedByConstructorParams

## What changes were proposed in this pull request?
I am testing DefinedByConstructorParams with branch 1.6. Do not merge it.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yhuai/spark defineByParams-1.6

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/14964.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #14964


commit 546742a70e88751a82bf81ee1141c6761dfd2e0e
Author: Yin Huai 
Date:   2016-09-05T19:04:35Z

Test DefinedByConstructorParams




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14964: [DO NOT MERGE] Test DefinedByConstructorParams

[yhuai]

Github user yhuai commented on the issue:

https://github.com/apache/spark/pull/14964
  
test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org