[GitHub] spark pull request: [SPARK-15449][MLlib][Example]:Wrong Data Forma...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13301#issuecomment-221696827
  
**[Test build #59292 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59292/consoleFull)**
 for PR 13301 at commit 
[`5bf30dd`](https://github.com/apache/spark/commit/5bf30dd7e9049c4bb52daff1ff33ce06f2c47e08).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15168][PySpark][ML] Add missing params ...

[holdenk]

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/12943#issuecomment-221696163
  
So I simplified the test down a fair amount, didn't switch to printing the 
model weights since that seems like it could be flaky with floats (I can of 
course use ... in doctests if we want but I don't think it adds much).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15269][SQL] Removes unexpected empty ta...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/13270#discussion_r64645557
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -68,7 +69,8 @@ private[spark] class HiveExternalCatalog(client: 
HiveClient) extends ExternalCat
   body
 } catch {
   case NonFatal(e) if isClientException(e) =>
-throw new AnalysisException(e.getClass.getCanonicalName + ": " + 
e.getMessage)
+throw new AnalysisException(
+  e.getClass.getCanonicalName + ": " + e.getMessage, cause = 
Some(e))
--- End diff --

Preserve the original exception so that we can see Hive internal stack 
trace.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15449][MLlib][Example]:Wrong Data Forma...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13301#issuecomment-221692709
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15449][MLlib][Example]:Wrong Data Forma...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13301#issuecomment-221692715
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59291/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15449][MLlib][Example]:Wrong Data Forma...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13301#issuecomment-221692582
  
**[Test build #59291 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59291/consoleFull)**
 for PR 13301 at commit 
[`fa3656e`](https://github.com/apache/spark/commit/fa3656e2aab980c0413357699d3774faf8372b0e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15449][MLlib][Example]:Wrong Data Forma...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13301#issuecomment-221690057
  
**[Test build #59291 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59291/consoleFull)**
 for PR 13301 at commit 
[`fa3656e`](https://github.com/apache/spark/commit/fa3656e2aab980c0413357699d3774faf8372b0e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15449][MLlib][Example]:Wrong Data Forma...

[wangmiao1981]

GitHub user wangmiao1981 opened a pull request:

https://github.com/apache/spark/pull/13301

[SPARK-15449][MLlib][Example]:Wrong Data Format - Documentation Issue

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)
In the MLLib naivebayes example, scala and python example doesn't use 
libsvm data, but Java does.

I make changes in scala and python example to use the libsvm data as the 
same as Java example.

## How was this patch tested?

Manual tests




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wangmiao1981/spark example

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/13301.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #13301


commit fa3656e2aab980c0413357699d3774faf8372b0e
Author: wm...@hotmail.com 
Date:   2016-05-25T19:55:18Z

change data source for mllib naivebayes example




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15168][PySpark][ML] Add missing params ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12943#issuecomment-221686698
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15451][build] Use jdk7's rt.jar when av...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13272#issuecomment-221686186
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59284/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15451][build] Use jdk7's rt.jar when av...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13272#issuecomment-221685732
  
**[Test build #59284 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59284/consoleFull)**
 for PR 13272 at commit 
[`865a1e0`](https://github.com/apache/spark/commit/865a1e0ef0f0c2168622b5de0a009c1a57c37423).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15168][PySpark][ML] Add missing params ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12943#issuecomment-221686538
  
**[Test build #59290 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59290/consoleFull)**
 for PR 12943 at commit 
[`ba6f81c`](https://github.com/apache/spark/commit/ba6f81cdd2f1a8a3e5cf4cd441528e75c4813253).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15451][build] Use jdk7's rt.jar when av...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13272#issuecomment-221686181
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15168][PySpark][ML] Add missing params ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12943#issuecomment-221686699
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59290/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15493][SQL] default QuoteEscapingEnable...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13267


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15492][ML][DOC]:Binarization scala exam...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13266#issuecomment-221684412
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15492][ML][DOC]:Binarization scala exam...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13266#issuecomment-221682051
  
**[Test build #59289 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59289/consoleFull)**
 for PR 13266 at commit 
[`09baceb`](https://github.com/apache/spark/commit/09baceb4f00c8b634f5bacea8d0bb37aaa92129e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15525][SQL][BUILD] Upgrade ANTLR4 SBT p...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13299#issuecomment-221685313
  
**[Test build #3019 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3019/consoleFull)**
 for PR 13299 at commit 
[`3b042b5`](https://github.com/apache/spark/commit/3b042b546cce4d3aacbfa83f5ee3b560f3e18f4c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15493][SQL] default QuoteEscapingEnable...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/13267#issuecomment-221684056
  
Thanks - merging in master/2.0.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15492][ML][DOC]:Binarization scala exam...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13266#issuecomment-221684272
  
**[Test build #59289 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59289/consoleFull)**
 for PR 13266 at commit 
[`09baceb`](https://github.com/apache/spark/commit/09baceb4f00c8b634f5bacea8d0bb37aaa92129e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15492][ML][DOC]:Binarization scala exam...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13266#issuecomment-221684417
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59289/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15168][PySpark][ML] Add missing params ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12943#issuecomment-221683374
  
**[Test build #59290 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59290/consoleFull)**
 for PR 12943 at commit 
[`ba6f81c`](https://github.com/apache/spark/commit/ba6f81cdd2f1a8a3e5cf4cd441528e75c4813253).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15492][ML][DOC]:Binarization scala exam...

[wangmiao1981]

Github user wangmiao1981 commented on the pull request:

https://github.com/apache/spark/pull/13266#issuecomment-221681098
  
@MLnick Done. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15525][SQL][BUILD] Upgrade ANTLR4 SBT p...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13299#issuecomment-221680302
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15431][SQL][HOTFIX] ignore 'list' comma...

[xwu0226]

Github user xwu0226 commented on the pull request:

https://github.com/apache/spark/pull/13276#issuecomment-221680340
  
Adding to the above observation:
1. `LIST FILES` command output is not captured on all 
`spark-branch-2.0-test-*` jenkins jobs. 
2. `LIST FILE 
file:/home/jenkins/workspace/spark-master-test-maven-hadoop-/sql/hive-thriftserver/target/scala-2.11/test-classes/data/files/small_kv.txt`
 command output is not captured on all `spark-master-test-maven-*` jenkins 
jobs.  This tells that the first command `LIST FILES` have passed the test. 
3. The test cases passed on `spark-master-test-sbt-*` jenkins jobs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15525][SQL][BUILD] Upgrade ANTLR4 SBT p...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13299#issuecomment-221680304
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59286/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15525][SQL][BUILD] Upgrade ANTLR4 SBT p...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13299#issuecomment-22167
  
**[Test build #59286 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59286/consoleFull)**
 for PR 13299 at commit 
[`3b042b5`](https://github.com/apache/spark/commit/3b042b546cce4d3aacbfa83f5ee3b560f3e18f4c).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13271#issuecomment-221677507
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59285/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13271#issuecomment-221677309
  
**[Test build #59285 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59285/consoleFull)**
 for PR 13271 at commit 
[`f09032c`](https://github.com/apache/spark/commit/f09032c0c7b6fb3042c428ed5b397603100d7f91).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13271#issuecomment-221677504
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/9192#issuecomment-221676892
  
Thanks @felixcheung for the update. I left some minor comments inline.

It seems unfortunate that we need to do some amount of code duplication to 
get this to work (i.e. define `read.df` and `read.df.default` etc.) But I think 
thats fine for two reasons (a) this is an internal code issue and we can 
continue to clean it up (b) i dont think we are adding a lot of methods there 
-- in fact we should remove some of the unused ones.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

[shivaram]

Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/9192#discussion_r64633002
  
--- Diff: R/pkg/R/SQLContext.R ---
@@ -37,6 +37,37 @@ getInternalType <- function(x) {
  stop(paste("Unsupported type for SparkDataFrame:", class(x
 }
 
+#' Temporary function to reroute old S3 Method call to new
+#' We need to check the class of x to ensure it is SQLContext before 
dispatching
+dispatchFunc <- function(newFuncSig, x, ...) {
--- End diff --

can we move this to utils.R. Also some function level comments on what the 
arguments mean would be useful (for example `numFuncSig` is only used to print 
the deprecation warning from what i see)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15525][SQL][BUILD] Upgrade ANTLR4 SBT p...

[MLnick]

Github user MLnick commented on the pull request:

https://github.com/apache/spark/pull/13299#issuecomment-221675752
  
Confirmed `build/sbt package` works and the plugin dep resolves.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

[shivaram]

Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/9192#discussion_r64633530
  
--- Diff: R/pkg/inst/tests/testthat/test_sparkSQL.R ---
@@ -169,48 +169,50 @@ test_that("create DataFrame from RDD", {
   error = function(err) {
 skip("Hive is not build with SparkSQL, skipped")
   })
-  sql(hiveCtx, "CREATE TABLE people (name string, age double, height 
float)")
-  df <- read.df(hiveCtx, jsonPathNa, "json", schema)
+  assign(".sparkRHivesc", hiveCtx, envir = .sparkREnv)
--- End diff --

minor nit: we should add a new method to create the test hive context that 
also does this assignment. seems like something that other test cases might 
forget to do


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15483][SQL] IncrementalExecution should...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13261


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10903] [SPARKR] R - Simplify SQLContext...

[shivaram]

Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/9192#discussion_r64633209
  
--- Diff: R/pkg/R/SQLContext.R ---
@@ -254,6 +301,7 @@ jsonFile <- function(sqlContext, path) {
 #' df <- jsonRDD(sqlContext, rdd)
 #'}
 
+# TODO: remove - this method is no longer exported
--- End diff --

Can we open a JIRA for this ? Would be good to clean up this file as I 
think a bunch of functions are not exported here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15483][SQL] IncrementalExecution should...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/13261#issuecomment-221673769
  
Thanks, merging to master and 2.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13271#issuecomment-221672349
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13271#issuecomment-221672353
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59283/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13271#issuecomment-221672103
  
**[Test build #59283 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59283/consoleFull)**
 for PR 13271 at commit 
[`ea7d883`](https://github.com/apache/spark/commit/ea7d883d7f9305937bc2b542df9d1bf603b3bf51).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/13283#discussion_r64631597
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
 ---
@@ -108,21 +108,20 @@ case class DataSource(
   dataSource
 case Failure(error) =>
   if (error.isInstanceOf[ClassNotFoundException]) {
-val className = error.getMessage
-if (spark2RemovedClasses.contains(className)) {
-  throw new ClassNotFoundException(s"$className is removed 
in Spark 2.0. " +
+// error.getMessage is the class name of provider2. 
Instead, we use provider here.
--- End diff --

This is for link issues. But it will be `NoClassDefFoundError`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15294][SPARKR][MINOR] Add pivot functio...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/13295#issuecomment-221670088
  
Jenkins, ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/13283#discussion_r64631743
  
--- Diff: python/pyspark/sql/utils.py ---
@@ -77,6 +83,8 @@ def deco(*a, **kw):
 raise QueryExecutionException(s.split(': ', 1)[1], 
stackTrace)
 if s.startswith('java.lang.IllegalArgumentException: '):
 raise IllegalArgumentException(s.split(': ', 1)[1], 
stackTrace)
+if s.startswith('java.lang.NoClassDefFoundError: '):
--- End diff --

The Python changes are not necessary. Right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-12922][SparkR][WIP] Implement gapply() ...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/12836#issuecomment-221669659
  
Hmm - What is the difference between `dapply_row` and SQL row UDF ? anyways 
this discussion probably belongs in a new JIRA and not in this PR


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15294][SPARKR][MINOR] Add pivot functio...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13295#issuecomment-221671264
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15294][SPARKR][MINOR] Add pivot functio...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13295#issuecomment-221671233
  
**[Test build #59288 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59288/consoleFull)**
 for PR 13295 at commit 
[`b276420`](https://github.com/apache/spark/commit/b276420f4aa3d75583f9b825c31f3eae48ed6e24).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15294][SPARKR][MINOR] Add pivot functio...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/13295#issuecomment-221670324
  
Thanks @mhnatiuk for opening this PR. Could we also add a unit test in 
`test_sparkSQL.R` for this ? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15525][SQL][BUILD] Upgrade ANTLR4 SBT p...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/13299#issuecomment-221669383
  
LGTM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/13283#discussion_r64631435
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
 ---
@@ -108,21 +108,20 @@ case class DataSource(
   dataSource
 case Failure(error) =>
   if (error.isInstanceOf[ClassNotFoundException]) {
-val className = error.getMessage
-if (spark2RemovedClasses.contains(className)) {
-  throw new ClassNotFoundException(s"$className is removed 
in Spark 2.0. " +
+// error.getMessage is the class name of provider2. 
Instead, we use provider here.
--- End diff --

In a second thought, I don't think we need this `if` branch. Could you just 
remove it?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15294][SPARKR][MINOR] Add pivot functio...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13295#issuecomment-221671267
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59288/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15492][ML][DOC]:Binarization scala exam...

[wangmiao1981]

Github user wangmiao1981 commented on the pull request:

https://github.com/apache/spark/pull/13266#issuecomment-221670964
  
@MLnick Sure. I will do it soon. Now, I am debugging a R bug. Thanks! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15294][SPARKR][MINOR] Add pivot functio...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13295#issuecomment-221671259
  
**[Test build #59288 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59288/consoleFull)**
 for PR 13295 at commit 
[`b276420`](https://github.com/apache/spark/commit/b276420f4aa3d75583f9b825c31f3eae48ed6e24).
 * This patch **fails some tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...

[wangmiao1981]

Github user wangmiao1981 commented on the pull request:

https://github.com/apache/spark/pull/13284#issuecomment-221668874
  
@shivaram I am debugging and try to find a hint. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7481][build][WIP] Add Hadoop 2.6+ spark...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12004#issuecomment-221668693
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7481][build][WIP] Add Hadoop 2.6+ spark...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12004#issuecomment-221668685
  
**[Test build #59287 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59287/consoleFull)**
 for PR 12004 at commit 
[`6b3812b`](https://github.com/apache/spark/commit/6b3812b24ca819997d6cd11c28a6d0b9a4402a2d).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `trait AzureTestSetup extends CloudSuite `
  * `trait S3aTestSetup extends CloudSuite `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15500][DOC][ML][PYSPARK] Remove default...

[MLnick]

Github user MLnick commented on the pull request:

https://github.com/apache/spark/pull/13277#issuecomment-221668407
  
Merged to master/branch-2.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7481][build][WIP] Add Hadoop 2.6+ spark...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12004#issuecomment-221668697
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59287/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15500][DOC][ML][PYSPARK] Remove default...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13277


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7481][build][WIP] Add Hadoop 2.6+ spark...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12004#issuecomment-221668290
  
**[Test build #59287 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59287/consoleFull)**
 for PR 12004 at commit 
[`6b3812b`](https://github.com/apache/spark/commit/6b3812b24ca819997d6cd11c28a6d0b9a4402a2d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15492][ML][DOC]:Binarization scala exam...

[MLnick]

Github user MLnick commented on the pull request:

https://github.com/apache/spark/pull/13266#issuecomment-221668080
  
@wangmiao1981 could you do the same for the `OneVsRestExample`? ie remove 
`DataFrame` type annotation and import. You can do that in this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/13283#issuecomment-221665525
  
@gatorsmile I saw it reports `org.apache.spark.sql.AnalysisException: 
Failed to find data source: mydatabase. Please find packages at 
http://spark-packages.org;; line 1 pos 15` for the following statement: 
`sql("select id from `mydatabase`.`file_path`")`. I'm not familiar with the 
table name resolution. Is it correct?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: typos fix for files in [mllib] [streaming] and...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13298#issuecomment-22145
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59281/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: typos fix for files in [mllib] [streaming] and...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13298#issuecomment-221666300
  
**[Test build #59281 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59281/consoleFull)**
 for PR 13298 at commit 
[`678e707`](https://github.com/apache/spark/commit/678e707d1edd3e7bee3d333920fd4c6f4e8cb599).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: typos fix for files in [mllib] [streaming] and...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13298#issuecomment-22144
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8603][SPARKR] Incorrect file separator ...

[shivaram]

Github user shivaram commented on a diff in the pull request:

https://github.com/apache/spark/pull/13165#discussion_r64628244
  
--- Diff: R/pkg/R/client.R ---
@@ -60,6 +60,15 @@ generateSparkSubmitArgs <- function(args, sparkHome, 
jars, sparkSubmitOpts, pack
   combinedArgs
 }
 
+determineLauncher <- function(sparkSubmitBin, combinedArgs, capture = 
FALSE) {
--- End diff --

You can use `nolint` - See `context.R` for an example. BTW what is the lint 
error here ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/13283#discussion_r64626689
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
 ---
@@ -132,12 +131,11 @@ case class DataSource(
   }
   }
 } catch {
-  case e: NoClassDefFoundError => // This one won't be caught by 
Scala NonFatal
-// NoClassDefFoundError's class name uses "/" rather than "." 
for packages
-val className = e.getMessage.replaceAll("/", ".")
-if (spark2RemovedClasses.contains(className)) {
-  throw new ClassNotFoundException(s"$className was removed in 
Spark 2.0. " +
-"Please check if your library is compatible with Spark 
2.0", e)
+  case e: NoClassDefFoundError =>
+// e.getMessage is the class name of provider2. Instead, we 
use provider here.
+if (spark2RemovedClasses.contains(provider)) {
--- End diff --

Will revert the changes for `NoClassDefFoundError`. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...

[shivaram]

Github user shivaram commented on the pull request:

https://github.com/apache/spark/pull/13284#issuecomment-221663730
  
@wangmiao1981 Lets continue the pipeRDD debugging on the JIRA.  This change 
LGTM for the subset and the masking tests

@felixcheung any other comments ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/13283#issuecomment-221662515
  
> ISSUE 3: Unable to detect incompatibility libraries for Spark 2.0 in Data 
Source Resolution. We report a strange error message:
> SQL Example:
>   select id from 
`org.apache.spark.sql.sources.HadoopFsRelationProvider`.`file_path`
> Error Message: 
>   Table or view not found: 
`org.apache.spark.sql.sources.HadoopFsRelationProvider`.`file_path`

This is not an issue you need to fix. `HadoopFsRelationProvider` is just an 
interface in Spark 1.6. The user should not use it like this. If someone sees 
HadoopFsRelationProvider is not found, it's usually a link issue. E.g., 
`com.databricks.spark.avro.DefaultSource` extends `HadoopFsRelationProvider`, 
however, `HadoopFsRelationProvider` has been removed in 2.0, so when loading 
`com.databricks.spark.avro.DefaultSource`, it will throw 
`NoClassDefFoundError(HadoopFsRelationProvider)` instead of 
`ClassNotFoundException(com.databricks.spark.avro.DefaultSource)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15463][SQL] support creating dataframe ...

[xwu0226]

Github user xwu0226 commented on the pull request:

https://github.com/apache/spark/pull/13300#issuecomment-221661115
  
@HyukjinKwon @falaki  Could you review the PR? Thanks! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/13283#discussion_r64622938
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
 ---
@@ -132,12 +131,11 @@ case class DataSource(
   }
   }
 } catch {
-  case e: NoClassDefFoundError => // This one won't be caught by 
Scala NonFatal
-// NoClassDefFoundError's class name uses "/" rather than "." 
for packages
-val className = e.getMessage.replaceAll("/", ".")
-if (spark2RemovedClasses.contains(className)) {
-  throw new ClassNotFoundException(s"$className was removed in 
Spark 2.0. " +
-"Please check if your library is compatible with Spark 
2.0", e)
+  case e: NoClassDefFoundError =>
+// e.getMessage is the class name of provider2. Instead, we 
use provider here.
+if (spark2RemovedClasses.contains(provider)) {
--- End diff --

You should not change this. If `provider` is not found, `loadClass` will 
throw `ClassNotFoundException`. If a class used by provider are not found, 
`NoClassDefFoundError` will be thrown.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15515] [SPARK-15514] [SQL] Error Handli...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/13283#discussion_r64622417
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
 ---
@@ -132,12 +131,11 @@ case class DataSource(
   }
   }
 } catch {
-  case e: NoClassDefFoundError => // This one won't be caught by 
Scala NonFatal
-// NoClassDefFoundError's class name uses "/" rather than "." 
for packages
-val className = e.getMessage.replaceAll("/", ".")
-if (spark2RemovedClasses.contains(className)) {
-  throw new ClassNotFoundException(s"$className was removed in 
Spark 2.0. " +
-"Please check if your library is compatible with Spark 
2.0", e)
+  case e: NoClassDefFoundError =>
+// e.getMessage is the class name of provider2. Instead, we 
use provider here.
+if (spark2RemovedClasses.contains(provider)) {
--- End diff --

In my previous PR, I want to provide a better message for e.g., 
`org.apache.spark.sql.DataFrame` not found. It usually happens when calling 
some method (has a `org.apache.spark.sql.DataFrame` parameter) in a class that 
is compiled with an old Spark. Obviously, here `provider` won't be 
`org.apache.spark.sql.DataFrame`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15439][SparkR]:Failed to run unit test ...

[wangmiao1981]

Github user wangmiao1981 commented on the pull request:

https://github.com/apache/spark/pull/13284#issuecomment-221656579
  
Re-tested on Ubuntu, the pipedRDD test case still fails. R version 3.3.0 
beta (2016-03-30 r70404)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR][CORE] Fix a HadoopRDD log message and ...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the pull request:

https://github.com/apache/spark/pull/13294#issuecomment-221656682
  
Thank you, @andrewor14 and @srowen !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: typos fix for files in [mllib] [streaming] and...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13298


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR][CORE] Fix a HadoopRDD log message and ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13294


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: typos fix for files in [mllib] [streaming] and...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/13298#issuecomment-221654254
  
OK, merging into master 2.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15463][SQL] support creating dataframe ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13300#issuecomment-221654079
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13289


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [MINOR][CORE] Fix a HadoopRDD log message and ...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/13294#issuecomment-221653771
  
m2.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/13289#discussion_r64620005
  
--- Diff: python/pyspark/sql/session.py ---
@@ -138,24 +138,37 @@ def getOrCreate(self):
 """Gets an existing :class:`SparkSession` or, if there is no 
existing one, creates a
 new one based on the options set in this builder.
 
-This method first checks whether there is a valid thread-local 
SparkSession,
-and if yes, return that one. It then checks whether there is a 
valid global
-default SparkSession, and if yes, return that one. If no valid 
global default
-SparkSession exists, the method creates a new SparkSession and 
assigns the
-newly created SparkSession as the global default.
+This method first checks whether there is a valid global 
default SparkSession, and if
+yes, return that one. If no valid global default SparkSession 
exists, the method
+creates a new SparkSession and assigns the newly created 
SparkSession as the global
+default.
+
+>>> s1 = SparkSession.builder.config("k1", "v1").getOrCreate()
+>>> s1.conf.get("k1") == "v1"
+True
 
 In case an existing SparkSession is returned, the config 
options specified
 in this builder will be applied to the existing SparkSession.
+
+>>> s2 = SparkSession.builder.config("k2", "v2").getOrCreate()
+>>> s1.conf.get("k1") == s2.conf.get("k1")
+True
+>>> s1.conf.get("k2") == s2.conf.get("k2")
+True
 """
 with self._lock:
-from pyspark.conf import SparkConf
 from pyspark.context import SparkContext
-from pyspark.sql.context import SQLContext
-sparkConf = SparkConf()
+from pyspark.conf import SparkConf
+session = SparkSession._instantiatedContext
+if session is None:
+sparkConf = SparkConf()
+for key, value in self._options.items():
+sparkConf.set(key, value)
+sc = SparkContext.getOrCreate(sparkConf)
+session = SparkSession(sc)
--- End diff --

actually before this line we might have to explicitly set the confs through 
`sc.conf.set`, since the `SparkContext` may be an existing one. There was a 
patch that did this for scala recently: 01e7b9c85bb84924e279021f9748774dce9702c8


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15345][SQL][PYSPARK]. SparkSession's co...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13160


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15520] [SQL] SparkSession builder in py...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/13289#issuecomment-221653189
  
Looks good. Merging into master 2.0.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15463][SQL] support creating dataframe ...

[xwu0226]

GitHub user xwu0226 opened a pull request:

https://github.com/apache/spark/pull/13300

[SPARK-15463][SQL] support creating dataframe out of RDD[String] for csv 
data

## What changes were proposed in this pull request?
Currently only `DataFrameReader.json(rdd: RDD[String]): DataFrame` is 
supported for converting RDD[String] to a dataframe. CSV content is similar to 
this, where users's application could have RDD[String] containing csv rows, we 
can also convert it to DataFrame, that is DataSet[Row]. 

This PR is to add the API `DataFrameReader.csv(rdd: RDD[String]): 
DataFrame`. Also in order to easily invoke the helper functions that are 
already implemented for csv parsing, I moved some of the private methods from 
`csv.DefaultSource` to `CSVRelation`.

## How was this patch tested?
A test case is added to load csv files to RDD[String] and covert to 
DataFrame and check the results. 
Regression test is run. 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xwu0226/spark SPARK-15463

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/13300.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #13300


commit 0b8b2fc3283b0b1c37ef83ed0a70c6beb55a1b25
Author: Xin Wu 
Date:   2016-05-25T08:18:08Z

SPARK-15463: support creating dataframe out of RDD[String] for csv data




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15345][SQL][PYSPARK]. SparkSession's co...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/13160#issuecomment-221652457
  
LGTM2. Merging into master 2.0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15525][SQL][BUILD] Upgrade ANTLR4 SBT p...

[hvanhovell]

Github user hvanhovell commented on the pull request:

https://github.com/apache/spark/pull/13299#issuecomment-221651230
  
cc @rxin @MLnick @vanzin (could you take a look at the build)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15525][SQL][BUILD] Upgrade ANTLR4 SBT p...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13299#issuecomment-221651306
  
**[Test build #59286 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59286/consoleFull)**
 for PR 13299 at commit 
[`3b042b5`](https://github.com/apache/spark/commit/3b042b546cce4d3aacbfa83f5ee3b560f3e18f4c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15525][SQL][BUILD] Upgrade ANTLR4 SBT p...

[hvanhovell]

GitHub user hvanhovell opened a pull request:

https://github.com/apache/spark/pull/13299

[SPARK-15525][SQL][BUILD] Upgrade ANTLR4 SBT plugin

## What changes were proposed in this pull request?
The ANTLR4 SBT plugin has been moved from its own repo to one on bintray. 
The version was also changed from `0.7.10` to `0.7.11`. The latter actually 
broke our build (@ihji has fixed this by also adding `0.7.10` and others to the 
bin-tray repo).

This PR upgrades the SBT-ANTLR4 plugin and ANTLR4 to their most recent 
versions (`0.7.11`/`4.5.3`). I have also removed a few obsolete build 
configurations.

## How was this patch tested?
Manually running SBT/Maven builds.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hvanhovell/spark SPARK-15525

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/13299.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #13299


commit 45d5bcf81a70021d05f39c714075f939a54fc4c7
Author: Herman van Hovell 
Date:   2016-05-25T17:31:56Z

Update ANTLR4 plugin, and remove old Maven plugin.

commit 3b042b546cce4d3aacbfa83f5ee3b560f3e18f4c
Author: Herman van Hovell 
Date:   2016-05-25T17:32:44Z

Merge remote-tracking branch 'apache-github/master' into SPARK-15525




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13271#issuecomment-221649899
  
**[Test build #59285 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59285/consoleFull)**
 for PR 13271 at commit 
[`f09032c`](https://github.com/apache/spark/commit/f09032c0c7b6fb3042c428ed5b397603100d7f91).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15451][build] Use jdk7's rt.jar when av...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13272#issuecomment-221645693
  
**[Test build #59284 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59284/consoleFull)**
 for PR 13272 at commit 
[`865a1e0`](https://github.com/apache/spark/commit/865a1e0ef0f0c2168622b5de0a009c1a57c37423).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[clockfly]

Github user clockfly commented on a diff in the pull request:

https://github.com/apache/spark/pull/13269#discussion_r64617059
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -42,17 +42,9 @@ class KeyValueGroupedDataset[K, V] private[sql](
 private val dataAttributes: Seq[Attribute],
 private val groupingAttributes: Seq[Attribute]) extends Serializable {
 
-  // Similar to [[Dataset]], we use unresolved encoders for later 
composition and resolved encoders
-  // when constructing new logical plans that will operate on the output 
of the current
-  // queryexecution.
-
-  private implicit val unresolvedKEncoder = encoderFor(kEncoder)
-  private implicit val unresolvedVEncoder = encoderFor(vEncoder)
-
-  private val resolvedKEncoder =
-unresolvedKEncoder.resolve(groupingAttributes, OuterScopes.outerScopes)
-  private val resolvedVEncoder =
-unresolvedVEncoder.resolve(dataAttributes, OuterScopes.outerScopes)
+  // Similar to [[Dataset]], we turn the passed in encoder to 
`ExpressionEncoder` explicitly.
+  private implicit val kEnc = encoderFor(kEncoder)
--- End diff --

Is it better to use the full name like keyEncoder?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-9044 Fix "Storage" tab in UI so that it ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13264


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-9044 Fix "Storage" tab in UI so that it ...

[zsxwing]

Github user zsxwing commented on the pull request:

https://github.com/apache/spark/pull/13264#issuecomment-221646004
  
LGTM. Merging to master / 2.0. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15436][SQL] Remove DescribeFunction and...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13292


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15451][build] Use jdk7's rt.jar when av...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/13272#issuecomment-221644960
  
@JoshRosen @srowen this is ready for review now, you can check the failed 
builds to see that it's working.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15436][SQL] Remove DescribeFunction and...

[hvanhovell]

Github user hvanhovell commented on the pull request:

https://github.com/apache/spark/pull/13292#issuecomment-221644166
  
merging to master & 2.0 thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15523][ML][MLLIB] Update JPMML to 1.2.1...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13297#issuecomment-221643566
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59280/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15523][ML][MLLIB] Update JPMML to 1.2.1...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13297#issuecomment-221643563
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15523][ML][MLLIB] Update JPMML to 1.2.1...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13297#issuecomment-221643262
  
**[Test build #59280 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59280/consoleFull)**
 for PR 13297 at commit 
[`3046f10`](https://github.com/apache/spark/commit/3046f10101676dd9a3a93e40e30cda5866edd5a2).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15495][SQL][WIP] Improve the explain ou...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13271#issuecomment-221641642
  
**[Test build #59283 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59283/consoleFull)**
 for PR 13271 at commit 
[`ea7d883`](https://github.com/apache/spark/commit/ea7d883d7f9305937bc2b542df9d1bf603b3bf51).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15451][build] Use jdk7's rt.jar when av...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13272#issuecomment-221641404
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org