date:20170626

[GitHub] spark issue #12257: [SPARK-14483][WEBUI] Display user name for each job and ...

2017-06-26 Thread sarutak

Github user sarutak commented on the issue:

https://github.com/apache/spark/pull/12257
  
Sorry for my late reply. I'll close this PR for now and might open another 
PR in the near future. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #12257: [SPARK-14483][WEBUI] Display user name for each j...

2017-06-26 Thread sarutak

Github user sarutak closed the pull request at:

https://github.com/apache/spark/pull/12257


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18429: [SPARK-21222] Move elimination of Distinct clause from a...

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18429
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18429: [SPARK-21222] Move elimination of Distinct clause from a...

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18429
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78682/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18429: [SPARK-21222] Move elimination of Distinct clause from a...

2017-06-26 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18429
  
**[Test build #78682 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78682/testReport)**
 for PR 18429 at commit 
[`2f89499`](https://github.com/apache/spark/commit/2f894997498c472f4da1f44e77157683da17041d).
 * This patch **fails PySpark pip packaging tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class EliminateDistinctSuite extends PlanTest `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15821: [SPARK-13534][PySpark] Using Apache Arrow to increase pe...

2017-06-26 Thread MaheshIBM

Github user MaheshIBM commented on the issue:

https://github.com/apache/spark/pull/15821
  
That lends me to believe that the download request could be resolving to 
different hosts every time, can it happen if there is a CDN working in the 
background?  Not all hosts are configured to use the bad certificate. While one 
(or more possibly) are using a certificate with DN of conda.binstar.org and 
responding to the domain name in the hostname of the url from where the package 
download is attempted. 

If there is a way for configuring pip to ignore ssl errors (only for 
purpose of troubleshooting and find root cause of the problem here), then that 
is one possible direction to take. I am looking for ways to ignore ssl errors 
when using pip, will update the comment if i find something. 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17937: Reload credentials file config when app starts with chec...

2017-06-26 Thread jerryshao

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/17937
  
This is already fix in https://github.com/apache/spark/pull/18230 CC 
@gatorsmile .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16995: [SPARK-19340][SQL] CSV file will result in an exception ...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16995
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17211: [SPARK-19869][SQL]move table related ddl from ddl.scala ...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17211
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17237: [SPARK-19852][PYSPARK][ML] Update Python API setHandleIn...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17237
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17181: [SPARK-19824][Core] Standalone master JSON not showing c...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17181
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17235: [SPARK-19320][MESOS][WIP]allow specifying a hard limit o...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17235
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17248: [SPARK-19909][SS] Batches will fail in case that tempora...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17248
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17341: [SPARK-20013][SQL]add a newTablePath parameter for renam...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17341
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17708: [SPARK-20413] Add new query hint NO_COLLAPSE.

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17708
  
We are closing the inactive PRs. After you run more test, please do reopen 
if you still hit this issue. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18429: [SPARK-21222] Move elimination of Distinct clause from a...

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18429
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78680/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18429: [SPARK-21222] Move elimination of Distinct clause from a...

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18429
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18416
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17721: [SPARK-20013][SQL]merge renameTable to alterTable in Ext...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17721
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18416
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78677/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18429: [SPARK-21222] Move elimination of Distinct clause from a...

2017-06-26 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18429
  
**[Test build #78680 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78680/testReport)**
 for PR 18429 at commit 
[`892f50a`](https://github.com/apache/spark/commit/892f50a3cfda91918961cf81e13a769f2591173c).
 * This patch **fails PySpark pip packaging tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17937: Reload credentials file config when app starts with chec...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17937
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16692: [SPARK-19335] Introduce UPSERT feature to SPARK

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16692
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16685: [SPARK-19335] Introduce insert, update, and upsert comma...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16685
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18416: [SPARK-21204][SQL][WIP] Add support for Scala Set collec...

2017-06-26 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18416
  
**[Test build #78677 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78677/testReport)**
 for PR 18416 at commit 
[`53b1dc8`](https://github.com/apache/spark/commit/53b1dc8e15b20c55b96def5d7e48fcec86e7482a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16225: [SPARK-14932][SQL] Allow DataFrame.replace() to replace ...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16225
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15839: [SPARK-18397][SQL]cannot create table by using the hive ...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/15839
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14807: [SPARK-17256][Deploy, Windows]Check before adding double...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/14807
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15258: [SPARK-17689][SQL][STREAMING] added excludeFiles option ...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/15258
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13506: [SPARK-15763][SQL] Support DELETE FILE command natively

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/13506
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14330: [SPARK-16693][SPARKR] Remove methods deprecated in 2.0.0...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/14330
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14191: [SPARK-16217][SQL] Support SELECT INTO statement

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/14191
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14198: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/14198
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13028: [SPARK-9860][SQL] Join: Determine the join strategy (bro...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/13028
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12257: [SPARK-14483][WEBUI] Display user name for each job and ...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/12257
  
Really appreciate your contribution! Sorry, based on the comment, we might 
need to close this PR, but please submit more PRs in the future. Thanks again! 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #12356: [SPARK-14557][SQL] Reading textfile (created though CTAS...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/12356
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #9366: [SPARK-11057] [SQL] Add correlation and covariance matric...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/9366
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #11420: [SPARK-13493][SQL] Enable case sensitiveness in json sch...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/11420
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #9202: [SPARK-10857] [SQL] Block SQL injection vulnerabilities u...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/9202
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #10861: SPARK-12948. [SQL]. Consider reducing size of broadcasts...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/10861
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14739: [SPARK-17176][WEB UI]set default task sort column to "St...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/14739
  
Really appreciate your contribution! Sorry, based on the comment, we might 
need to close this PR, but please submit more PRs in the future. Thanks again! 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #8927: [SPARK-10796][CORE]Resubmit stage while lost task in Zomb...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/8927
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #7075: [SPARK-8674] [MLlib] Implementation of a 2 sample Kolmogo...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/7075
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17139: [SPARK-19486][CORE](try 3) Investigate using multiple th...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17139
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18359: Update Word2Vec.scala

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18359
  
Really appreciate your contribution! Sorry, based on the comment, we might 
need to close this PR, but please submit more PRs in the future. Thanks again! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17738: [SPARK-20422][Spark Core] Worker registration retries sh...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17738
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16458: [SPARK-19001] [Deploy]don't submit sendHeartbeat task ag...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16458
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16508: [spark-18806] [core] the processors DriverWrapper and Co...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16508
  
We are closing it due to inactivity. please do reopen if you want to push 
it forward. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17714: [SPARK-20428][Core]REST interface about 'v1/submissions/...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17714
  
Really appreciate your contribution! Sorry, based on the comment, we might 
need to close this PR, but please submit more PRs in the future. Thanks again! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15821: [SPARK-13534][PySpark] Using Apache Arrow to increase pe...

2017-06-26 Thread BryanCutler

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/15821

It's not looking like the SSL Verification Error is the issue, there are a
handful of recent builds that have passed after getting that same error,
see below.  Maybe something else is timing out?

From
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78669
```
prepending /tmp/tmp.87E7MDUu95/3.5/bin to PATH

Fetching package metadata: ..SSL verification error: hostname
'conda.binstar.org' doesn't match either of 'anaconda.com',
'anacondacloud.com', 'anacondacloud.org', 'binstar.org', 'wakari.io'
.SSL verification error: hostname 'conda.binstar.org' doesn't match
either of 'anaconda.com', 'anacondacloud.com', 'anacondacloud.org',
'binstar.org', 'wakari.io'
...
Solving package specifications: .

Package plan for installation in environment /tmp/tmp.87E7MDUu95/3.5:

The following NEW packages will be INSTALLED:

arrow-cpp:   0.4.1-np112py35_2  (soft-link)
certifi: 2017.4.17-py35_0   (soft-link)
jemalloc:5.0.0-1(soft-link)
ncurses: 5.9-10 (soft-link)
parquet-cpp: 1.1.0-2(soft-link)
pyarrow: 0.4.0-np112py35_0  (soft-link)

```

On Jun 26, 2017 9:23 PM, "Mahesh Sawaiker"  wrote:

> This does not seem like a timeout issue, the certificate CN and the what
> is used as the hostname are not matching. So clearly the client downloads
> the certificate but is not able to verify (no timeout). If anything it may
> be possible to configure the code/command to ignore ssl cert errors.
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18411: [SPARK-18004][SQL] Make sure the date or timestam...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18411#discussion_r124187044
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/jdbc/OracleDialect.scala ---
@@ -68,5 +69,13 @@ private case object OracleDialect extends JdbcDialect {
 case _ => None
   }
 
+  override def beforeFetch(connection: Connection, properties: Map[String, 
String]): Unit = {
+// Set general date and timestamp format before query.
+val stmt = connection.createStatement()
+stmt.execute("alter session set NLS_DATE_FORMAT = '-MM-DD'")
+stmt.execute("alter session set NLS_TIMESTAMP_FORMAT = '-MM-DD 
HH24:MI:SS.FF'")
--- End diff --

Will this impact the existing users?
> val sqlText = s"SELECT $columnList FROM ${options.table} $myWhereClause"

`${options.table}` could be a subquery containing the timestamp/date, right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18405: [SPARK-21194][SQL] Fail the putNullmethod when containsN...

2017-06-26 Thread jinxing64

Github user jinxing64 commented on the issue:

https://github.com/apache/spark/pull/18405
  
The test is complaining below, which I think is irrelevant.
Fetching package metadata: ..SSL verification error: hostname 
'conda.binstar.org' doesn't match either of 'anaconda.com', 
'anacondacloud.com', 'anacondacloud.org', 'binstar.org', 'wakari.io'
.SSL verification error: hostname 'conda.binstar.org' doesn't match either 
of 'anaconda.com', 'anacondacloud.com', 'anacondacloud.org', 'binstar.org', 
'wakari.io'
...
Solving package specifications: .
Error:  Package missing in current linux-64 channels: 
  - pyarrow 0.4|0.4.0*

You can search for this package on anaconda.org with

anaconda search -t conda pyarrow 0.4|0.4.0*

You may need to install the anaconda-client command line client with

conda install anaconda-client
Cleaning up temporary directory - /tmp/tmp.MqyoDpuoee
[error] running 
/home/jenkins/workspace/SparkPullRequestBuilder/dev/run-pip-tests ; received 
return code 1
Attempting to post to Github...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18428: [Spark-21221][ML] CrossValidator and TrainValidat...

2017-06-26 Thread jkbradley

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/18428#discussion_r124184937
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/tuning/ValidatorParams.scala ---
@@ -126,10 +126,22 @@ private[ml] object ValidatorParams {
   extraMetadata: Option[JObject] = None): Unit = {
 import org.json4s.JsonDSL._
 
+var numParamsNotJson = 0
 val estimatorParamMapsJson = compact(render(
   instance.getEstimatorParamMaps.map { case paramMap =>
 paramMap.toSeq.map { case ParamPair(p, v) =>
-  Map("parent" -> p.parent, "name" -> p.name, "value" -> 
p.jsonEncode(v))
+  v match {
+case writeableObj: MLWritable =>
+  numParamsNotJson += 1
--- End diff --

nit: move this down 1 line to index from 0


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18428: [Spark-21221][ML] CrossValidator and TrainValidat...

2017-06-26 Thread jkbradley

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/18428#discussion_r124161422
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/tuning/CrossValidatorSuite.scala ---
@@ -156,6 +156,46 @@ class CrossValidatorSuite
 CrossValidatorSuite.compareParamMaps(cv.getEstimatorParamMaps, 
cv2.getEstimatorParamMaps)
   }
 
+  test("read/write: CrossValidator with nested estimator") {
+val ova = new OneVsRest()
+  .setClassifier(new LogisticRegression)
+val evaluator = new BinaryClassificationEvaluator()
+  .setMetricName("areaUnderPR")  // not default metric
+
--- End diff --

style: remove extra newline


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18428: [Spark-21221][ML] CrossValidator and TrainValidat...

2017-06-26 Thread jkbradley

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/18428#discussion_r124185775
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/tuning/ValidatorParams.scala ---
@@ -183,8 +195,14 @@ private[ml] object ValidatorParams {
   val paramPairs = pMap.map { case pInfo: Map[String, String] =>
 val est = uidToParams(pInfo("parent"))
 val param = est.getParam(pInfo("name"))
-val value = param.jsonDecode(pInfo("value"))
-param -> value
+if (pInfo("isJson").toBoolean.booleanValue()) {
+  val value = param.jsonDecode(pInfo("value"))
+  param -> value
+} else {
+  val path = param.jsonDecode(pInfo("value")).toString
+  val value = 
DefaultParamsReader.loadParamsInstance[MLWritable](path, sc)
--- End diff --

This is OK with me for now since it will address all cases I've seen.  In 
the future, it'd be great to make this more general by allowing it to read any 
MLReadable type (not just DefaultParamsReadable).  I'll comment in the save() 
section above about this too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18428: [Spark-21221][ML] CrossValidator and TrainValidat...

2017-06-26 Thread jkbradley

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/18428#discussion_r124185314
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/tuning/ValidatorParams.scala ---
@@ -126,10 +126,22 @@ private[ml] object ValidatorParams {
   extraMetadata: Option[JObject] = None): Unit = {
 import org.json4s.JsonDSL._
 
+var numParamsNotJson = 0
 val estimatorParamMapsJson = compact(render(
   instance.getEstimatorParamMaps.map { case paramMap =>
 paramMap.toSeq.map { case ParamPair(p, v) =>
-  Map("parent" -> p.parent, "name" -> p.name, "value" -> 
p.jsonEncode(v))
+  v match {
+case writeableObj: MLWritable =>
+  numParamsNotJson += 1
+  val paramPath = new Path(path, "param" + p.name + 
numParamsNotJson).toString
--- End diff --

How about changing the prefix "param" -> "epm_"?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18428: [Spark-21221][ML] CrossValidator and TrainValidat...

2017-06-26 Thread jkbradley

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/18428#discussion_r124161463
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/tuning/CrossValidatorSuite.scala ---
@@ -156,6 +156,46 @@ class CrossValidatorSuite
 CrossValidatorSuite.compareParamMaps(cv.getEstimatorParamMaps, 
cv2.getEstimatorParamMaps)
   }
 
+  test("read/write: CrossValidator with nested estimator") {
+val ova = new OneVsRest()
+  .setClassifier(new LogisticRegression)
+val evaluator = new BinaryClassificationEvaluator()
+  .setMetricName("areaUnderPR")  // not default metric
--- End diff --

Is this needed for this unit test?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18428: [Spark-21221][ML] CrossValidator and TrainValidat...

2017-06-26 Thread jkbradley

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/18428#discussion_r124180242
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/tuning/TrainValidationSplitSuite.scala 
---
@@ -136,6 +136,29 @@ class TrainValidationSplitSuite
 assert(tvs.getSeed === tvs2.getSeed)
   }
 
+  test("read/write: TrainValidationSplit with nested estimator") {
+val ova = new OneVsRest()
+  .setClassifier(new LogisticRegression)
+val evaluator = new BinaryClassificationEvaluator()
+  .setMetricName("areaUnderPR")  // not default metric
+val classifier1 = new LogisticRegression().setRegParam(2.0)
+val classifier2 = new LogisticRegression().setRegParam(3.0)
+val paramMaps = new ParamGridBuilder()
+  .addGrid(ova.classifier, Array(classifier1, classifier2))
+  .build()
+val tvs = new TrainValidationSplit()
+  .setEstimator(ova)
+  .setEvaluator(evaluator)
+  .setTrainRatio(0.5)
+  .setEstimatorParamMaps(paramMaps)
+  .setSeed(42L)
+
+val tvs2 = testDefaultReadWrite(tvs, testParams = false)
+
+assert(tvs.getTrainRatio === tvs2.getTrainRatio)
+assert(tvs.getSeed === tvs2.getSeed)
--- End diff --

check classifier in paramMaps too


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18428: [Spark-21221][ML] CrossValidator and TrainValidat...

2017-06-26 Thread jkbradley

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/18428#discussion_r124168033
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/tuning/CrossValidatorSuite.scala ---
@@ -156,6 +156,46 @@ class CrossValidatorSuite
 CrossValidatorSuite.compareParamMaps(cv.getEstimatorParamMaps, 
cv2.getEstimatorParamMaps)
   }
 
+  test("read/write: CrossValidator with nested estimator") {
+val ova = new OneVsRest()
+  .setClassifier(new LogisticRegression)
+val evaluator = new BinaryClassificationEvaluator()
+  .setMetricName("areaUnderPR")  // not default metric
+
+
+val classifier1 = new LogisticRegression().setRegParam(2.0)
+val classifier2 = new LogisticRegression().setRegParam(3.0)
+val paramMaps = new ParamGridBuilder()
+  .addGrid(ova.classifier, Array(classifier1, classifier2))
+  .build()
+val cv = new CrossValidator()
+  .setEstimator(ova)
+  .setEvaluator(evaluator)
+  .setNumFolds(20)
+  .setEstimatorParamMaps(paramMaps)
+
+val cv2 = testDefaultReadWrite(cv, testParams = false)
+
+assert(cv.uid === cv2.uid)
+assert(cv.getNumFolds === cv2.getNumFolds)
+assert(cv.getSeed === cv2.getSeed)
+
+assert(cv2.getEvaluator.isInstanceOf[BinaryClassificationEvaluator])
+val evaluator2 = 
cv2.getEvaluator.asInstanceOf[BinaryClassificationEvaluator]
+assert(evaluator.uid === evaluator2.uid)
+assert(evaluator.getMetricName === evaluator2.getMetricName)
+
+cv2.getEstimator match {
+  case ova2: OneVsRest =>
+assert(ova.uid === ova2.uid)
+
assert(ova.getClassifier.asInstanceOf[LogisticRegression].getMaxIter
--- End diff --

Check type of classifier before casting


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18428: [Spark-21221][ML] CrossValidator and TrainValidat...

2017-06-26 Thread jkbradley

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/18428#discussion_r124161468
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/tuning/CrossValidatorSuite.scala ---
@@ -156,6 +156,46 @@ class CrossValidatorSuite
 CrossValidatorSuite.compareParamMaps(cv.getEstimatorParamMaps, 
cv2.getEstimatorParamMaps)
   }
 
+  test("read/write: CrossValidator with nested estimator") {
+val ova = new OneVsRest()
+  .setClassifier(new LogisticRegression)
--- End diff --

style: fix indentation


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18428: [Spark-21221][ML] CrossValidator and TrainValidat...

2017-06-26 Thread jkbradley

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/18428#discussion_r124185896
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/tuning/ValidatorParams.scala ---
@@ -126,10 +126,22 @@ private[ml] object ValidatorParams {
   extraMetadata: Option[JObject] = None): Unit = {
 import org.json4s.JsonDSL._
 
+var numParamsNotJson = 0
 val estimatorParamMapsJson = compact(render(
   instance.getEstimatorParamMaps.map { case paramMap =>
 paramMap.toSeq.map { case ParamPair(p, v) =>
-  Map("parent" -> p.parent, "name" -> p.name, "value" -> 
p.jsonEncode(v))
+  v match {
+case writeableObj: MLWritable =>
--- End diff --

Per my comment below in the load() section, this should be restricted to 
DefaultParamsWritable for now.  Could you please do so, but also add a check 
which throws an error if `v` is MLWritable but not DefaultParamsWritable?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18428: [Spark-21221][ML] CrossValidator and TrainValidat...

2017-06-26 Thread jkbradley

Github user jkbradley commented on a diff in the pull request:

https://github.com/apache/spark/pull/18428#discussion_r124161588
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/tuning/CrossValidatorSuite.scala ---
@@ -156,6 +156,46 @@ class CrossValidatorSuite
 CrossValidatorSuite.compareParamMaps(cv.getEstimatorParamMaps, 
cv2.getEstimatorParamMaps)
   }
 
+  test("read/write: CrossValidator with nested estimator") {
+val ova = new OneVsRest()
+  .setClassifier(new LogisticRegression)
+val evaluator = new BinaryClassificationEvaluator()
+  .setMetricName("areaUnderPR")  // not default metric
+
+
+val classifier1 = new LogisticRegression().setRegParam(2.0)
+val classifier2 = new LogisticRegression().setRegParam(3.0)
+val paramMaps = new ParamGridBuilder()
+  .addGrid(ova.classifier, Array(classifier1, classifier2))
--- End diff --

Add comment that it is important to test Param values which inherit from 
Params.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17681: [SPARK-20383][SQL] Supporting Create [temporary] ...

2017-06-26 Thread ouyangxiaochen

Github user ouyangxiaochen commented on a diff in the pull request:

https://github.com/apache/spark/pull/17681#discussion_r124183913
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala 
---
@@ -46,27 +46,53 @@ case class CreateFunctionCommand(
 functionName: String,
 className: String,
 resources: Seq[FunctionResource],
-isTemp: Boolean)
+isTemp: Boolean,
+ifNotExists: Boolean,
+replace: Boolean)
   extends RunnableCommand {
 
+  if (ifNotExists && replace) {
+throw new AnalysisException("CREATE FUNCTION with both IF NOT EXISTS 
and REPLACE" +
+  " is not allowed.")
+  }
+
+  // Disallows 'CREATE TEMPORARY FUNCTION IF NOT EXISTS' to be consistent
+  // with 'CREATE TEMPORARY FUNCTION'
+  if (ifNotExists && isTemp) {
+throw new AnalysisException(
+  "It is not allowed to define a TEMPORARY function with IF NOT 
EXISTS.")
+  }
+
+  // Temporary function names should not contain database prefix like 
"database.function"
+  if (databaseName.isDefined && isTemp) {
+throw new AnalysisException(s"Specifying a database in CREATE 
TEMPORARY FUNCTION " +
+  s"is not allowed: '${databaseName.get}'")
+  }
+
   override def run(sparkSession: SparkSession): Seq[Row] = {
 val catalog = sparkSession.sessionState.catalog
-val func = CatalogFunction(FunctionIdentifier(functionName, 
databaseName), className, resources)
+val func = FunctionIdentifier(functionName, databaseName)
 if (isTemp) {
-  if (databaseName.isDefined) {
-throw new AnalysisException(s"Specifying a database in CREATE 
TEMPORARY FUNCTION " +
-  s"is not allowed: '${databaseName.get}'")
-  }
   // We first load resources and then put the builder in the function 
registry.
   // Please note that it is allowed to overwrite an existing temp 
function.
   catalog.loadFunctionResources(resources)
-  catalog.registerFunction(func, ignoreIfExists = false)
+  // Handles `CREATE OR REPLACE TEMPORARY FUNCTION AS ... USING ...`
+  // We drop the temp function in FunctionRegistry firstly and then 
create a new one
+  if (replace && catalog.functionExists(func)) {
+catalog.dropTempFunction(functionName, ignoreIfNotExists = true)
+  }
+  catalog.registerFunction(CatalogFunction(func, className, 
resources), ignoreIfExists = false)
 } else {
-  // For a permanent, we will store the metadata into underlying 
external catalog.
-  // This function will be loaded into the FunctionRegistry when a 
query uses it.
-  // We do not load it into FunctionRegistry right now.
-  // TODO: should we also parse "IF NOT EXISTS"?
-  catalog.createFunction(func, ignoreIfExists = false)
+  // Handles `CREATE OR REPLACE FUNCTION AS ... USING ...`
+  if (replace && catalog.functionExists(func)) {
+// alter the function in the metastore
+catalog.alterFunction(CatalogFunction(func, className, resources))
--- End diff --

I added the test case in `SessionCatalogSuite` at line 1230.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18429: [SPARK-21222] Move elimination of Distinct clause from a...

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18429
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18429: [SPARK-21222] Move elimination of Distinct clause from a...

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18429
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78676/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18429: [SPARK-21222] Move elimination of Distinct clause from a...

2017-06-26 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18429
  
**[Test build #78676 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78676/testReport)**
 for PR 18429 at commit 
[`7604811`](https://github.com/apache/spark/commit/7604811863567cc81778b0f0cb39c1385564781c).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18174: [SPARK-20950][CORE]add a new config to diskWriteB...

2017-06-26 Thread heary-cao

Github user heary-cao commented on a diff in the pull request:

https://github.com/apache/spark/pull/18174#discussion_r124182713
  
--- Diff: 
core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillWriter.java
 ---
@@ -61,6 +65,9 @@ public UnsafeSorterSpillWriter(
 this.file = spilledFileInfo._2();
 this.blockId = spilledFileInfo._1();
 this.numRecordsToWrite = numRecordsToWrite;
+this.diskWriteBufferSize =
--- End diff --

Do you want to initialize as a member variable?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16028
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16028
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78681/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override

2017-06-26 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16028
  
**[Test build #78681 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78681/testReport)**
 for PR 16028 at commit 
[`5e85dc6`](https://github.com/apache/spark/commit/5e85dc672c8e1c25c225a23c6992aee25d4fd5da).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18182: [SPARK-20959][CORE]Add a parameter to UnsafeExternalSort...

2017-06-26 Thread heary-cao

Github user heary-cao commented on the issue:

https://github.com/apache/spark/pull/18182
  
@kiszk 
yes,
I tried other SQL algorithms, and the results showed that,
But for this hard coding, it is still modified to be configurable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17681: [SPARK-20383][SQL] Supporting Create [temporary] Functio...

2017-06-26 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17681
  
**[Test build #78683 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78683/testReport)**
 for PR 17681 at commit 
[`b787509`](https://github.com/apache/spark/commit/b787509033a3e30b0074a015f5cc5a8317bd3056).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17681: [SPARK-20383][SQL] Supporting Create [temporary] Functio...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17681
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17681: [SPARK-20383][SQL] Supporting Create [temporary] ...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17681#discussion_r124181223
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala 
---
@@ -46,27 +46,53 @@ case class CreateFunctionCommand(
 functionName: String,
 className: String,
 resources: Seq[FunctionResource],
-isTemp: Boolean)
+isTemp: Boolean,
+ifNotExists: Boolean,
+replace: Boolean)
   extends RunnableCommand {
 
+  if (ifNotExists && replace) {
+throw new AnalysisException("CREATE FUNCTION with both IF NOT EXISTS 
and REPLACE" +
+  " is not allowed.")
+  }
+
+  // Disallows 'CREATE TEMPORARY FUNCTION IF NOT EXISTS' to be consistent
+  // with 'CREATE TEMPORARY FUNCTION'
+  if (ifNotExists && isTemp) {
+throw new AnalysisException(
+  "It is not allowed to define a TEMPORARY function with IF NOT 
EXISTS.")
+  }
+
+  // Temporary function names should not contain database prefix like 
"database.function"
+  if (databaseName.isDefined && isTemp) {
+throw new AnalysisException(s"Specifying a database in CREATE 
TEMPORARY FUNCTION " +
+  s"is not allowed: '${databaseName.get}'")
+  }
+
   override def run(sparkSession: SparkSession): Seq[Row] = {
 val catalog = sparkSession.sessionState.catalog
-val func = CatalogFunction(FunctionIdentifier(functionName, 
databaseName), className, resources)
+val func = FunctionIdentifier(functionName, databaseName)
 if (isTemp) {
-  if (databaseName.isDefined) {
-throw new AnalysisException(s"Specifying a database in CREATE 
TEMPORARY FUNCTION " +
-  s"is not allowed: '${databaseName.get}'")
-  }
   // We first load resources and then put the builder in the function 
registry.
   // Please note that it is allowed to overwrite an existing temp 
function.
   catalog.loadFunctionResources(resources)
-  catalog.registerFunction(func, ignoreIfExists = false)
+  // Handles `CREATE OR REPLACE TEMPORARY FUNCTION AS ... USING ...`
+  // We drop the temp function in FunctionRegistry firstly and then 
create a new one
+  if (replace && catalog.functionExists(func)) {
+catalog.dropTempFunction(functionName, ignoreIfNotExists = true)
+  }
+  catalog.registerFunction(CatalogFunction(func, className, 
resources), ignoreIfExists = false)
 } else {
-  // For a permanent, we will store the metadata into underlying 
external catalog.
-  // This function will be loaded into the FunctionRegistry when a 
query uses it.
-  // We do not load it into FunctionRegistry right now.
-  // TODO: should we also parse "IF NOT EXISTS"?
-  catalog.createFunction(func, ignoreIfExists = false)
+  // Handles `CREATE OR REPLACE FUNCTION AS ... USING ...`
+  if (replace && catalog.functionExists(func)) {
+// alter the function in the metastore
+catalog.alterFunction(CatalogFunction(func, className, resources))
--- End diff --

No test case to cover the logics verification? We need to check whether the 
function is replaced.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16028
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16028
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78679/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16028: [SPARK-18518][ML] HasSolver supports override

2017-06-26 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16028
  
**[Test build #78679 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78679/testReport)**
 for PR 16028 at commit 
[`b3683c1`](https://github.com/apache/spark/commit/b3683c1b10b1e09fd301029e42b2ca84b9ac6563).
 * This patch **fails due to an unknown error code, -10**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  final val thresholds: DoubleArrayParam = new DoubleArrayParam(this, 
\"thresholds\", \"thresholds in multi-class classification to adjust the 
probability of predicting each class. Array must have length equal to the 
number of classes, with values > 0 excepting that at most one value may be 0. 
The class with largest value p/t is predicted, where p is the original 
probability of that class and t is the class's threshold\", (t: Array[Double]) 
=> t.forall(_ >= 0) && t.count(_ == 0) <= 1)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18418: [SPARK-19104][SQL] Lambda variables should work when par...

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18418
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18418: [SPARK-19104][SQL] Lambda variables should work when par...

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18418
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78678/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18418: [SPARK-19104][SQL] Lambda variables should work when par...

2017-06-26 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18418
  
**[Test build #78678 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78678/testReport)**
 for PR 18418 at commit 
[`d6161d5`](https://github.com/apache/spark/commit/d6161d5f4206c8e3cda63383905f9f3db9474d50).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18426: [SPARK-21216][SS] Hive strategies missed in Structured S...

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18426
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18426: [SPARK-21216][SS] Hive strategies missed in Structured S...

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18426
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78670/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18426: [SPARK-21216][SS] Hive strategies missed in Structured S...

2017-06-26 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18426
  
**[Test build #78670 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78670/testReport)**
 for PR 18426 at commit 
[`c0968ed`](https://github.com/apache/spark/commit/c0968ed4f1880d848a67f6b9164771f29669fa26).
 * This patch **fails from timeout after a configured wait of \`250m\`**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18417: [INFRA] Close stale PRs

2017-06-26 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/18417
  
(17211 is a dupe)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17681: [SPARK-20383][SQL] Supporting Create [temporary] ...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17681#discussion_r124179677
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala 
---
@@ -1458,6 +1458,8 @@ abstract class DDLSuite extends QueryTest with 
SQLTestUtils {
 }
   }
 
+
+
--- End diff --

Nit: revert them back?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18417: [INFRA] Close stale PRs

2017-06-26 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/18417
  
Sure.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18405: [SPARK-21194][SQL] Fail the putNullmethod when containsN...

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18405
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18405: [SPARK-21194][SQL] Fail the putNullmethod when containsN...

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18405
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78673/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18405: [SPARK-21194][SQL] Fail the putNullmethod when containsN...

2017-06-26 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18405
  
**[Test build #78673 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78673/testReport)**
 for PR 18405 at commit 
[`255c50a`](https://github.com/apache/spark/commit/255c50a87051df42933bbd83aea14ccd54c18826).
 * This patch **fails PySpark pip packaging tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18417: [INFRA] Close stale PRs

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18417
  
Three more PRs?
https://github.com/apache/spark/pull/17830
https://github.com/apache/spark/pull/14742
https://github.com/apache/spark/pull/17211


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18421: [SPARK-21213][SQL] Support collecting partition-l...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18421#discussion_r124178671
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/SparkSqlParserSuite.scala
 ---
@@ -239,18 +239,20 @@ class SparkSqlParserSuite extends AnalysisTest {
   AnalyzeTableCommand(TableIdentifier("t"), noscan = false))
 assertEqual("analyze table t compute statistics noscan",
   AnalyzeTableCommand(TableIdentifier("t"), noscan = true))
-assertEqual("analyze table t partition (a) compute statistics nOscAn",
+assertEqual("analyze table t compute statistics nOscAn",
   AnalyzeTableCommand(TableIdentifier("t"), noscan = true))
 
-// Partitions specified - we currently parse them but don't do 
anything with it
+// Partitions specified
 assertEqual("ANALYZE TABLE t PARTITION(ds='2008-04-09', hr=11) COMPUTE 
STATISTICS",
-  AnalyzeTableCommand(TableIdentifier("t"), noscan = false))
+  AnalyzeTableCommand(TableIdentifier("t"), noscan = false,
+partitionSpec = Some(Map("ds" -> "2008-04-09", "hr" -> "11"
 assertEqual("ANALYZE TABLE t PARTITION(ds='2008-04-09', hr=11) COMPUTE 
STATISTICS noscan",
-  AnalyzeTableCommand(TableIdentifier("t"), noscan = true))
-assertEqual("ANALYZE TABLE t PARTITION(ds, hr) COMPUTE STATISTICS",
-  AnalyzeTableCommand(TableIdentifier("t"), noscan = false))
-assertEqual("ANALYZE TABLE t PARTITION(ds, hr) COMPUTE STATISTICS 
noscan",
-  AnalyzeTableCommand(TableIdentifier("t"), noscan = true))
+  AnalyzeTableCommand(TableIdentifier("t"), noscan = true,
+partitionSpec = Some(Map("ds" -> "2008-04-09", "hr" -> "11"
+intercept("ANALYZE TABLE t PARTITION(ds, hr) COMPUTE STATISTICS",
--- End diff --

This should be legal based on the description of Hive? 

https://cwiki.apache.org/confluence/display/Hive/StatsDev


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18429: [SPARK-21222] Move elimination of Distinct clause from a...

2017-06-26 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18429
  
**[Test build #78682 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78682/testReport)**
 for PR 18429 at commit 
[`2f89499`](https://github.com/apache/spark/commit/2f894997498c472f4da1f44e77157683da17041d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18429: [SPARK-21222] Move elimination of Distinct clause...

2017-06-26 Thread gengliangwang

Github user gengliangwang commented on a diff in the pull request:

https://github.com/apache/spark/pull/18429#discussion_r124178339
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/EliminateDistinceSuite.scala
 ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.dsl.expressions._
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.plans.PlanTest
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Expand, 
LocalRelation, LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.RuleExecutor
+
+class EliminateDistinceSuite extends PlanTest {
--- End diff --

Typo corrected. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/11994
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78672/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-06-26 Thread SparkQA

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/11994
  
**[Test build #78672 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78672/testReport)**
 for PR 11994 at commit 
[`dd981ba`](https://github.com/apache/spark/commit/dd981ba1db4066109d61af1cfb18a06819b4bed5).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2017-06-26 Thread AmplabJenkins

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/11994
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18421: [SPARK-21213][SQL] Support collecting partition-l...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18421#discussion_r124178194
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTableCommand.scala
 ---
@@ -24,18 +24,23 @@ import scala.util.control.NonFatal
 import org.apache.hadoop.fs.{FileSystem, Path}
 
 import org.apache.spark.internal.Logging
-import org.apache.spark.sql.{AnalysisException, Row, SparkSession}
+import org.apache.spark.sql.{AnalysisException, Column, Row, SparkSession}
 import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.catalog.{CatalogStatistics, 
CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute
+import org.apache.spark.sql.catalyst.catalog.{CatalogStatistics, 
CatalogTable, CatalogTablePartition, CatalogTableType}
+import 
org.apache.spark.sql.catalyst.catalog.CatalogTypes.TablePartitionSpec
+import org.apache.spark.sql.catalyst.expressions.{And, EqualTo, 
Expression, Literal}
 import org.apache.spark.sql.internal.SessionState
 
 
 /**
- * Analyzes the given table to generate statistics, which will be used in 
query optimizations.
+ * Analyzes the given table or partition to generate statistics, which 
will be used in
+ * query optimizations.
--- End diff --

Could you please add the description about `partitionSpec`? 
>  If certain partition specs are specified, then statistics are gathered 
for only those partitions. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18429: [SPARK-21222] Move elimination of Distinct clause...

2017-06-26 Thread rxin

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/18429#discussion_r124177929
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/EliminateDistinceSuite.scala
 ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.dsl.expressions._
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.plans.PlanTest
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Expand, 
LocalRelation, LogicalPlan}
+import org.apache.spark.sql.catalyst.rules.RuleExecutor
+
+class EliminateDistinceSuite extends PlanTest {
--- End diff --

Distinct. not Distince.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18421: [SPARK-21213][SQL] Support collecting partition-l...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18421#discussion_r124177691
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -101,19 +101,30 @@ class SparkSqlAstBuilder(conf: SQLConf) extends 
AstBuilder(conf) {
* }}}
--- End diff --

Please also update the above description with newly supported the partition 
spec. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18421: [SPARK-21213][SQL] Support collecting partition-l...

2017-06-26 Thread gatorsmile

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18421#discussion_r124177638
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -101,19 +101,30 @@ class SparkSqlAstBuilder(conf: SQLConf) extends 
AstBuilder(conf) {
* }}}
*/
   override def visitAnalyze(ctx: AnalyzeContext): LogicalPlan = 
withOrigin(ctx) {
-if (ctx.partitionSpec != null) {
-  logWarning(s"Partition specification is ignored: 
${ctx.partitionSpec.getText}")
-}
-if (ctx.identifier != null) {
+val noscan = if (ctx.identifier != null) {
   if (ctx.identifier.getText.toLowerCase(Locale.ROOT) != "noscan") {
 throw new ParseException(s"Expected `NOSCAN` instead of 
`${ctx.identifier.getText}`", ctx)
   }
-  AnalyzeTableCommand(visitTableIdentifier(ctx.tableIdentifier))
-} else if (ctx.identifierSeq() == null) {
-  AnalyzeTableCommand(visitTableIdentifier(ctx.tableIdentifier), 
noscan = false)
+  true
+} else {
+  false
+}
+
+val partitionSpec = if (ctx.partitionSpec != null) {
+  Option(ctx.partitionSpec).map(visitNonOptionalPartitionSpec)
 } else {
+  None
+}
--- End diff --

```val partitionSpec = 
Option(ctx.partitionSpec).map(visitNonOptionalPartitionSpec)```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

1 2 3 4 5 6 7 >

1 - 100 of 609 matches

Mail list logo