date:20170703

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18507
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18159
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79078/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17758
  
**[Test build #79075 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79075/testReport)**
 for PR 17758 at commit 
[`12159c4`](https://github.com/apache/spark/commit/12159c403955f54066ed8c532ed991f829edfc1f).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class PreprocessDDLCommands(sparkSession: SparkSession) extends 
Rule[LogicalPlan] `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18174
  
**[Test build #79074 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79074/testReport)**
 for PR 18174 at commit 
[`f6d895c`](https://github.com/apache/spark/commit/f6d895c944c514b7e51db19388ef00016671dddb).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18481
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79079/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18159
  
**[Test build #79078 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79078/testReport)**
 for PR 18159 at commit 
[`f575f0c`](https://github.com/apache/spark/commit/f575f0c999647d1f7fda22652ea480f7a0d07047).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `trait DataWritingCommand extends RunnableCommand `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18502: [SPARK-21278][PYSPARK][WIP] Upgrade to Py4J 0.10.5

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18502
  
**[Test build #79076 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79076/testReport)**
 for PR 18502 at commit 
[`f708dde`](https://github.com/apache/spark/commit/f708ddec38917867f9f13c7136ecef28c46af3a1).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18507
  
**[Test build #79077 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79077/testReport)**
 for PR 18507 at commit 
[`9788b19`](https://github.com/apache/spark/commit/9788b19d06800cce243a79acc189c3424912f393).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18507
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79077/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18501: [SPARK-20256][SQL] SessionState should be created more l...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18501
  
**[Test build #79080 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79080/testReport)**
 for PR 18501 at commit 
[`8a1a64f`](https://github.com/apache/spark/commit/8a1a64f1d1c429709799c00087dabfb97f4ca8b7).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18501: [SPARK-20256][SQL] SessionState should be created more l...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18501
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18481
  
**[Test build #79079 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79079/testReport)**
 for PR 18481 at commit 
[`6d5adf0`](https://github.com/apache/spark/commit/6d5adf0f10338168943542e8f2177761d2140438).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18481
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18159
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18413: [SPARK-21205][SQL] pmod(number, 0) should be null.

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18413
  
**[Test build #79073 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79073/testReport)**
 for PR 18413 at commit 
[`da037c8`](https://github.com/apache/spark/commit/da037c810a8c121d7075b741478419ffb77202d8).
 * This patch **fails due to an unknown error code, -9**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18502: [SPARK-21278][PYSPARK][WIP] Upgrade to Py4J 0.10.5

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18502
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79076/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18501: [SPARK-20256][SQL] SessionState should be created more l...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18501
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79080/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17758
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79075/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18174
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18413: [SPARK-21205][SQL] pmod(number, 0) should be null.

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18413
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17758
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18413: [SPARK-21205][SQL] pmod(number, 0) should be null.

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18413
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79073/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18174
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79074/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18502: [SPARK-21278][PYSPARK][WIP] Upgrade to Py4J 0.10.5

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18502
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...

2017-07-03 Thread 10110346

Github user 10110346 commented on the issue:

https://github.com/apache/spark/pull/18507
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...

2017-07-03 Thread actuaryzhang

Github user actuaryzhang commented on the issue:

https://github.com/apache/spark/pull/18481
  
jenkins, retest this please 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18509: [SS][MINOR] Make EventTimeWatermarkExec explicitl...

2017-07-03 Thread jaceklaskowski

GitHub user jaceklaskowski opened a pull request:

https://github.com/apache/spark/pull/18509

[SS][MINOR] Make EventTimeWatermarkExec explicitly UnaryExecNode

## What changes were proposed in this pull request?

Making EventTimeWatermarkExec explicitly UnaryExecNode

/cc @tdas @zsxwing 

## How was this patch tested?

Local build.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jaceklaskowski/spark 
EventTimeWatermarkExec-UnaryExecNode

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18509.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18509


commit 2cc9fdca381f50cb405bf5fbcaa7229652749a83
Author: Jacek Laskowski 
Date:   2017-07-03T07:07:12Z

[SS][MINOR] Make EventTimeWatermarkExec explicitly UnaryExecNode




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18509: [SS][MINOR] Make EventTimeWatermarkExec explicitly Unary...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18509
  
**[Test build #79081 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79081/testReport)**
 for PR 18509 at commit 
[`2cc9fdc`](https://github.com/apache/spark/commit/2cc9fdca381f50cb405bf5fbcaa7229652749a83).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18507
  
**[Test build #79082 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79082/testReport)**
 for PR 18507 at commit 
[`9788b19`](https://github.com/apache/spark/commit/9788b19d06800cce243a79acc189c3424912f393).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18481
  
**[Test build #79083 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79083/testReport)**
 for PR 18481 at commit 
[`6d5adf0`](https://github.com/apache/spark/commit/6d5adf0f10338168943542e8f2177761d2140438).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17227: [SPARK-19507][PySpark][SQL] Show field name in _verify_t...

2017-07-03 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/17227
  
@dgingrich, I tried to address my comments at my best here - 
https://github.com/dgingrich/spark/pull/1. Could you review that change and 
merge it if it looks good to you so that the change is merged into this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

2017-07-03 Thread maropu

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/17758
  
Who calls `DDLCommand.validate`? Do you assume a new analyzer rule like 
`DDLCommandAnalysis`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...

2017-07-03 Thread heary-cao

Github user heary-cao commented on the issue:

https://github.com/apache/spark/pull/18174
  
retest this please



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18159
  
**[Test build #79084 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79084/testReport)**
 for PR 18159 at commit 
[`8380023`](https://github.com/apache/spark/commit/8380023ab96fa6febe19918da9284fe14aea1f53).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18491: [SPARK-21268][MLLIB] Move center calculations to a distr...

Github user MLnick commented on the issue:

https://github.com/apache/spark/pull/18491
  
Is there any real performance benefit here? Doesn't seem like it unless you 
have really large `k`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18159
  
**[Test build #79085 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79085/testReport)**
 for PR 18159 at commit 
[`dfaa4f1`](https://github.com/apache/spark/commit/dfaa4f11f35e2615ce627f29b1a60ab27dfd5477).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18481
  
**[Test build #79083 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79083/testReport)**
 for PR 18481 at commit 
[`6d5adf0`](https://github.com/apache/spark/commit/6d5adf0f10338168943542e8f2177761d2140438).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18481
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18481: [SPARK-20889][SparkR] Grouped documentation for WINDOW c...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18481
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79083/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18510: [SPARK-21284][SQL] rename SessionCatalog.register...

GitHub user cloud-fan opened a pull request:

https://github.com/apache/spark/pull/18510

[SPARK-21284][SQL] rename SessionCatalog.registerFunction parameter name

## What changes were proposed in this pull request?

Looking at the code in `SessionCatalog.registerFunction`, the parameter 
`ignoreIfExists` has a wrong name. When `ignoreIfExists` is true, we will 
override the function if it already exists. So `overrideIfExists` should be the 
corrected name.

## How was this patch tested?

N/A

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cloud-fan/spark minor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18510.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18510


commit a518bdffb951c93418e2e4ea3fd4734255808ead
Author: Wenchen Fan 
Date:   2017-07-03T07:52:58Z

rename SessionCatalog.registerFunction parameter name




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18510: [SPARK-21284][SQL] rename SessionCatalog.registerFunctio...

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18510
  
cc @gatorsmile 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18510: [SPARK-21284][SQL] rename SessionCatalog.register...

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/18510#discussion_r125228460
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -1104,10 +1104,10 @@ class SessionCatalog(
*/
   def registerFunction(
--- End diff --

@ouyangxiaochen , you can call this method in 
https://github.com/apache/spark/pull/17681/files#diff-d2a203f08c862bd762e6740c16e972f7R84


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18510: [SPARK-21284][SQL] rename SessionCatalog.registerFunctio...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18510
  
**[Test build #79086 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79086/testReport)**
 for PR 18510 at commit 
[`a518bdf`](https://github.com/apache/spark/commit/a518bdffb951c93418e2e4ea3fd4734255808ead).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18510: [SPARK-21284][SQL] rename SessionCatalog.registerFunctio...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18510
  
**[Test build #79086 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79086/testReport)**
 for PR 18510 at commit 
[`a518bdf`](https://github.com/apache/spark/commit/a518bdffb951c93418e2e4ea3fd4734255808ead).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18510: [SPARK-21284][SQL] rename SessionCatalog.registerFunctio...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18510
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18510: [SPARK-21284][SQL] rename SessionCatalog.registerFunctio...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18510
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79086/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/17758
  
Actually after more thoughts, I think it's ok to do the validation in 
`RunnableCommand.run`, as it's still done in the driver side.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18413: [SPARK-21205][SQL] pmod(number, 0) should be null.

2017-07-03 Thread wangyum

Github user wangyum commented on the issue:

https://github.com/apache/spark/pull/18413
  
Jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18413: [SPARK-21205][SQL] pmod(number, 0) should be null.

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18413
  
**[Test build #79087 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79087/testReport)**
 for PR 18413 at commit 
[`da037c8`](https://github.com/apache/spark/commit/da037c810a8c121d7075b741478419ffb77202d8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18507
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18474: [SPARK-21235][TESTS] UTest should clear temp results whe...

2017-07-03 Thread wangjiaochun

Github user wangjiaochun commented on the issue:

https://github.com/apache/spark/pull/18474
  
1.  Test environment and test methodï¼IDEA project direct Run 
BlockManagerSuite.scala.
2.  I test this case again use step through,find this case 
encryptionTest("on-disk storage") Runs a test twice, if SparkConf object with 
encryption off(false), the disk blocks will clear. if encryption is on(set 
ture), disk blocks not clearã


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18507: [SPARK-21283][core]FileOutputStream should be cre...

2017-07-03 Thread jiangxb1987

Github user jiangxb1987 commented on a diff in the pull request:

https://github.com/apache/spark/pull/18507#discussion_r125237381
  
--- Diff: core/src/test/scala/org/apache/spark/util/FileAppenderSuite.scala 
---
@@ -52,10 +52,12 @@ class FileAppenderSuite extends SparkFunSuite with 
BeforeAndAfter with Logging {
   test("basic file appender") {
 val testString = (1 to 1000).mkString(", ")
 val inputStream = new 
ByteArrayInputStream(testString.getBytes(StandardCharsets.UTF_8))
+val header = "Add header"
--- End diff --

Please add comment before this to explain the change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18511: [SPARK-21286][Test] Modified a unit test

2017-07-03 Thread Geek-He

GitHub user Geek-He opened a pull request:

https://github.com/apache/spark/pull/18511

[SPARK-21286][Test] Modified a unit test

## What changes were proposed in this pull request?
The old unit test not effect


## How was this patch tested?
unit test



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Geek-He/spark dev_0703

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18511.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18511


commit 5c4279ddca49f36635658c8fbe8b37ecb31ecca4
Author: he.qiao 
Date:   2017-07-03T08:12:55Z

ä¿®æ¹ä¸å¤ååæµè¯éè¯¯




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18510: [SPARK-21284][SQL] rename SessionCatalog.register...

2017-07-03 Thread viirya

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/18510#discussion_r125238823
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala 
---
@@ -60,7 +60,7 @@ case class CreateFunctionCommand(
   // We first load resources and then put the builder in the function 
registry.
   // Please note that it is allowed to overwrite an existing temp 
function.
   catalog.loadFunctionResources(resources)
-  catalog.registerFunction(func, ignoreIfExists = false)
+  catalog.registerFunction(func, overrideIfExists = false)
--- End diff --

This looks a bit weird. The above comment says `Please note that it is 
allowed to overwrite an existing temp function.`. But we don't override it if 
it exists.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18511: [SPARK-21286][Test] Modified a unit test

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18511
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17985: Add "full_outer" name to join types

2017-07-03 Thread BartekH

Github user BartekH commented on the issue:

https://github.com/apache/spark/pull/17985
  
It has failed again. It's not my fault, please retest it one more time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18484: [SPARK-21264][PYTHON] Call cross join path in joi...

2017-07-03 Thread HyukjinKwon

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/18484#discussion_r125240048
  
--- Diff: python/pyspark/sql/tests.py ---
@@ -2021,6 +2021,11 @@ def test_toDF_with_schema_string(self):
 self.assertEqual(df.schema.simpleString(), "struct")
 self.assertEqual(df.collect(), [Row(key=i) for i in range(100)])
 
+def test_join_without_on(self):
+self.assertRaises(
+AnalysisException,
+lambda: self.spark.range(1).join(self.spark.range(1), 
how="inner").collect())
--- End diff --

Sure. Thank you.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18484: [SPARK-21264][PYTHON] Call cross join path in join witho...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18484
  
**[Test build #79088 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79088/testReport)**
 for PR 18484 at commit 
[`48a92d1`](https://github.com/apache/spark/commit/48a92d1166e9ab13ef8926885b12c5e513148115).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18499: [SPARK-21176][WEB UI] Use a single ProxyServlet to proxy...

2017-07-03 Thread jiangxb1987

Github user jiangxb1987 commented on the issue:

https://github.com/apache/spark/pull/18499
  
cc @gurvindersingh Could you look at this please?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18308: [SPARK-21099][Spark Core] INFO Log Message Using Incorre...

2017-07-03 Thread jerryshao

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/18308
  
Is this the final modified code @ihazem , why do you check 
`hasCachedBlocks` both inside and outside of logInfo statement? Also the code 
is too long.

Can you please at least do a round of self-review before pushing the 
changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18509: [SS][MINOR] Make EventTimeWatermarkExec explicitly Unary...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18509
  
**[Test build #79081 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79081/testReport)**
 for PR 18509 at commit 
[`2cc9fdc`](https://github.com/apache/spark/commit/2cc9fdca381f50cb405bf5fbcaa7229652749a83).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18509: [SS][MINOR] Make EventTimeWatermarkExec explicitly Unary...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18509
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18509: [SS][MINOR] Make EventTimeWatermarkExec explicitly Unary...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18509
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79081/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18484: [SPARK-21264][PYTHON] Call cross join path in join witho...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18484
  
**[Test build #79088 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79088/testReport)**
 for PR 18484 at commit 
[`48a92d1`](https://github.com/apache/spark/commit/48a92d1166e9ab13ef8926885b12c5e513148115).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18484: [SPARK-21264][PYTHON] Call cross join path in join witho...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18484
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79088/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18512: [SPARK-20946][SQL] simplify the config setting lo...

GitHub user dongjoon-hyun opened a pull request:

https://github.com/apache/spark/pull/18512

[SPARK-20946][SQL] simplify the config setting logic in 
SparkSession.getOrCreate

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)

Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dongjoon-hyun/spark SPARK-20946

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18512.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18512


commit fe7bdad1530dadefb0283fac597137a3db2538c1
Author: Wenchen Fan 
Date:   2017-06-01T05:17:05Z

simplify the config setting logic in SparkSession.getOrCreate

commit b84806b78c96564fb7f093b679979ef4cd733880
Author: Wenchen Fan 
Date:   2017-06-02T03:41:42Z

address comments

commit 705ccc007817bdcb9b87ef0141b16ae03e760ef6
Author: Dongjoon Hyun 
Date:   2017-07-03T09:30:31Z

fix




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18484: [SPARK-21264][PYTHON] Call cross join path in join witho...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18484
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18512: [SPARK-20946][SQL] simplify the config setting logic in ...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18512
  
**[Test build #79089 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79089/testReport)**
 for PR 18512 at commit 
[`705ccc0`](https://github.com/apache/spark/commit/705ccc007817bdcb9b87ef0141b16ae03e760ef6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18441: [SPARK-21137][CORE] Spark reads many small files ...

2017-07-03 Thread kiszk

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/18441#discussion_r125247939
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala ---
@@ -35,8 +36,12 @@ private[spark] class BinaryFileRDD[T](
   extends NewHadoopRDD[String, T](sc, inputFormatClass, keyClass, 
valueClass, conf) {
 
   override def getPartitions: Array[Partition] = {
-val inputFormat = inputFormatClass.newInstance
 val conf = getConf
+// setMinPartitions below will call FileInputFormat.listStatus(), 
which can be quite slow when
+// traversing a large number of directories and files. Parallelize it.
+conf.setIfUnset(FileInputFormat.LIST_STATUS_NUM_THREADS,
+  Runtime.getRuntime.availableProcessors().toString)
--- End diff --

+1. 
This code aggressively tries to use all of the cores on all of the Spark 
instances on a node. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18507
  
**[Test build #79090 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79090/testReport)**
 for PR 18507 at commit 
[`139e16c`](https://github.com/apache/spark/commit/139e16c04f7532101c8fd44983fe42306c7bb5fc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17681: [SPARK-20383][SQL] Supporting Create [temporary] ...

2017-07-03 Thread ouyangxiaochen

Github user ouyangxiaochen commented on a diff in the pull request:

https://github.com/apache/spark/pull/17681#discussion_r125250502
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -1056,6 +1056,27 @@ class SessionCatalog(
   }
 
   /**
+   * overwirte a metastore function in the database specified in 
`funcDefinition`..
+   * If no database is specified, assume the function is in the current 
database.
+   */
+  def alterFunction(funcDefinition: CatalogFunction): Unit = {
+val db = 
formatDatabaseName(funcDefinition.identifier.database.getOrElse(getCurrentDatabase))
+requireDbExists(db)
+val identifier = 
FunctionIdentifier(funcDefinition.identifier.funcName, Some(db))
+val newFuncDefinition = funcDefinition.copy(identifier = identifier)
+if (functionExists(identifier)) {
+  if (functionRegistry.functionExists(identifier)) {
+// If we have loaded this function into the FunctionRegistry,
+// also drop it from there.
+// For a permanent function, because we loaded it to the 
FunctionRegistry
+// when it's first used, we also need to drop it from the 
FunctionRegistry.
+functionRegistry.dropFunction(identifier)
+  }
+  externalCatalog.alterFunction(db, newFuncDefinition)
+}
--- End diff --

yes


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18507: [SPARK-21283][core]FileOutputStream should be cre...

2017-07-03 Thread 10110346

Github user 10110346 commented on a diff in the pull request:

https://github.com/apache/spark/pull/18507#discussion_r125250432
  
--- Diff: core/src/test/scala/org/apache/spark/util/FileAppenderSuite.scala 
---
@@ -52,10 +52,12 @@ class FileAppenderSuite extends SparkFunSuite with 
BeforeAndAfter with Logging {
   test("basic file appender") {
 val testString = (1 to 1000).mkString(", ")
 val inputStream = new 
ByteArrayInputStream(testString.getBytes(StandardCharsets.UTF_8))
+val header = "Add header"
--- End diff --

ok,thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18507
  
**[Test build #79091 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79091/testReport)**
 for PR 18507 at commit 
[`a22951f`](https://github.com/apache/spark/commit/a22951f341901381480c04eae035e06bf9465bbe).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18513: [SPARK-13969][ML] Add FeatureHasher transformer

GitHub user MLnick opened a pull request:

https://github.com/apache/spark/pull/18513

[SPARK-13969][ML] Add FeatureHasher transformer

This PR adds a `FeatureHasher` transformer, modeled on 
[scikit-learn](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html)
 and [Vowpal 
wabbit](https://github.com/JohnLangford/vowpal_wabbit/wiki/Feature-Hashing-and-Extraction).

The transformer operates on multiple input columns in one pass. Current 
behavior is:
* for numerical columns, the values are assumed to be real values and the 
feature index is `hash(columnName)` while feature value is `feature_value`
* for string columns, the values are assumed to be categorical and the 
feature index is `hash(column_name=feature_value)`, while feature value is `1.0`
* For hash collisions, feature values will be summed
* `null` (missing) values are ignored

The following dataframe illustrates the basic semantics:
```

+---+--+-+-+--+-+
|int|double|float|stringNum|string|features 
|

+---+--+-+-+--+-+
|3  |4.0   |5.0  |1|foo   
|(16,[0,8,11,12,15],[5.0,3.0,1.0,4.0,1.0])|
|6  |7.0   |8.0  |2|bar   
|(16,[0,8,11,12,15],[8.0,6.0,1.0,7.0,1.0])|

+---+--+-+-+--+-+
```

## How was this patch tested?

New unit tests and manual experiments.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MLnick/spark FeatureHasher

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18513.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18513


commit 6ab19a963f35de29af0a6b7b1598d5add78f200a
Author: Nick Pentreath 
Date:   2016-08-23T10:29:06Z

initial WIP

commit ebd2cbf3467f26121c602f7c77c2018253cbdf18
Author: Nick Pentreath 
Date:   2017-02-01T10:43:07Z

Further work

commit ba255bfda792d58aaded892e49c6cf48f0391159
Author: Nick Pentreath 
Date:   2017-06-22T10:52:12Z

Clean up

commit 0be1e6572110d7d550f69fd86d3dd4e96660fde6
Author: Nick Pentreath 
Date:   2017-06-22T10:52:37Z

Add tests

commit 2f3ea21e2e1835d7218e8c7bd096cc0787ed595c
Author: Nick Pentreath 
Date:   2017-06-22T13:08:26Z

Copy, save/load, clean up

commit 7d678fbf5f88d377b79153212a3e0a2596039b17
Author: Nick Pentreath 
Date:   2017-06-26T12:38:02Z

Move numFeatures to HasNumFeatures shared trait

commit 60572776de80ebcf1782c3d7def749557c8bec61
Author: Nick Pentreath 
Date:   2017-07-03T07:18:25Z

Update shared params from codegen run

commit 9edb3bda8cbc4e00f05b91718249edf2750fc028
Author: Nick Pentreath 
Date:   2017-07-03T09:32:32Z

Update tests. Null values ignored in feature hashing.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18491: [SPARK-21268][MLLIB] Move center calculations to a distr...

2017-07-03 Thread srowen

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/18491
  
I doubt it makes much difference, yeah. It pushes a little computation off 
the driver, and computation that happens on each iteration. It seemed like a 
tiny win that didn't complicate things.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17227: [SPARK-19507][PySpark][SQL] Show field name in _verify_t...

2017-07-03 Thread HyukjinKwon

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/17227
  
Wait ... this is possibly a hot path that requires row-per-operation ... 
@ueshin and @dgingrich, I think we should re-write this to avoid 
record-per-type dispatch ... For me, I wouldn't mind if we go and merge my 
approach, dgingrich#1 here for now. I will make a follow up right after it gets 
merged.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18512: [SPARK-20946][SQL] Simplify the config setting logic in ...

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/18512
  
Hi, @cloud-fan and @gatorsmile .
Could you review this PR?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18513
  
**[Test build #79092 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79092/testReport)**
 for PR 18513 at commit 
[`9edb3bd`](https://github.com/apache/spark/commit/9edb3bda8cbc4e00f05b91718249edf2750fc028).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18512: [SPARK-20946][SQL] Simplify the config setting lo...

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/18512#discussion_r125252781
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -904,22 +906,14 @@ object SparkSession {
 
 // No active nor global default session. Create a new one.
 val sparkContext = userSuppliedContext.getOrElse {
-  // set app name if not given
-  val randomAppName = java.util.UUID.randomUUID().toString
   val sparkConf = new SparkConf()
   options.foreach { case (k, v) => sparkConf.set(k, v) }
--- End diff --

This line is changed. `options` are copied into new SparkConf.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17681: [SPARK-20383][SQL] Supporting Create [temporary] ...

2017-07-03 Thread ouyangxiaochen

Github user ouyangxiaochen commented on a diff in the pull request:

https://github.com/apache/spark/pull/17681#discussion_r125253313
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
 ---
@@ -590,6 +590,14 @@ class InMemoryCatalog(
 catalog(db).functions.remove(funcName)
   }
 
+  override protected def doAlterFunction(db: String, func: 
CatalogFunction): Unit = synchronized {
+requireDbExists(db)
+requireFunctionExists(db, func.identifier.funcName)
+catalog(db).functions.remove(func.identifier.funcName)
+requireFunctionNotExists(db, func.identifier.funcName)
--- End diff --

Ah, there is really no need to do this. OK. remove it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18507
  
**[Test build #79082 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79082/testReport)**
 for PR 18507 at commit 
[`9788b19`](https://github.com/apache/spark/commit/9788b19d06800cce243a79acc189c3424912f393).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18507
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79082/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18507
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18513
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79092/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18513
  
**[Test build #79092 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79092/testReport)**
 for PR 18513 at commit 
[`9edb3bd`](https://github.com/apache/spark/commit/9edb3bda8cbc4e00f05b91718249edf2750fc028).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18513
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer

Github user MLnick commented on the issue:

https://github.com/apache/spark/pull/18513

**Note 1**: this is distinct from `HashingTF` which handles vectorizing
text to term frequencies (analogous to
[HashingVectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.HashingVectorizer.html)).
Thie feature hasher _could_ be extended to also handle `Seq[String]` input
columns. But I feel it conflates concerns - e.g. `HashingTF` handles min term
frequencies, binarization etc.

However we could later add basic support for `Seq[String]` columns - this
would handle raw text in a similar way to Vowpal Wabbit, i.e. it all gets
hashed into one feature vector (can be combined with namespaces later).

**Note 2**: some potential follow ups:
* support specifying categorical columns explicitly. This would be to allow
forcing some columns that are in numerical format to be treated as categorical.
Strings would still be treated as categorical.
* support using the sign of hashed value as sign of feature value, and then
support `non_negative` param (see
[scikit-learn](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html))
* support feature namespaces and feature interactions similar to [Vowpal
Wabbit](https://github.com/JohnLangford/vowpal_wabbit/wiki/Feature-interactions)
(see [here](https://gist.github.com/luoq/b4c374b5cbabe3ae76ffacdac22750af) for
an outline of the code used). This could provide an efficient and scalable form
of `PolynomialExpansion`.

cc @srowen @jkbradley @sethah @hhbyyh @yanboliang @BryanCutler @holdenk

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer

Github user MLnick commented on the issue:

https://github.com/apache/spark/pull/18513
  
I've moved `HashingTF` `numFeatures` param to `sharedParams` which results 
in the MiMa failure since it would now be marked `final`. Can't quite recall 
what we've done previously in this case - whether we accept that it breaks user 
code, but that in most cases users should not have really been extending or 
overriding these params. Or leave it as is.

I'm ok with the latter - `numFeatures` is not really that necessary to be a 
shared param. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18491: [SPARK-21268][MLLIB] Move center calculations to ...

Github user MLnick commented on a diff in the pull request:

https://github.com/apache/spark/pull/18491#discussion_r125256928
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala ---
@@ -292,15 +292,16 @@ class KMeans private (
   }.reduceByKey { case ((sum1, count1), (sum2, count2)) =>
 axpy(1.0, sum2, sum1)
 (sum1, count1 + count2)
+  }.mapValues { case (sum, count) =>
+scal(1.0 / count, sum)
+new VectorWithNorm(sum)
   }.collectAsMap()
 
   bcCenters.destroy(blocking = false)
 
   // Update the cluster centers and costs
   converged = true
-  totalContribs.foreach { case (j, (sum, count)) =>
-scal(1.0 / count, sum)
-val newCenter = new VectorWithNorm(sum)
+  totalContribs.foreach { case (j, newCenter) =>
--- End diff --

perhaps the variable name should change from `totalContribs` to 
`newCenters`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18491: [SPARK-21268][MLLIB] Move center calculations to a distr...

Github user MLnick commented on the issue:

https://github.com/apache/spark/pull/18491
  
Fair enough. LGTM with small comment about possibly changing the variable 
name?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18502: [SPARK-21278][PYSPARK][WIP] Upgrade to Py4J 0.10.5

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/18502
  
Retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18460: [SPARK-21247][SQL] Allow case-insensitive type equality ...

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/18460
  
Hi, @cloud-fan .
Could you review this, too?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18502: [SPARK-21278][PYSPARK][WIP] Upgrade to Py4J 0.10.5

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18502
  
**[Test build #79093 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79093/testReport)**
 for PR 18502 at commit 
[`f708dde`](https://github.com/apache/spark/commit/f708ddec38917867f9f13c7136ecef28c46af3a1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18159
  
**[Test build #79084 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79084/testReport)**
 for PR 18159 at commit 
[`8380023`](https://github.com/apache/spark/commit/8380023ab96fa6febe19918da9284fe14aea1f53).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18510: [SPARK-21284][SQL] rename SessionCatalog.register...

2017-07-03 Thread ouyangxiaochen

Github user ouyangxiaochen commented on a diff in the pull request:

https://github.com/apache/spark/pull/18510#discussion_r125258324
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ---
@@ -1104,10 +1104,10 @@ class SessionCatalog(
*/
   def registerFunction(
--- End diff --

OK, I will do this change after your PR is merged into master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18159
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18159
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79084/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18159
  
**[Test build #79085 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79085/testReport)**
 for PR 18159 at commit 
[`dfaa4f1`](https://github.com/apache/spark/commit/dfaa4f11f35e2615ce627f29b1a60ab27dfd5477).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18159: [SPARK-20703][SQL] Associate metrics with data writes on...