[GitHub] spark issue #19037: SPARK-21825: change Set(5, 3) to Seq(5, 3, 5) & Set(2, 3...

[iamhumanbeing]

Github user iamhumanbeing commented on the issue:

https://github.com/apache/spark/pull/19037
  
ok. I will close this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19037: SPARK-21825: change Set(5, 3) to Seq(5, 3, 5) & S...

[iamhumanbeing]

Github user iamhumanbeing closed the pull request at:

https://github.com/apache/spark/pull/19037


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19049: [WEB-UI]Add the 'master' column to identify the type of ...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19049
  
When would the history server receive events from clusters with different 
resource managers?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19029: [SPARK-21818][ML][MLLIB] Fix bug of MultivariateOnlineSu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19029
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81118/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19029: [SPARK-21818][ML][MLLIB] Fix bug of MultivariateOnlineSu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19029
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19029: [SPARK-21818][ML][MLLIB] Fix bug of MultivariateOnlineSu...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19029
  
**[Test build #81118 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81118/testReport)**
 for PR 19029 at commit 
[`c24292c`](https://github.com/apache/spark/commit/c24292ccad700d39892a576390cff2559c4f3b9a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19049: [WEB-UI]Add the 'master' column to identify the type of ...

[guoxiaolongzte]

Github user guoxiaolongzte commented on the issue:

https://github.com/apache/spark/pull/19049
  
Sorry，I do not understand what you mean.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19049: [WEB-UI]Add the 'master' column to identify the type of ...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19049
  
This still isn't addressing the point: when would they not _all_ be YARN?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19049: [WEB-UI]Add the 'master' column to identify the type of ...

[guoxiaolongzte]

Github user guoxiaolongzte commented on the issue:

https://github.com/apache/spark/pull/19049
  
For example, I would like to view some of the applications of the resource 
schedule of yarn, I can sort the 'master' column.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19049: [WEB-UI]Add the 'master' column to identify the type of ...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19049
  
What does knowing the resource scheduler help? but my point is, it's 
already known in general.
A history server most certainly collects history from different users' 
apps, but not from different resource managers. It's possible to mix history 
logs from different ones but it's exceptional.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19049: [WEB-UI]Add the 'master' column to identify the type of ...

[guoxiaolongzte]

Github user guoxiaolongzte commented on the issue:

https://github.com/apache/spark/pull/19049
  
1.It is more convenient for the observer to more easily know the resource 
scheduling mode of the current spark application.

2.For the 'Spark User' column. It would be identical in almost all use 
cases.But it still very important.So I think the 'master' column is equally 
important, This shows the content more clearly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18488: [SPARK-21255][SQL][WIP] Fixed NPE when creating e...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18488


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18488: [SPARK-21255][SQL][WIP] Fixed NPE when creating encoder ...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/18488
  
Merged to master


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19051: [MINOR][BUILD] Fix build warnings and Java lint errors

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19051
  
**[Test build #81120 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81120/testReport)**
 for PR 19051 at commit 
[`29fd7d7`](https://github.com/apache/spark/commit/29fd7d7ec0425427b2e1fefea51deb882a445cc9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19051: [MINOR][BUILD] Fix build warnings and Java lint e...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/19051#discussion_r135190132
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/tuning/CrossValidatorSuite.scala ---
@@ -187,14 +187,13 @@ class CrossValidatorSuite
 cv2.getEstimator match {
   case ova2: OneVsRest =>
 assert(ova.uid === ova2.uid)
-val classifier = ova2.getClassifier
-classifier match {
+ova2.getClassifier match {
--- End diff --

Another weird one where the construct is equivalent but ended up causing 
scalac to construe this as an existential type


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19051: [MINOR][BUILD] Fix build warnings and Java lint e...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/19051#discussion_r135190085
  
--- Diff: 
common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStoreIterator.java 
---
@@ -31,7 +32,7 @@
  * 
  */
 @Private
-public interface KVStoreIterator extends Iterator, AutoCloseable {
+public interface KVStoreIterator extends Iterator, Closeable {
--- End diff --

Weird one, but it warns on extending `AutoCloseable` without explicitly 
narrowing the exceptions that `close()` throws to not include 
`InterruptedException`. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19051: [MINOR][BUILD] Fix build warnings and Java lint e...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/19051#discussion_r135190194
  
--- Diff: pom.xml ---
@@ -2058,7 +2058,7 @@
   ${java.version}
   -target
   ${java.version}
-  -Xlint:all,-serial,-path
+  -Xlint:all,-serial,-path,-try
--- End diff --

Avoids many harmless warnings about not referencing the variable define in 
try-with-resources


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19051: [MINOR][BUILD] Fix build warnings and Java lint e...

[srowen]

GitHub user srowen opened a pull request:

https://github.com/apache/spark/pull/19051

[MINOR][BUILD] Fix build warnings and Java lint errors

## What changes were proposed in this pull request?

Fix build warnings and Java lint errors. This just helps a bit in 
evaluating (new) warnings in another PR I have open.

## How was this patch tested?

Existing tests

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/srowen/spark JavaWarnings

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19051.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19051


commit 29fd7d7ec0425427b2e1fefea51deb882a445cc9
Author: Sean Owen 
Date:   2017-08-25T06:18:34Z

Fix build warnings and Java lint errors




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19049: [WEB-UI]Add the 'master' column to identify the type of ...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19049
  
What's the value of this column? In just about all cases it will have the 
same value in every row, like 'yarn'. Even if not, why is it useful?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19049: [WEB-UI]Add the 'master' column to identify the type of ...

[guoxiaolongzte]

Github user guoxiaolongzte commented on the issue:

https://github.com/apache/spark/pull/19049
  
Sorry, I did not understand what you meant.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15435
  
**[Test build #81119 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81119/testReport)**
 for PR 15435 at commit 
[`1395de2`](https://github.com/apache/spark/commit/1395de2c5ab85ea76c690b786c55d459180f1b44).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...

[WeichenXu123]

Github user WeichenXu123 commented on the issue:

https://github.com/apache/spark/pull/15435
  
Jenkins test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19049: [WEB-UI]Add the 'master' column to identify the type of ...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19049
  
Why? It would be identical in almost all use cases. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19037: SPARK-21825: change Set(5, 3) to Seq(5, 3, 5) & Set(2, 3...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19037
  
Same, why do you think the test can or should be more specific? There is no 
motivation


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18991: [SPARK-21783][SQL] Turn on ORC filter push-down by defau...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18991
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18991: [SPARK-21783][SQL] Turn on ORC filter push-down by defau...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18991
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81114/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18991: [SPARK-21783][SQL] Turn on ORC filter push-down by defau...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18991
  
**[Test build #81114 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81114/testReport)**
 for PR 18991 at commit 
[`2bc2b17`](https://github.com/apache/spark/commit/2bc2b17aba5231c6ac3e0ab7c830acc56790df9f).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17357: [SPARK-20025][CORE] Ignore SPARK_LOCAL* env, whil...

[ScrapCodes]

Github user ScrapCodes commented on a diff in the pull request:

https://github.com/apache/spark/pull/17357#discussion_r135187410
  
--- Diff: 
core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala ---
@@ -139,7 +139,9 @@ private[rest] class StandaloneSubmitRequestServlet(
 val driverExtraLibraryPath = 
sparkProperties.get("spark.driver.extraLibraryPath")
 val superviseDriver = sparkProperties.get("spark.driver.supervise")
 val appArgs = request.appArgs
-val environmentVariables = request.environmentVariables
+// Filter SPARK_LOCAL environment variables from being set on the 
remote system.
+val environmentVariables =
+  
request.environmentVariables.filterNot(_._1.startsWith("SPARK_LOCAL"))
--- End diff --

Alright, I will check how it is used across the project. 
Just noted, In `LocalDirsSuite`, comments in `test("SPARK_LOCAL_DIRS 
override also affects driver") ` seems to corroborate with my intentions here.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19029: [SPARK-21818][ML][MLLIB] Fix bug of MultivariateOnlineSu...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19029
  
**[Test build #81118 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81118/testReport)**
 for PR 19029 at commit 
[`c24292c`](https://github.com/apache/spark/commit/c24292ccad700d39892a576390cff2559c4f3b9a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18956: [SPARK-21726][SQL] Check for structural integrity of the...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/18956
  
`RewritePredicateSubquery` fails structural integrity check because it can 
produce `Join` with conflicting attributes in its left and right plans.

I submitted #19050 to fix it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19050: [SPARK-21835][SQL][WIP] RewritePredicateSubquery should ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19050
  
**[Test build #81117 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81117/testReport)**
 for PR 19050 at commit 
[`edb6271`](https://github.com/apache/spark/commit/edb6271807a1ad8704f5d1715592af79a7c0b63a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19050: [SPARK-21835][SQL][WIP] RewritePredicateSubquery ...

[viirya]

GitHub user viirya opened a pull request:

https://github.com/apache/spark/pull/19050

[SPARK-21835][SQL][WIP] RewritePredicateSubquery should not produce 
unresolved query plans

## What changes were proposed in this pull request?

`RewritePredicateSubquery` rewrites correlated subquery to join operations. 
`RewritePredicateSubquery` can produce unresolved query plans due to 
conflicting attributes. The unresolved query plans will fail structural 
integrity check. We should deduplicate the conflicting attributes in the 
rewritten join. 

## How was this patch tested?

WIP to add test later

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/viirya/spark-1 SPARK-21835

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19050.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19050


commit edb6271807a1ad8704f5d1715592af79a7c0b63a
Author: Liang-Chi Hsieh 
Date:   2017-08-25T05:40:17Z

Dedup join after RewritePredicateSubquery.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19029: [SPARK-21818][ML][MLLIB] Fix bug of MultivariateO...

[WeichenXu123]

Github user WeichenXu123 commented on a diff in the pull request:

https://github.com/apache/spark/pull/19029#discussion_r135186430
  
--- Diff: mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala ---
@@ -438,6 +438,10 @@ private[ml] object SummaryBuilderImpl extends Logging {
 while (i < len) {
   realVariance(i) = (currM2n(i) + deltaMean(i) * deltaMean(i) * 
weightSum(i) *
 (totalWeightSum - weightSum(i)) / totalWeightSum) / denominator
+  // Because of numerical error, it is possible to get negative 
real variance
+  if (realVariance(i) < 0.0) {
--- End diff --

Hmm.. `WeightedLeastSquares` use another way to compute variance `Var(X) = 
E(X^2) - E(X)^2`. But it seems also possible to have this problem.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19049: [WEB-UI]Add the 'master' column to identify the type of ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19049
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19049: [WEB-UI]Add the 'master' column to identify the t...

[guoxiaolongzte]

GitHub user guoxiaolongzte opened a pull request:

https://github.com/apache/spark/pull/19049

[WEB-UI]Add the 'master' column to identify the type of resource management 
for the spark job, in the history server web ui.

## What changes were proposed in this pull request?
Add the 'master' column to identify the type of resource management for the 
spark job, in the history server web ui.

such as:
      --master yarn, which means that the resource type of the blurs 
job is yarn.
      --master ip: 7077, that the spark resource management type is 
standalone.
      --master local, which indicates that the resource type of the 
blurs job is local.

Screenshots are as follows:

![1](https://user-images.githubusercontent.com/26266482/29700661-329e3516-8999-11e7-8ebc-479b7986c489.png)

(Please fill in changes proposed in this fix)

## How was this patch tested?

manual tests
(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)

Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guoxiaolongzte/spark historyserver_add_master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19049.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19049


commit bd8ab9c830eb032561ae14876ab905a46e4b2e32
Author: guoxiaolong 
Date:   2017-08-25T05:22:25Z

[WEB-UI]Add the 'master' column to identify the type of resource management 
for the spark job, in the history server web ui.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Expression...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19044
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Expression...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19044
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81113/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Expression...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19044
  
**[Test build #81113 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81113/testReport)**
 for PR 19044 at commit 
[`e042621`](https://github.com/apache/spark/commit/e042621003edca43501eef9e6321f2448f343cdd).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18730: [SPARK-21527][CORE] Use buffer limit in order to use JAV...

[caneGuy]

Github user caneGuy commented on the issue:

https://github.com/apache/spark/pull/18730
  
@cloud-fan can we retest this?Thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18730: [SPARK-21527][CORE] Use buffer limit in order to use JAV...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18730
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81112/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18730: [SPARK-21527][CORE] Use buffer limit in order to use JAV...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18730
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18730: [SPARK-21527][CORE] Use buffer limit in order to use JAV...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18730
  
**[Test build #81112 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81112/testReport)**
 for PR 18730 at commit 
[`14ca824`](https://github.com/apache/spark/commit/14ca824794ffd543aa169327e78de95f23b1102d).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15435
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15435
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/8/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19048: [SPARK-21834] Incorrect executor request in case of dyna...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19048
  
**[Test build #81116 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81116/testReport)**
 for PR 19048 at commit 
[`e000db3`](https://github.com/apache/spark/commit/e000db31f7ce620a004aa74841861abb4a0631af).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15435
  
**[Test build #8 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/8/testReport)**
 for PR 15435 at commit 
[`1395de2`](https://github.com/apache/spark/commit/1395de2c5ab85ea76c690b786c55d459180f1b44).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19048: [SPARK-21834] Incorrect executor request in case of dyna...

[sitalkedia]

Github user sitalkedia commented on the issue:

https://github.com/apache/spark/pull/19048
  
cc - @markhamstra , @sameeragarwal, @rxin, @vanzin,  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18953: [SPARK-20682][SQL] Update ORC data source based on Apach...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/18953
  
Hi, @cloud-fan .
Could you review this again when you have sometime?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18966: [SPARK-21751][SQL] CodeGeneraor.splitExpressions counts ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18966
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18966: [SPARK-21751][SQL] CodeGeneraor.splitExpressions counts ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18966
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81109/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18966: [SPARK-21751][SQL] CodeGeneraor.splitExpressions counts ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18966
  
**[Test build #81109 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81109/testReport)**
 for PR 18966 at commit 
[`ea9fea4`](https://github.com/apache/spark/commit/ea9fea48bf30dc53305fbe524a11dd1dee57ca82).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17886: [SPARK-13983][SQL] Fix HiveThriftServer2 can not get "--...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17886
  
**[Test build #81115 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81115/testReport)**
 for PR 17886 at commit 
[`e752d42`](https://github.com/apache/spark/commit/e752d423254e75b460e50c92afad4b63a7f60aea).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17886: [SPARK-13983][SQL] Fix HiveThriftServer2 can not get "--...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17886
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81115/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17886: [SPARK-13983][SQL] Fix HiveThriftServer2 can not get "--...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17886
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19048: [SPARK-21834] Incorrect executor request in case of dyna...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19048
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81110/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19048: [SPARK-21834] Incorrect executor request in case of dyna...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19048
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19048: [SPARK-21834] Incorrect executor request in case of dyna...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19048
  
**[Test build #81110 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81110/testReport)**
 for PR 19048 at commit 
[`e30bbac`](https://github.com/apache/spark/commit/e30bbac45d346e96f1004578234324cb3426b3b3).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18991: [SPARK-21783][SQL] Turn on ORC filter push-down by defau...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/18991
  
Hi, @cloud-fan, @gatorsmile , @sameeragarwal , @rxin , @mridulm .
Could you reivew this one liner PR about ORC PPD configuration?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18991: [SPARK-21783][SQL] Turn on ORC filter push-down by defau...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18991
  
**[Test build #81114 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81114/testReport)**
 for PR 18991 at commit 
[`2bc2b17`](https://github.com/apache/spark/commit/2bc2b17aba5231c6ac3e0ab7c830acc56790df9f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17886: [SPARK-13983][SQL] Fix HiveThriftServer2 can not get "--...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17886
  
**[Test build #81115 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81115/testReport)**
 for PR 17886 at commit 
[`e752d42`](https://github.com/apache/spark/commit/e752d423254e75b460e50c92afad4b63a7f60aea).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18991: [SPARK-21783][SQL] Turn on ORC filter push-down by defau...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/18991
  
Retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Expression...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19044
  
**[Test build #81113 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81113/testReport)**
 for PR 19044 at commit 
[`e042621`](https://github.com/apache/spark/commit/e042621003edca43501eef9e6321f2448f343cdd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19034: [SPARK-21821][CORE]Support to force kill the CoarseGrain...

[jerryshao]

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/19034
  
Agree with @vanzin , if you really want to fix this issue I think you 
should find out the root cause and fix the code in Spark.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Expression...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/19044
  
Retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18315: [SPARK-21108] [ML] convert LinearSVC to aggregato...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18315


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18315: [SPARK-21108] [ML] convert LinearSVC to aggregator frame...

[yanboliang]

Github user yanboliang commented on the issue:

https://github.com/apache/spark/pull/18315
  
Merged into master. Thanks for all.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18315: [SPARK-21108] [ML] convert LinearSVC to aggregato...

[yanboliang]

Github user yanboliang commented on a diff in the pull request:

https://github.com/apache/spark/pull/18315#discussion_r135172322
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/optim/aggregator/HingeAggregatorSuite.scala
 ---
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.ml.optim.aggregator
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.ml.feature.Instance
+import org.apache.spark.ml.linalg.{BLAS, Vector, Vectors}
+import org.apache.spark.ml.util.TestingUtils._
+import org.apache.spark.mllib.util.MLlibTestSparkContext
+
+class HingeAggregatorSuite extends SparkFunSuite with 
MLlibTestSparkContext {
+
+  import DifferentiableLossAggregatorSuite.getClassificationSummarizers
+
+  @transient var instances: Array[Instance] = _
+  @transient var instancesConstantFeature: Array[Instance] = _
+
+  override def beforeAll(): Unit = {
+super.beforeAll()
+instances = Array(
+  Instance(0.0, 0.1, Vectors.dense(1.0, 2.0)),
+  Instance(1.0, 0.5, Vectors.dense(1.5, 1.0)),
+  Instance(0.0, 0.3, Vectors.dense(4.0, 0.5))
+)
+instancesConstantFeature = Array(
+  Instance(0.0, 0.1, Vectors.dense(1.0, 2.0)),
+  Instance(1.0, 0.5, Vectors.dense(1.0, 1.0)),
+  Instance(1.0, 0.3, Vectors.dense(1.0, 0.5)))
+  }
+
+   /** Get summary statistics for some data and create a new 
HingeAggregator. */
+  private def getNewAggregator(
+  instances: Array[Instance],
+  coefficients: Vector,
+  fitIntercept: Boolean): HingeAggregator = {
+val (featuresSummarizer, ySummarizer) =
+  
DifferentiableLossAggregatorSuite.getClassificationSummarizers(instances)
+val featuresStd = featuresSummarizer.variance.toArray.map(math.sqrt)
+val bcFeaturesStd = spark.sparkContext.broadcast(featuresStd)
+val bcCoefficients = spark.sparkContext.broadcast(coefficients)
+new HingeAggregator(bcFeaturesStd, fitIntercept)(bcCoefficients)
+  }
+
+  test("aggregator add method input size") {
+val coefArray = Array(1.0, 2.0)
+val interceptArray = Array(2.0)
--- End diff --

Okay.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19039: [SPARK-21829][CORE] Enable config to permanently blackli...

[jerryshao]

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/19039
  
The changes you made in `BlacklistTracker` seems break the design purpose 
of backlist. The blacklist in Spark as well as in MR/TEZ assumes bad 
nodes/executors will be back to normal in several hours, so it always has a 
timeout for blacklist.

In your case, the problem is not bad nodes/executors, it is that you don't 
what to start executors on some nodes (like slow nodes). This is more like a 
cluster manager problem rather than Spark problem. To summarize your problem, 
you want your Spark application runs on some specific nodes.

To solve your problem, for YARN you could use node label and Spark on YARN 
already support node label. You could google node label to know the details.

For standalone, simply you should not start worker on such nodes you don't 
want.

For Mesos I'm not sure, I guess it should also has similar approaches.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18730: [SPARK-21527][CORE] Use buffer limit in order to use JAV...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18730
  
**[Test build #81112 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81112/testReport)**
 for PR 18730 at commit 
[`14ca824`](https://github.com/apache/spark/commit/14ca824794ffd543aa169327e78de95f23b1102d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Expression...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19044
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81107/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Expression...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19044
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Expression...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19044
  
**[Test build #81107 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81107/testReport)**
 for PR 19044 at commit 
[`e042621`](https://github.com/apache/spark/commit/e042621003edca43501eef9e6321f2448f343cdd).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15435
  
**[Test build #8 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/8/testReport)**
 for PR 15435 at commit 
[`1395de2`](https://github.com/apache/spark/commit/1395de2c5ab85ea76c690b786c55d459180f1b44).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19046: [SPARK-18769][yarn] Limit resource requests based on RM'...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19046
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19046: [SPARK-18769][yarn] Limit resource requests based on RM'...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19046
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81108/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19046: [SPARK-18769][yarn] Limit resource requests based on RM'...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19046
  
**[Test build #81108 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81108/testReport)**
 for PR 19046 at commit 
[`03477f5`](https://github.com/apache/spark/commit/03477f5282ba00b40915bae32dc5bd48c946e27c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19048: [SPARK-21834] Incorrect executor request in case of dyna...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19048
  
**[Test build #81110 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81110/testReport)**
 for PR 19048 at commit 
[`e30bbac`](https://github.com/apache/spark/commit/e30bbac45d346e96f1004578234324cb3426b3b3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Expression...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/19044
  
Thank you for review, @gatorsmile !


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19048: [SPARK-21834] Incorrect executor request in case ...

[sitalkedia]

GitHub user sitalkedia opened a pull request:

https://github.com/apache/spark/pull/19048

[SPARK-21834] Incorrect executor request in case of dynamic allocation

## What changes were proposed in this pull request?

killExecutor api currently does not allow killing an executor without 
updating the total number of executors needed. In case of dynamic allocation is 
turned on and the allocator tries to kill an executor, the scheduler reduces 
the total number of executors needed ( see 
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L635)
 which is incorrect because the allocator already takes care of setting the 
required number of executors itself.

## How was this patch tested?

Ran a job on the cluster and made sure the executor request is correct


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sitalkedia/spark 
skedia/oss_fix_executor_allocation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19048.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19048


commit 120f3833c86776bd90c4e1db66846d7a9035f29a
Author: Sital Kedia 
Date:   2017-08-25T00:24:53Z

[SPARK-21834] Incorrect executor request in case of dynamic allocation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19031: [SPARK-21603][SQL][FOLLOW-UP] Use -1 to disable maxLines...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/19031
  
ping


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18966: [SPARK-21751][SQL] CodeGeneraor.splitExpressions counts ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18966
  
**[Test build #81109 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81109/testReport)**
 for PR 18966 at commit 
[`ea9fea4`](https://github.com/apache/spark/commit/ea9fea48bf30dc53305fbe524a11dd1dee57ca82).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16422: [SPARK-17642] [SQL] support DESC EXTENDED/FORMATTED tabl...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16422
  
ping @wzhfy 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18966: [SPARK-21751][SQL] CodeGeneraor.splitExpressions ...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/18966#discussion_r135164815
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/BenchmarkWideTable.scala
 ---
@@ -42,12 +42,16 @@ class BenchmarkWideTable extends BenchmarkBase {
 /**
  * Here are some numbers with different split threshold:
  *
- *  Split threshold  methods   Rate(M/s)   Per Row(ns)
- *  10   400   0.4 2279
- *  100  200   0.6 1554
- *  1k   370.9 1116
- *  8k   5 0.5 2025
- *  64k  1 0.021649
+ *  Split threshold  Rate(M/s)   Per Row(ns)
+ *  10   0.5 2131.3
+ *  20   0.5 2073.7
+ *  40   0.5 2085.2
+ *  64   0.5 2012.2
+ *  80   0.5 2112.2
+ *  100  0.5 1984.0
+ *  128  0.5 2097.9
+ *  256  0.5 2038.9
+ *  1024 0.5 2045.2
--- End diff --

@gatorsmile Sorry, I updated the result. This threshold impacts this test 
case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19046: [SPARK-18769][yarn] Limit resource requests based on RM'...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19046
  
**[Test build #81108 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81108/testReport)**
 for PR 19046 at commit 
[`03477f5`](https://github.com/apache/spark/commit/03477f5282ba00b40915bae32dc5bd48c946e27c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Expression...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/19044
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Expression...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19044
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81106/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Expression...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19044
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Expression...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19044
  
**[Test build #81106 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81106/testReport)**
 for PR 19044 at commit 
[`bc906c8`](https://github.com/apache/spark/commit/bc906c879787c0d55f7336c3646f20e4df2ba192).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19037: SPARK-21825: change Set(5, 3) to Seq(5, 3, 5) & Set(2, 3...

[iamhumanbeing]

Github user iamhumanbeing commented on the issue:

https://github.com/apache/spark/pull/19037
  
@srowen  How about 
"assert(exchanges.map(_.outputPartitioning.numPartitions).sorted === List(2, 2, 
2, 3))"?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19040: [SPARK-21826][SQL][2.1][2.0] outer broadcast hash...

[cloud-fan]

Github user cloud-fan closed the pull request at:

https://github.com/apache/spark/pull/19040


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19045: [WIP][SPARK-20628][CORE] Keep track of nodes (/ spot ins...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19045
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81103/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19045: [WIP][SPARK-20628][CORE] Keep track of nodes (/ spot ins...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19045
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19045: [WIP][SPARK-20628][CORE] Keep track of nodes (/ spot ins...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19045
  
**[Test build #81103 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81103/testReport)**
 for PR 19045 at commit 
[`65a29c1`](https://github.com/apache/spark/commit/65a29c12c1740c285ff7b06f3788cd2a92ce87f1).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18704: [SPARK-20783][SQL] Create ColumnVector to abstract exist...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/18704
  
@cloud-fan could you please review this again?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18837: [Spark-20812][Mesos] Add secrets support to the dispatch...

[ArtRand]

Github user ArtRand commented on the issue:

https://github.com/apache/spark/pull/18837
  
Jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18956: [SPARK-21726][SQL] Check for structural integrity of the...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/18956
  
Seems there are other issues caused by `RewritePredicateSubquery` rule. 
I'll investigate and fix it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Exp...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/19044#discussion_r135158510
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/catalyst/ExpressionSQLBuilderSuite.scala
 ---
@@ -19,12 +19,28 @@ package org.apache.spark.sql.catalyst
 
 import java.sql.Timestamp
 
-import org.apache.spark.sql.catalyst.dsl.expressions._
-import org.apache.spark.sql.catalyst.expressions.{If, Literal, 
SpecifiedWindowFrame, TimeAdd,
-  TimeSub, WindowSpecDefinition}
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.hive.test.TestHiveSingleton
 import org.apache.spark.unsafe.types.CalendarInterval
 
-class ExpressionSQLBuilderSuite extends SQLBuilderTest {
+class ExpressionSQLBuilderSuite extends QueryTest with TestHiveSingleton {
+  protected def checkSQL(e: Expression, expectedSQL: String): Unit = {
+val actualSQL = e.sql
+try {
+  assert(actualSQL === expectedSQL)
--- End diff --

The PR is updated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Expression...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19044
  
**[Test build #81107 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81107/testReport)**
 for PR 19044 at commit 
[`e042621`](https://github.com/apache/spark/commit/e042621003edca43501eef9e6321f2448f343cdd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19044: [SPARK-21832][TEST] Merge SQLBuilderTest into Exp...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/19044#discussion_r135157926
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/catalyst/ExpressionSQLBuilderSuite.scala
 ---
@@ -19,12 +19,28 @@ package org.apache.spark.sql.catalyst
 
 import java.sql.Timestamp
 
-import org.apache.spark.sql.catalyst.dsl.expressions._
-import org.apache.spark.sql.catalyst.expressions.{If, Literal, 
SpecifiedWindowFrame, TimeAdd,
-  TimeSub, WindowSpecDefinition}
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.hive.test.TestHiveSingleton
 import org.apache.spark.unsafe.types.CalendarInterval
 
-class ExpressionSQLBuilderSuite extends SQLBuilderTest {
+class ExpressionSQLBuilderSuite extends QueryTest with TestHiveSingleton {
+  protected def checkSQL(e: Expression, expectedSQL: String): Unit = {
+val actualSQL = e.sql
+try {
+  assert(actualSQL === expectedSQL)
--- End diff --

I see. Yep. That's better.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18966: [SPARK-21751][SQL] CodeGeneraor.splitExpressions ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18966#discussion_r135157982
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/BenchmarkWideTable.scala
 ---
@@ -42,12 +42,16 @@ class BenchmarkWideTable extends BenchmarkBase {
 /**
  * Here are some numbers with different split threshold:
  *
- *  Split threshold  methods   Rate(M/s)   Per Row(ns)
- *  10   400   0.4 2279
- *  100  200   0.6 1554
- *  1k   370.9 1116
- *  8k   5 0.5 2025
- *  64k  1 0.021649
+ *  Split threshold  Rate(M/s)   Per Row(ns)
+ *  10   0.5 2131.3
+ *  20   0.5 2073.7
+ *  40   0.5 2085.2
+ *  64   0.5 2012.2
+ *  80   0.5 2112.2
+ *  100  0.5 1984.0
+ *  128  0.5 2097.9
+ *  256  0.5 2038.9
+ *  1024 0.5 2045.2
--- End diff --

Does this threshold impact this test case?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org