[GitHub] spark issue #19939: [SPARK-20557] [SQL] Only support TIMESTAMP WITH TIME ZON...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19939
  
**[Test build #84692 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84692/testReport)**
 for PR 19939 at commit 
[`69c6e3e`](https://github.com/apache/spark/commit/69c6e3ef97b6450da5250171d47cf6c7f0faa9ff).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19939: [SPARK-20557] [SQL] Only support TIMESTAMP WITH TIME ZON...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/19939
  
cc @cloud-fan @ueshin @srowen @JannikArndt 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19939: [SPARK-20557] [SQL] Only support TIMESTAMP WITH T...

[gatorsmile]

GitHub user gatorsmile opened a pull request:

https://github.com/apache/spark/pull/19939

[SPARK-20557] [SQL] Only support TIMESTAMP WITH TIME ZONE for Oracle Dialect

## What changes were proposed in this pull request?
In the previous PRs, https://github.com/apache/spark/pull/17832 and 
https://github.com/apache/spark/pull/17835 , we convert `TIMESTAMP WITH TIME 
ZONE` and `TIME WITH TIME ZONE` to `TIMESTAMP` for all the JDBC sources. 
However, this conversion could be risky since it does not respect our SQL 
configuration `spark.sql.session.timeZone`. 

In addition, each vendor might have different semantics for these two 
types. For example, Postgres simply returns `TIMESTAMP` types for `TIMESTAMP 
WITH TIME ZONE`. For such supports, we should do it case by case. This PR 
reverts the general support of `TIMESTAMP WITH TIME ZONE` and `TIME WITH TIME 
ZONE` for JDBC sources, except ORACLE Dialect.

When supporting the ORACLE's `TIMESTAMP WITH TIME ZONE`, we only support it 
when the JVM default timezone is the same as the user-specified configuration 
`spark.sql.session.timeZone`. Now, we still treat `TIMESTAMP WITH TIME ZONE` as 
`TIMESTAMP` when fetching the values via the Oracle JDBC connector, whose 
client converts the timestamp values with time zone to the timestamp values 
using the local JVM default timezone (a test case is added to 
`OracleIntegrationSuite.scala` in this PR for showing the behavior). Thus, to 
avoid any future behavior change, we will not support it if JVM default 
timezone is different from `spark.sql.session.timeZone`

No regression because the previous two PRs were just merged to be 
unreleased master branch. 

## How was this patch tested?
Added the test cases

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gatorsmile/spark timezoneUpdate

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19939.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19939


commit 69c6e3ef97b6450da5250171d47cf6c7f0faa9ff
Author: gatorsmile 
Date:   2017-12-10T07:06:37Z

fix




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19938: [SPARK-22747][SQL] Localize lifetime of mutable states i...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19938
  
**[Test build #84691 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84691/testReport)**
 for PR 19938 at commit 
[`ac65dd2`](https://github.com/apache/spark/commit/ac65dd21b975481f5c4feb5f0745a2c45d000728).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19938: [SPARK-22747][SQL] Localize lifetime of mutable s...

[kiszk]

GitHub user kiszk opened a pull request:

https://github.com/apache/spark/pull/19938

[SPARK-22747][SQL] Localize lifetime of mutable states in HashAggregateExec

## What changes were proposed in this pull request?

This PR localizes lifetime of mutable states, which are used for `isNull` 
and `value` of aggregation results, in generated code by `HashAggregateExec`. 

These status are passed to successor operations thru `consume()` method. It 
may violate this assumption at #19865 when operations that uses these variables 
are split.  In the following example, `agg_localBufValue` and 
`agg_localBufisNull` are passed to an successor operation (`projection`).

This PR is based on @cloud-fan 's 
[suggestion](https://github.com/apache/spark/pull/19865#issuecomment-348776654).

Without this PR
```
/* 005 */ final class GeneratedIterator extends 
org.apache.spark.sql.execution.BufferedRowIterator {
/* 006 */   private Object[] references;
/* 007 */   private scala.collection.Iterator[] inputs;
/* 008 */   private boolean agg_initAgg;
/* 009 */   private boolean agg_bufIsNull;
/* 010 */   private long agg_bufValue;
/* 011 */   private scala.collection.Iterator inputadapter_input;
...
/* 039 */   private void agg_doAggregateWithoutKey() throws 
java.io.IOException {
/* 040 */ // initialize aggregation buffer
/* 041 */ final long agg_value = -1L;
/* 042 */ agg_bufIsNull = true;
/* 043 */ agg_bufValue = agg_value;
/* 044 */
/* 045 */ while (inputadapter_input.hasNext() && !stopEarly()) {
/* 046 */   InternalRow inputadapter_row = (InternalRow) 
inputadapter_input.next();
/* 047 */   boolean inputadapter_isNull = inputadapter_row.isNullAt(0);
/* 048 */   long inputadapter_value = inputadapter_isNull ? -1L : 
(inputadapter_row.getLong(0));
...
/* 100 */   } while (false);
/* 101 */   final boolean agg_isNull3 = agg_coalesceTmpIsNull;
/* 102 */   // update aggregation buffer
/* 103 */   agg_bufIsNull = agg_isNull3;
/* 104 */   agg_bufValue = agg_value3;
/* 105 */   if (shouldStop()) return;
/* 106 */ }
/* 107 */
/* 108 */   }
/* 109 */
/* 110 */   protected void processNext() throws java.io.IOException {
/* 111 */ while (!agg_initAgg) {
/* 112 */   agg_initAgg = true;
/* 113 */   long agg_beforeAgg = System.nanoTime();
/* 114 */   agg_doAggregateWithoutKey();
/* 115 */   agg_aggTime.add((System.nanoTime() - agg_beforeAgg) / 
100);
/* 116 */
/* 117 */   // output the result
/* 118 */
/* 119 */   agg_numOutputRows.add(1);
/* 120 */   agg_rowWriter.zeroOutNullBytes();
/* 121 */
/* 122 */   if (agg_bufisNull) {
/* 123 */ agg_rowWriter.setNullAt(0);
/* 124 */   } else {
/* 125 */ agg_rowWriter.write(0, agg_bufValue);
/* 126 */   }
/* 127 */   append(agg_result);
/* 128 */ }
/* 129 */   }
```

With this PR
```
/* 005 */ final class GeneratedIterator extends 
org.apache.spark.sql.execution.BufferedRowIterator {
/* 006 */   private Object[] references;
/* 007 */   private scala.collection.Iterator[] inputs;
/* 008 */   private boolean agg_initAgg;
/* 009 */   private boolean agg_bufIsNull;
/* 010 */   private long agg_bufValue;
/* 011 */   private scala.collection.Iterator inputadapter_input;
...
/* 039 */   private void agg_doAggregateWithoutKey() throws 
java.io.IOException {
/* 040 */ // initialize aggregation buffer
/* 041 */ final long agg_value = -1L;
/* 042 */ agg_bufIsNull = true;
/* 043 */ agg_bufValue = agg_value;
/* 044 */
/* 045 */ while (inputadapter_input.hasNext() && !stopEarly()) {
/* 046 */   InternalRow inputadapter_row = (InternalRow) 
inputadapter_input.next();
/* 047 */   boolean inputadapter_isNull = inputadapter_row.isNullAt(0);
/* 048 */   long inputadapter_value = inputadapter_isNull ? -1L : 
(inputadapter_row.getLong(0));
...
/* 100 */   } while (false);
/* 101 */   final boolean agg_isNull3 = agg_coalesceTmpIsNull;
/* 102 */   // update aggregation buffer
/* 103 */   agg_bufIsNull = agg_isNull3;
/* 104 */   agg_bufValue = agg_value3;
/* 105 */   if (shouldStop()) return;
/* 106 */ }
/* 107 */
/* 108 */   }
/* 109 */
/* 110 */   protected void processNext() throws java.io.IOException {
/* 111 */ while (!agg_initAgg) {
/* 112 */   agg_initAgg = true;
/* 113 */   long agg_beforeAgg = System.nanoTime();
/* 114 */   agg_doAggregateWithoutKey();
/* 115 */   agg_aggTime.add((System.nanoTime() - agg_beforeAgg) / 
100);
/* 116 */
/* 117 */   // output the result
/* 118 */
/* 119 */

[GitHub] spark pull request #19918: [SPARK-22726] [TEST] Basic tests for Binary Compa...

[wangyum]

Github user wangyum commented on a diff in the pull request:

https://github.com/apache/spark/pull/19918#discussion_r155939348
  
--- Diff: 
sql/core/src/test/resources/sql-tests/inputs/typeCoercion/native/binaryComparison.sql
 ---
@@ -0,0 +1,287 @@
+--
+--   Licensed to the Apache Software Foundation (ASF) under one or more
+--   contributor license agreements.  See the NOTICE file distributed with
+--   this work for additional information regarding copyright ownership.
+--   The ASF licenses this file to You under the Apache License, Version 
2.0
+--   (the "License"); you may not use this file except in compliance with
+--   the License.  You may obtain a copy of the License at
+--
+--  http://www.apache.org/licenses/LICENSE-2.0
+--
+--   Unless required by applicable law or agreed to in writing, software
+--   distributed under the License is distributed on an "AS IS" BASIS,
+--   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+--   See the License for the specific language governing permissions and
+--   limitations under the License.
+--
+
+-- Binary Comparison
+
+CREATE TEMPORARY VIEW t AS SELECT 1;
+
+SELECT cast(1 as binary) = '1' FROM t;
--- End diff --

Seems binary comparison without 
[<=>](https://github.com/apache/spark/blob/ced6ccf0d6f362e299f270ed2a474f2e14f845da/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala#L594).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19933: [SPARK-22744][CORE] Add a configuration to show the appl...

[LantaoJin]

Github user LantaoJin commented on the issue:

https://github.com/apache/spark/pull/19933
  
I think you miss my point. Please image that I am an infra team member in a 
company, and spark users are almost using spark sql or just offering an 
application jar. What I want is get the submit host to help them to debug 
environment issue or to control the spark version or to limit the submitting 
hosts. And I cannot finger out where the bad apps submitted from if the user 
didn’t log it. Yes, I just need a read only info at beginning. So I ask if it 
can be changed to a system property instread of spark conf. And why I finally 
choose using a configuration is there is one situation that the submitting host 
is not the original submitting host from user, it could be a platform server.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19764: [SPARK-22539][SQL] Add second order for rangepartitioner...

[caneGuy]

Github user caneGuy commented on the issue:

https://github.com/apache/spark/pull/19764
  
Ping 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19932: [SPARK-22745][SQL] read partition stats from Hive

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19932
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84689/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19932: [SPARK-22745][SQL] read partition stats from Hive

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19932
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19932: [SPARK-22745][SQL] read partition stats from Hive

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19932
  
**[Test build #84689 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84689/testReport)**
 for PR 19932 at commit 
[`09a7c05`](https://github.com/apache/spark/commit/09a7c0594507ae6f14f3f016fdc407477e320107).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/19865
  
I confirmed that this failure does not occur after merging #19937 in my 
environment.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19928: [SPARK-22267][SQL][TEST] Spark SQL incorrectly reads ORC...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/19928
  
Thank you for review and approval, @HyukjinKwon !


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19527
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84690/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #84690 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84690/testReport)**
 for PR 19527 at commit 
[`32318fa`](https://github.com/apache/spark/commit/32318faebd118509bdd0c0100e84c4755182ea27).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19594: [SPARK-21984] [SQL] Join estimation based on equi-height...

[wzhfy]

Github user wzhfy commented on the issue:

https://github.com/apache/spark/pull/19594
  
ping @cloud-fan 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19527
  
**[Test build #84690 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84690/testReport)**
 for PR 19527 at commit 
[`32318fa`](https://github.com/apache/spark/commit/32318faebd118509bdd0c0100e84c4755182ea27).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19932: [SPARK-22745][SQL] read partition stats from Hive

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19932
  
**[Test build #84689 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84689/testReport)**
 for PR 19932 at commit 
[`09a7c05`](https://github.com/apache/spark/commit/09a7c0594507ae6f14f3f016fdc407477e320107).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19932: [SPARK-22745][SQL] read partition stats from Hive

[wzhfy]

Github user wzhfy commented on a diff in the pull request:

https://github.com/apache/spark/pull/19932#discussion_r155936167
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala ---
@@ -353,15 +374,6 @@ class StatisticsSuite extends 
StatisticsCollectionTestBase with TestHiveSingleto
   createPartition("2010-01-02", 11,
 "SELECT '1', 'A' from src UNION ALL SELECT '1', 'A' from src")
 
-  sql(s"ANALYZE TABLE $tableName PARTITION (ds='2010-01-01') COMPUTE 
STATISTICS NOSCAN")
-
-  assertPartitionStats("2010-01-01", "10", rowCount = None, 
sizeInBytes = 2000)
-  assertPartitionStats("2010-01-01", "11", rowCount = None, 
sizeInBytes = 2000)
-  assert(queryStats("2010-01-02", "10") === None)
-  assert(queryStats("2010-01-02", "11") === None)
--- End diff --

After the change, these checks are not right as we read hive stats. So I 
remove them.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19932: [SPARK-22745][SQL] read partition stats from Hive

[wzhfy]

Github user wzhfy commented on a diff in the pull request:

https://github.com/apache/spark/pull/19932#discussion_r155936087
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala ---
@@ -213,6 +213,29 @@ class StatisticsSuite extends 
StatisticsCollectionTestBase with TestHiveSingleto
 }
   }
 
+  test("SPARK- - read Hive's statistics for partition") {
--- End diff --

oh, I forgot it, thanks!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19754: [BUILD] update release scripts

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19754
  
(The package build failure, which began Nov 26, is:)

```
gpg: skipped 
"/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-FLng1qGV/gpg.tmp":
 No secret key
gpg: signing failed: No secret key
Deleting credential directory 
/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-FLng1qGV
Build step 'Execute shell' marked build as failure
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
Finished: FAILURE
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19754: [BUILD] update release scripts

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19754
  
The package build was failing for a while, but the docs build has a 
probably-related new error:

```
ATTENTION!  Your password for authentication realm:

    ASF Committers

can only be stored to disk unencrypted!  You are advised to configure
your system so that Subversion can store passwords encrypted, if
possible.  See the documentation for details.

You can avoid future appearances of this warning by setting the value
of the 'store-plaintext-passwords' option to either 'yes' or 'no' in
'/home/jenkins/.subversion/servers'.
---
Store password unencrypted (yes/no)? svn: Commit failed (details follow):
svn: Can't read stdin: End of file found
Deleting credential directory 
/home/jenkins/workspace/spark-master-docs/spark-utils/new-release-scripts/jenkins/jenkins-credentials-wmz3q7pN
Build step 'Execute shell' marked build as failure
[WS-CLEANUP] Deleting project workspace...[WS-CLEANUP] done
Finished: FAILURE
```

Was this expected and needs a Jenkins config change, or not expected?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19934: [SPARK-3685][CORE] Prints explicit warnings when configu...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19934
  
Hi @liancheng, @steveloughran and @srowen.

What do you think about the current approach? The JIRA seems targeting 
"Spark's local dir should accept only local paths" and it's my best effort to 
resolve it. I don't feel strongly about this.

If this sounds too much or you guys are not sure for now, I could close 
this PR and the resolve JIRA for now. It works to me as well.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19932: [SPARK-22745][SQL] read partition stats from Hive

[wangyum]

Github user wangyum commented on a diff in the pull request:

https://github.com/apache/spark/pull/19932#discussion_r155935430
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala ---
@@ -213,6 +213,29 @@ class StatisticsSuite extends 
StatisticsCollectionTestBase with TestHiveSingleto
 }
   }
 
+  test("SPARK- - read Hive's statistics for partition") {
--- End diff --

SPARK- -> SPARK-22745?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

[liyinan926]

Github user liyinan926 commented on a diff in the pull request:

https://github.com/apache/spark/pull/19717#discussion_r155935022
  
--- Diff: 
resource-managers/kubernetes/docker/src/main/dockerfiles/spark-base/Dockerfile 
---
@@ -0,0 +1,43 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8-alpine
--- End diff --

Got it. Should we defer updating the licensing information later when we 
have clearer on how we will maintain and release the docker images?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19717: [SPARK-22646] [Submission] Spark on Kubernetes - ...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/19717#discussion_r155934917
  
--- Diff: 
resource-managers/kubernetes/docker/src/main/dockerfiles/spark-base/Dockerfile 
---
@@ -0,0 +1,43 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8-alpine
--- End diff --

I was referring to the docker image openjdk
 
https://github.com/docker-library/openjdk/blob/b4f29ba829765552239bd18f272fcdaf09eca259/LICENSE

of course, the openjdk bin/bit is not under MIT license. I'm actually not 
quite sure how "releasing" the docker image goes from the licensing point of 
view...


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19937: [SPARK-22746][SQL] Avoid the generation of useles...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/19937#discussion_r155934663
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
 ---
@@ -617,6 +619,7 @@ case class SortMergeJoinExec(
 
 s"""
|while (findNextInnerJoinRows($leftInput, $rightInput)) {
+   |  ${leftVarDecl.mkString("\n")}
--- End diff --

Do you think what microbench is relevant for this measurement? I would 
appreciate your thought.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19717: [SPARK-22646] [Submission] Spark on Kubernetes - basic s...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/19717
  
sounds like a good to go from @vanzin and a couple of others too.
any other comment?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16578: [SPARK-4502][SQL] Parquet nested column pruning

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16578
  
yes!
sorry about the delay, I think there's a lot of interests in this PR.
@gatorsmile @viirya ?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19754: [BUILD] update release scripts

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/19754
  
yes, specifically this changes the output location for

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-docs/

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-master-package/

and later

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-release-docs/

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20Packaging/job/spark-release-package/

and it shows up like this

https://dist.apache.org/repos/dist/dev/spark/2.3.0-SNAPSHOT-2017_12_09_08_01-ab1b6ee-docs/



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19805: [SPARK-22649][PYTHON][SQL] Adding localCheckpoint...

[holdenk]

Github user holdenk commented on a diff in the pull request:

https://github.com/apache/spark/pull/19805#discussion_r155934026
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -537,9 +537,48 @@ class Dataset[T] private[sql](
*/
   @Experimental
   @InterfaceStability.Evolving
-  def checkpoint(eager: Boolean): Dataset[T] = {
--- End diff --

So we already test checkpoint in DatasetSuite


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19865
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84688/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19865
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19865
  
**[Test build #84688 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84688/testReport)**
 for PR 19865 at commit 
[`c37311f`](https://github.com/apache/spark/commit/c37311f8d0385c979054fb800bbc5b1a784d199a).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19865
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84687/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19865
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19865
  
**[Test build #84687 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84687/testReport)**
 for PR 19865 at commit 
[`4e83f9f`](https://github.com/apache/spark/commit/4e83f9f40e514600ce4e56b40c7e41e74e998ba8).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19676: [SPARK-14516][FOLLOWUP] Adding ClusteringEvaluato...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/19676#discussion_r155929522
  
--- Diff: 
examples/src/main/java/org/apache/spark/examples/ml/JavaKMeansExample.java ---
@@ -51,9 +52,14 @@ public static void main(String[] args) {
 KMeans kmeans = new KMeans().setK(2).setSeed(1L);
 KMeansModel model = kmeans.fit(dataset);
 
-// Evaluate clustering by computing Within Set Sum of Squared Errors.
-double WSSSE = model.computeCost(dataset);
-System.out.println("Within Set Sum of Squared Errors = " + WSSSE);
+// Make predictions
+Dataset predictions = model.transform(dataset);
+
+// Evaluate clustering by computing Silhouette score
+ClusteringEvaluator evaluator = new ClusteringEvaluator();
+
+double silhouette = evaluator.evaluate(predictions);
+System.out.println("Silhouette with squared euclidean distance = " + 
silhouette);
--- End diff --

Thanks, I don't think I am changing the code again, but I can fix this 
grammatical error if you want.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19676: [SPARK-14516][FOLLOWUP] Adding ClusteringEvaluato...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/19676#discussion_r155928871
  
--- Diff: 
examples/src/main/java/org/apache/spark/examples/ml/JavaKMeansExample.java ---
@@ -51,9 +52,14 @@ public static void main(String[] args) {
 KMeans kmeans = new KMeans().setK(2).setSeed(1L);
 KMeansModel model = kmeans.fit(dataset);
 
-// Evaluate clustering by computing Within Set Sum of Squared Errors.
-double WSSSE = model.computeCost(dataset);
-System.out.println("Within Set Sum of Squared Errors = " + WSSSE);
+// Make predictions
+Dataset predictions = model.transform(dataset);
+
+// Evaluate clustering by computing Silhouette score
+ClusteringEvaluator evaluator = new ClusteringEvaluator();
+
+double silhouette = evaluator.evaluate(predictions);
+System.out.println("Silhouette with squared euclidean distance = " + 
silhouette);
--- End diff --

euclidean -> Euclidean, but not important to change unless you're touching 
the code again anyway


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19937: [SPARK-22746][SQL] Avoid the generation of useless mutab...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19937
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84685/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19937: [SPARK-22746][SQL] Avoid the generation of useless mutab...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19937
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19937: [SPARK-22746][SQL] Avoid the generation of useless mutab...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19937
  
**[Test build #84685 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84685/testReport)**
 for PR 19937 at commit 
[`6d48924`](https://github.com/apache/spark/commit/6d489248f4b2c2e5b40f51524e21992a59226ae8).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19525: [SPARK-22289] [ML] Add JSON support for Matrix parameter...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19525
  
**[Test build #84686 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84686/testReport)**
 for PR 19525 at commit 
[`5799456`](https://github.com/apache/spark/commit/579945668e6f7e93d76102185bdf1d274e6c7773).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19525: [SPARK-22289] [ML] Add JSON support for Matrix parameter...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19525
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84686/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19525: [SPARK-22289] [ML] Add JSON support for Matrix parameter...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19525
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19937: [SPARK-22746][SQL] Avoid the generation of useles...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/19937#discussion_r155927167
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
 ---
@@ -617,6 +619,7 @@ case class SortMergeJoinExec(
 
 s"""
|while (findNextInnerJoinRows($leftInput, $rightInput)) {
+   |  ${leftVarDecl.mkString("\n")}
--- End diff --

I am not sure about the amount of overhead introduced honestly. But I think 
that there might be, even if low. Therefore my suggestion was to avoid the 
overhead, since IMHO it is feasible and it reflects the previous situation. 
Honestly I don't think that the readability of the generated code is a big 
point, because I think that the generated code is already nearly impossible to 
be read as of now (without reading and knowing the code which generates it).

But maybe we can create a test and evaluate the overhead if you think this 
is the best option. WDYT?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19911: [SPARK-22729][SQL] Add getTruncateQuery to JdbcDialect

[danielvdende]

Github user danielvdende commented on the issue:

https://github.com/apache/spark/pull/19911
  
@dongjoon-hyun @gatorsmile As @gatorsmile pointed out, the 
`isCascadingTruncateTable` is a method in the public API, so we can't just drop 
it. I've made changes again, now the truncate query method is defined for all 
dialects by default, with only the PostgresDialect overriding the definition 
(for reasons we discussed before). Have also reinstated the test I removed 
before (to test the `isCascadingTruncateTable` function). Let me know what you 
think, once you're ok with the changes, I'll squash the commits into 1.

Thanks for helping out :), much appreciated.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19865
  
**[Test build #84688 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84688/testReport)**
 for PR 19865 at commit 
[`c37311f`](https://github.com/apache/spark/commit/c37311f8d0385c979054fb800bbc5b1a784d199a).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19937: [SPARK-22746][SQL] Avoid the generation of useles...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/19937#discussion_r155927018
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
 ---
@@ -617,6 +619,7 @@ case class SortMergeJoinExec(
 
 s"""
|while (findNextInnerJoinRows($leftInput, $rightInput)) {
+   |  ${leftVarDecl.mkString("\n")}
--- End diff --

Since they are local variable, it takes almost no cost. If they are on CPU 
registers, there is no cost. If they are in stack frame, it is up to one 
instruction to increase or decrease stack frame size.

WDYT? Did you see huge overhead to create and destroy local variables?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19937: [SPARK-22746][SQL] Avoid the generation of useles...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/19937#discussion_r155926973
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
 ---
@@ -617,6 +619,7 @@ case class SortMergeJoinExec(
 
 s"""
|while (findNextInnerJoinRows($leftInput, $rightInput)) {
+   |  ${leftVarDecl.mkString("\n")}
--- End diff --

I think that the advantage would be to reuse them, avoiding creating and 
destroying them at every loop.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19865: [SPARK-22668][SQL] Ensure no global variables in argumen...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19865
  
**[Test build #84687 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84687/testReport)**
 for PR 19865 at commit 
[`4e83f9f`](https://github.com/apache/spark/commit/4e83f9f40e514600ce4e56b40c7e41e74e998ba8).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19937: [SPARK-22746][SQL] Avoid the generation of useles...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/19937#discussion_r155926812
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
 ---
@@ -617,6 +619,7 @@ case class SortMergeJoinExec(
 
 s"""
|while (findNextInnerJoinRows($leftInput, $rightInput)) {
+   |  ${leftVarDecl.mkString("\n")}
--- End diff --

Since I have no strong preference, I would like to hear opinions from 
others.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19934: [SPARK-3685][CORE] Prints explicit warnings when configu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19934
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19934: [SPARK-3685][CORE] Prints explicit warnings when configu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19934
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84684/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19934: [SPARK-3685][CORE] Prints explicit warnings when configu...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19934
  
**[Test build #84684 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84684/testReport)**
 for PR 19934 at commit 
[`0db4bf1`](https://github.com/apache/spark/commit/0db4bf1c1b447ce39f790d7c81fc3bb2619e156a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19937: [SPARK-22746][SQL] Avoid the generation of useles...

[kiszk]

Github user kiszk commented on a diff in the pull request:

https://github.com/apache/spark/pull/19937#discussion_r155926316
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
 ---
@@ -617,6 +619,7 @@ case class SortMergeJoinExec(
 
 s"""
|while (findNextInnerJoinRows($leftInput, $rightInput)) {
+   |  ${leftVarDecl.mkString("\n")}
--- End diff --

It could be. Would it be possible to let us know advantages compare to the 
current method?  
I think that to shorten lifetime of variables (i.e. current approach) make 
generated code more readable.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19936: Branch 0.5

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19936
  
@khanm002, could you close this please?

If you are going to propose a change, please follow this linke - 
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19931: [SPARK-22672][SQL][TEST][FOLLOWUP] Fix to use `spark.con...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/19931
  
Oh, thank you so much, @HyukjinKwon .


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19525: [SPARK-22289] [ML] Add JSON support for Matrix parameter...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19525
  
**[Test build #84686 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84686/testReport)**
 for PR 19525 at commit 
[`5799456`](https://github.com/apache/spark/commit/579945668e6f7e93d76102185bdf1d274e6c7773).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19937: [SPARK-22746][SQL] Avoid the generation of useles...

[mgaido91]

Github user mgaido91 commented on a diff in the pull request:

https://github.com/apache/spark/pull/19937#discussion_r155925941
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
 ---
@@ -617,6 +619,7 @@ case class SortMergeJoinExec(
 
 s"""
|while (findNextInnerJoinRows($leftInput, $rightInput)) {
+   |  ${leftVarDecl.mkString("\n")}
--- End diff --

can't we define them before the loop and reuse them?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19865: [SPARK-22668][SQL] Assert to ensure no global variables ...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/19865
  
@cloud-fan unfortunately, #19878 did not fix this issue. #19937 will fix 
this issue.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19937: [SPARK-22746][SQL] Avoid the generation of useless mutab...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19937
  
**[Test build #84685 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84685/testReport)**
 for PR 19937 at commit 
[`6d48924`](https://github.com/apache/spark/commit/6d489248f4b2c2e5b40f51524e21992a59226ae8).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19527: [SPARK-13030][ML] Create OneHotEncoderEstimator for OneH...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/19527
  
Ok, I understood. In other words, the extra category is added as the last 
category and `dropLast` option works as before. It makes sense to me.






---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19937: [SPARK-22746][SQL] Avoid the generation of useles...

[kiszk]

GitHub user kiszk opened a pull request:

https://github.com/apache/spark/pull/19937

[SPARK-22746][SQL] Avoid the generation of useless mutable states by 
SortMergeJoin

## What changes were proposed in this pull request?

This PR reduce the number of global mutable variables in generated code of 
`SortMergeJoin`.

## How was this patch tested?

Existing test cases

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kiszk/spark SPARK-22746

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19937.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19937


commit 6d489248f4b2c2e5b40f51524e21992a59226ae8
Author: Kazuaki Ishizaki 
Date:   2017-12-09T15:27:24Z

initial commit




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19754: [BUILD] update release scripts

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19754


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19754: [BUILD] update release scripts

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19754
  
merged to master


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19935: Branch 0.6

[khanm002]

Github user khanm002 closed the pull request at:

https://github.com/apache/spark/pull/19935


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19933: [SPARK-22744][CORE] Add a configuration to show the appl...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19933
  
You're just using this as a write-only config to pass info, and it's for 
your own app-specific purposes. It's easier to just pass this info or log it in 
your app.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19935: Branch 0.6

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19935
  
@khanm002 close this


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19936: Branch 0.5

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19936
  
@khanm002 close this


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19936: Branch 0.5

[khanm002]

Github user khanm002 commented on a diff in the pull request:

https://github.com/apache/spark/pull/19936#discussion_r155924835
  
--- Diff: repl/src/main/scala/spark/repl/SparkILoop.scala ---
@@ -200,7 +200,7 @@ class SparkILoop(in0: Option[BufferedReader], val out: 
PrintWriter, val master:
     __  
  / __/__  ___ _/ /__
 _\ \/ _ \/ _ `/ __/  '_/
-   /___/ .__/\_,_/_/ /_/\_\   version 0.5.2-SNAPSHOT
--- End diff --

#-   /___/ .__/\_,_/_/ /_/\_\   version 0.5.2-SNAPSHOT


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19936: Branch 0.5

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19936
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19935: Branch 0.6

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19935
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19936: Branch 0.5

[khanm002]

GitHub user khanm002 opened a pull request:

https://github.com/apache/spark/pull/19936

Branch 0.5

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)

Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/spark branch-0.5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19936.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19936


commit 1b9bba0e9aea82c0390d4c43098c08940d3b0309
Author: Thomas Dudziak 
Date:   2012-10-09T22:21:38Z

Support for Hadoop 2 distributions such as cdh4

Conflicts:

core/src/main/scala/spark/NewHadoopRDD.scala
core/src/main/scala/spark/PairRDDFunctions.scala
project/SparkBuild.scala

commit 8eec96fa5436902d2aa24cf8700b4424aff2005a
Author: Matei Zaharia 
Date:   2012-11-21T02:23:34Z

Change version to 0.5.2

commit 5b021ce0990ec675afc6939cc2c06f041c973d17
Author: Matei Zaharia 
Date:   2012-11-23T00:26:15Z

Change version to 0.5.3-SNAPSHOT




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19935: Branch 0.6

[khanm002]

GitHub user khanm002 opened a pull request:

https://github.com/apache/spark/pull/19935

Branch 0.6

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)

Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/spark branch-0.6

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19935.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19935


commit ce143d64ed68e956248c3ba5cc310a63a79f33c8
Author: Matei Zaharia 
Date:   2012-10-25T04:52:13Z

Strip leading mesos:// in URLs passed to Mesos

commit d3387427ce6608f53df371f9365c49062ae0dee5
Author: root 
Date:   2012-10-26T07:31:08Z

Don't throw an error in the block manager when a block is cached on the 
master due to
a locally computed operation

commit d2b2fc229e488f961f029846c34660552468dda4
Author: Matei Zaharia 
Date:   2012-11-06T23:57:38Z

Made Akka timeout and message frame size configurable, and upped the 
defaults

commit 222355e0584f52be2b0285257151f7c3f1f3f3fa
Author: Thomas Dudziak 
Date:   2012-10-22T20:10:47Z

Tweaked run file to live more happily with typesafe's debian package

commit ef683d4e01bc0ff3fb783bd6b1308b5e4ecd7ece
Author: Josh Rosen 
Date:   2012-10-23T20:49:52Z

Fix minor typos in quick start guide.

commit cf0bf73d07600f92f24af7b97a2f60b12d1e4f96
Author: Josh Rosen 
Date:   2012-10-18T17:01:38Z

Allow EC2 script to stop/destroy cluster after master/slave failures.

commit 43465e92a934a7fc93154c97e397074707d8d803
Author: Josh Rosen 
Date:   2012-11-04T00:02:47Z

Fix check for existing instances during EC2 launch.

commit d20142b105acedc7074dc4edb743ea78cd851d7f
Author: Shivaram Venkataraman 
Date:   2012-11-01T17:46:38Z

Remove unnecessary hash-map put in MemoryStore

commit 171e97af5b67dd322f787655d70baa40318dbb87
Author: Josh Rosen 
Date:   2012-10-31T06:32:38Z

Cancel spot instance requests when exiting spark-ec2.

commit 5acd753876eab712a3e8fbf3ae33fb4c0b978abd
Author: Matei Zaharia 
Date:   2012-10-21T06:33:37Z

Various fixes to standalone mode and web UI:

- Don't report a job as finishing multiple times
- Don't show state of workers as LOADING when they're running
- Show start and finish times in web UI
- Sort web UI tables by ID and time by default

commit 4fe0d808b0d211d7e00341a3ba95e83792c01681
Author: Imran Rashid 
Date:   2012-11-07T23:35:51Z

fix bug in getting slave id out of mesos

commit a24540887c6968353db3bb9c28b23eb48a68da75
Author: Matei Zaharia 
Date:   2012-11-08T08:10:13Z

Merge pull request #300 from enachb/mesos_slavelost

fix bug in getting slave id out of mesos

commit b3b52c995a37385fc08af5837feea18bddee55a0
Author: Matei Zaharia 
Date:   2012-11-08T17:53:40Z

Fix for connections not being reused (from Josh Rosen)

commit bb2b9ff37cd2503cc6ea82c5dd395187b0910af0
Author: Matei Zaharia 
Date:   2012-11-09T07:13:12Z

Added an option to spread out jobs in the standalone mode.

commit e870ca50c6dbbc7bc951bb8432c4eb9b7c816c5e
Author: Tathagata Das 
Date:   2012-11-09T22:09:37Z

Fixed deadlock in BlockManager.
1. Changed the lock structure of BlockManager by replacing the 337 
coarse-grained locks to use BlockInfo objects as per-block fine-grained locks.
2. Changed the MemoryStore lock structure by making the block putting 
threads lock on a different object (not the memory store) thus making sure 
putting threads minimally blocks to the getting treads.
3. Added spark.storage.ThreadingTest to stress test the BlockManager using 
5 block producer and 5 block consumer threads.

commit 9d5740f6bfdeb52747f70a4b6c7cb82c57b225d4
Author: Tathagata Das 
Date:   2012-11-09T23:46:15Z

Incorporated Matei's suggestions. Tested with 5 producer(consumer) threads 
each doing 50k puts (gets), took 15 minutes to run, no errors or deadlocks.

commit dc84ce72190f2910bced98a504fac20f305871a4
Author: root 
Date:   2012-11-11T07:05:22Z

Fix K-means 

[GitHub] spark issue #19594: [SPARK-21984] [SQL] Join estimation based on equi-height...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19594
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84683/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19594: [SPARK-21984] [SQL] Join estimation based on equi-height...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19594
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19594: [SPARK-21984] [SQL] Join estimation based on equi-height...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19594
  
**[Test build #84683 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84683/testReport)**
 for PR 19594 at commit 
[`e69e213`](https://github.com/apache/spark/commit/e69e21348b4cde2abaec9dbb46381caf1ed3a1a4).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19934: [SPARK-3685][CORE] Prints explicit warnings when configu...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19934
  
**[Test build #84684 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84684/testReport)**
 for PR 19934 at commit 
[`0db4bf1`](https://github.com/apache/spark/commit/0db4bf1c1b447ce39f790d7c81fc3bb2619e156a).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19934: [SPARK-3685][CORE] Prints explicit warnings when ...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/19934#discussion_r155922544
  
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -829,7 +829,18 @@ private[spark] object Utils extends Logging {
   }
 
   private def getOrCreateLocalRootDirsImpl(conf: SparkConf): Array[String] 
= {
--- End diff --

Note that, this seems only being called via 
`Utils.getOrCreateLocalRootDirs` which caches `localRootDirs`. So, I think this 
won't mess up logs.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19934: [SPARK-3685][CORE] Prints explicit warnings when ...

[HyukjinKwon]

GitHub user HyukjinKwon opened a pull request:

https://github.com/apache/spark/pull/19934

[SPARK-3685][CORE] Prints explicit warnings when configured local 
directories are set to URIs

## What changes were proposed in this pull request?

This PR proposes to print warnings before creating local by `java.io.File`. 

I think we can't just simply disallow and throw an exception for such cases 
of `hdfs:/tmp/foo` case because it might break compatibility. Note that 
`hdfs:/tmp/foo` creates a directory called `hdfs:/`.

There were many discussion here about whether we should support this in 
other file systems or now; however, since the JIRA targets "Spark's local dir 
should accept only local paths", here, I tried to simply print warnings. 

I think we could open another JIRA and design doc if this is something we 
should support, separately.

**Before**

```
./bin/spark-shell --conf spark.local.dir=file:/a/b/c
```

This creates a local directory as below:

```
 file:/
└── a
└── b
└── c
...
```

**After**

```bash
./bin/spark-shell --conf spark.local.dir=file:/a/b/c
```

Now, it prints a warning as below:

```
...
17/12/09 21:58:49 WARN Utils: The configured local directories are not 
expected to be URIs; however, got suspicious values [file:/a/b/c]. Please check 
your configured local directories.
...
```

```bash
./bin/spark-shell --conf spark.local.dir=file:/a/b/c,/tmp/a/b/c,hdfs:/a/b/c
```

It also works with comma-separated ones:

```
...
17/12/09 22:05:01 WARN Utils: The configured local directories are not 
expected to be URIs; however, got suspicious values [file:/a/b/c, hdfs:/a/b/c]. 
Please check your configured local directories.
...
 ```


## How was this patch tested?

 Manually tested:

 ```
 ./bin/spark-shell --conf spark.local.dir=C:\\a\\b\\c
 ./bin/spark-shell --conf spark.local.dir=/tmp/a/b/c
 ./bin/spark-shell --conf spark.local.dir=a/b/c
 ./bin/spark-shell --conf spark.local.dir=a/b/c,/tmp/a/b/c,C:\\a\\b\\c
 ```

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HyukjinKwon/spark SPARK-3685

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/19934.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19934


commit 0db4bf1c1b447ce39f790d7c81fc3bb2619e156a
Author: hyukjinkwon 
Date:   2017-12-09T13:10:00Z

Prints explicit warnings when configured local directories are set to URIs




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19933: [SPARK-22744][CORE] Add a configuration to show the appl...

[LantaoJin]

Github user LantaoJin commented on the issue:

https://github.com/apache/spark/pull/19933
  
How about change it to system property and it can show in Spark UI. It 
really can help user (platform team) to trace, control and support application.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19932: [SPARK-22745][SQL] read partition stats from Hive

[wzhfy]

Github user wzhfy commented on a diff in the pull request:

https://github.com/apache/spark/pull/19932#discussion_r155921370
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 
---
@@ -413,32 +413,7 @@ private[hive] class HiveClientImpl(
 case (key, _) => excludedTableProperties.contains(key)
   }
   val comment = properties.get("comment")
-
-  // Here we are reading statistics from Hive.
-  // Note that this statistics could be overridden by Spark's 
statistics if that's available.
-  val totalSize = 
properties.get(StatsSetupConst.TOTAL_SIZE).map(BigInt(_))
--- End diff --

The code path is moved to the method `readHiveStats`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19594: [SPARK-21984] [SQL] Join estimation based on equi-height...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19594
  
**[Test build #84683 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84683/testReport)**
 for PR 19594 at commit 
[`e69e213`](https://github.com/apache/spark/commit/e69e21348b4cde2abaec9dbb46381caf1ed3a1a4).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19932: [SPARK-22745][SQL] read partition stats from Hive

[wzhfy]

Github user wzhfy commented on the issue:

https://github.com/apache/spark/pull/19932
  
cc @cloud-fan @gatorsmile 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19594: [SPARK-21984] [SQL] Join estimation based on equi-height...

[wzhfy]

Github user wzhfy commented on the issue:

https://github.com/apache/spark/pull/19594
  
retest this please..


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19594: [SPARK-21984] [SQL] Join estimation based on equi-height...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19594
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84682/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19594: [SPARK-21984] [SQL] Join estimation based on equi-height...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19594
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19594: [SPARK-21984] [SQL] Join estimation based on equi-height...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19594
  
**[Test build #84682 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84682/testReport)**
 for PR 19594 at commit 
[`e69e213`](https://github.com/apache/spark/commit/e69e21348b4cde2abaec9dbb46381caf1ed3a1a4).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19931: [SPARK-22672][SQL][TEST][FOLLOWUP] Fix to use `sp...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/19931


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19931: [SPARK-22672][SQL][TEST][FOLLOWUP] Fix to use `spark.con...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19931
  
Merged to master.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19931: [SPARK-22672][SQL][TEST][FOLLOWUP] Fix to use `spark.con...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19931
  
@dongjoon-hyun mind double checking if there is anything we should do as a 
followup before we go merging this?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19932: [SPARK-22745][SQL] read partition stats from Hive

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19932
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84680/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19932: [SPARK-22745][SQL] read partition stats from Hive

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19932
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19932: [SPARK-22745][SQL] read partition stats from Hive

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19932
  
**[Test build #84680 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84680/testReport)**
 for PR 19932 at commit 
[`48b81b5`](https://github.com/apache/spark/commit/48b81b5065808ffeff99142a03cd59bf54a9ea5d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19933: [SPARK-22744][CORE] Add a configuration to show the appl...

[srowen]

Github user srowen commented on the issue:

https://github.com/apache/spark/pull/19933
  
Your app can already log what it likes. This isn't a config, just an 
attempt to pass a piece of info. I think this should be closed. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19676: [SPARK-14516][FOLLOWUP] Adding ClusteringEvaluator to ex...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19676
  
**[Test build #84681 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84681/testReport)**
 for PR 19676 at commit 
[`feb619d`](https://github.com/apache/spark/commit/feb619d657f6ff66dec240ee4619e6f53208ac18).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19676: [SPARK-14516][FOLLOWUP] Adding ClusteringEvaluator to ex...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19676
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84681/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19676: [SPARK-14516][FOLLOWUP] Adding ClusteringEvaluator to ex...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19676
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org