[GitHub] spark pull request #21369: [SPARK-22713][CORE] ExternalAppendOnlyMap leaks w...

[JerryLead]

Github user JerryLead commented on a diff in the pull request:

https://github.com/apache/spark/pull/21369#discussion_r189451809
  
--- Diff: 
core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala
 ---
@@ -585,17 +592,15 @@ class ExternalAppendOnlyMap[K, V, C](
   } else {
 logInfo(s"Task ${context.taskAttemptId} force spilling in-memory 
map to disk and " +
   s"it will release 
${org.apache.spark.util.Utils.bytesToString(getUsed())} memory")
-nextUpstream = spillMemoryIteratorToDisk(upstream)
+val nextUpstream = spillMemoryIteratorToDisk(upstream)
+assert(!upstream.hasNext)
 hasSpilled = true
+upstream = nextUpstream
--- End diff --

Thanks for fixing this issues. I think the potential solution is to change 
the `upstream` reference, but I have not tested if this change is sufficient 
and safe.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21342: [SPARK-24294] Throw SparkException when OOM in Broadcast...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21342
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3380/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21342: [SPARK-24294] Throw SparkException when OOM in Broadcast...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21342
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21342: [SPARK-24294] Throw SparkException when OOM in Broadcast...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21342
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90847/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21342: [SPARK-24294] Throw SparkException when OOM in Broadcast...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21342
  
**[Test build #90847 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90847/testReport)**
 for PR 21342 at commit 
[`bd24746`](https://github.com/apache/spark/commit/bd2474672c1090ed748182e6d89c4bec7f940328).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21342: [SPARK-24294] Throw SparkException when OOM in Broadcast...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21342
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21342: [SPARK-24294] Throw SparkException when OOM in Broadcast...

[jinxing64]

Github user jinxing64 commented on the issue:

https://github.com/apache/spark/pull/21342
  
@kiszk @cloud-fan 
Thanks a  lot for comments.
I tested manually and found that the bug exists in all fatal throwable.
In current change, I catch all fatal throwable and wrap in 
SparkFatalException，ThreadUtils will catch SparkFatalException and re-throw 
the wrapped throwable.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21342: [SPARK-24294] Throw SparkException when OOM in Broadcast...

[jinxing64]

Github user jinxing64 commented on the issue:

https://github.com/apache/spark/pull/21342
  
I will update the pr title if the change is on the right 
direction.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21342: [SPARK-24294] Throw SparkException when OOM in Broadcast...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21342
  
**[Test build #90847 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90847/testReport)**
 for PR 21342 at commit 
[`bd24746`](https://github.com/apache/spark/commit/bd2474672c1090ed748182e6d89c4bec7f940328).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21342: [SPARK-24294] Throw SparkException when OOM in Broadcast...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21342
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21342: [SPARK-24294] Throw SparkException when OOM in Broadcast...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21342
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90846/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21342: [SPARK-24294] Throw SparkException when OOM in Broadcast...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21342
  
**[Test build #90846 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90846/testReport)**
 for PR 21342 at commit 
[`bb5a74b`](https://github.com/apache/spark/commit/bb5a74b1e2770c475d0639e0751de7e8bfae8e73).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `public final class SparkOutOfMemoryException extends Exception `


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21342: [SPARK-24294] Throw SparkException when OOM in Broadcast...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21342
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3379/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21342: [SPARK-24294] Throw SparkException when OOM in Broadcast...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21342
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21342: [SPARK-24294] Throw SparkException when OOM in Broadcast...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21342
  
**[Test build #90846 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90846/testReport)**
 for PR 21342 at commit 
[`bb5a74b`](https://github.com/apache/spark/commit/bb5a74b1e2770c475d0639e0751de7e8bfae8e73).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21371
  
**[Test build #90845 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90845/testReport)**
 for PR 21371 at commit 
[`a823841`](https://github.com/apache/spark/commit/a823841fd419541137fd02d9a02822b1fa36f5ae).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21371
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21371
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3378/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/21371
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/21371
  
retest this please.

On Sun, May 20, 2018, 12:24 PM Apache Spark QA 
wrote:

> *Test build #90844 has finished
> 
*
> for PR 21371 at commit a823841
> 

> .
>
>- This patch *fails Spark unit tests*.
>- This patch merges cleanly.
>- This patch adds no public classes.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21371
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90844/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21371
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21371
  
**[Test build #90844 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90844/testReport)**
 for PR 21371 at commit 
[`a823841`](https://github.com/apache/spark/commit/a823841fd419541137fd02d9a02822b1fa36f5ae).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20761: [SPARK-20327][CORE][YARN] Add CLI support for YAR...

[galv]

Github user galv commented on a diff in the pull request:

https://github.com/apache/spark/pull/20761#discussion_r189449133
  
--- Diff: 
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ResourceTypeHelperSuite.scala
 ---
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.yarn
+
+import org.apache.hadoop.yarn.api.records.Resource
+import org.apache.hadoop.yarn.util.Records
+import org.scalatest.{BeforeAndAfterAll, Matchers}
+
+import org.apache.spark.SparkFunSuite
+import 
org.apache.spark.deploy.yarn.TestYarnResourceTypeHelper.ResourceInformation
+
+class ResourceTypeHelperSuite extends SparkFunSuite with Matchers with 
BeforeAndAfterAll {
+
+  private val CUSTOM_RES_1 = "custom-resource-type-1"
+  private val CUSTOM_RES_2 = "custom-resource-type-2"
+
+  override def beforeAll(): Unit = {
+super.beforeAll()
+  }
+
+  private def getExpectedUnmatchedErrorMessage(value: String) = {
+"Value of resource type should match pattern " +
+  s"([0-9]+)([A-Za-z]*), unmatched value: $value"
+  }
+
+  test("resource type value does not match pattern") {
+assume(ResourceTypeHelper.isYarnResourceTypesAvailable())
+TestYarnResourceTypeHelper.initializeResourceTypes(List())
+
+val resourceTypes = Map(CUSTOM_RES_1 -> "**@#")
+
+val thrown = intercept[IllegalArgumentException] {
+  ResourceTypeHelper.setResourceInfoFromResourceTypes(resourceTypes, 
createAResource)
+}
+thrown.getMessage should equal 
(getExpectedUnmatchedErrorMessage("**@#"))
+  }
+
+  test("resource type just unit defined") {
+assume(ResourceTypeHelper.isYarnResourceTypesAvailable())
+TestYarnResourceTypeHelper.initializeResourceTypes(List())
+
+val resourceTypes = Map(CUSTOM_RES_1 -> "m")
+
+val thrown = intercept[IllegalArgumentException] {
+  ResourceTypeHelper.setResourceInfoFromResourceTypes(resourceTypes, 
createAResource)
+}
+thrown.getMessage should equal (getExpectedUnmatchedErrorMessage("m"))
+  }
+
+  test("resource type with null value should not be allowed") {
+assume(ResourceTypeHelper.isYarnResourceTypesAvailable())
+TestYarnResourceTypeHelper.initializeResourceTypes(List())
+
+val resourceTypes = Map(CUSTOM_RES_1 -> "123")
+
+val thrown = intercept[IllegalArgumentException] {
+  ResourceTypeHelper.setResourceInfoFromResourceTypes(resourceTypes, 
null)
+}
+thrown.getMessage should equal ("Resource parameter should not be 
null!")
+  }
+
+  test("resource type with valid value and invalid unit") {
+assume(ResourceTypeHelper.isYarnResourceTypesAvailable())
+TestYarnResourceTypeHelper.initializeResourceTypes(List(CUSTOM_RES_1))
+
+val resourceTypes = Map(CUSTOM_RES_1 -> "123ppp")
+val resource = createAResource
+
+val thrown = intercept[IllegalArgumentException] {
+  ResourceTypeHelper.setResourceInfoFromResourceTypes(resourceTypes, 
resource)
+}
+thrown.getMessage should fullyMatch regex
+  """Unknown unit 'ppp'\. Known units are \[.*\]"""
+  }
+
+  test("resource type with valid value and without unit") {
+assume(ResourceTypeHelper.isYarnResourceTypesAvailable())
+TestYarnResourceTypeHelper.initializeResourceTypes(List(CUSTOM_RES_1))
+
+val resourceTypes = Map(CUSTOM_RES_1 -> "123")
+val resource = createAResource
+
+ResourceTypeHelper.setResourceInfoFromResourceTypes(resourceTypes, 
resource)
+val customResource: ResourceInformation = TestYarnResourceTypeHelper
+  .getResourceInformationByName(resource, CUSTOM_RES_1)
+customResource.name should equal (CUSTOM_RES_1)
+customResource.value should be (123)
+customResource.units should be ("")
+  }
+
+  test("resource type with valid value and unit") {
+

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/21374
  
@dongjoon-hyun thank you, l will kick this later.



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21371
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error and flaky...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21371
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3377/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/21371
  
@hvanhovell the test is flaky and I propose a fixing for it now. Please 
take a look if you have time. Thanks.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/21371#discussion_r189448749
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonInferSchema.scala
 ---
@@ -66,8 +69,12 @@ private[sql] object JsonInferSchema {
 s"Parse Mode: ${FailFastMode.name}.", e)
   }
 }
-  }
-}.fold(StructType(Nil))(
+  }.fold(StructType(Nil))(
+compatibleRootType(columnNameOfCorruptRecord, parseMode))
+  Iterator(typeInPartition)
+}.collect()
--- End diff --

Previously two foldings with `compatibleRootType` on executors and local in 
one `RDD.fold` call all use default SQL config values. Now on executors SQL 
config can work, so inconsistency happens because local folding uses default 
SQL config.




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21371
  
**[Test build #90844 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90844/testReport)**
 for PR 21371 at commit 
[`a823841`](https://github.com/apache/spark/commit/a823841fd419541137fd02d9a02822b1fa36f5ae).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21294: [SPARK-24197][SparkR][SQL] Adding array_sort function to...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/21294
  
looked into a bit not sure why..
- no related changes to testthat
- nothing in the R releases around this
- no config about default type of NA etc.
- tested this out but not seeing the test failure

perhaps this is something on the JVM? or Spark scala side changes?



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21371
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21371
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90840/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21375: [HOT-FIX][SQL] Fix: SQLConf.scala:1757: not found...

[wangyum]

Github user wangyum closed the pull request at:

https://github.com/apache/spark/pull/21375


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21371
  
**[Test build #90840 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90840/testReport)**
 for PR 21371 at commit 
[`7b193f6`](https://github.com/apache/spark/commit/7b193f6bd55838210936716f2c428ae2cc26489d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21375: [HOT-FIX][SQL] Fix: SQLConf.scala:1757: not found...

[wangyum]

GitHub user wangyum opened a pull request:

https://github.com/apache/spark/pull/21375

[HOT-FIX][SQL] Fix: SQLConf.scala:1757: not found: value Utils

## What changes were proposed in this pull request?

Fix: `SQLConf.scala:1757: not found: value Utils`

## How was this patch tested?

manual tests


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wangyum/spark hot-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/21375.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #21375


commit 5375965ca1de10e97ca6f2b0b984dbe0e9306c68
Author: Yuming Wang 
Date:   2018-05-20T00:30:58Z

Fix: SQLConf.scala:1757: not found: value Utils




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21371
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90841/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21371
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/21374
  
Hi, @kiszk and @felixcheung .
Unfortunately, the master branch build is broken by 
https://github.com/apache/spark/pull/21299 .



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21371
  
**[Test build #90841 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90841/testReport)**
 for PR 21371 at commit 
[`d010c17`](https://github.com/apache/spark/commit/d010c17f94e031e18a2edab682648bcaef692d5b).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/21370#discussion_r189447446
  
--- Diff: python/pyspark/sql/dataframe.py ---
@@ -78,6 +78,12 @@ def __init__(self, jdf, sql_ctx):
 self.is_cached = False
 self._schema = None  # initialized lazily
 self._lazy_rdd = None
+self._eager_eval = sql_ctx.getConf(
+"spark.jupyter.eagerEval.enabled", "false").lower() == "true"
--- End diff --

let's add all these to documentation too


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/21370#discussion_r189447423
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -3056,7 +3059,6 @@ class Dataset[T] private[sql](
* view, e.g. `SELECT * FROM global_temp.view1`.
*
* @throws AnalysisException if the view name is invalid or already 
exists
-   *
--- End diff --

why all these comment changes? could you revert them


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/21370#discussion_r189447465
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -237,9 +236,13 @@ class Dataset[T] private[sql](
* @param truncate If set to more than 0, truncates strings to 
`truncate` characters and
*   all cells will be aligned right.
* @param vertical If set to true, prints output rows vertically (one 
line per column value).
+   * @param html If set to true, return output as html table.
*/
   private[sql] def showString(
-  _numRows: Int, truncate: Int = 20, vertical: Boolean = false): 
String = {
+  _numRows: Int,
+  truncate: Int = 20,
--- End diff --

when the output is truncated, does jupyter handle that properly?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21370: [SPARK-24215][PySpark] Implement _repr_html_ for ...

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/21370#discussion_r189447477
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -237,9 +236,13 @@ class Dataset[T] private[sql](
* @param truncate If set to more than 0, truncates strings to 
`truncate` characters and
*   all cells will be aligned right.
* @param vertical If set to true, prints output rows vertically (one 
line per column value).
+   * @param html If set to true, return output as html table.
*/
   private[sql] def showString(
-  _numRows: Int, truncate: Int = 20, vertical: Boolean = false): 
String = {
+  _numRows: Int,
+  truncate: Int = 20,
--- End diff --

or does that depend on the html output on L300?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21374
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3376/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21374
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21374
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21374
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90843/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21374
  
**[Test build #90843 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90843/testReport)**
 for PR 21374 at commit 
[`a922b88`](https://github.com/apache/spark/commit/a922b88ecf70ba7815cdc2e50b58563f41489e64).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21370: [SPARK-24215][PySpark] Implement _repr_html_ for datafra...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/21370
  
we will wait for the tests to be fixed first.

@xuanyuanking could you update the PR description to clarify which is 
"before" which is "after"?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21370: [SPARK-24215][PySpark] Implement _repr_html_ for datafra...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/21370
  
@HyukjinKwon @holdenk @ueshin 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21374
  
**[Test build #90843 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90843/testReport)**
 for PR 21374 at commit 
[`a922b88`](https://github.com/apache/spark/commit/a922b88ecf70ba7815cdc2e50b58563f41489e64).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/21374
  
Jenkins, retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21374
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21374
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3375/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21374
  
**[Test build #90842 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90842/testReport)**
 for PR 21374 at commit 
[`a922b88`](https://github.com/apache/spark/commit/a922b88ecf70ba7815cdc2e50b58563f41489e64).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21374
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90842/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21374
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21374: [SPARK-24323][SQL] Fix lint-java errors

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21374
  
**[Test build #90842 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90842/testReport)**
 for PR 21374 at commit 
[`a922b88`](https://github.com/apache/spark/commit/a922b88ecf70ba7815cdc2e50b58563f41489e64).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21374: [SPARK-24323][SQL] Fix lint-java errors

[kiszk]

GitHub user kiszk opened a pull request:

https://github.com/apache/spark/pull/21374

[SPARK-24323][SQL] Fix lint-java errors

## What changes were proposed in this pull request?

This PR fixes the following errors reported by `lint-java`
```
% dev/lint-java
Using `mvn` from path: /usr/bin/mvn
Checkstyle checks failed at following occurrences:
[ERROR] 
src/main/java/org/apache/spark/sql/sources/v2/reader/InputPartition.java:[39] 
(sizes) LineLength: Line is longer than 100 characters (found 104).
[ERROR] 
src/main/java/org/apache/spark/sql/sources/v2/reader/InputPartitionReader.java:[26]
 (sizes) LineLength: Line is longer than 100 characters (found 110).
[ERROR] 
src/main/java/org/apache/spark/sql/sources/v2/reader/InputPartitionReader.java:[30]
 (sizes) LineLength: Line is longer than 100 characters (found 104).
```

## How was this patch tested?

Run `lint-java` manually.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kiszk/spark SPARK-24323

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/21374.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #21374


commit a922b88ecf70ba7815cdc2e50b58563f41489e64
Author: Kazuaki Ishizaki 
Date:   2018-05-19T23:21:56Z

fix lint errors




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21367: [SPARK-24321][SQL] Extract common code from Divid...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/21367#discussion_r189446823
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala
 ---
@@ -297,13 +279,39 @@ case class Divide(left: Expression, right: 
Expression) extends BinaryArithmetic
   if (${eval1.isNull}) {
 ${ev.isNull} = true;
   } else {
-${ev.value} = $divide;
+${ev.value} = $operation;
   }
 }""")
 }
   }
 }
 
+// scalastyle:off line.size.limit
+@ExpressionDescription(
+  usage = "expr1 _FUNC_ expr2 - Returns `expr1`/`expr2`. It always 
performs floating point division.",
+  examples = """
+Examples:
+  > SELECT 3 _FUNC_ 2;
+   1.5
+  > SELECT 2L _FUNC_ 2L;
+   1.0
+  """)
+// scalastyle:on line.size.limit
+case class Divide(left: Expression, right: Expression) extends DivModLike {
+
+  override def inputType: AbstractDataType = TypeCollection(DoubleType, 
DecimalType)
+
+  override def symbol: String = "/"
+  override def decimalMethod: String = "$div"
+  override def nullable: Boolean = true
--- End diff --

redundant?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21367: [SPARK-24321][SQL] Extract common code from Divide/Remai...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/21367
  
LGTM


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21373: [SPARK-24250][SQL][FOLLOWUP] Fix build failure

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21373
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90839/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21373: [SPARK-24250][SQL][FOLLOWUP] Fix build failure

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21373
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21373: [SPARK-24250][SQL][FOLLOWUP] Fix build failure

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21373
  
**[Test build #90839 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90839/testReport)**
 for PR 21373 at commit 
[`a3c0035`](https://github.com/apache/spark/commit/a3c0035a73c6ea1ce5228ec21be3334573a65678).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[mgaido91]

Github user mgaido91 commented on the issue:

https://github.com/apache/spark/pull/21371
  
LGTM


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21371
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3374/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21371
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/21371#discussion_r189446146
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/internal/ExecutorSideSQLConfSuite.scala
 ---
@@ -59,7 +59,8 @@ class ExecutorSideSQLConfSuite extends SparkFunSuite with 
SQLTestUtils {
 val pathString = path.getCanonicalPath
 spark.range(10).select('id.as("ID")).write.json(pathString)
 spark.range(10).write.mode("append").json(pathString)
-assert(spark.read.json(pathString).columns.toSet == Set("id", 
"ID"))
+val df = spark.read.option("samplingRatio", 1).json(pathString)
--- End diff --

oh, I just found `samplingRatio` is 1.0 by default. The flaky test should 
not because of this.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21371
  
**[Test build #90841 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90841/testReport)**
 for PR 21371 at commit 
[`d010c17`](https://github.com/apache/spark/commit/d010c17f94e031e18a2edab682648bcaef692d5b).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21373: [SPARK-24250][SQL][FOLLOWUP] Fix build failure

[mgaido91]

Github user mgaido91 commented on the issue:

https://github.com/apache/spark/pull/21373
  
@viirya sorry, I haven't seen it. Thanks for fixing it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21373: [SPARK-24250][SQL][FOLLOWUP] Fix build failure

[mgaido91]

Github user mgaido91 closed the pull request at:

https://github.com/apache/spark/pull/21373


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21371
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21371
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3373/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21371
  
**[Test build #90840 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90840/testReport)**
 for PR 21371 at commit 
[`7b193f6`](https://github.com/apache/spark/commit/7b193f6bd55838210936716f2c428ae2cc26489d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/21299
  
@viirya and @mgaido91 are working on the build failure, but there seem to 
be test failures, too.
Maybe, we had better revert this first to unblock all the PRs.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/21371
  
@hvanhovell I ran locally and seems no such test failure? Checking.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21371: [SPARK-24250][SQL][FollowUp] Fix compile error by adding...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/21371
  
retest this please.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21373: [SPARK-24250][SQL][FOLLOWUP] Fix build failure

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/21373
  
@mgaido91 ah, I've fired one earlier #21371. But seems there still a test 
failure needed to be addressed. I'm looking into it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21373: [SPARK-24250][SQL][FOLLOWUP] Fix build failure

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21373
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21373: [SPARK-24250][SQL][FOLLOWUP] Fix build failure

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21373
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3372/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21373: [SPARK-24250][SQL][FOLLOWUP] Fix build failure

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21373
  
**[Test build #90839 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90839/testReport)**
 for PR 21373 at commit 
[`a3c0035`](https://github.com/apache/spark/commit/a3c0035a73c6ea1ce5228ec21be3334573a65678).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21373: [SPARK-24250][SQL][FOLLOWUP] Fix build failure

[mgaido91]

Github user mgaido91 commented on the issue:

https://github.com/apache/spark/pull/21373
  
cc @cloud-fan  @dongjoon-hyun @gatorsmile 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #21373: [SPARK-24250][SQL][FOLLOWUP] Fix build failure

[mgaido91]

GitHub user mgaido91 opened a pull request:

https://github.com/apache/spark/pull/21373

[SPARK-24250][SQL][FOLLOWUP] Fix build failure

## What changes were proposed in this pull request?

SPARK-24250 is causing a build failure because it removed an import which 
was not used in that patch but was used in SPARK-23850, which was committed 
meanwhile.

## How was this patch tested?

building the project


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mgaido91/spark SPARK-24250_followup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/21373.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #21373


commit a3c0035a73c6ea1ce5228ec21be3334573a65678
Author: Marco Gaido 
Date:   2018-05-19T19:10:06Z

[SPARK-24250][SQL][FOLLOWUP] Fix build failure




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21372: [SPARK-24322][BUILD] Upgrade Apache ORC to 1.4.4

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/21372
  
The master branch failure is due to 
https://github.com/apache/spark/pull/21299 .


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21372: [SPARK-24322][BUILD] Upgrade Apache ORC to 1.4.4

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21372
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21372: [SPARK-24322][BUILD] Upgrade Apache ORC to 1.4.4

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21372
  
**[Test build #90838 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90838/testReport)**
 for PR 21372 at commit 
[`2d11cdc`](https://github.com/apache/spark/commit/2d11cdced5a53a7504cb2e52df67ad9256870d90).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21372: [SPARK-24322][BUILD] Upgrade Apache ORC to 1.4.4

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21372
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90838/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21299: [SPARK-24250][SQL] support accessing SQLConf inside task...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/21299
  
Hi, @cloud-fan.
This seems to cause a build failure. Could you take a look?

cc @gatorsmile 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21372: [SPARK-24322][BUILD] Upgrade Apache ORC to 1.4.4

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21372
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3371/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21372: [SPARK-24322][BUILD] Upgrade Apache ORC to 1.4.4

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21372
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21372: [SPARK-24322][BUILD] Upgrade Apache ORC to 1.4.4

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21372
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3370/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16171: [SPARK-18739][ML][PYSPARK] Classification and regression...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16171
  
Build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16171: [SPARK-18739][ML][PYSPARK] Classification and regression...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16171
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 

https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3369/
Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21372: [SPARK-24322][BUILD] Upgrade Apache ORC to 1.4.4

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21372
  
Merged build finished. Test PASSed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21372: [SPARK-24322][BUILD] Upgrade Apache ORC to 1.4.4

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21372
  
**[Test build #90838 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90838/testReport)**
 for PR 21372 at commit 
[`2d11cdc`](https://github.com/apache/spark/commit/2d11cdced5a53a7504cb2e52df67ad9256870d90).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21372: [SPARK-24322][BUILD] Upgrade Apache ORC to 1.4.4

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/21372
  
Retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21372: [SPARK-24322][BUILD] Upgrade Apache ORC to 1.4.4

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21372
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90837/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21372: [SPARK-24322][BUILD] Upgrade Apache ORC to 1.4.4

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/21372
  
**[Test build #90837 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90837/testReport)**
 for PR 21372 at commit 
[`2d11cdc`](https://github.com/apache/spark/commit/2d11cdced5a53a7504cb2e52df67ad9256870d90).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21372: [SPARK-24322][BUILD] Upgrade Apache ORC to 1.4.4

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21372
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org