[GitHub] spark issue #23099: [SPARK-25954][SS] Upgrade to Kafka 2.1.0
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23099 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23099: [SPARK-25954][SS] Upgrade to Kafka 2.1.0
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23099 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99093/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23099: [SPARK-25954][SS] Upgrade to Kafka 2.1.0
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23099 **[Test build #99093 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99093/testReport)** for PR 23099 at commit [`7e9a65c`](https://github.com/apache/spark/commit/7e9a65c8f5227abe967cfc053781f91a3b742a07). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user tasanuma commented on the issue: https://github.com/apache/spark/pull/23101 Updated PR adding jetty-sslengine to `dev/deps/spark-deps-hadoop-2.7`. It is required by [HADOOP-12765](https://issues.apache.org/jira/browse/HADOOP-12765). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5214/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23101 **[Test build #99100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99100/testReport)** for PR 23101 at commit [`32c6f5b`](https://github.com/apache/spark/commit/32c6f5beebca465a602fb8ecbcf48aadd293f037). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22149: [SPARK-25158][SQL]Executor accidentally exit beca...
Github user LuciferYang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235276659
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
---
@@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread(
}
threwException = false
} catch {
+ // TaskKilledException should not be thrown again, otherwise it will
be captured by
+ // SparkUncaughtExceptionHandler, then Executor will exit because of
TaskKilledException.
+ case e: TaskKilledException =>
--- End diff --
Cannot argree more ~
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23070 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23070 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99094/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23070 **[Test build #99094 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99094/testReport)** for PR 23070 at commit [`bd2debc`](https://github.com/apache/spark/commit/bd2debcc2237ad178ef00b762bcdc80b63d1ecb7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21732 last comment, LGTM otherwise --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21732: [SPARK-24762][SQL] Enable Option of Product encod...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21732#discussion_r235273262
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala
---
@@ -257,6 +251,11 @@ case class ExpressionEncoder[T](
*/
def isSerializedAsStruct: Boolean =
objSerializer.dataType.isInstanceOf[StructType]
+ /**
+ * Returns true if the type `T` is `Option`.
+ */
+ def isOptionType: Boolean =
classOf[Option[_]].isAssignableFrom(clsTag.runtimeClass)
--- End diff --
regarding maintainability, I think it's better to add a
```
// maybe there is a better naming
def isSerializedAsStructForTopLevel: Boolean = {
isSerializedAsStruct && isOption
}
```
The benefit is, when people call `isSerializedAsStruct`, they will see this
similar method and think carefully which one he should use. Otherwise, people
may miss to check `isOptionType` easily.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23027 @wangyum, why did you close this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23101 **[Test build #99099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99099/testReport)** for PR 23101 at commit [`3627e4f`](https://github.com/apache/spark/commit/3627e4f7cbd4a873fa2b06fd50d6d371a4e82817). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99099/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5213/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23101 **[Test build #99099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99099/testReport)** for PR 23101 at commit [`3627e4f`](https://github.com/apache/spark/commit/3627e4f7cbd4a873fa2b06fd50d6d371a4e82817). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23058: [SPARK-25905][CORE] When getting a remote block, avoid f...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23058 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23058: [SPARK-25905][CORE] When getting a remote block, avoid f...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23058 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99091/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user tasanuma commented on the issue: https://github.com/apache/spark/pull/23101 Thanks all for your support. Updated PR addressing @ueshin's comment. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23058: [SPARK-25905][CORE] When getting a remote block, avoid f...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23058 **[Test build #99091 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99091/testReport)** for PR 23058 at commit [`125d746`](https://github.com/apache/spark/commit/125d746d25d02e32a593d60535ce1b4936543805). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/23101 also cc @wangyum for the owner of [SPARK-25330](https://issues.apache.org/jira/browse/SPARK-25330). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23101 cc @dbtsai --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/23101 cc @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/23101 I guess we need to update `dev/deps/spark-deps-hadoop-2.7` file as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23101 **[Test build #99098 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99098/testReport)** for PR 23101 at commit [`24e4942`](https://github.com/apache/spark/commit/24e49422cafa064a053827923fdd8154499fc53c). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99098/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23101 **[Test build #99098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99098/testReport)** for PR 23101 at commit [`24e4942`](https://github.com/apache/spark/commit/24e49422cafa064a053827923fdd8154499fc53c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5212/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22149: [SPARK-25158][SQL]Executor accidentally exit beca...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235266375
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
---
@@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread(
}
threwException = false
} catch {
+ // TaskKilledException should not be thrown again, otherwise it will
be captured by
+ // SparkUncaughtExceptionHandler, then Executor will exit because of
TaskKilledException.
+ case e: TaskKilledException =>
--- End diff --
I see. So `ScriptTransformationExec` is special because
1. it starts a new thread
2. the new thread is very likely to throw `TaskKilledException`, when
speculation is on.
I think we should not kill the executor just because
`ScriptTransformationWriterThread` fails. We should log the error, instead of
throwing it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/23101 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23054 **[Test build #99097 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99097/testReport)** for PR 23054 at commit [`d784142`](https://github.com/apache/spark/commit/d784142a0ee7ecf4300e542ddf2647e059217902). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5211/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/23054 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21732 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21732 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99090/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21732 **[Test build #99090 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99090/testReport)** for PR 21732 at commit [`4e4718a`](https://github.com/apache/spark/commit/4e4718ab2af448e5a039c5e564c6e33e0584e07c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to ...
GitHub user tasanuma opened a pull request: https://github.com/apache/spark/pull/23101 [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix java.version problem ## What changes were proposed in this pull request? Hadoop-2.7.3 has a issue that fails to parse some java.version. Please see [SPARK-26134](https://issues.apache.org/jira/browse/SPARK-26134) for the detail. ## How was this patch tested? You can merge this pull request into a Git repository by running: $ git pull https://github.com/tasanuma/spark SPARK-26134 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23101.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23101 commit 24e49422cafa064a053827923fdd8154499fc53c Author: Takanobu Asanuma Date: 2018-11-21T06:01:11Z upgrade hadoop from 2.7.3 to 2.7.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22683: [SPARK-25696] The storage memory displayed on spark Appl...
Github user httfighter commented on the issue: https://github.com/apache/spark/pull/22683 @srowen Thank you for your review. I agree with you, and I will make changes in the near future. @wangyum Thank you for your help. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23086: [SPARK-25528][SQL] data source v2 API refactor (batch re...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23086 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23086: [SPARK-25528][SQL] data source v2 API refactor (batch re...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23086 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99088/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23086: [SPARK-25528][SQL] data source v2 API refactor (batch re...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23086 **[Test build #99088 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99088/testReport)** for PR 23086 at commit [`188be4f`](https://github.com/apache/spark/commit/188be4ff3d21677893ebcac4c3d59e948dbf4494). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class DataSourceV2StreamingScanExec(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5210/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99096/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23054 **[Test build #99096 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99096/testReport)** for PR 23054 at commit [`d784142`](https://github.com/apache/spark/commit/d784142a0ee7ecf4300e542ddf2647e059217902). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22092: [SPARK-25101][CORE]Creating leaderLatch with id for gett...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22092 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Github user wangyum commented on a diff in the pull request:
https://github.com/apache/spark/pull/23027#discussion_r235259340
--- Diff: sql/core/benchmarks/FilterPushdownBenchmark-results.txt ---
@@ -2,669 +2,809 @@
Pushdown for many distinct value case
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row (value IS NULL): Best/Avg Time(ms)Rate(M/s)
Per Row(ns) Relative
-Parquet Vectorized 11405 / 11485 1.4
725.1 1.0X
-Parquet Vectorized (Pushdown) 675 / 690 23.3
42.9 16.9X
-Native ORC Vectorized 7127 / 7170 2.2
453.1 1.6X
-Native ORC Vectorized (Pushdown) 519 / 541 30.3
33.0 22.0X
+Parquet Vectorized7823 / 7996 2.0
497.4 1.0X
+Parquet Vectorized (Pushdown) 460 / 468 34.2
29.2 17.0X
+Native ORC Vectorized 5412 / 5550 2.9
344.1 1.4X
+Native ORC Vectorized (Pushdown) 551 / 563 28.6
35.0 14.2X
+InMemoryTable Vectorized 6 /6 2859.1
0.31422.0X
+InMemoryTable Vectorized (Pushdown) 5 /6 3023.0
0.31503.6X
-OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
-Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
+Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
+Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
-Parquet Vectorized 11457 / 11473 1.4
728.4 1.0X
-Parquet Vectorized (Pushdown) 656 / 686 24.0
41.7 17.5X
-Native ORC Vectorized 7328 / 7342 2.1
465.9 1.6X
-Native ORC Vectorized (Pushdown) 539 / 565 29.2
34.2 21.3X
+Parquet Vectorized 8322 / 11160 1.9
529.1 1.0X
+Parquet Vectorized (Pushdown) 463 / 472 34.0
29.4 18.0X
+Native ORC Vectorized 5622 / 5635 2.8
357.4 1.5X
+Native ORC Vectorized (Pushdown) 563 / 595 27.9
35.8 14.8X
+InMemoryTable Vectorized 4831 / 4881 3.3
307.2 1.7X
+InMemoryTable Vectorized (Pushdown) 1980 / 2027 7.9
125.9 4.2X
--- End diff --
I think the reason is
[SPARK-22599](https://issues.apache.org/jira/browse/SPARK-22599). But if we
cached all data to memory, the result is:
```
Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms)
Rate(M/s) Per Row(ns) Relative
Parquet Vectorized6491 / 6716 2.4
412.7 1.0X
Parquet Vectorized (Pushdown) 491 / 496 32.0
31.2 13.2X
Native ORC Vectorized 5849 / 6103 2.7
371.9 1.1X
Native ORC Vectorized (Pushdown) 533 / 572 29.5
33.9 12.2X
InMemoryTable Vectorized 2788 / 2854 5.6
177.2 2.3X
InMemoryTable Vectorized (Pushdown)370 / 408 42.5
23.5 17.5X
```
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Github user wangyum closed the pull request at: https://github.com/apache/spark/pull/23027 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22879: [SPARK-25872][SQL][TEST] Add an optimizer tracker...
Github user wangyum closed the pull request at: https://github.com/apache/spark/pull/22879 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22149: [SPARK-25158][SQL]Executor accidentally exit beca...
Github user LuciferYang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235257700
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
---
@@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread(
}
threwException = false
} catch {
+ // TaskKilledException should not be thrown again, otherwise it will
be captured by
+ // SparkUncaughtExceptionHandler, then Executor will exit because of
TaskKilledException.
+ case e: TaskKilledException =>
--- End diff --
Sorry, maybe I didn't express it clearly.
There is no essential difference between `ScriptTransformationExec` and
other operators in exception processing, exceptions handle by `catch` block of
`Executor.TaskRunner#run` method.
The key point is`ScriptTransformationExec` create a new thread named
`ScriptTransformationWriterThread `, it belongs to main threadgroup,
Exception throw from this thread cannot catch by `catch` block of
`Executor.TaskRunner#run` method, and it will handle by
`SparkUncaughtExceptionHandler` which we register in `Executor` at noLocal run
model, `SparkUncaughtExceptionHandler` will call
`System.exit(SparkExitCode.UNCAUGHT_EXCEPTION)` when `case _ if
exitOnUncaughtException`
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22149: [SPARK-25158][SQL]Executor accidentally exit beca...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235253094
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
---
@@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread(
}
threwException = false
} catch {
+ // TaskKilledException should not be thrown again, otherwise it will
be captured by
+ // SparkUncaughtExceptionHandler, then Executor will exit because of
TaskKilledException.
+ case e: TaskKilledException =>
--- End diff --
> Actually ScriptTransformationExec like a streaming pipe, but other
operators basically no such characteristics.
can you elaborate on it? How `TaskKilledException` is handled in other
operators?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21688 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21688 **[Test build #99084 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99084/testReport)** for PR 21688 at commit [`8f6efbd`](https://github.com/apache/spark/commit/8f6efbda4ad6b7fc4c86f4a2ae43e49847243826). * This patch **fails from timeout after a configured wait of `400m`**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21688 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99084/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22305: [SPARK-24561][SQL][Python] User-defined window aggregati...
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/22305 @BryanCutler @HyukjinKwon @ueshin I have addressed all the comments so far. Could you please take another look? Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22149: [SPARK-25158][SQL]Executor accidentally exit beca...
Github user LuciferYang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235248435
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
---
@@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread(
}
threwException = false
} catch {
+ // TaskKilledException should not be thrown again, otherwise it will
be captured by
+ // SparkUncaughtExceptionHandler, then Executor will exit because of
TaskKilledException.
+ case e: TaskKilledException =>
--- End diff --
On the other hand perhaps `ScriptTransformationWriterThread` should not
throw any Exception because the Exception mark and handle by
`ScriptTransformationExec` , but I'm not sure.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23054 **[Test build #99096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99096/testReport)** for PR 23054 at commit [`d784142`](https://github.com/apache/spark/commit/d784142a0ee7ecf4300e542ddf2647e059217902). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5209/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23054: [SPARK-26085][SQL] Key attribute of non-struct ty...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/23054#discussion_r235247826
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -1594,6 +1594,15 @@ object SQLConf {
"WHERE, which does not follow SQL standard.")
.booleanConf
.createWithDefault(false)
+
+ val ALIAS_NON_STRUCT_GROUPING_KEY_AS_VALUE =
+buildConf("spark.sql.legacy.dataset.aliasNonStructGroupingKeyAsValue")
+ .internal()
+ .doc("When set to true, the key attribute resulted from running
`Dataset.groupByKey` " +
+"for non-struct key type, will be named as `value`, following the
behavior of Spark " +
--- End diff --
`nameNonStructGroupingKeyAsValue` is fine to me.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23100 **[Test build #99095 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99095/testReport)** for PR 23100 at commit [`ee3de58`](https://github.com/apache/spark/commit/ee3de5862e975f9659af474c43133294ec5ce369). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23099: [SPARK-25954][SS] Upgrade to Kafka 2.1.0
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23099 Thank you for retriggering, @HyukjinKwon . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5208/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22149: [SPARK-25158][SQL]Executor accidentally exit beca...
Github user LuciferYang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235247176
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
---
@@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread(
}
threwException = false
} catch {
+ // TaskKilledException should not be thrown again, otherwise it will
be captured by
+ // SparkUncaughtExceptionHandler, then Executor will exit because of
TaskKilledException.
+ case e: TaskKilledException =>
--- End diff --
Actually ` ScriptTransformationExec` like a streaming pipe, but other
operators basically no such characteristics.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22149: [SPARK-25158][SQL]Executor accidentally exit beca...
Github user LuciferYang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22149#discussion_r235246544
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala
---
@@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread(
}
threwException = false
} catch {
+ // TaskKilledException should not be thrown again, otherwise it will
be captured by
+ // SparkUncaughtExceptionHandler, then Executor will exit because of
TaskKilledException.
+ case e: TaskKilledException =>
--- End diff --
Thanks for your reply @cloud-fan , the error log and problem analysis of
one bad case I paste in https://issues.apache.org/jira/browse/SPARK-25158, in
that case `.UnsafeInMemorySorter$SortedIterator.loadNext` throw
`TaskKilledException`, and in another case `TaskKilledException` throw by
`InterruptibleIterator.hasNext` , and from the error stack line 281
`iter.map(outputProjection).foreach ` is the stack root.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99092/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23070 **[Test build #99094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99094/testReport)** for PR 23070 at commit [`bd2debc`](https://github.com/apache/spark/commit/bd2debcc2237ad178ef00b762bcdc80b63d1ecb7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23100 **[Test build #99092 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99092/testReport)** for PR 23100 at commit [`cf6da4b`](https://github.com/apache/spark/commit/cf6da4b72d04ab109739400dcbf6d75a9d34625e). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23070 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23070 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5207/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23098: [WIP][SPARK-26132][BUILD][CORE] Remove support for Scala...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23098 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23043: [SPARK-26021][SQL] replace minus zero with zero in Platf...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/23043 Please also update the PR description to reflect latest change. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23098: [WIP][SPARK-26132][BUILD][CORE] Remove support for Scala...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23098 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99087/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22939: [SPARK-25446][R] Add schema_of_json() and schema_of_csv(...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22939 gentle ping, @felixcheung. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23098: [WIP][SPARK-26132][BUILD][CORE] Remove support for Scala...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23098 **[Test build #99087 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99087/testReport)** for PR 23098 at commit [`1c1f000`](https://github.com/apache/spark/commit/1c1f0006c4a9439459d07132180b8f7fbd26b879). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23043: [SPARK-26021][SQL] replace minus zero with zero i...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/23043#discussion_r235244695
--- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
---
@@ -128,6 +133,11 @@ public static double getDouble(Object object, long
offset) {
}
public static void putDouble(Object object, long offset, double value) {
+if (Double.isNaN(value)) {
+ value = Double.NaN;
+} else if(value == -0.0d) {
--- End diff --
ditto.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23043: [SPARK-26021][SQL] replace minus zero with zero i...
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/23043#discussion_r235244686
--- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java
---
@@ -120,6 +120,11 @@ public static float getFloat(Object object, long
offset) {
}
public static void putFloat(Object object, long offset, float value) {
+if (Float.isNaN(value)) {
+ value = Float.NaN;
+} else if(value == -0.0f) {
--- End diff --
Add a space after if for consistent style.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23099: [WIP][SPARK-25954][SS] Upgrade to Kafka 2.1.0
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23099 **[Test build #99093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99093/testReport)** for PR 23099 at commit [`7e9a65c`](https://github.com/apache/spark/commit/7e9a65c8f5227abe967cfc053781f91a3b742a07). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23070 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5205/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23099: [WIP][SPARK-25954][SS] Upgrade to Kafka 2.1.0
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23099 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5206/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23071: [SPARK-26102][SQL][TEST] Extracting common CSV/JSON func...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23071 I think we don't need this for now. Let's do this when more `from/to_...` functions are added later. The amount of codes increases actually. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23099: [WIP][SPARK-25954][SS] Upgrade to Kafka 2.1.0
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23099 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23080: [SPARK-26108][SQL] Support custom lineSep in CSV datasou...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23080 @MaxGekk, let's rebase this one accordingly with encoding support. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23080: [SPARK-26108][SQL] Support custom lineSep in CSV ...
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/23080#discussion_r235244407
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
---
@@ -192,6 +192,20 @@ class CSVOptions(
*/
val emptyValueInWrite = emptyValue.getOrElse("\"\"")
+ /**
+ * A string between two consecutive JSON records.
+ */
+ val lineSeparator: Option[String] = parameters.get("lineSep").map { sep
=>
+require(sep.nonEmpty, "'lineSep' cannot be an empty string.")
+require(sep.length <= 2, "'lineSep' can contain 1 or 2 characters.")
--- End diff --
Hm, I see.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23085: [Docs] Added csv, orc, and text output format options to...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23085 @mrandrewandrade, let's close this for now. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
