[GitHub] spark issue #23099: [SPARK-25954][SS] Upgrade to Kafka 2.1.0
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23099 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23099: [SPARK-25954][SS] Upgrade to Kafka 2.1.0
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23099 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99093/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23099: [SPARK-25954][SS] Upgrade to Kafka 2.1.0
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23099 **[Test build #99093 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99093/testReport)** for PR 23099 at commit [`7e9a65c`](https://github.com/apache/spark/commit/7e9a65c8f5227abe967cfc053781f91a3b742a07). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user tasanuma commented on the issue: https://github.com/apache/spark/pull/23101 Updated PR adding jetty-sslengine to `dev/deps/spark-deps-hadoop-2.7`. It is required by [HADOOP-12765](https://issues.apache.org/jira/browse/HADOOP-12765). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5214/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23101 **[Test build #99100 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99100/testReport)** for PR 23101 at commit [`32c6f5b`](https://github.com/apache/spark/commit/32c6f5beebca465a602fb8ecbcf48aadd293f037). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22149: [SPARK-25158][SQL]Executor accidentally exit beca...
Github user LuciferYang commented on a diff in the pull request: https://github.com/apache/spark/pull/22149#discussion_r235276659 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala --- @@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread( } threwException = false } catch { + // TaskKilledException should not be thrown again, otherwise it will be captured by + // SparkUncaughtExceptionHandler, then Executor will exit because of TaskKilledException. + case e: TaskKilledException => --- End diff -- Cannot argree more ~ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23070 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23070 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99094/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23070 **[Test build #99094 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99094/testReport)** for PR 23070 at commit [`bd2debc`](https://github.com/apache/spark/commit/bd2debcc2237ad178ef00b762bcdc80b63d1ecb7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21732 last comment, LGTM otherwise --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #21732: [SPARK-24762][SQL] Enable Option of Product encod...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21732#discussion_r235273262 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala --- @@ -257,6 +251,11 @@ case class ExpressionEncoder[T]( */ def isSerializedAsStruct: Boolean = objSerializer.dataType.isInstanceOf[StructType] + /** + * Returns true if the type `T` is `Option`. + */ + def isOptionType: Boolean = classOf[Option[_]].isAssignableFrom(clsTag.runtimeClass) --- End diff -- regarding maintainability, I think it's better to add a ``` // maybe there is a better naming def isSerializedAsStructForTopLevel: Boolean = { isSerializedAsStruct && isOption } ``` The benefit is, when people call `isSerializedAsStruct`, they will see this similar method and think carefully which one he should use. Otherwise, people may miss to check `isOptionType` easily. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark add InM...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23027 @wangyum, why did you close this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23101 **[Test build #99099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99099/testReport)** for PR 23101 at commit [`3627e4f`](https://github.com/apache/spark/commit/3627e4f7cbd4a873fa2b06fd50d6d371a4e82817). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99099/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5213/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23101 **[Test build #99099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99099/testReport)** for PR 23101 at commit [`3627e4f`](https://github.com/apache/spark/commit/3627e4f7cbd4a873fa2b06fd50d6d371a4e82817). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23058: [SPARK-25905][CORE] When getting a remote block, avoid f...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23058 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23058: [SPARK-25905][CORE] When getting a remote block, avoid f...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23058 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99091/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user tasanuma commented on the issue: https://github.com/apache/spark/pull/23101 Thanks all for your support. Updated PR addressing @ueshin's comment. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23058: [SPARK-25905][CORE] When getting a remote block, avoid f...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23058 **[Test build #99091 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99091/testReport)** for PR 23058 at commit [`125d746`](https://github.com/apache/spark/commit/125d746d25d02e32a593d60535ce1b4936543805). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/23101 also cc @wangyum for the owner of [SPARK-25330](https://issues.apache.org/jira/browse/SPARK-25330). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23101 cc @dbtsai --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/23101 cc @srowen --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/23101 I guess we need to update `dev/deps/spark-deps-hadoop-2.7` file as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23101 **[Test build #99098 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99098/testReport)** for PR 23101 at commit [`24e4942`](https://github.com/apache/spark/commit/24e49422cafa064a053827923fdd8154499fc53c). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99098/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23101 **[Test build #99098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99098/testReport)** for PR 23101 at commit [`24e4942`](https://github.com/apache/spark/commit/24e49422cafa064a053827923fdd8154499fc53c). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5212/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22149: [SPARK-25158][SQL]Executor accidentally exit beca...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22149#discussion_r235266375 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala --- @@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread( } threwException = false } catch { + // TaskKilledException should not be thrown again, otherwise it will be captured by + // SparkUncaughtExceptionHandler, then Executor will exit because of TaskKilledException. + case e: TaskKilledException => --- End diff -- I see. So `ScriptTransformationExec` is special because 1. it starts a new thread 2. the new thread is very likely to throw `TaskKilledException`, when speculation is on. I think we should not kill the executor just because `ScriptTransformationWriterThread` fails. We should log the error, instead of throwing it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/23101 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23054 **[Test build #99097 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99097/testReport)** for PR 23054 at commit [`d784142`](https://github.com/apache/spark/commit/d784142a0ee7ecf4300e542ddf2647e059217902). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5211/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/23054 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21732 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21732 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99090/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21732: [SPARK-24762][SQL] Enable Option of Product encoders
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21732 **[Test build #99090 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99090/testReport)** for PR 21732 at commit [`4e4718a`](https://github.com/apache/spark/commit/4e4718ab2af448e5a039c5e564c6e33e0584e07c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix jav...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23101 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23101: [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to ...
GitHub user tasanuma opened a pull request: https://github.com/apache/spark/pull/23101 [SPARK-26134][CORE] Upgrading Hadoop to 2.7.4 to fix java.version problem ## What changes were proposed in this pull request? Hadoop-2.7.3 has a issue that fails to parse some java.version. Please see [SPARK-26134](https://issues.apache.org/jira/browse/SPARK-26134) for the detail. ## How was this patch tested? You can merge this pull request into a Git repository by running: $ git pull https://github.com/tasanuma/spark SPARK-26134 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/23101.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #23101 commit 24e49422cafa064a053827923fdd8154499fc53c Author: Takanobu Asanuma Date: 2018-11-21T06:01:11Z upgrade hadoop from 2.7.3 to 2.7.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22683: [SPARK-25696] The storage memory displayed on spark Appl...
Github user httfighter commented on the issue: https://github.com/apache/spark/pull/22683 @srowen Thank you for your review. I agree with you, and I will make changes in the near future. @wangyum Thank you for your help. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23086: [SPARK-25528][SQL] data source v2 API refactor (batch re...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23086 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23086: [SPARK-25528][SQL] data source v2 API refactor (batch re...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23086 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99088/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23086: [SPARK-25528][SQL] data source v2 API refactor (batch re...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23086 **[Test build #99088 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99088/testReport)** for PR 23086 at commit [`188be4f`](https://github.com/apache/spark/commit/188be4ff3d21677893ebcac4c3d59e948dbf4494). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class DataSourceV2StreamingScanExec(` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5210/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20958: [SPARK-23844][SS] Fix socket source honors recovered off...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20958 Build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99096/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23054 **[Test build #99096 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99096/testReport)** for PR 23054 at commit [`d784142`](https://github.com/apache/spark/commit/d784142a0ee7ecf4300e542ddf2647e059217902). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22092: [SPARK-25101][CORE]Creating leaderLatch with id for gett...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22092 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/23027#discussion_r235259340 --- Diff: sql/core/benchmarks/FilterPushdownBenchmark-results.txt --- @@ -2,669 +2,809 @@ Pushdown for many distinct value case -OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64 -Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz +Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz Select 0 string row (value IS NULL): Best/Avg Time(ms)Rate(M/s) Per Row(ns) Relative -Parquet Vectorized 11405 / 11485 1.4 725.1 1.0X -Parquet Vectorized (Pushdown) 675 / 690 23.3 42.9 16.9X -Native ORC Vectorized 7127 / 7170 2.2 453.1 1.6X -Native ORC Vectorized (Pushdown) 519 / 541 30.3 33.0 22.0X +Parquet Vectorized7823 / 7996 2.0 497.4 1.0X +Parquet Vectorized (Pushdown) 460 / 468 34.2 29.2 17.0X +Native ORC Vectorized 5412 / 5550 2.9 344.1 1.4X +Native ORC Vectorized (Pushdown) 551 / 563 28.6 35.0 14.2X +InMemoryTable Vectorized 6 /6 2859.1 0.31422.0X +InMemoryTable Vectorized (Pushdown) 5 /6 3023.0 0.31503.6X -OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64 -Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz +Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6 +Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative -Parquet Vectorized 11457 / 11473 1.4 728.4 1.0X -Parquet Vectorized (Pushdown) 656 / 686 24.0 41.7 17.5X -Native ORC Vectorized 7328 / 7342 2.1 465.9 1.6X -Native ORC Vectorized (Pushdown) 539 / 565 29.2 34.2 21.3X +Parquet Vectorized 8322 / 11160 1.9 529.1 1.0X +Parquet Vectorized (Pushdown) 463 / 472 34.0 29.4 18.0X +Native ORC Vectorized 5622 / 5635 2.8 357.4 1.5X +Native ORC Vectorized (Pushdown) 563 / 595 27.9 35.8 14.8X +InMemoryTable Vectorized 4831 / 4881 3.3 307.2 1.7X +InMemoryTable Vectorized (Pushdown) 1980 / 2027 7.9 125.9 4.2X --- End diff -- I think the reason is [SPARK-22599](https://issues.apache.org/jira/browse/SPARK-22599). But if we cached all data to memory, the result is: ``` Java HotSpot(TM) 64-Bit Server VM 1.8.0_191-b12 on Mac OS X 10.12.6 Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz Select 0 string row ('7864320' < value < '7864320'): Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative Parquet Vectorized6491 / 6716 2.4 412.7 1.0X Parquet Vectorized (Pushdown) 491 / 496 32.0 31.2 13.2X Native ORC Vectorized 5849 / 6103 2.7 371.9 1.1X Native ORC Vectorized (Pushdown) 533 / 572 29.5 33.9 12.2X InMemoryTable Vectorized 2788 / 2854 5.6 177.2 2.3X InMemoryTable Vectorized (Pushdown)370 / 408 42.5 23.5 17.5X ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23027: [SPARK-26049][SQL][TEST] FilterPushdownBenchmark ...
Github user wangyum closed the pull request at: https://github.com/apache/spark/pull/23027 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22879: [SPARK-25872][SQL][TEST] Add an optimizer tracker...
Github user wangyum closed the pull request at: https://github.com/apache/spark/pull/22879 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22149: [SPARK-25158][SQL]Executor accidentally exit beca...
Github user LuciferYang commented on a diff in the pull request: https://github.com/apache/spark/pull/22149#discussion_r235257700 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala --- @@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread( } threwException = false } catch { + // TaskKilledException should not be thrown again, otherwise it will be captured by + // SparkUncaughtExceptionHandler, then Executor will exit because of TaskKilledException. + case e: TaskKilledException => --- End diff -- Sorry, maybe I didn't express it clearly. There is no essential difference between `ScriptTransformationExec` and other operators in exception processing, exceptions handle by `catch` block of `Executor.TaskRunner#run` method. The key point is`ScriptTransformationExec` create a new thread named `ScriptTransformationWriterThread `, it belongs to main threadgroup, Exception throw from this thread cannot catch by `catch` block of `Executor.TaskRunner#run` method, and it will handle by `SparkUncaughtExceptionHandler` which we register in `Executor` at noLocal run model, `SparkUncaughtExceptionHandler` will call `System.exit(SparkExitCode.UNCAUGHT_EXCEPTION)` when `case _ if exitOnUncaughtException` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22149: [SPARK-25158][SQL]Executor accidentally exit beca...
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22149#discussion_r235253094 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala --- @@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread( } threwException = false } catch { + // TaskKilledException should not be thrown again, otherwise it will be captured by + // SparkUncaughtExceptionHandler, then Executor will exit because of TaskKilledException. + case e: TaskKilledException => --- End diff -- > Actually ScriptTransformationExec like a streaming pipe, but other operators basically no such characteristics. can you elaborate on it? How `TaskKilledException` is handled in other operators? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21688 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21688 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99084/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21688: [SPARK-21809] : Change Stage Page to use datatables to s...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21688 **[Test build #99084 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99084/testReport)** for PR 21688 at commit [`8f6efbd`](https://github.com/apache/spark/commit/8f6efbda4ad6b7fc4c86f4a2ae43e49847243826). * This patch **fails from timeout after a configured wait of `400m`**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22305: [SPARK-24561][SQL][Python] User-defined window aggregati...
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/22305 @BryanCutler @HyukjinKwon @ueshin I have addressed all the comments so far. Could you please take another look? Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22149: [SPARK-25158][SQL]Executor accidentally exit beca...
Github user LuciferYang commented on a diff in the pull request: https://github.com/apache/spark/pull/22149#discussion_r235248435 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala --- @@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread( } threwException = false } catch { + // TaskKilledException should not be thrown again, otherwise it will be captured by + // SparkUncaughtExceptionHandler, then Executor will exit because of TaskKilledException. + case e: TaskKilledException => --- End diff -- On the other hand perhaps `ScriptTransformationWriterThread` should not throw any Exception because the Exception mark and handle by `ScriptTransformationExec` , but I'm not sure. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23054 **[Test build #99096 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99096/testReport)** for PR 23054 at commit [`d784142`](https://github.com/apache/spark/commit/d784142a0ee7ecf4300e542ddf2647e059217902). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23054: [SPARK-26085][SQL] Key attribute of non-struct type unde...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23054 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5209/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23054: [SPARK-26085][SQL] Key attribute of non-struct ty...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/23054#discussion_r235247826 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -1594,6 +1594,15 @@ object SQLConf { "WHERE, which does not follow SQL standard.") .booleanConf .createWithDefault(false) + + val ALIAS_NON_STRUCT_GROUPING_KEY_AS_VALUE = +buildConf("spark.sql.legacy.dataset.aliasNonStructGroupingKeyAsValue") + .internal() + .doc("When set to true, the key attribute resulted from running `Dataset.groupByKey` " + +"for non-struct key type, will be named as `value`, following the behavior of Spark " + --- End diff -- `nameNonStructGroupingKeyAsValue` is fine to me. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23100 **[Test build #99095 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99095/testReport)** for PR 23100 at commit [`ee3de58`](https://github.com/apache/spark/commit/ee3de5862e975f9659af474c43133294ec5ce369). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23099: [SPARK-25954][SS] Upgrade to Kafka 2.1.0
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/23099 Thank you for retriggering, @HyukjinKwon . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5208/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22149: [SPARK-25158][SQL]Executor accidentally exit beca...
Github user LuciferYang commented on a diff in the pull request: https://github.com/apache/spark/pull/22149#discussion_r235247176 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala --- @@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread( } threwException = false } catch { + // TaskKilledException should not be thrown again, otherwise it will be captured by + // SparkUncaughtExceptionHandler, then Executor will exit because of TaskKilledException. + case e: TaskKilledException => --- End diff -- Actually ` ScriptTransformationExec` like a streaming pipe, but other operators basically no such characteristics. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #22149: [SPARK-25158][SQL]Executor accidentally exit beca...
Github user LuciferYang commented on a diff in the pull request: https://github.com/apache/spark/pull/22149#discussion_r235246544 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformationExec.scala --- @@ -308,6 +308,12 @@ private class ScriptTransformationWriterThread( } threwException = false } catch { + // TaskKilledException should not be thrown again, otherwise it will be captured by + // SparkUncaughtExceptionHandler, then Executor will exit because of TaskKilledException. + case e: TaskKilledException => --- End diff -- Thanks for your reply @cloud-fan , the error log and problem analysis of one bad case I paste in https://issues.apache.org/jira/browse/SPARK-25158, in that case `.UnsafeInMemorySorter$SortedIterator.loadNext` throw `TaskKilledException`, and in another case `TaskKilledException` throw by `InterruptibleIterator.hasNext` , and from the error stack line 281 `iter.map(outputProjection).foreach ` is the stack root. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99092/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23070 **[Test build #99094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99094/testReport)** for PR 23070 at commit [`bd2debc`](https://github.com/apache/spark/commit/bd2debcc2237ad178ef00b762bcdc80b63d1ecb7). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23100 **[Test build #99092 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99092/testReport)** for PR 23100 at commit [`cf6da4b`](https://github.com/apache/spark/commit/cf6da4b72d04ab109739400dcbf6d75a9d34625e). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23070 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23070 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5207/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23098: [WIP][SPARK-26132][BUILD][CORE] Remove support for Scala...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23098 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23043: [SPARK-26021][SQL] replace minus zero with zero in Platf...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/23043 Please also update the PR description to reflect latest change. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23098: [WIP][SPARK-26132][BUILD][CORE] Remove support for Scala...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23098 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99087/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22939: [SPARK-25446][R] Add schema_of_json() and schema_of_csv(...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22939 gentle ping, @felixcheung. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23098: [WIP][SPARK-26132][BUILD][CORE] Remove support for Scala...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23098 **[Test build #99087 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99087/testReport)** for PR 23098 at commit [`1c1f000`](https://github.com/apache/spark/commit/1c1f0006c4a9439459d07132180b8f7fbd26b879). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23043: [SPARK-26021][SQL] replace minus zero with zero i...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/23043#discussion_r235244695 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java --- @@ -128,6 +133,11 @@ public static double getDouble(Object object, long offset) { } public static void putDouble(Object object, long offset, double value) { +if (Double.isNaN(value)) { + value = Double.NaN; +} else if(value == -0.0d) { --- End diff -- ditto. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23043: [SPARK-26021][SQL] replace minus zero with zero i...
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/23043#discussion_r235244686 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java --- @@ -120,6 +120,11 @@ public static float getFloat(Object object, long offset) { } public static void putFloat(Object object, long offset, float value) { +if (Float.isNaN(value)) { + value = Float.NaN; +} else if(value == -0.0f) { --- End diff -- Add a space after if for consistent style. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23099: [WIP][SPARK-25954][SS] Upgrade to Kafka 2.1.0
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/23099 **[Test build #99093 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/99093/testReport)** for PR 23099 at commit [`7e9a65c`](https://github.com/apache/spark/commit/7e9a65c8f5227abe967cfc053781f91a3b742a07). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23070: [SPARK-26099][SQL] Verification of the corrupt column in...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23070 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5205/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23100: [WIP][SPARK-26133][ML] Remove deprecated OneHotEncoder a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23100 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23099: [WIP][SPARK-25954][SS] Upgrade to Kafka 2.1.0
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23099 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5206/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23071: [SPARK-26102][SQL][TEST] Extracting common CSV/JSON func...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23071 I think we don't need this for now. Let's do this when more `from/to_...` functions are added later. The amount of codes increases actually. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23099: [WIP][SPARK-25954][SS] Upgrade to Kafka 2.1.0
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/23099 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23080: [SPARK-26108][SQL] Support custom lineSep in CSV datasou...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23080 @MaxGekk, let's rebase this one accordingly with encoding support. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #23080: [SPARK-26108][SQL] Support custom lineSep in CSV ...
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/23080#discussion_r235244407 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala --- @@ -192,6 +192,20 @@ class CSVOptions( */ val emptyValueInWrite = emptyValue.getOrElse("\"\"") + /** + * A string between two consecutive JSON records. + */ + val lineSeparator: Option[String] = parameters.get("lineSep").map { sep => +require(sep.nonEmpty, "'lineSep' cannot be an empty string.") +require(sep.length <= 2, "'lineSep' can contain 1 or 2 characters.") --- End diff -- Hm, I see. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #23085: [Docs] Added csv, orc, and text output format options to...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/23085 @mrandrewandrade, let's close this for now. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org