[GitHub] spark issue #18660: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8Str...
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/18660 Also merged to branch-2.2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18660: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8Str...
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/18660 thanks @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18660: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8Str...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18660 thanks, merging to master! @brkyvz I think it's fine, this bug is very obvious. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18660: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8Str...
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/18660 I couldn't write an easy reproduction for the bug :( --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18660: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8Str...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18660 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18660: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8Str...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18660 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79689/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18660: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8Str...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18660 **[Test build #79689 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79689/testReport)** for PR 18660 at commit [`d220290`](https://github.com/apache/spark/commit/d2202903518b3dfa0f4a719a0b9cb5431088ed66). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` public static class LongWrapper implements Serializable ` * ` public static class IntWrapper implements Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18660: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8Str...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18660 good catch! LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18660: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8Str...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/18660 > I don't think we're actually trying to ship these values anywhere. I see. They are static classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18660: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8Str...
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/18660 cc @cloud-fan and @tejasapatil who have more info based on: https://github.com/apache/spark/commit/c96d14abae5962a7b15239319c2a151b95f7db94 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18660: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8Str...
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/18660 I don't think we're actually trying to ship these values anywhere. We're failing to serialize the task during WholeStageCodegenExec: ``` at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:298) at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288) at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:108) at org.apache.spark.SparkContext.clean(SparkContext.scala:2310) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1.apply(RDD.scala:841) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1.apply(RDD.scala:840) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) at org.apache.spark.rdd.RDD.mapPartitionsWithIndex(RDD.scala:840) at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:389) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18660: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8Str...
Github user zsxwing commented on the issue: https://github.com/apache/spark/pull/18660 Is it safe to just ignore them? Maybe we should recover them `readExternal/read` method? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18660: [SPARK-21445] Make IntWrapper and LongWrapper in UTF8Str...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18660 **[Test build #79689 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79689/testReport)** for PR 18660 at commit [`d220290`](https://github.com/apache/spark/commit/d2202903518b3dfa0f4a719a0b9cb5431088ed66). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
