[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user zsxwing closed the pull request at: https://github.com/apache/spark/pull/10656 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/10656#issuecomment-169951385 Merging this! Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10656#issuecomment-169899417 CC @tdas --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10656#issuecomment-169897706 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48997/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10656#issuecomment-169897704 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10656#issuecomment-169897645 **[Test build #48997 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48997/consoleFull)** for PR 10656 at commit [`78763de`](https://github.com/apache/spark/commit/78763de3777edd3f4a69abf4c2caf15d22ab03c2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10656#issuecomment-169881498 **[Test build #48997 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48997/consoleFull)** for PR 10656 at commit [`78763de`](https://github.com/apache/spark/commit/78763de3777edd3f4a69abf4c2caf15d22ab03c2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/10656 [SPARK-12591][Streaming]Register OpenHashMapBasedStateMap for Kryo (branch 1.6) backport #10609 to branch 1.6 You can merge this pull request into a Git repository by running: $ git pull https://github.com/zsxwing/spark SPARK-12591-branch-1.6 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10656.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10656 commit 78763de3777edd3f4a69abf4c2caf15d22ab03c2 Author: Shixiong Zhu Date: 2016-01-08T01:46:24Z [SPARK-12591][STREAMING] Register OpenHashMapBasedStateMap for Kryo The default serializer in Kryo is FieldSerializer and it ignores transient fields and never calls `writeObject` or `readObject`. So we should register OpenHashMapBasedStateMap using `DefaultSerializer` to make it work with Kryo. Author: Shixiong Zhu Closes #10609 from zsxwing/SPARK-12591. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10609 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169863068 LGTM. Merging this to master. Please send another PR for 1.6 ASAP. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169855045 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48979/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169855043 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169854898 **[Test build #48979 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48979/consoleFull)** for PR 10609 at commit [`4e4e9a1`](https://github.com/apache/spark/commit/4e4e9a136ffae30665979df7307a6175188690f7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169843942 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169843686 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48987/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169843683 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169840672 By the way, I will send another PR for branch 1.6 due to the conflicts of MimaExcludes.scala. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169833544 **[Test build #48979 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48979/consoleFull)** for PR 10609 at commit [`4e4e9a1`](https://github.com/apache/spark/commit/4e4e9a136ffae30665979df7307a6175188690f7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169831500 retest this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169828273 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48967/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169828271 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169828020 **[Test build #48967 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48967/consoleFull)** for PR 10609 at commit [`33368be`](https://github.com/apache/spark/commit/33368be36f1f7b3163cad1d67f6e816cf48e7750). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169822732 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169822736 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48973/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169820761 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48963/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169820758 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169820507 **[Test build #48963 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48963/consoleFull)** for PR 10609 at commit [`bf0892c`](https://github.com/apache/spark/commit/bf0892c7e2fffa21f5eac75bfc32d35d7d21353d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169818726 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169817535 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48970/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169817531 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169811230 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169809796 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169809785 **[Test build #48969 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48969/consoleFull)** for PR 10609 at commit [`4e4e9a1`](https://github.com/apache/spark/commit/4e4e9a136ffae30665979df7307a6175188690f7). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169809797 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48969/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169809439 **[Test build #48969 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48969/consoleFull)** for PR 10609 at commit [`4e4e9a1`](https://github.com/apache/spark/commit/4e4e9a136ffae30665979df7307a6175188690f7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/10609#discussion_r49127069 --- Diff: core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala --- @@ -426,6 +439,7 @@ private[serializer] class KryoOutputDataOutputBridge(output: KryoOutput) extends override def writeChar(v: Int): Unit = output.writeChar(v.toChar) override def writeLong(v: Long): Unit = output.writeLong(v) override def writeByte(v: Int): Unit = output.writeByte(v) + override def writeObject(obj: AnyRef): Unit = kryo.writeClassAndObject(output, obj) --- End diff -- done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/10609#discussion_r49125114 --- Diff: core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala --- @@ -426,6 +439,7 @@ private[serializer] class KryoOutputDataOutputBridge(output: KryoOutput) extends override def writeChar(v: Int): Unit = output.writeChar(v.toChar) override def writeLong(v: Long): Unit = output.writeLong(v) override def writeByte(v: Int): Unit = output.writeByte(v) + override def writeObject(obj: AnyRef): Unit = kryo.writeClassAndObject(output, obj) --- End diff -- Should there be a new unit test in the KryoSerializerSuite to test this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169803306 LGTM! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169803074 **[Test build #48967 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48967/consoleFull)** for PR 10609 at commit [`33368be`](https://github.com/apache/spark/commit/33368be36f1f7b3163cad1d67f6e816cf48e7750). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/10609#discussion_r49122800
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/StateMapSuite.scala ---
@@ -321,4 +336,59 @@ class StateMapSuite extends SparkFunSuite {
}
}
}
+
+ test("OpenHashMapBasedStateMap - serializing and deserializing with
KryoSerializable states") {
+val map = new OpenHashMapBasedStateMap[KryoState, KryoState]()
+map.put(new KryoState("a"), new KryoState("b"), 1)
+testSerialization(
+ new KryoSerializer(conf), map, "error deserializing and serialized
KryoSerializable states")
+ }
+
+ test("EmptyStateMap - serializing and deserializing") {
+val map = StateMap.empty[KryoState, KryoState]
+// Since EmptyStateMap doesn't contains any date, KryoState won't
break JavaSerializer.
+assert(serializeAndDeserialize(new JavaSerializer(conf), map).
+ isInstanceOf[EmptyStateMap[KryoState, KryoState]])
+assert(serializeAndDeserialize(new KryoSerializer(conf), map).
+ isInstanceOf[EmptyStateMap[KryoState, KryoState]])
+ }
+
+ test("MapWithStateRDDRecord - serializing and deserializing with
KryoSerializable states") {
+val map = new OpenHashMapBasedStateMap[KryoState, KryoState]()
+map.put(new KryoState("a"), new KryoState("b"), 1)
+
+val record =
+ MapWithStateRDDRecord[KryoState, KryoState, KryoState](map, Seq(new
KryoState("c")))
+val deserRecord = serializeAndDeserialize(new KryoSerializer(conf),
record)
+assert(!(record eq deserRecord))
+assert(record.stateMap.getAll().toSeq ===
deserRecord.stateMap.getAll().toSeq)
+assert(record.mappedData === deserRecord.mappedData)
+ }
+
+ private def serializeAndDeserialize[T: ClassTag](serializer: Serializer,
t: T): T = {
--- End diff --
Yes. `deserialize` needs the ClassTag
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/10609#discussion_r49122591
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/StateMapSuite.scala ---
@@ -321,4 +336,59 @@ class StateMapSuite extends SparkFunSuite {
}
}
}
+
+ test("OpenHashMapBasedStateMap - serializing and deserializing with
KryoSerializable states") {
+val map = new OpenHashMapBasedStateMap[KryoState, KryoState]()
+map.put(new KryoState("a"), new KryoState("b"), 1)
+testSerialization(
+ new KryoSerializer(conf), map, "error deserializing and serialized
KryoSerializable states")
+ }
+
+ test("EmptyStateMap - serializing and deserializing") {
+val map = StateMap.empty[KryoState, KryoState]
+// Since EmptyStateMap doesn't contains any date, KryoState won't
break JavaSerializer.
+assert(serializeAndDeserialize(new JavaSerializer(conf), map).
+ isInstanceOf[EmptyStateMap[KryoState, KryoState]])
+assert(serializeAndDeserialize(new KryoSerializer(conf), map).
+ isInstanceOf[EmptyStateMap[KryoState, KryoState]])
+ }
+
+ test("MapWithStateRDDRecord - serializing and deserializing with
KryoSerializable states") {
+val map = new OpenHashMapBasedStateMap[KryoState, KryoState]()
+map.put(new KryoState("a"), new KryoState("b"), 1)
+
+val record =
+ MapWithStateRDDRecord[KryoState, KryoState, KryoState](map, Seq(new
KryoState("c")))
+val deserRecord = serializeAndDeserialize(new KryoSerializer(conf),
record)
+assert(!(record eq deserRecord))
+assert(record.stateMap.getAll().toSeq ===
deserRecord.stateMap.getAll().toSeq)
+assert(record.mappedData === deserRecord.mappedData)
+ }
+
+ private def serializeAndDeserialize[T: ClassTag](serializer: Serializer,
t: T): T = {
--- End diff --
Does this need the ClassTag
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/10609#discussion_r49121790
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -408,9 +409,11 @@ private[serializer] class
KryoInputDataInputBridge(input: KryoInput) extends Dat
override def readBoolean(): Boolean = input.readBoolean()
override def readUnsignedByte(): Int = input.readByteUnsigned()
override def readDouble(): Double = input.readDouble()
+ override def readObject(): AnyRef = kryo.readClassAndObject(input)
}
-private[serializer] class KryoOutputDataOutputBridge(output: KryoOutput)
extends DataOutput {
+private[spark] class KryoOutputObjectOutputBridge(
--- End diff --
Can you put some docs on this class to explain what this does?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169793287 **[Test build #48963 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48963/consoleFull)** for PR 10609 at commit [`bf0892c`](https://github.com/apache/spark/commit/bf0892c7e2fffa21f5eac75bfc32d35d7d21353d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169788654 **[Test build #48960 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48960/consoleFull)** for PR 10609 at commit [`a65ab45`](https://github.com/apache/spark/commit/a65ab45cefdf697763c85bbf2b4806ca8b618dd5). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169788713 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169785460 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48958/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169784539 **[Test build #48960 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48960/consoleFull)** for PR 10609 at commit [`a65ab45`](https://github.com/apache/spark/commit/a65ab45cefdf697763c85bbf2b4806ca8b618dd5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169781200 **[Test build #48958 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48958/consoleFull)** for PR 10609 at commit [`a65ab45`](https://github.com/apache/spark/commit/a65ab45cefdf697763c85bbf2b4806ca8b618dd5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169780118 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/10609#discussion_r49112106
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala ---
@@ -77,21 +81,26 @@ private[streaming] class EmptyStateMap[K: ClassTag, S:
ClassTag] extends StateMa
}
/** Implementation of StateMap based on Spark's
[[org.apache.spark.util.collection.OpenHashMap]] */
-private[streaming] class OpenHashMapBasedStateMap[K: ClassTag, S:
ClassTag](
+private[streaming] class OpenHashMapBasedStateMap[K, S](
@transient @volatile var parentStateMap: StateMap[K, S],
-initialCapacity: Int = DEFAULT_INITIAL_CAPACITY,
-deltaChainThreshold: Int = DELTA_CHAIN_LENGTH_THRESHOLD
- ) extends StateMap[K, S] { self =>
+private var initialCapacity: Int = DEFAULT_INITIAL_CAPACITY,
+private var deltaChainThreshold: Int = DELTA_CHAIN_LENGTH_THRESHOLD
+ )(implicit private var keyClassTag: ClassTag[K], private var
stateClassTag: ClassTag[S])
--- End diff --
Add `keyClassTag` and `stateClassTag` so that we can recover them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/10609#discussion_r49111736 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala --- @@ -77,6 +80,7 @@ private[streaming] class EmptyStateMap[K: ClassTag, S: ClassTag] extends StateMa } /** Implementation of StateMap based on Spark's [[org.apache.spark.util.collection.OpenHashMap]] */ +@DefaultSerializer(classOf[KryoJavaSerializer]) --- End diff -- Actually, this is not a correct fix because the user can put objects that only support Kryo into this Map. KryoJavaSerializer cannot handle them. Fixed this issue in my latest commit. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/10609#discussion_r49034162 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/util/StateMap.scala --- @@ -77,6 +80,7 @@ private[streaming] class EmptyStateMap[K: ClassTag, S: ClassTag] extends StateMa } /** Implementation of StateMap based on Spark's [[org.apache.spark.util.collection.OpenHashMap]] */ +@DefaultSerializer(classOf[KryoJavaSerializer]) --- End diff -- Can you explain what this does? Just for some context? Also will this serialize using Kryo or just embed Java serialization within Kryo? Maybe it is a good idea to make it as efficient in Kryo as possible. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user tdas commented on a diff in the pull request:
https://github.com/apache/spark/pull/10609#discussion_r49034101
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/StateMapSuite.scala ---
@@ -267,12 +270,17 @@ class StateMapSuite extends SparkFunSuite {
assertMap(stateMap, refMap.toMap, time, "Final state map does not
match reference map")
}
- private def testSerialization[MapType <: StateMap[Int, Int]](
+ private def testSerialization[MapType <: StateMap[Int, Int] : ClassTag](
map: MapType, msg: String): MapType = {
-val deserMap = Utils.deserialize[MapType](
- Utils.serialize(map), Thread.currentThread().getContextClassLoader)
-assertMap(deserMap, map, 1, msg)
-deserMap
+val deserMaps = Array(new JavaSerializer(conf), new
KryoSerializer(conf)).map {
--- End diff --
Easier to read if this is made a function and called with two different
serializers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169519483 @tdas updated as we discussed offline. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169516182 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48884/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169516181 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169516066 **[Test build #48884 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48884/consoleFull)** for PR 10609 at commit [`bf5632e`](https://github.com/apache/spark/commit/bf5632e7a3a1d0c8309d430e81a188f3dd439b6f). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169508519 **[Test build #48884 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48884/consoleFull)** for PR 10609 at commit [`bf5632e`](https://github.com/apache/spark/commit/bf5632e7a3a1d0c8309d430e81a188f3dd439b6f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/10609#discussion_r48993619
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -166,6 +166,15 @@ class KryoSerializer(conf: SparkConf)
kryo.register(Utils.classForName("scala.collection.immutable.$colon$colon"))
kryo.register(classOf[ArrayBuffer[Any]])
+// Add classes for Streaming
+try {
+ kryo.register(
+
Utils.classForName("org.apache.spark.streaming.util.OpenHashMapBasedStateMap"),
--- End diff --
Maybe we should add an internal API to let other projects register their
classes to KryoSerializer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/10609#discussion_r48993279
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -166,6 +166,15 @@ class KryoSerializer(conf: SparkConf)
kryo.register(Utils.classForName("scala.collection.immutable.$colon$colon"))
kryo.register(classOf[ArrayBuffer[Any]])
+// Add classes for Streaming
+try {
+ kryo.register(
+
Utils.classForName("org.apache.spark.streaming.util.OpenHashMapBasedStateMap"),
--- End diff --
There is a tricky thing here.
Kryo assigns an unique id to each registered class and only writes its id,
so the register order of classes should be same. Otherwise, the ids won't be
matched and deserialization will fail.
However, for tests that start a local cluster, their executors have
`OpenHashMapBasedStateMap` but the driver doesn't. So I added
`OpenHashMapBasedStateMap` at the last class to make sure other classes's ids
are same.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169412353 CC @tdas --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169270419 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48843/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169270414 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169270236 **[Test build #48843 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48843/consoleFull)** for PR 10609 at commit [`d587f0a`](https://github.com/apache/spark/commit/d587f0ad2132855188d31cc15148802c59eae8cf). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169253775 **[Test build #48843 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48843/consoleFull)** for PR 10609 at commit [`d587f0a`](https://github.com/apache/spark/commit/d587f0ad2132855188d31cc15148802c59eae8cf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169210523 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48808/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169210522 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169210384 **[Test build #48808 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48808/consoleFull)** for PR 10609 at commit [`0228eef`](https://github.com/apache/spark/commit/0228eef185e379e80cd3622194e785187f673bce). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169196126 **[Test build #48808 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48808/consoleFull)** for PR 10609 at commit [`0228eef`](https://github.com/apache/spark/commit/0228eef185e379e80cd3622194e785187f673bce). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169191999 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169192001 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/48798/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169191880 **[Test build #48798 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48798/consoleFull)** for PR 10609 at commit [`7466a50`](https://github.com/apache/spark/commit/7466a50a08ab68c75fb12405584043df40b17e5c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10609#issuecomment-169175648 **[Test build #48798 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48798/consoleFull)** for PR 10609 at commit [`7466a50`](https://github.com/apache/spark/commit/7466a50a08ab68c75fb12405584043df40b17e5c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-12591][Streaming]Register OpenHashMapBa...
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/10609 [SPARK-12591][Streaming]Register OpenHashMapBasedStateMap for Kryo The default serializer in Kryo is FieldSerializer and it ignores transient fields and never calls `writeObject` or `readObject`. So we should register OpenHashMapBasedStateMap using `KryoJavaSerializer` to make it work with Kryo. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zsxwing/spark SPARK-12591 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/10609.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #10609 commit 7466a50a08ab68c75fb12405584043df40b17e5c Author: Shixiong Zhu Date: 2016-01-05T23:55:12Z Register OpenHashMapBasedStateMap for Kryo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
