[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60501805 Since merging this PR, I've started noticing some OOM failures in a test that uses SnappyOutputStream: ``` - partial aggregation without spill *** RUN ABORTED *** java.lang.OutOfMemoryError: Java heap space at org.xerial.snappy.buffer.CachedBufferAllocator.allocate(CachedBufferAllocator.java:48) at org.xerial.snappy.SnappyOutputStream.init(SnappyOutputStream.java:96) at org.xerial.snappy.SnappyOutputStream.init(SnappyOutputStream.java:83) at org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:125) at org.apache.spark.storage.BlockManager.wrapForCompression(BlockManager.scala:1084) at org.apache.spark.storage.BlockManager$$anonfun$7.apply(BlockManager.scala:579) at org.apache.spark.storage.BlockManager$$anonfun$7.apply(BlockManager.scala:579) at org.apache.spark.storage.DiskBlockObjectWriter.open(BlockObjectWriter.scala:126) at org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:192) at org.apache.spark.util.collection.ExternalSorter.spillToMergeableFile(ExternalSorter.scala:300) ``` @xerial do you think there could be a memory leak in the new CachedBufferAllocator? https://github.com/xerial/snappy-java/compare/1.1.1.3...develop --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60501893 I've reverted this in commit 898b22ab1fe90e8a3935b19566465046f2256fa6 until we figure out what's going on. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60348540 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22114/consoleFull) for PR 2911 at commit [`adec96c`](https://github.com/apache/spark/commit/adec96cde4785c542d4b89b906c56e070afd5dc9). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60349709 Hmm, looks like that might have been a transient failure. Just to be sure, though, I'm going to run this one more time to make sure that it still passes, then merge it (since I don't think that any issues we'll observe will be caused by this small snappy-java version bump). Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60411094 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60411671 [Test build #22143 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22143/consoleFull) for PR 2911 at commit [`adec96c`](https://github.com/apache/spark/commit/adec96cde4785c542d4b89b906c56e070afd5dc9). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60415381 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22143/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60415564 Jenkins, retest this please (sorry I had to abort this to clean the workspace). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60416066 [Test build #22144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22144/consoleFull) for PR 2911 at commit [`adec96c`](https://github.com/apache/spark/commit/adec96cde4785c542d4b89b906c56e070afd5dc9). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60423873 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22144/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60423859 [Test build #22144 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22144/consoleFull) for PR 2911 at commit [`adec96c`](https://github.com/apache/spark/commit/adec96cde4785c542d4b89b906c56e070afd5dc9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60430329 The IOException: unexpected exception type error here is actually masking the real error; this is due to us throwing something other than IOException from read/writeExternal; I've opened https://issues.apache.org/jira/browse/SPARK-4080 to fix this (working on a patch now). I'm pretty sure that this isn't caused by Snappy, but instead is an instance of some longer-standing non-deterministic serialization issue in our code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60457129 Now that I've fixed the IOException issue via #2932, let's retest this to see if I get a more informative error message. I'm almost positive that this isn't a Snappy issue; I'm just curious to see what happens. Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60460533 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60460723 [Test build #22178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22178/consoleFull) for PR 2911 at commit [`adec96c`](https://github.com/apache/spark/commit/adec96cde4785c542d4b89b906c56e070afd5dc9). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60463860 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22178/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60463855 [Test build #22178 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22178/consoleFull) for PR 2911 at commit [`adec96c`](https://github.com/apache/spark/commit/adec96cde4785c542d4b89b906c56e070afd5dc9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60464908 Those failures are related to known issues from the Hive 13 PR, not this, so I'm going to merge this across all of our backport branches in order to improve our error reporting. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2911 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60331762 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22100/consoleFull) for PR 2911 at commit [`adec96c`](https://github.com/apache/spark/commit/adec96cde4785c542d4b89b906c56e070afd5dc9). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60332096 @xerial Thanks for fixing that so quickly! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60338998 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22100/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60338993 **[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22100/consoleFull)** for PR 2911 at commit [`adec96c`](https://github.com/apache/spark/commit/adec96cde4785c542d4b89b906c56e070afd5dc9) after a configured wait of `120m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60339557 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60339789 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22106/consoleFull) for PR 2911 at commit [`adec96c`](https://github.com/apache/spark/commit/adec96cde4785c542d4b89b906c56e070afd5dc9). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60342597 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22106/consoleFull) for PR 2911 at commit [`adec96c`](https://github.com/apache/spark/commit/adec96cde4785c542d4b89b906c56e070afd5dc9). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60342600 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22106/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user xerial commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60343444 @JoshRosen If you have the stack trace of this error, please let me know. I would like to check it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60344869 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22114/consoleFull) for PR 2911 at commit [`adec96c`](https://github.com/apache/spark/commit/adec96cde4785c542d4b89b906c56e070afd5dc9). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60344903 @xerial Here's a link to the exception from that most recent test failure: https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22106/testReport/junit/org.apache.spark.util.collection/ExternalAppendOnlyMapSuite/spilling_with_compression/ In case that link breaks, here's the driver stacktrace: ``` sbt.ForkMain$ForkError: Test failed with compression using codec org.apache.spark.io.SnappyCompressionCodec: Job aborted due to stage failure: Task 0 in stage 4.0 failed 4 times, most recent failure: Lost task 0.3 in stage 4.0 (TID 14, localhost): java.io.IOException: unexpected exception type java.io.ObjectStreamClass.throwMiscException(ObjectStreamClass.java:1538) java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1025) java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62) org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87) org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:163) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) java.lang.Thread.run(Thread.java:745) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1191) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1180) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1179) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1179) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:694) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:694) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:694) at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1397) at akka.actor.Actor$class.aroundReceive(Actor.scala:465) at org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1352) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) at akka.actor.ActorCell.invoke(ActorCell.scala:487) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) at akka.dispatch.Mailbox.run(Mailbox.scala:220) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ``` This isn't the full stacktrace of the actual error, which took place in an executor. Until we merge #2845, I don't think that I'll have an easy way to grab the full executor logs from Jenkins. I wasn't able to reproduce this failure locally. I'm going to try our new experimental Deflake build Jenkins button, which reruns only the failing tests, in order to see if I can reproduce this. If so, I'll SSH in and grab the full logs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-4056] Upgrade snappy-java to 1.1.1.5
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2911#issuecomment-60344962 Actually, I don't think that deflake build plugin will necessarily work as expected given all of the customization in our build; I guess it was added for another project that shares the Jenkins server with us. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
