[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22114 The tests are failing because they're now using too much direct memory: ``` 18/08/17 09:57:35.396 rpc-server-4647-3 WARN TransportChannelHandler: Exception in connection from /192.168.10.22:36322 io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 2852126720, max: 2863661056) ``` That's the first test in jenkins; the others seem to fail because of improper cleanup in that first test. But it seems that either we can't just use direct buffers here, or maybe the test might need some tweaking. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22114 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94900/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22114 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22114 **[Test build #94900 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94900/testReport)** for PR 22114 at commit [`c2f9ed1`](https://github.com/apache/spark/commit/c2f9ed10776842ffe0746fcc89b157675fa6c455). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22114 FYI I wrote a small micro bench to pound on the pool in both the old way (mixed on heap and off heap allocations) and new way (only off heap) and it actually seems faster if all allocations use the `buffer()` call. So I guess we're safe here. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22114 **[Test build #94900 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94900/testReport)** for PR 22114 at commit [`c2f9ed1`](https://github.com/apache/spark/commit/c2f9ed10776842ffe0746fcc89b157675fa6c455). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22114 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user NiharS commented on the issue: https://github.com/apache/spark/pull/22114 They pass on my machine :( --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22114 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22114 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94867/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22114 **[Test build #94867 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94867/testReport)** for PR 22114 at commit [`c2f9ed1`](https://github.com/apache/spark/commit/c2f9ed10776842ffe0746fcc89b157675fa6c455). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22114 **[Test build #94867 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94867/testReport)** for PR 22114 at commit [`c2f9ed1`](https://github.com/apache/spark/commit/c2f9ed10776842ffe0746fcc89b157675fa6c455). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22114 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user NiharS commented on the issue: https://github.com/apache/spark/pull/22114 Tried with a significantly larger input, both with and without the change. They ran in just about the same time. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user NiharS commented on the issue: https://github.com/apache/spark/pull/22114 Working on a better test now! With my small program there was definitely no noticeable change in runtime. The memory monitor was running too, though, so I did have some external factors. I'll try with a significantly larger input and see if there's anything that stands out --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22114 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22114 Did you get a chance to measure whether this has any negative effect of performance? Since it's using a buffer pool I wouldn't expect any, but you never know... --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/22114 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22114 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22114: [SPARK-24938][Core] Prevent Netty from using onheap memo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22114 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
