[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-22 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18543 ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the fea

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18543 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-05 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18543 Seems reasonable, though I don't know this code well. There is a similar pattern in KVComparator. I wonder if it's even more efficient to have a `clear()` method to reset state of an `UnsafeRow`? -

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-05 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/18543 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18543 **[Test build #79247 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79247/testReport)** for PR 18543 at commit [`5e7a935`](https://github.com/apache/spark/commit/5e

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18543 **[Test build #79247 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79247/testReport)** for PR 18543 at commit [`5e7a935`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18543 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79247/ Test PASSed. ---

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18543 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-06 Thread j-baker
Github user j-baker commented on the issue: https://github.com/apache/spark/pull/18543 Can confirm that my job which was exhibiting OOMs no longer OOMs after this change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. I

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-09 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18543 @davies or @rxin might be worth a look. The only possible downside here is the extra cycles to clear the reference, so I raise it in case you think we need to find a more optimized way to deal with t

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-10 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18543 Hi @j-baker , do you have any code snippet to reproduce this memory leak? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pr

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-12 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18543 [This program](https://gist.github.com/kiszk/29ce3558564ccb0b8797dec37b29eb12) works well with this PR while it causes an infinite loop in org.apache.spark.memory.TaskMemoryManager.allocatePage with

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18543 **[Test build #79756 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79756/testReport)** for PR 18543 at commit [`05fc618`](https://github.com/apache/spark/commit/05

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18543 **[Test build #79757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79757/testReport)** for PR 18543 at commit [`51ec9a0`](https://github.com/apache/spark/commit/51

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-19 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18543 BTW, since the GC root is `TaskContext`, is it possible to update https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.ja

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18543 **[Test build #79756 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79756/testReport)** for PR 18543 at commit [`05fc618`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18543 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79756/ Test PASSed. ---

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-19 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18543 **[Test build #79757 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79757/testReport)** for PR 18543 at commit [`51ec9a0`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18543 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18543 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18543: [SPARK-21319][SQL] Fix memory leak in UnsafeExternalRowS...

2017-07-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18543 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79757/ Test PASSed. ---