[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178793683 Quoting from the JIRA ticket: > In our particular case, this behavior manifests since the currentTaskAttemptId() method is returning -1 for each Spark receiver task. This in and of itself could be a bug and is something I'm going to look into. I think this is definitely a bug. I believe that the intent here was that we'd return a dummy `taskAttemptId` on the driver but that any code running in a task should have a valid `TaskContext` thread local and thus a valid task attempt id. `TaskContext` isn't an inheritable thread-local, though, so we'll have to explicitly propagate it from the top-level task thread to the receiver threads in order to address this. Even if we did fix the `TaskContext` propagation issue, the fix in this patch would still be necessary because we'd still have to be properly thread-safe in case a multi-threaded receiver was storing blocks. Intuitively, the idea of adding extra synchronization here seems right to me, although I'd like to take a closer look at the changes here to see whether this will introduce performance problems: my guess is that the under-synchronization might have been caused by a desire to avoid holding monitors/locks during expensive operations. @zsxwing, do you know why the streaming longevity / memory leak tests didn't catch this leak? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user nongli commented on a diff in the pull request: https://github.com/apache/spark/pull/11012#discussion_r51628572 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -304,10 +309,9 @@ private[spark] class MemoryStore(blockManager: BlockManager, memoryManager: Memo // release the unroll memory yet. Instead, we transfer it to pending unroll memory // so `tryToPut` can further transfer it to normal storage memory later. // TODO: we can probably express this without pending unroll memory (SPARK-10907) - val amountToTransferToPending = currentUnrollMemoryForThisTask - previousMemoryReserved --- End diff -- This is the race you are addressing right? The issue is that previousMemoryReserved is out of date. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178785689 > An alternative option might be to track unroll/pending unroll memory based on block ID rather than task attempt ID. I think you need to account for this on a per-task rather than per-block basis in order to enforce fair sharing of memory between concurrently-running tasks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178782148 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178782116 /cc @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178790263 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50583/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178790259 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178819290 > @zsxwing, do you know why the streaming longevity / memory leak tests didn't catch this leak? If I understand correctly, this issue only happens when a receiver starts multiple threads. The memory leak tests I did only use one thread per receiver. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user nongli commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178824893 This looks right to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/11012#discussion_r51633108 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -304,10 +309,9 @@ private[spark] class MemoryStore(blockManager: BlockManager, memoryManager: Memo // release the unroll memory yet. Instead, we transfer it to pending unroll memory // so `tryToPut` can further transfer it to normal storage memory later. // TODO: we can probably express this without pending unroll memory (SPARK-10907) - val amountToTransferToPending = currentUnrollMemoryForThisTask - previousMemoryReserved --- End diff -- Yeah, it seems like the current PR description doesn't quite describe the changes here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user budde commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178853877 From Jenkins output: >Fetching upstream changes from https://github.com/apache/spark.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/apache/spark.git +refs/pull/11012/*:refs/remotes/origin/pr/11012/* # timeout=15 ERROR: Timeout after 15 minutes ERROR: Error fetching remote repo 'origin' --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user budde commented on a diff in the pull request: https://github.com/apache/spark/pull/11012#discussion_r51645315 --- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala --- @@ -304,10 +309,9 @@ private[spark] class MemoryStore(blockManager: BlockManager, memoryManager: Memo // release the unroll memory yet. Instead, we transfer it to pending unroll memory // so `tryToPut` can further transfer it to normal storage memory later. // TODO: we can probably express this without pending unroll memory (SPARK-10907) - val amountToTransferToPending = currentUnrollMemoryForThisTask - previousMemoryReserved --- End diff -- @nongli â the problem is that the original implementation assumes that previousMemoryReserved is an invariant representing the number of unroll bytes allocated for the process besides the pending bytes allocated during the unroll, but no synchronization exists to enforce this invariant. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user budde commented on a diff in the pull request:
https://github.com/apache/spark/pull/11012#discussion_r51643796
--- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala ---
@@ -304,10 +309,9 @@ private[spark] class MemoryStore(blockManager:
BlockManager, memoryManager: Memo
// release the unroll memory yet. Instead, we transfer it to
pending unroll memory
// so `tryToPut` can further transfer it to normal storage
memory later.
// TODO: we can probably express this without pending unroll
memory (SPARK-10907)
- val amountToTransferToPending = currentUnrollMemoryForThisTask -
previousMemoryReserved
--- End diff --
Per my earlier comment, I updated the PR to use to use a var named
previousMemoryReserved to manually track the number of unroll bytes allocated
during a given invocation of unrollSafely rather than relying on
unrollMemoryMap(taskAttemptId) not being modified outside of the given thread
between the assignment to previousMemoryReserved and the memory maps being
updated in the finally { } block. This should remove the need to make the whole
method synchronized.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178892226 **[Test build #50612 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50612/consoleFull)** for PR 11012 at commit [`5207fb4`](https://github.com/apache/spark/commit/5207fb4756dc563c27d8e71e6fa4a4d2dbcf24ba). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178887920 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178935410 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50612/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178935407 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178935050 **[Test build #50612 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50612/consoleFull)** for PR 11012 at commit [`5207fb4`](https://github.com/apache/spark/commit/5207fb4756dc563c27d8e71e6fa4a4d2dbcf24ba). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178945649 **[Test build #50629 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50629/consoleFull)** for PR 11012 at commit [`5207fb4`](https://github.com/apache/spark/commit/5207fb4756dc563c27d8e71e6fa4a4d2dbcf24ba). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user budde commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178940544 Looks like a bunch of Spark SQL/Hive tests are failing due to this error: >Caused by: sbt.ForkMain$ForkError: org.apache.spark.SparkException: Job aborted due to stage failure: Task 19 in stage 4892.0 failed 1 times, most recent failure: Lost task 19.0 in stage 4892.0 (TID 69335, localhost): java.lang.RuntimeException: Stream '/jars/TestUDTF.jar' was not found. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user budde commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178929153 Latest change is looking good on my end. No unroll memory is being leaked. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178944716 **[Test build #50627 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50627/consoleFull)** for PR 11012 at commit [`5207fb4`](https://github.com/apache/spark/commit/5207fb4756dc563c27d8e71e6fa4a4d2dbcf24ba). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user budde commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178766741 Updated PR with new implementation that uses a counter variable instead of requiring the whole method to be atomic. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178776979 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178776981 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50578/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178918227 @budde thanks for providing so much detail in the JIRA and fixing this issue. I believe the latest changes are correct and seem more performant than what you had earlier. It seems that this issue only happens if we have multiple threads running the same task. In your case, this is the receiver task, though in any case we shouldn't be getting -1 for `currentTaskAttemptId` unless we're running the receiver on the driver (i.e. in local mode). Have you had a chance to run your test against the latest changes to prove that it fixes the leak? If so, I will go ahead and merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178932261 **[Test build #2498 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2498/consoleFull)** for PR 11012 at commit [`5207fb4`](https://github.com/apache/spark/commit/5207fb4756dc563c27d8e71e6fa4a4d2dbcf24ba). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178932447 **[Test build #2499 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2499/consoleFull)** for PR 11012 at commit [`5207fb4`](https://github.com/apache/spark/commit/5207fb4756dc563c27d8e71e6fa4a4d2dbcf24ba). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178941583 no it's unrelated. I manually triggered a few extra builds. retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178980161 **[Test build #50629 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50629/consoleFull)** for PR 11012 at commit [`5207fb4`](https://github.com/apache/spark/commit/5207fb4756dc563c27d8e71e6fa4a4d2dbcf24ba). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178980260 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50629/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178982671 Merged into master and 1.6. Thanks again for catching this issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/11012 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178978030 **[Test build #2499 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2499/consoleFull)** for PR 11012 at commit [`5207fb4`](https://github.com/apache/spark/commit/5207fb4756dc563c27d8e71e6fa4a4d2dbcf24ba). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178965433 **[Test build #2498 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2498/consoleFull)** for PR 11012 at commit [`5207fb4`](https://github.com/apache/spark/commit/5207fb4756dc563c27d8e71e6fa4a4d2dbcf24ba). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178979130 **[Test build #50627 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50627/consoleFull)** for PR 11012 at commit [`5207fb4`](https://github.com/apache/spark/commit/5207fb4756dc563c27d8e71e6fa4a4d2dbcf24ba). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178979204 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50627/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178979203 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user budde commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178314141 Pinging @andrewor14 , the original implementor of unrollSafely(), for any potential feedback. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
GitHub user budde opened a pull request:
https://github.com/apache/spark/pull/11012
[SPARK-13122] Fix race condition in MemoryStore.unrollSafely()
https://issues.apache.org/jira/browse/SPARK-13122
A race condition can occur in MemoryStore's unrollSafely() method if two
threads that
return the same value for currentTaskAttemptId() execute this method
concurrently. This
change makes the operation of reading the initial amount of unroll memory
used, performing
the unroll, and updating the associated memory maps atomic in order to
avoid this race
condition.
Initial proposed fix wraps all of unrollSafely() in a
memoryManager.synchronized { } block. A cleaner approach might be introduce a
mechanism that synchronizes based on task attempt ID. An alternative option
might be to track unroll/pending unroll memory based on block ID rather than
task attempt ID.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/budde/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11012.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11012
commit 6e0156c6e3c2fd137005ce3d55cecdf7070795da
Author: Adam Budde
Date: 2016-02-02T01:04:45Z
[SPARK-13122] Fix race condition in MemoryStore.unrollSafely()
https://issues.apache.org/jira/browse/SPARK-13122
A race condition can occur in MemoryStore's unrollSafely() method if two
threads that
return the same value for currentTaskAttemptId() execute this method
concurrently. This
change makes the operation of reading the initial amount of unroll memory
used, performing
the unroll, and updating the associated memory maps atomic in order to
avoid this race
condition.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178299115 **[Test build #50520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50520/consoleFull)** for PR 11012 at commit [`6e0156c`](https://github.com/apache/spark/commit/6e0156c6e3c2fd137005ce3d55cecdf7070795da). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178346293 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178346294 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50520/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-13122] Fix race condition in MemoryStor...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11012#issuecomment-178345954 **[Test build #50520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50520/consoleFull)** for PR 11012 at commit [`6e0156c`](https://github.com/apache/spark/commit/6e0156c6e3c2fd137005ce3d55cecdf7070795da). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
