[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user suyanNone commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-138480264 sorry too late to see that, I will update it today --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4887#discussion_r38582913
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -152,6 +152,9 @@ private[spark] class CacheManager(blockManager:
BlockManager) extends Logging {
*/
updatedBlocks ++=
blockManager.putIterator(key, values, level, tellMaster = true,
effectiveStorageLevel)
+ if (level.useMemory) {
+blockManager.memoryStore.releasePendingUnrollMemoryForThisThread()
+ }
--- End diff --
I think this is correct, but it's a confusing place to do this. It requires
the reader of the code to understand the difference between `level` and
`putLevel`. I would move this check to after the `putInBlockManager` in L185 of
the old code instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4887#discussion_r38582236
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -152,6 +152,9 @@ private[spark] class CacheManager(blockManager:
BlockManager) extends Logging {
*/
updatedBlocks ++=
blockManager.putIterator(key, values, level, tellMaster = true,
effectiveStorageLevel)
+ if (level.useMemory) {
+blockManager.memoryStore.releasePendingUnrollMemoryForThisThread()
+ }
--- End diff --
@suyanNone this only happens if we failed to unroll in memory. In this case
we can't release the pending unroll memory because we created an `Array` that
contains some (but not all) of the values in this partitions. We need this
`Array` to return an iterator to the caller of this method so we can't release
it until after the task ends.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4887#discussion_r38583096
--- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala ---
@@ -152,6 +152,9 @@ private[spark] class CacheManager(blockManager:
BlockManager) extends Logging {
*/
updatedBlocks ++=
blockManager.putIterator(key, values, level, tellMaster = true,
effectiveStorageLevel)
+ if (level.useMemory) {
+blockManager.memoryStore.releasePendingUnrollMemoryForThisThread()
+ }
--- End diff --
Then I would add a comment for this block down there:
```
// After putting this on disk, we no longer need the Array that stored the
partial values, so we
// should release the unroll memory occupied in the process for other tasks
(SPARK-6157).
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-137240977 > In the previous situation, it is aim to resolve memory_and_disk level block, first unroll failed, but it will reserved unroll memory for this task, and that unroll memory part should release after that block already put into disk, because nobody will use value from that unroll arrays. So it no necessary to reserved that unroll memory part. Oh I see, the idea is that once it's on disk we no longer need the `Array`, so we can just release the unroll memory since the `Array` will be garbage collected. This makes sense. Would you mind updating the title and description to be something more succinct? ``` [SPARK-6157] [CORE] Unrolling with MEMORY_AND_DISK should always release memory ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-137243514 @suyanNone once you address the comments I'll merge this. Thanks for catching this optimization. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-136819519 Sure, I'll have a look later today. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user suyanNone commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-136661103 @srowen Ok, I check the current code, it change to memory threshold per task instead per thread. so that problem is still exist... " In the previous situation, it is aim to resolve memory_and_disk level block, first unroll failed, but it will reserved unroll memory for this thread, and that unroll memory part should release after that block already put into disk, because nobody will use value for that unroll arrays. So it no necessary to reserved that unroll memory part. " Do you agree that description or do I make that understandable? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-136663671 @andrewor14 can you look at this again, given you wrote https://github.com/apache/spark/commit/01125a1162b7bf42ae907cb6616616cc4ffb6fa9 ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user suyanNone commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-136657671 @srowen @nitin2goyal I not sure it is need because there is a big change for memory threshold per thread. In the previous situation, it is aim to resolve memory_and_disk level block, first unroll failed, but it will reserved unroll memory for this thread, and that unroll memory part should release after that block already put into disk. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user nitin2goyal commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-136281829 Any update on this patch? @srowen @suyanNone --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-136287081 From the discussions, I'm not sure this change has been sufficiently explained or motivated. I am not sure I follow and the related PR/JIRA has been closed. Without further update I think this PR should be closed. The way forward would be to more clearly illustrate why as of the current master there's an issue and why this solves it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user suyanNone commented on a diff in the pull request:
https://github.com/apache/spark/pull/4887#discussion_r32811415
--- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala ---
@@ -295,9 +296,9 @@ private[spark] class MemoryStore(blockManager:
BlockManager, maxMemory: Long)
// In this case, we should release the memory after we cache the
block there.
// Otherwise, if we return an iterator, we release the memory
reserved here
// later when the task finishes.
- if (keepUnrolling) {
-accountingLock.synchronized {
- val amountToRelease = currentUnrollMemoryForThisThread -
previousMemoryReserved
+ accountingLock.synchronized {
--- End diff --
@srowen releasePendingUnrollMemoryForThisThread means: we will reserved the
unroll memory reserved for the unroll block, we need to release that part of
memory when we not use any more, like: we put the unroll successful block to
memoryStore, or we put unroll unsuccessful block into the disk, or we complete
the task.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-112673738 [Test build #35025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35025/consoleFull) for PR 4887 at commit [`f364f07`](https://github.com/apache/spark/commit/f364f07730b66bb79f06fc092738876c45b83299). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-112673468 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-112673028 [Test build #35023 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35023/consoleFull) for PR 4887 at commit [`26c4e86`](https://github.com/apache/spark/commit/26c4e8658520b9391cee6dad07a9ab69a009d1f6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-112673457 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-112713787 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-112713758 [Test build #35023 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35023/console) for PR 4887 at commit [`26c4e86`](https://github.com/apache/spark/commit/26c4e8658520b9391cee6dad07a9ab69a009d1f6). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-112672362 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-112714878 [Test build #35025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35025/console) for PR 4887 at commit [`f364f07`](https://github.com/apache/spark/commit/f364f07730b66bb79f06fc092738876c45b83299). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-112714933 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-112672318 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-112553970 Let's close this PR at this point. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user suyanNone commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-112642422 @srowen eh...give me some time to review this patch... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4887#discussion_r28416270
--- Diff: core/src/main/scala/org/apache/spark/storage/MemoryStore.scala ---
@@ -295,9 +296,9 @@ private[spark] class MemoryStore(blockManager:
BlockManager, maxMemory: Long)
// In this case, we should release the memory after we cache the
block there.
// Otherwise, if we return an iterator, we release the memory
reserved here
// later when the task finishes.
- if (keepUnrolling) {
-accountingLock.synchronized {
- val amountToRelease = currentUnrollMemoryForThisThread -
previousMemoryReserved
+ accountingLock.synchronized {
--- End diff --
This grabs the lock unconditionally now; why not put it inside the if block?
I'm not as qualified to review this but it seems like you are always
unrolling if the storage level includes disk, not just if it includes memory,
which aren't the same thing. I think you might also have to explain the call to
`blockManager.memoryStore.releasePendingUnrollMemoryForThisThread()` above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-92668018 [Test build #30226 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30226/consoleFull) for PR 4887 at commit [`e58280a`](https://github.com/apache/spark/commit/e58280ad2169760ab4f2ea03f80c8e455ce5551b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-92696888 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30226/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-92696876 [Test build #30226 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30226/consoleFull) for PR 4887 at commit [`e58280a`](https://github.com/apache/spark/commit/e58280ad2169760ab4f2ea03f80c8e455ce5551b). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. * This patch does not change any dependencies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user suyanNone commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-77791515 @srowen also be my fault, for lazy to make description more clear... I will update the description to make more sense --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-77338343 Eh, it may be my fault for calling them duplicate. They began with identical descriptions. If there are really two separable, logical changes here, I apologize and perhaps we can reopen the second one with a clearly distinguished description. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-77248119 Hey @suyanNone how is this related to #4886? The JIRA for this PR (SPARK-6157) is marked as duplicate of the JIRA for that PR (SPARK-6156), so are these two trying to solve the same issue? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6157][CORE]Unroll unsuccessful memory_a...
Github user suyanNone commented on the pull request: https://github.com/apache/spark/pull/4887#issuecomment-77295859 @andrewor14 , i thinks is not duplicate. Put a memory_and_disk level block. 1) Try to put in memory store, unroll fails. 2) Put into disk success. 3) return values through get(blockID). SPARK-6156 is aim to resolve not try to cache block again in process 3 SPARK-6157 is aim to resolve that situation that have two RDD cache in one Stage, both tow RDD storage level is memory_and_disk. first RDD(large), put memory unroll fails, then put disk success, it will hold an unroll memory part. second RDD(small), also want to put in memory, but may first RDD hold its unroll memory(may large), then second RDD cant put in memory again, may it in actually could. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
