[GitHub] spark issue #16387: [SPARK-18986][Core] ExternalAppendOnlyMap shouldn't fail...
Github user samkum commented on the issue:
https://github.com/apache/spark/pull/16387
I can't commit on the timelines. I don't want to block things at my end.
-Sameer.
On Thu, Feb 16, 2017 at 1:23 AM, Marcelo Vanzin
wrote:
> @samkum <https://github.com/samkum> if you can't get back to us with the
> information we asked for, I'll just assume that the problem is on your
side
> and push this change. It's been sitting here already reviewed for long
> enough.
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/16387#issuecomment-280119943>, or
mute
> the thread
>
<https://github.com/notifications/unsubscribe-auth/AARBhsYxfqSP4u5z5F2Iad_-84ufK-ZSks5rc1fRgaJpZM4LUlyt>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16387: [SPARK-18986][Core] ExternalAppendOnlyMap shouldn't fail...
Github user samkum commented on the issue: https://github.com/apache/spark/pull/16387 Hi Marcelo, Are you referring to following commit. 5645533( https://github.com/apache/spark/pull/16387/commits/5645533ede8bb37aa316afbc0d003d042bf5d52d ) -Sameer. On Wed, Feb 8, 2017 at 11:35 PM, Marcelo Vanzin wrote: > @samkum <https://github.com/samkum> the code is in this PR! Just revert > the last two commits. > > â > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/spark/pull/16387#issuecomment-278411836>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/AARBhrLi2YaA1B6C8a203ComRgZ5mMgPks5ragP-gaJpZM4LUlyt> > . > --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16387: [SPARK-18986][Core] ExternalAppendOnlyMap shouldn't fail...
Github user samkum commented on the issue: https://github.com/apache/spark/pull/16387 Please share the commit id...i think that will make it clear. I can't share the code as its propriety to organization,. but i am just reducing a map phase, reduce by key and reducebykeyandwindow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16387: [SPARK-18986][Core] ExternalAppendOnlyMap shouldn't fail...
Github user samkum commented on the issue:
https://github.com/apache/spark/pull/16387
Hi liang,
I got pulled into another project. Will need few more days to catch up on
bandwidth.
Just for confirmation, I will testing with changes just to TaskManager
class.
-Sameer.
On Feb 6, 2017 7:29 PM, "Liang-Chi Hsieh" wrote:
> @samkum <https://github.com/samkum> Any update?
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/16387#issuecomment-277688500>, or
mute
> the thread
>
<https://github.com/notifications/unsubscribe-auth/AARBhjWvXCEKp7lqrStBywqNjY35BL8aks5rZyc9gaJpZM4LUlyt>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16387: [SPARK-18986][Core] ExternalAppendOnlyMap shouldn't fail...
Github user samkum commented on the issue: https://github.com/apache/spark/pull/16387 @mridulm <https://github.com/mridulm> - need one clarification: if I test it with out ut @viirya <https://github.com/viirya> commit b1ef9ec,wouldnt the commit fail at the assertion.( assert(readingIterator != null). Would the taskManager change take care of this. On Thu, Jan 26, 2017 at 10:43 AM, Sameer Kumar wrote: > ok..I will get back to you in next couple of days. > > -Sameer. > > On Thu, Jan 26, 2017 at 3:56 AM, Marcelo Vanzin > wrote: > >> Nope, I didn't tested it in isolation. >> >> Could you do that? To make sure that it's really caused by this change? >> >> â >> You are receiving this because you were mentioned. >> Reply to this email directly, view it on GitHub >> <https://github.com/apache/spark/pull/16387#issuecomment-275252690>, or mute >> the thread >> <https://github.com/notifications/unsubscribe-auth/AARBhtcSjke249vrHE-oOyZXKTPSbob3ks5rV8wmgaJpZM4LUlyt> >> . >> > > --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16387: [SPARK-18986][Core] ExternalAppendOnlyMap shouldn't fail...
Github user samkum commented on the issue: https://github.com/apache/spark/pull/16387 ok..I will get back to you in next couple of days. -Sameer. On Thu, Jan 26, 2017 at 3:56 AM, Marcelo Vanzin wrote: > Nope, I didn't tested it in isolation. > > Could you do that? To make sure that it's really caused by this change? > > â > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/spark/pull/16387#issuecomment-275252690>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/AARBhtcSjke249vrHE-oOyZXKTPSbob3ks5rV8wmgaJpZM4LUlyt> > . > --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16387: [SPARK-18986][Core] ExternalAppendOnlyMap shouldn't fail...
Github user samkum commented on the issue: https://github.com/apache/spark/pull/16387 Nope, I didn't tested it in isolation. -Sameer. On Jan 24, 2017 10:09 PM, "Marcelo Vanzin" wrote: > No the question is whether you tested without @viirya > <https://github.com/viirya> commit b1ef9ec (the last one that forces > spills of in-memory maps), or just the very last version of the patch. > > â > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/spark/pull/16387#issuecomment-274859256>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/AARBhjRZruTZJcLIkrBMF-x9tQ6GcF-_ks5rVilUgaJpZM4LUlyt> > . > --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16387: [SPARK-18986][Core] ExternalAppendOnlyMap shouldn't fail...
Github user samkum commented on the issue:
https://github.com/apache/spark/pull/16387
Hi,
No, I havent test this with out the ExternalAppendOnlyMap fix if this is
what you are asking. I tested it with changes to both the classes:
TaskMemoryManager and ExternalAppendOnlyMap.
-Sameer.
On Tue, Jan 24, 2017 at 3:17 AM, Marcelo Vanzin
wrote:
> @samkum <https://github.com/samkum> can you reply to the questions above?
> This PR is basically blocked on your feedback that it makes things worse.
>
> @samkum <https://github.com/samkum> did you try this patch without the
> "spill map on every call" behavior (basically returning "false" instead of
> spilling the in-memory map when the iterator doesn't exist)?
>
> If that case also results in increased gc, maybe your app just needs more
> memory. That would be the case where the assert would trigger before.
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/16387#issuecomment-274627803>, or
mute
> the thread
>
<https://github.com/notifications/unsubscribe-auth/AARBhqyMAq1IRIMLHFwU5atf2IYdWs88ks5rVSALgaJpZM4LUlyt>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...
Github user samkum commented on the issue:
https://github.com/apache/spark/pull/16603
Combined with 16387.
On Jan 18, 2017 3:54 PM, "Liang-Chi Hsieh" wrote:
> @samkum <https://github.com/samkum> Just this fix or combined with #16387
> <https://github.com/apache/spark/pull/16387>?
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/16603#issuecomment-273438586>, or
mute
> the thread
>
<https://github.com/notifications/unsubscribe-auth/AARBhjjcmTAapQoC20Ih-1-5gid-2gg2ks5rTehFgaJpZM4Lkjq2>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...
Github user samkum commented on the issue: https://github.com/apache/spark/pull/16603 This fix also has GC issues. Please find the screenshot attached.  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16387: [SPARK-18986][Core] ExternalAppendOnlyMap shouldn't fail...
Github user samkum commented on the issue: https://github.com/apache/spark/pull/16387 The fix is better but still produces visible GC issues. After removing the fix, GC is no longer a problem.  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16387: [SPARK-18986][Core] ExternalAppendOnlyMap shouldn't fail...
Github user samkum commented on the issue: https://github.com/apache/spark/pull/16387 I have tested this, but I found a very strange observation. GC frequency has increased many folds...and majority of the time is spend in GC. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
