[
https://issues.apache.org/jira/browse/PIG-1466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898296#action_12898296
]
Thejas M Nair commented on PIG-1466:
------------------------------------
bq. It would also be nice to know when GC is called but we can make message to
reflect that
Olga, Are you suggesting that we should log everytime the memory manager
handler is called or when the memory manager invokes GC after spilling enough
memory ?
I am not sure if it is useful to log every call to the memory manager handler,
maybe we can log the first time for each type of threshold has been exceeded
and then every time we actually spill something to disk.
> Improve log messages for memory usage
> -------------------------------------
>
> Key: PIG-1466
> URL: https://issues.apache.org/jira/browse/PIG-1466
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Affects Versions: 0.7.0
> Reporter: Ashutosh Chauhan
> Assignee: Thejas M Nair
> Priority: Minor
> Fix For: 0.8.0
>
>
> For anything more then a moderately sized dataset Pig usually spits following
> messages:
> {code}
> 2010-05-27 18:28:31,659 INFO org.apache.pig.impl.util.SpillableMemoryManager:
> low memory handler called (Usage
> threshold exceeded) init = 4194304(4096K) used = 672012960(656262K) committed
> = 954466304(932096K) max =
> 954466304(932096K)
> 2010-05-27 18:10:52,653 INFO org.apache.pig.impl.util.SpillableMemoryManager:
> low memory handler called (Collection
> threshold exceeded) init = 4194304(4096K) used = 954466304(932096K) committed
> = 954466304(932096K) max =
> 954466304(932096K)
> {code}
> This seems to confuse users a lot. Once these messages are printed, users
> tend to believe that Pig is having hard time with memory, is spilling to disk
> etc. but in fact Pig might be cruising along at ease. We should be little
> more careful what to print in logs. Currently these are printed when a
> notification is sent by JVM and some other conditions are met which may not
> necessarily indicate low memory condition. Furthermore, with
> {{InternalCachedBag}} embraced everywhere in favor of {{DefaultBag}}, these
> messages have lost their usefulness. At the every least, we should lower the
> log level at which these are printed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
