[
https://issues.apache.org/jira/browse/HADOOP-15726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16615493#comment-16615493
]
Erik Krogen commented on HADOOP-15726:
--------------------------------------
Thanks for taking a look Chen!
# It can definitely be used for text-only info; you would just call log() with
no arguments (no values specified). Then, you can use {{getCount()}} to
indicate how many lines were throttled. I believe this addresses your
parenthetical thought as well. Take a look at the [most recent
patch|https://issues.apache.org/jira/secure/attachment/12938732/HDFS-13791-HDFS-12943.003.patch]
in HDFS-13791 for examples of this.
# I considered this as well... Especially while writing the Javadoc, I felt the
confusion between "logging" to the throttler, and actually logging. I have a
few ideas, though I'm not thrilled about any of them. Can you let me know your
thoughts? (cc [~csun]):
** checkLog() - indicate that you are *checking* whether you should log rather
than actually logging. This doesn't indicate to me that you are updating the
values stored in the helper, though.
** store() - indicate that you store values. This may be too value-centric and
not be accommodating to the text-only situations.
** update() - fairly generic, indicate that you are updating the state of the
helper both in terms of values and whether or not logging should occur
# The name is only necessary because the API which accepts a user-specified
time value also accepts a logger name. I wanted to avoid too many method
overloads with various argument combinations so I did one short version
({{log(double...)}}) and one long version which accepts both options (this has
the additional benefit of avoiding confusion between {{log(long, double...)}}
and {{log(double...)}}, since something intended as a long could also be
interpreted as a double). If you look at other examples in HDFS-13791, most of
the time, only the short version of the API is used, so there is none of this
confusion. So far we have only one use case which requires multiple logs on a
single helper, and that was a primary/dependent situation, so we made the
dependent case easy. I see your reasoning that it may be less common, but with
only one use case for multiple loggers so far, I think I would prefer to wait
and see which ends up being more common? Let me know if this makes sense to you.
> Create utility to limit frequency of log statements
> ---------------------------------------------------
>
> Key: HADOOP-15726
> URL: https://issues.apache.org/jira/browse/HADOOP-15726
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, util
> Reporter: Erik Krogen
> Assignee: Erik Krogen
> Priority: Major
> Attachments: HADOOP-15726.000.patch, HADOOP-15726.001.patch
>
>
> There is a common pattern of logging a behavior that is normally extraneous.
> Under some circumstances, such a behavior becomes common, flooding the logs
> and making it difficult to see what else is going on in the system. Under
> such situations it is beneficial to limit how frequently the extraneous
> behavior is logged, while capturing some summary information about the
> suppressed log statements.
> This is currently implemented in {{FSNamesystemLock}} (in HDFS-10713). We
> have additional use cases for this in HDFS-13791, so this is a good time to
> create a common utility for different sites to share this logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
