[
https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olivér Szabó updated AMBARI-20378:
----------------------------------
Description:
Add de-duplication support to logfeeder.
For handling duplications, it will be an LRU cache for every input.
It can be set up at input config block. For example (the values below are the
defaults):
{code:java}
{
"input": [
{
...
"cache_enabled" : "true",
"cache_size" : "100",
"cache_dedup_interval" : "1000",
"cache_last_dedup_enabled" : "false",
"cache_key_field" : "log_message"
}
] ...
{code}
{{cache_dedup_interval}} : if the interval (timestamp - date difference) is not
reached between 2 log messages (same content), then the new log will be
dropped.
{{cache_last_dedup_enabled}}: if its enabled and the new log message is the
same as the last one, the new log will be dropped. (dedup interval wont have
impact on that feature)
{{cache_key_field}} : field of the log message which will be compared and
stored in the cache (as keys)
> LogFeeder: Add de-duplication support
> -------------------------------------
>
> Key: AMBARI-20378
> URL: https://issues.apache.org/jira/browse/AMBARI-20378
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
> Affects Versions: 2.5.0
> Reporter: Olivér Szabó
> Assignee: Olivér Szabó
> Fix For: trunk
>
> Attachments: AMBARI-20378.patch
>
>
> Add de-duplication support to logfeeder.
> For handling duplications, it will be an LRU cache for every input.
> It can be set up at input config block. For example (the values below are the
> defaults):
> {code:java}
> {
> "input": [
> {
> ...
> "cache_enabled" : "true",
> "cache_size" : "100",
> "cache_dedup_interval" : "1000",
> "cache_last_dedup_enabled" : "false",
> "cache_key_field" : "log_message"
> }
> ] ...
> {code}
> {{cache_dedup_interval}} : if the interval (timestamp - date difference) is
> not reached between 2 log messages (same content), then the new log will be
> dropped.
> {{cache_last_dedup_enabled}}: if its enabled and the new log message is the
> same as the last one, the new log will be dropped. (dedup interval wont have
> impact on that feature)
> {{cache_key_field}} : field of the log message which will be compared and
> stored in the cache (as keys)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
