[ https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olivér Szabó updated AMBARI-20378: ---------------------------------- Resolution: Fixed Status: Resolved (was: Patch Available) committed to trunk: {code:java} commit f8931a925f18dccdfe617641ae912fd93cf5f3e1 Author: oleewere <oleew...@gmail.com> Date: Fri Mar 10 14:45:13 2017 +0100 AMBARI-20378. Logfeeder: add de-duplication support (oleewere) {code} > LogFeeder: Add de-duplication support > ------------------------------------- > > Key: AMBARI-20378 > URL: https://issues.apache.org/jira/browse/AMBARI-20378 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch > Affects Versions: 2.5.0 > Reporter: Olivér Szabó > Assignee: Olivér Szabó > Fix For: trunk > > Attachments: AMBARI-20378.patch > > > Add de-duplication support to logfeeder. > For handling duplications, it will be an LRU cache for every input. > It can be set up at logfeeder.properties (global setting). For example (the > values below are the defaults): > {code:java} > logfeeder.cache.enabled=false > logfeeder.cache.size=100 > logfeeder.cache.key.field=log_message > logfeeder.cache.dedup.interval=1000 > logfeeder.cache.last.dedup.enabled=false > {code} > can be overriden in input config block (per monitored file) > {code:java} > { > "input": [ > { > ... > "cache_enabled" : "true", > "cache_size" : "100", > "cache_dedup_interval" : "1000", > "cache_last_dedup_enabled" : "false", > "cache_key_field" : "log_message" > } > ] ... > {code} > {{cache_dedup_interval}} : if the interval (timestamp - date difference) is > not reached between 2 log messages (same content), then the new log will be > dropped. > {{cache_last_dedup_enabled}}: if its enabled and the new log message is the > same as the last one, the new log will be dropped. (dedup interval wont have > impact on that feature) > {{cache_key_field}} : field of the log message which will be compared and > stored in the cache (as keys) -- This message was sent by Atlassian JIRA (v6.3.15#6346)