[ https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olivér Szabó updated AMBARI-20378: ---------------------------------- Description: Add de-duplication support to logfeeder. For handling duplications, it will be an LRU cache for every input. It can be set up at logfeeder.properties. For example (the values below are the defaults): {code:java} logfeeder.cache.enabled=false logfeeder.cache.size=100 logfeeder.cache.key.field=log_message logfeeder.cache.dedup.interval=1000 logfeeder.cache.last.dedup.enabled=false {code} can be overriden in input config block (per monitored file) {code:java} { "input": [ { ... "cache_enabled" : "true", "cache_size" : "100", "cache_dedup_interval" : "1000", "cache_last_dedup_enabled" : "false", "cache_key_field" : "log_message" } ] ... {code} {{cache_dedup_interval}} : if the interval (timestamp - date difference) is not reached between 2 log messages (same content), then the new log will be dropped. {{cache_last_dedup_enabled}}: if its enabled and the new log message is the same as the last one, the new log will be dropped. (dedup interval wont have impact on that feature) {{cache_key_field}} : field of the log message which will be compared and stored in the cache (as keys) was: Add de-duplication support to logfeeder. For handling duplications, it will be an LRU cache for every input. It can be set up at input config block. For example (the values below are the defaults): {code:java} { "input": [ { ... "cache_enabled" : "true", "cache_size" : "100", "cache_dedup_interval" : "1000", "cache_last_dedup_enabled" : "false", "cache_key_field" : "log_message" } ] ... {code} {{cache_dedup_interval}} : if the interval (timestamp - date difference) is not reached between 2 log messages (same content), then the new log will be dropped. {{cache_last_dedup_enabled}}: if its enabled and the new log message is the same as the last one, the new log will be dropped. (dedup interval wont have impact on that feature) {{cache_key_field}} : field of the log message which will be compared and stored in the cache (as keys) > LogFeeder: Add de-duplication support > ------------------------------------- > > Key: AMBARI-20378 > URL: https://issues.apache.org/jira/browse/AMBARI-20378 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch > Affects Versions: 2.5.0 > Reporter: Olivér Szabó > Assignee: Olivér Szabó > Fix For: trunk > > Attachments: AMBARI-20378.patch > > > Add de-duplication support to logfeeder. > For handling duplications, it will be an LRU cache for every input. > It can be set up at logfeeder.properties. For example (the values below are > the defaults): > {code:java} > logfeeder.cache.enabled=false > logfeeder.cache.size=100 > logfeeder.cache.key.field=log_message > logfeeder.cache.dedup.interval=1000 > logfeeder.cache.last.dedup.enabled=false > {code} > can be overriden in input config block (per monitored file) > {code:java} > { > "input": [ > { > ... > "cache_enabled" : "true", > "cache_size" : "100", > "cache_dedup_interval" : "1000", > "cache_last_dedup_enabled" : "false", > "cache_key_field" : "log_message" > } > ] ... > {code} > {{cache_dedup_interval}} : if the interval (timestamp - date difference) is > not reached between 2 log messages (same content), then the new log will be > dropped. > {{cache_last_dedup_enabled}}: if its enabled and the new log message is the > same as the last one, the new log will be dropped. (dedup interval wont have > impact on that feature) > {{cache_key_field}} : field of the log message which will be compared and > stored in the cache (as keys) -- This message was sent by Atlassian JIRA (v6.3.15#6346)