[
https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olivér Szabó updated AMBARI-20378:
----------------------------------
Description:
Add de-duplication support to logfeeder.
For handling duplications, it will be an LRU cache for every input.
It can be set up at logfeeder.properties (global setting). For example (the
values below are the defaults):
{code:java}
logfeeder.cache.enabled=false
logfeeder.cache.size=100
logfeeder.cache.key.field=log_message
logfeeder.cache.dedup.interval=1000
logfeeder.cache.last.dedup.enabled=false
{code}
can be overriden in input config block (per monitored file)
{code:java}
{
"input": [
{
...
"cache_enabled" : "true",
"cache_size" : "100",
"cache_dedup_interval" : "1000",
"cache_last_dedup_enabled" : "false",
"cache_key_field" : "log_message"
}
] ...
{code}
{{cache_dedup_interval}} : if the interval (timestamp - date difference) is not
reached between 2 log messages (same content), then the new log will be
dropped.
{{cache_last_dedup_enabled}}: if its enabled and the new log message is the
same as the last one, the new log will be dropped. (dedup interval wont have
impact on that feature)
{{cache_key_field}} : field of the log message which will be compared and
stored in the cache (as keys)
was:
Add de-duplication support to logfeeder.
For handling duplications, it will be an LRU cache for every input.
It can be set up at logfeeder.properties. For example (the values below are the
defaults):
{code:java}
logfeeder.cache.enabled=false
logfeeder.cache.size=100
logfeeder.cache.key.field=log_message
logfeeder.cache.dedup.interval=1000
logfeeder.cache.last.dedup.enabled=false
{code}
can be overriden in input config block (per monitored file)
{code:java}
{
"input": [
{
...
"cache_enabled" : "true",
"cache_size" : "100",
"cache_dedup_interval" : "1000",
"cache_last_dedup_enabled" : "false",
"cache_key_field" : "log_message"
}
] ...
{code}
{{cache_dedup_interval}} : if the interval (timestamp - date difference) is not
reached between 2 log messages (same content), then the new log will be
dropped.
{{cache_last_dedup_enabled}}: if its enabled and the new log message is the
same as the last one, the new log will be dropped. (dedup interval wont have
impact on that feature)
{{cache_key_field}} : field of the log message which will be compared and
stored in the cache (as keys)
> LogFeeder: Add de-duplication support
> -------------------------------------
>
> Key: AMBARI-20378
> URL: https://issues.apache.org/jira/browse/AMBARI-20378
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
> Affects Versions: 2.5.0
> Reporter: Olivér Szabó
> Assignee: Olivér Szabó
> Fix For: trunk
>
> Attachments: AMBARI-20378.patch
>
>
> Add de-duplication support to logfeeder.
> For handling duplications, it will be an LRU cache for every input.
> It can be set up at logfeeder.properties (global setting). For example (the
> values below are the defaults):
> {code:java}
> logfeeder.cache.enabled=false
> logfeeder.cache.size=100
> logfeeder.cache.key.field=log_message
> logfeeder.cache.dedup.interval=1000
> logfeeder.cache.last.dedup.enabled=false
> {code}
> can be overriden in input config block (per monitored file)
> {code:java}
> {
> "input": [
> {
> ...
> "cache_enabled" : "true",
> "cache_size" : "100",
> "cache_dedup_interval" : "1000",
> "cache_last_dedup_enabled" : "false",
> "cache_key_field" : "log_message"
> }
> ] ...
> {code}
> {{cache_dedup_interval}} : if the interval (timestamp - date difference) is
> not reached between 2 log messages (same content), then the new log will be
> dropped.
> {{cache_last_dedup_enabled}}: if its enabled and the new log message is the
> same as the last one, the new log will be dropped. (dedup interval wont have
> impact on that feature)
> {{cache_key_field}} : field of the log message which will be compared and
> stored in the cache (as keys)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
