[jira] [Updated] (AMBARI-20378) LogFeeder: Add de-duplication support
[
https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olivér Szabó updated AMBARI-20378:
--
Fix Version/s: 2.5.1
> LogFeeder: Add de-duplication support
> -
>
> Key: AMBARI-20378
> URL: https://issues.apache.org/jira/browse/AMBARI-20378
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.5.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
> Fix For: trunk, 2.5.1
>
> Attachments: AMBARI-20378.patch
>
>
> Add de-duplication support to logfeeder.
> For handling duplications, it will be an LRU cache for every input.
> It can be set up at logfeeder.properties (global setting). For example (the
> values below are the defaults):
> {code:java}
> logfeeder.cache.enabled=false
> logfeeder.cache.size=100
> logfeeder.cache.key.field=log_message
> logfeeder.cache.dedup.interval=1000
> logfeeder.cache.last.dedup.enabled=false
> {code}
> can be overriden in input config block (per monitored file)
> {code:java}
> {
> "input": [
> {
> ...
> "cache_enabled" : "true",
> "cache_size" : "100",
> "cache_dedup_interval" : "1000",
> "cache_last_dedup_enabled" : "false",
> "cache_key_field" : "log_message"
> }
> ] ...
> {code}
> {{cache_dedup_interval}} : if the interval (timestamp - date difference) is
> not reached between 2 log messages (same content), then the new log will be
> dropped.
> {{cache_last_dedup_enabled}}: if its enabled and the new log message is the
> same as the last one, the new log will be dropped. (dedup interval wont have
> impact on that feature)
> {{cache_key_field}} : field of the log message which will be compared and
> stored in the cache (as keys)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AMBARI-20378) LogFeeder: Add de-duplication support
[
https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olivér Szabó updated AMBARI-20378:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
committed to trunk:
{code:java}
commit f8931a925f18dccdfe617641ae912fd93cf5f3e1
Author: oleewere
Date: Fri Mar 10 14:45:13 2017 +0100
AMBARI-20378. Logfeeder: add de-duplication support (oleewere)
{code}
> LogFeeder: Add de-duplication support
> -
>
> Key: AMBARI-20378
> URL: https://issues.apache.org/jira/browse/AMBARI-20378
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.5.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
> Fix For: trunk
>
> Attachments: AMBARI-20378.patch
>
>
> Add de-duplication support to logfeeder.
> For handling duplications, it will be an LRU cache for every input.
> It can be set up at logfeeder.properties (global setting). For example (the
> values below are the defaults):
> {code:java}
> logfeeder.cache.enabled=false
> logfeeder.cache.size=100
> logfeeder.cache.key.field=log_message
> logfeeder.cache.dedup.interval=1000
> logfeeder.cache.last.dedup.enabled=false
> {code}
> can be overriden in input config block (per monitored file)
> {code:java}
> {
> "input": [
> {
> ...
> "cache_enabled" : "true",
> "cache_size" : "100",
> "cache_dedup_interval" : "1000",
> "cache_last_dedup_enabled" : "false",
> "cache_key_field" : "log_message"
> }
> ] ...
> {code}
> {{cache_dedup_interval}} : if the interval (timestamp - date difference) is
> not reached between 2 log messages (same content), then the new log will be
> dropped.
> {{cache_last_dedup_enabled}}: if its enabled and the new log message is the
> same as the last one, the new log will be dropped. (dedup interval wont have
> impact on that feature)
> {{cache_key_field}} : field of the log message which will be compared and
> stored in the cache (as keys)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AMBARI-20378) LogFeeder: Add de-duplication support
[
https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olivér Szabó updated AMBARI-20378:
--
Description:
Add de-duplication support to logfeeder.
For handling duplications, it will be an LRU cache for every input.
It can be set up at logfeeder.properties (global setting). For example (the
values below are the defaults):
{code:java}
logfeeder.cache.enabled=false
logfeeder.cache.size=100
logfeeder.cache.key.field=log_message
logfeeder.cache.dedup.interval=1000
logfeeder.cache.last.dedup.enabled=false
{code}
can be overriden in input config block (per monitored file)
{code:java}
{
"input": [
{
...
"cache_enabled" : "true",
"cache_size" : "100",
"cache_dedup_interval" : "1000",
"cache_last_dedup_enabled" : "false",
"cache_key_field" : "log_message"
}
] ...
{code}
{{cache_dedup_interval}} : if the interval (timestamp - date difference) is not
reached between 2 log messages (same content), then the new log will be
dropped.
{{cache_last_dedup_enabled}}: if its enabled and the new log message is the
same as the last one, the new log will be dropped. (dedup interval wont have
impact on that feature)
{{cache_key_field}} : field of the log message which will be compared and
stored in the cache (as keys)
was:
Add de-duplication support to logfeeder.
For handling duplications, it will be an LRU cache for every input.
It can be set up at logfeeder.properties. For example (the values below are the
defaults):
{code:java}
logfeeder.cache.enabled=false
logfeeder.cache.size=100
logfeeder.cache.key.field=log_message
logfeeder.cache.dedup.interval=1000
logfeeder.cache.last.dedup.enabled=false
{code}
can be overriden in input config block (per monitored file)
{code:java}
{
"input": [
{
...
"cache_enabled" : "true",
"cache_size" : "100",
"cache_dedup_interval" : "1000",
"cache_last_dedup_enabled" : "false",
"cache_key_field" : "log_message"
}
] ...
{code}
{{cache_dedup_interval}} : if the interval (timestamp - date difference) is not
reached between 2 log messages (same content), then the new log will be
dropped.
{{cache_last_dedup_enabled}}: if its enabled and the new log message is the
same as the last one, the new log will be dropped. (dedup interval wont have
impact on that feature)
{{cache_key_field}} : field of the log message which will be compared and
stored in the cache (as keys)
> LogFeeder: Add de-duplication support
> -
>
> Key: AMBARI-20378
> URL: https://issues.apache.org/jira/browse/AMBARI-20378
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.5.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
> Fix For: trunk
>
> Attachments: AMBARI-20378.patch
>
>
> Add de-duplication support to logfeeder.
> For handling duplications, it will be an LRU cache for every input.
> It can be set up at logfeeder.properties (global setting). For example (the
> values below are the defaults):
> {code:java}
> logfeeder.cache.enabled=false
> logfeeder.cache.size=100
> logfeeder.cache.key.field=log_message
> logfeeder.cache.dedup.interval=1000
> logfeeder.cache.last.dedup.enabled=false
> {code}
> can be overriden in input config block (per monitored file)
> {code:java}
> {
> "input": [
> {
> ...
> "cache_enabled" : "true",
> "cache_size" : "100",
> "cache_dedup_interval" : "1000",
> "cache_last_dedup_enabled" : "false",
> "cache_key_field" : "log_message"
> }
> ] ...
> {code}
> {{cache_dedup_interval}} : if the interval (timestamp - date difference) is
> not reached between 2 log messages (same content), then the new log will be
> dropped.
> {{cache_last_dedup_enabled}}: if its enabled and the new log message is the
> same as the last one, the new log will be dropped. (dedup interval wont have
> impact on that feature)
> {{cache_key_field}} : field of the log message which will be compared and
> stored in the cache (as keys)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AMBARI-20378) LogFeeder: Add de-duplication support
[
https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olivér Szabó updated AMBARI-20378:
--
Attachment: (was: AMBARI-20378.patch)
> LogFeeder: Add de-duplication support
> -
>
> Key: AMBARI-20378
> URL: https://issues.apache.org/jira/browse/AMBARI-20378
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.5.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
> Fix For: trunk
>
> Attachments: AMBARI-20378.patch
>
>
> Add de-duplication support to logfeeder.
> For handling duplications, it will be an LRU cache for every input.
> It can be set up at logfeeder.properties (global setting). For example (the
> values below are the defaults):
> {code:java}
> logfeeder.cache.enabled=false
> logfeeder.cache.size=100
> logfeeder.cache.key.field=log_message
> logfeeder.cache.dedup.interval=1000
> logfeeder.cache.last.dedup.enabled=false
> {code}
> can be overriden in input config block (per monitored file)
> {code:java}
> {
> "input": [
> {
> ...
> "cache_enabled" : "true",
> "cache_size" : "100",
> "cache_dedup_interval" : "1000",
> "cache_last_dedup_enabled" : "false",
> "cache_key_field" : "log_message"
> }
> ] ...
> {code}
> {{cache_dedup_interval}} : if the interval (timestamp - date difference) is
> not reached between 2 log messages (same content), then the new log will be
> dropped.
> {{cache_last_dedup_enabled}}: if its enabled and the new log message is the
> same as the last one, the new log will be dropped. (dedup interval wont have
> impact on that feature)
> {{cache_key_field}} : field of the log message which will be compared and
> stored in the cache (as keys)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AMBARI-20378) LogFeeder: Add de-duplication support
[
https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olivér Szabó updated AMBARI-20378:
--
Attachment: AMBARI-20378.patch
> LogFeeder: Add de-duplication support
> -
>
> Key: AMBARI-20378
> URL: https://issues.apache.org/jira/browse/AMBARI-20378
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.5.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
> Fix For: trunk
>
> Attachments: AMBARI-20378.patch
>
>
> Add de-duplication support to logfeeder.
> For handling duplications, it will be an LRU cache for every input.
> It can be set up at logfeeder.properties (global setting). For example (the
> values below are the defaults):
> {code:java}
> logfeeder.cache.enabled=false
> logfeeder.cache.size=100
> logfeeder.cache.key.field=log_message
> logfeeder.cache.dedup.interval=1000
> logfeeder.cache.last.dedup.enabled=false
> {code}
> can be overriden in input config block (per monitored file)
> {code:java}
> {
> "input": [
> {
> ...
> "cache_enabled" : "true",
> "cache_size" : "100",
> "cache_dedup_interval" : "1000",
> "cache_last_dedup_enabled" : "false",
> "cache_key_field" : "log_message"
> }
> ] ...
> {code}
> {{cache_dedup_interval}} : if the interval (timestamp - date difference) is
> not reached between 2 log messages (same content), then the new log will be
> dropped.
> {{cache_last_dedup_enabled}}: if its enabled and the new log message is the
> same as the last one, the new log will be dropped. (dedup interval wont have
> impact on that feature)
> {{cache_key_field}} : field of the log message which will be compared and
> stored in the cache (as keys)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AMBARI-20378) LogFeeder: Add de-duplication support
[
https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olivér Szabó updated AMBARI-20378:
--
Description:
Add de-duplication support to logfeeder.
For handling duplications, it will be an LRU cache for every input.
It can be set up at logfeeder.properties. For example (the values below are the
defaults):
{code:java}
logfeeder.cache.enabled=false
logfeeder.cache.size=100
logfeeder.cache.key.field=log_message
logfeeder.cache.dedup.interval=1000
logfeeder.cache.last.dedup.enabled=false
{code}
can be overriden in input config block (per monitored file)
{code:java}
{
"input": [
{
...
"cache_enabled" : "true",
"cache_size" : "100",
"cache_dedup_interval" : "1000",
"cache_last_dedup_enabled" : "false",
"cache_key_field" : "log_message"
}
] ...
{code}
{{cache_dedup_interval}} : if the interval (timestamp - date difference) is not
reached between 2 log messages (same content), then the new log will be
dropped.
{{cache_last_dedup_enabled}}: if its enabled and the new log message is the
same as the last one, the new log will be dropped. (dedup interval wont have
impact on that feature)
{{cache_key_field}} : field of the log message which will be compared and
stored in the cache (as keys)
was:
Add de-duplication support to logfeeder.
For handling duplications, it will be an LRU cache for every input.
It can be set up at input config block. For example (the values below are the
defaults):
{code:java}
{
"input": [
{
...
"cache_enabled" : "true",
"cache_size" : "100",
"cache_dedup_interval" : "1000",
"cache_last_dedup_enabled" : "false",
"cache_key_field" : "log_message"
}
] ...
{code}
{{cache_dedup_interval}} : if the interval (timestamp - date difference) is not
reached between 2 log messages (same content), then the new log will be
dropped.
{{cache_last_dedup_enabled}}: if its enabled and the new log message is the
same as the last one, the new log will be dropped. (dedup interval wont have
impact on that feature)
{{cache_key_field}} : field of the log message which will be compared and
stored in the cache (as keys)
> LogFeeder: Add de-duplication support
> -
>
> Key: AMBARI-20378
> URL: https://issues.apache.org/jira/browse/AMBARI-20378
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.5.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
> Fix For: trunk
>
> Attachments: AMBARI-20378.patch
>
>
> Add de-duplication support to logfeeder.
> For handling duplications, it will be an LRU cache for every input.
> It can be set up at logfeeder.properties. For example (the values below are
> the defaults):
> {code:java}
> logfeeder.cache.enabled=false
> logfeeder.cache.size=100
> logfeeder.cache.key.field=log_message
> logfeeder.cache.dedup.interval=1000
> logfeeder.cache.last.dedup.enabled=false
> {code}
> can be overriden in input config block (per monitored file)
> {code:java}
> {
> "input": [
> {
> ...
> "cache_enabled" : "true",
> "cache_size" : "100",
> "cache_dedup_interval" : "1000",
> "cache_last_dedup_enabled" : "false",
> "cache_key_field" : "log_message"
> }
> ] ...
> {code}
> {{cache_dedup_interval}} : if the interval (timestamp - date difference) is
> not reached between 2 log messages (same content), then the new log will be
> dropped.
> {{cache_last_dedup_enabled}}: if its enabled and the new log message is the
> same as the last one, the new log will be dropped. (dedup interval wont have
> impact on that feature)
> {{cache_key_field}} : field of the log message which will be compared and
> stored in the cache (as keys)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AMBARI-20378) LogFeeder: Add de-duplication support
[
https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olivér Szabó updated AMBARI-20378:
--
Attachment: AMBARI-20378.patch
> LogFeeder: Add de-duplication support
> -
>
> Key: AMBARI-20378
> URL: https://issues.apache.org/jira/browse/AMBARI-20378
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.5.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
> Fix For: trunk
>
> Attachments: AMBARI-20378.patch
>
>
> Add de-duplication support to logfeeder.
> For handling duplications, it will be an LRU cache for every input.
> It can be set up at input config block. For example (the values below are the
> defaults):
> {code:java}
> {
> "input": [
> {
> ...
> "cache_enabled" : "true",
> "cache_size" : "100",
> "cache_dedup_interval" : "1000",
> "cache_last_dedup_enabled" : "false",
> "cache_key_field" : "log_message"
> }
> ] ...
> {code}
> {{cache_dedup_interval}} : if the interval (timestamp - date difference) is
> not reached between 2 log messages (same content), then the new log will be
> dropped.
> {{cache_last_dedup_enabled}}: if its enabled and the new log message is the
> same as the last one, the new log will be dropped. (dedup interval wont have
> impact on that feature)
> {{cache_key_field}} : field of the log message which will be compared and
> stored in the cache (as keys)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AMBARI-20378) LogFeeder: Add de-duplication support
[
https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olivér Szabó updated AMBARI-20378:
--
Attachment: (was: AMBARI-20378.patch)
> LogFeeder: Add de-duplication support
> -
>
> Key: AMBARI-20378
> URL: https://issues.apache.org/jira/browse/AMBARI-20378
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.5.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
> Fix For: trunk
>
> Attachments: AMBARI-20378.patch
>
>
> Add de-duplication support to logfeeder.
> For handling duplications, it will be an LRU cache for every input.
> It can be set up at input config block. For example (the values below are the
> defaults):
> {code:java}
> {
> "input": [
> {
> ...
> "cache_enabled" : "true",
> "cache_size" : "100",
> "cache_dedup_interval" : "1000",
> "cache_last_dedup_enabled" : "false",
> "cache_key_field" : "log_message"
> }
> ] ...
> {code}
> {{cache_dedup_interval}} : if the interval (timestamp - date difference) is
> not reached between 2 log messages (same content), then the new log will be
> dropped.
> {{cache_last_dedup_enabled}}: if its enabled and the new log message is the
> same as the last one, the new log will be dropped. (dedup interval wont have
> impact on that feature)
> {{cache_key_field}} : field of the log message which will be compared and
> stored in the cache (as keys)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AMBARI-20378) LogFeeder: Add de-duplication support
[
https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Olivér Szabó updated AMBARI-20378:
--
Description:
Add de-duplication support to logfeeder.
For handling duplications, it will be an LRU cache for every input.
It can be set up at input config block. For example (the values below are the
defaults):
{code:java}
{
"input": [
{
...
"cache_enabled" : "true",
"cache_size" : "100",
"cache_dedup_interval" : "1000",
"cache_last_dedup_enabled" : "false",
"cache_key_field" : "log_message"
}
] ...
{code}
{{cache_dedup_interval}} : if the interval (timestamp - date difference) is not
reached between 2 log messages (same content), then the new log will be
dropped.
{{cache_last_dedup_enabled}}: if its enabled and the new log message is the
same as the last one, the new log will be dropped. (dedup interval wont have
impact on that feature)
{{cache_key_field}} : field of the log message which will be compared and
stored in the cache (as keys)
> LogFeeder: Add de-duplication support
> -
>
> Key: AMBARI-20378
> URL: https://issues.apache.org/jira/browse/AMBARI-20378
> Project: Ambari
> Issue Type: Bug
> Components: ambari-logsearch
>Affects Versions: 2.5.0
>Reporter: Olivér Szabó
>Assignee: Olivér Szabó
> Fix For: trunk
>
> Attachments: AMBARI-20378.patch
>
>
> Add de-duplication support to logfeeder.
> For handling duplications, it will be an LRU cache for every input.
> It can be set up at input config block. For example (the values below are the
> defaults):
> {code:java}
> {
> "input": [
> {
> ...
> "cache_enabled" : "true",
> "cache_size" : "100",
> "cache_dedup_interval" : "1000",
> "cache_last_dedup_enabled" : "false",
> "cache_key_field" : "log_message"
> }
> ] ...
> {code}
> {{cache_dedup_interval}} : if the interval (timestamp - date difference) is
> not reached between 2 log messages (same content), then the new log will be
> dropped.
> {{cache_last_dedup_enabled}}: if its enabled and the new log message is the
> same as the last one, the new log will be dropped. (dedup interval wont have
> impact on that feature)
> {{cache_key_field}} : field of the log message which will be compared and
> stored in the cache (as keys)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (AMBARI-20378) LogFeeder: Add de-duplication support
[ https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olivér Szabó updated AMBARI-20378: -- Status: Patch Available (was: Open) > LogFeeder: Add de-duplication support > - > > Key: AMBARI-20378 > URL: https://issues.apache.org/jira/browse/AMBARI-20378 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.5.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó > Fix For: trunk > > Attachments: AMBARI-20378.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AMBARI-20378) LogFeeder: Add de-duplication support
[ https://issues.apache.org/jira/browse/AMBARI-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olivér Szabó updated AMBARI-20378: -- Attachment: AMBARI-20378.patch > LogFeeder: Add de-duplication support > - > > Key: AMBARI-20378 > URL: https://issues.apache.org/jira/browse/AMBARI-20378 > Project: Ambari > Issue Type: Bug > Components: ambari-logsearch >Affects Versions: 2.5.0 >Reporter: Olivér Szabó >Assignee: Olivér Szabó > Fix For: trunk > > Attachments: AMBARI-20378.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
