[jira] [Commented] (AMBARI-21145) Allow wildcard for log directory folder in the path component of Logfeeder input
[
https://issues.apache.org/jira/browse/AMBARI-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251852#comment-16251852
]
Juanjo Marron commented on AMBARI-21145:
Thanks for the update!
> Allow wildcard for log directory folder in the path component of Logfeeder
> input
>
>
> Key: AMBARI-21145
> URL: https://issues.apache.org/jira/browse/AMBARI-21145
> Project: Ambari
> Issue Type: Improvement
> Components: logsearch
>Affects Versions: 2.5.0
>Reporter: Keta Patel
>Assignee: Olivér Szabó
>Priority: Minor
> Fix For: 2.6.1
>
>
> The wildcard processing in Logfeeder is carried out only for the last file
> component in the log path. For a few services, where there are multiple log
> folders and log files from each of the folders is desired to be parsed in
> Logsearch UI, the lack of a wildcard processing for the previous portions
> becomes an issue. Hence, introducing wildcard processing for the log
> directory part of the "path" in the Logfeeder's input for the service will
> help resolve this issue.
> The idea is if we have a folder pattern, we will craete a thread group, and
> there will be different threads for every matching input (now only for tail
> file)
> Features (new input block settings):
> - {{path_update_interval_min}}: the period in minutes for checking new files
> (default: 5, based on detach values, its possible that a new input wont be
> monitored)
> - {{detach_interval_min}}: the period in minutes for checking which files are
> too old (default: 300)
> - {{detach_time_min: the period}} in minutes when we flag a file is too old
> (default: 2000)
> - if a path contains a {{*}} before the last {{/}}, then logfeeder will know
> that the inputs should handled in a thread group, not in a standalone thread
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AMBARI-21145) Allow wildcard for log directory folder in the path component of Logfeeder input
[ https://issues.apache.org/jira/browse/AMBARI-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243670#comment-16243670 ] Olivér Szabó commented on AMBARI-21145: --- [~jmarron], yes that will be there from Ambari 2.6.1 obviously we will need some description how to use it properly & some cleanups as well > Allow wildcard for log directory folder in the path component of Logfeeder > input > > > Key: AMBARI-21145 > URL: https://issues.apache.org/jira/browse/AMBARI-21145 > Project: Ambari > Issue Type: Improvement > Components: logsearch >Affects Versions: 2.5.0 >Reporter: Keta Patel >Assignee: Olivér Szabó >Priority: Minor > > The wildcard processing in Logfeeder is carried out only for the last file > component in the log path. For a few services, where there are multiple log > folders and log files from each of the folders is desired to be parsed in > Logsearch UI, the lack of a wildcard processing for the previous portions > becomes an issue. Hence, introducing wildcard processing for the log > directory part of the "path" in the Logfeeder's input for the service will > help resolve this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMBARI-21145) Allow wildcard for log directory folder in the path component of Logfeeder input
[ https://issues.apache.org/jira/browse/AMBARI-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243130#comment-16243130 ] Juanjo Marron commented on AMBARI-21145: Thanks [~oleewere] for the branch-2.6 integration. It means it should be available on Ambari 2.6.1, right? > Allow wildcard for log directory folder in the path component of Logfeeder > input > > > Key: AMBARI-21145 > URL: https://issues.apache.org/jira/browse/AMBARI-21145 > Project: Ambari > Issue Type: Improvement > Components: logsearch >Affects Versions: 2.5.0 >Reporter: Keta Patel >Assignee: Olivér Szabó >Priority: Minor > > The wildcard processing in Logfeeder is carried out only for the last file > component in the log path. For a few services, where there are multiple log > folders and log files from each of the folders is desired to be parsed in > Logsearch UI, the lack of a wildcard processing for the previous portions > becomes an issue. Hence, introducing wildcard processing for the log > directory part of the "path" in the Logfeeder's input for the service will > help resolve this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMBARI-21145) Allow wildcard for log directory folder in the path component of Logfeeder input
[
https://issues.apache.org/jira/browse/AMBARI-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243005#comment-16243005
]
Olivér Szabó commented on AMBARI-21145:
---
committed to branch-2.6:
{code:java}
commit dadf55ff184db70a40bdef9f2e3cb60685d0bf83
Author: Oliver Szabo
Date: Mon Oct 9 21:17:38 2017 +0200
AMBARI-21145. Allow wildcard for log directory folder in the path component
of Logfeeder input (oleewere)
{code}
I am going to keep this open until its done for trunk as well (it will be a bit
different)
> Allow wildcard for log directory folder in the path component of Logfeeder
> input
>
>
> Key: AMBARI-21145
> URL: https://issues.apache.org/jira/browse/AMBARI-21145
> Project: Ambari
> Issue Type: Improvement
> Components: logsearch
>Affects Versions: 2.5.0
>Reporter: Keta Patel
>Assignee: Olivér Szabó
>Priority: Minor
>
> The wildcard processing in Logfeeder is carried out only for the last file
> component in the log path. For a few services, where there are multiple log
> folders and log files from each of the folders is desired to be parsed in
> Logsearch UI, the lack of a wildcard processing for the previous portions
> becomes an issue. Hence, introducing wildcard processing for the log
> directory part of the "path" in the Logfeeder's input for the service will
> help resolve this issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (AMBARI-21145) Allow wildcard for log directory folder in the path component of Logfeeder input
[ https://issues.apache.org/jira/browse/AMBARI-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243055#comment-16243055 ] Hudson commented on AMBARI-21145: - SUCCESS: Integrated in Jenkins build Ambari-branch-2.6 #433 (See [https://builds.apache.org/job/Ambari-branch-2.6/433/]) AMBARI-21145. Allow wildcard for log directory folder in the path (oleewere: [http://git-wip-us.apache.org/repos/asf?p=ambari.git=commit=dadf55ff184db70a40bdef9f2e3cb60685d0bf83]) * (edit) ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/AbstractInputFile.java * (edit) ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/Input.java * (edit) ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/util/FileUtil.java * (edit) ambari-logsearch/docker/logsearch-docker.sh * (edit) ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/InputFile.java * (add) ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/AbstractLogFileMonitor.java * (add) ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFileDetachMonitor.java * (edit) ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/logconfig/LogConfigHandler.java * (edit) ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/filter/FilterGrok.java * (edit) ambari-logsearch/docker/Dockerfile * (edit) ambari-logsearch/ambari-logsearch-logfeeder/pom.xml * (add) ambari-logsearch/ambari-logsearch-logfeeder/src/main/java/org/apache/ambari/logfeeder/input/monitor/LogFilePathUpdateMonitor.java > Allow wildcard for log directory folder in the path component of Logfeeder > input > > > Key: AMBARI-21145 > URL: https://issues.apache.org/jira/browse/AMBARI-21145 > Project: Ambari > Issue Type: Improvement > Components: logsearch >Affects Versions: 2.5.0 >Reporter: Keta Patel >Assignee: Olivér Szabó >Priority: Minor > > The wildcard processing in Logfeeder is carried out only for the last file > component in the log path. For a few services, where there are multiple log > folders and log files from each of the folders is desired to be parsed in > Logsearch UI, the lack of a wildcard processing for the previous portions > becomes an issue. Hence, introducing wildcard processing for the log > directory part of the "path" in the Logfeeder's input for the service will > help resolve this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMBARI-21145) Allow wildcard for log directory folder in the path component of Logfeeder input
[ https://issues.apache.org/jira/browse/AMBARI-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211452#comment-16211452 ] Olivér Szabó commented on AMBARI-21145: --- [~patel...@us.ibm.com], I have a patch for that issue: https://reviews.apache.org/r/63027/ , it wont go into 2.6.0 (probably 2.6.1) and it will be a bit different for 3.0 as well > Allow wildcard for log directory folder in the path component of Logfeeder > input > > > Key: AMBARI-21145 > URL: https://issues.apache.org/jira/browse/AMBARI-21145 > Project: Ambari > Issue Type: Improvement > Components: logsearch >Affects Versions: 2.5.0 >Reporter: Keta Patel >Assignee: Olivér Szabó >Priority: Minor > > The wildcard processing in Logfeeder is carried out only for the last file > component in the log path. For a few services, where there are multiple log > folders and log files from each of the folders is desired to be parsed in > Logsearch UI, the lack of a wildcard processing for the previous portions > becomes an issue. Hence, introducing wildcard processing for the log > directory part of the "path" in the Logfeeder's input for the service will > help resolve this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMBARI-21145) allow wildcard for log directory folder in the path component of Logfeeder input
[ https://issues.apache.org/jira/browse/AMBARI-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177309#comment-16177309 ] Juanjo Marron commented on AMBARI-21145: Thanks [~mgergely] for your answer. I would keep this JIRA as unresolved since, in the services we are installing on top of HDP, we are hitting the issue described by [~patel...@us.ibm.com] when trying to integrate into LogSearch. One of the logs for our service has different paths for diff nodes in the cluster (minor numeric change in one of the folders) and ideally they should be captured under an unique log id > allow wildcard for log directory folder in the path component of Logfeeder > input > > > Key: AMBARI-21145 > URL: https://issues.apache.org/jira/browse/AMBARI-21145 > Project: Ambari > Issue Type: Improvement > Components: logsearch >Affects Versions: 2.5.0 >Reporter: Keta Patel >Assignee: Keta Patel >Priority: Minor > > The wildcard processing in Logfeeder is carried out only for the last file > component in the log path. For a few services, where there are multiple log > folders and log files from each of the folders is desired to be parsed in > Logsearch UI, the lack of a wildcard processing for the previous portions > becomes an issue. Hence, introducing wildcard processing for the log > directory part of the "path" in the Logfeeder's input for the service will > help resolve this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMBARI-21145) allow wildcard for log directory folder in the path component of Logfeeder input
[ https://issues.apache.org/jira/browse/AMBARI-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172878#comment-16172878 ] Miklos Gergely commented on AMBARI-21145: - The issue in comment #1 is solved. The issue in the description, and the title ( wildcards in the directory folder ) is suggested by the Log Search developer team not to be allowed, the reasons are in comment #2. You may mark this jira as resolved. > allow wildcard for log directory folder in the path component of Logfeeder > input > > > Key: AMBARI-21145 > URL: https://issues.apache.org/jira/browse/AMBARI-21145 > Project: Ambari > Issue Type: Improvement > Components: logsearch >Affects Versions: 2.5.0 >Reporter: Keta Patel >Assignee: Keta Patel >Priority: Minor > > The wildcard processing in Logfeeder is carried out only for the last file > component in the log path. For a few services, where there are multiple log > folders and log files from each of the folders is desired to be parsed in > Logsearch UI, the lack of a wildcard processing for the previous portions > becomes an issue. Hence, introducing wildcard processing for the log > directory part of the "path" in the Logfeeder's input for the service will > help resolve this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMBARI-21145) allow wildcard for log directory folder in the path component of Logfeeder input
[ https://issues.apache.org/jira/browse/AMBARI-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16172403#comment-16172403 ] Juanjo Marron commented on AMBARI-21145: Can we confirm this JIRA has been resolved by AMBARI-21387? In that case it could be marked as resolved > allow wildcard for log directory folder in the path component of Logfeeder > input > > > Key: AMBARI-21145 > URL: https://issues.apache.org/jira/browse/AMBARI-21145 > Project: Ambari > Issue Type: Improvement > Components: logsearch >Affects Versions: 2.5.0 >Reporter: Keta Patel >Assignee: Keta Patel >Priority: Minor > > The wildcard processing in Logfeeder is carried out only for the last file > component in the log path. For a few services, where there are multiple log > folders and log files from each of the folders is desired to be parsed in > Logsearch UI, the lack of a wildcard processing for the previous portions > becomes an issue. Hence, introducing wildcard processing for the log > directory part of the "path" in the Logfeeder's input for the service will > help resolve this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMBARI-21145) allow wildcard for log directory folder in the path component of Logfeeder input
[ https://issues.apache.org/jira/browse/AMBARI-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076445#comment-16076445 ] Miklos Gergely commented on AMBARI-21145: - The issue in comment #1 is solved by this Jira. The patch is committed to trunk. > allow wildcard for log directory folder in the path component of Logfeeder > input > > > Key: AMBARI-21145 > URL: https://issues.apache.org/jira/browse/AMBARI-21145 > Project: Ambari > Issue Type: Improvement > Components: logsearch >Affects Versions: 2.5.0 >Reporter: Keta Patel >Assignee: Keta Patel >Priority: Minor > > The wildcard processing in Logfeeder is carried out only for the last file > component in the log path. For a few services, where there are multiple log > folders and log files from each of the folders is desired to be parsed in > Logsearch UI, the lack of a wildcard processing for the previous portions > becomes an issue. Hence, introducing wildcard processing for the log > directory part of the "path" in the Logfeeder's input for the service will > help resolve this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMBARI-21145) allow wildcard for log directory folder in the path component of Logfeeder input
[ https://issues.apache.org/jira/browse/AMBARI-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064936#comment-16064936 ] Miklos Gergely commented on AMBARI-21145: - About the problem in the description: In fact Log Feeder doesn't support wild card characters in the log folder names. Our assumption was that if there are log files in separate folders the users would surely don't want to load them under the same log id (called type in the input configuration). So you can create a separate input elements for each one of them, and they will be loaded each under their own log id. Please confirm if you really want to load input files from different folders under the same log id, and then we can take a look at the issue, but consider the suggestions above. About the problem in comment #1: if the "tail" parameter is "true", then Log Feeder loads only the first file found for a path with wildcard characters in it. If you set the "tail" parameter to "false", then it will load all of them, but will also stop after it loaded them i.e. it won't wait for new entries to come. We are planning to modify this behavior, and make it possible to load all the files and follow the first one. This is good for rolled over log files, where the user wants to load all the previous parts, but only follow the actual one. However if you still want to load log files from different directories, or specify a path with wild card in it in order to track multiple active log files, and follow all of them then it is not good. > allow wildcard for log directory folder in the path component of Logfeeder > input > > > Key: AMBARI-21145 > URL: https://issues.apache.org/jira/browse/AMBARI-21145 > Project: Ambari > Issue Type: Improvement > Components: logsearch >Affects Versions: 2.5.0 >Reporter: Keta Patel >Assignee: Keta Patel >Priority: Minor > > The wildcard processing in Logfeeder is carried out only for the last file > component in the log path. For a few services, where there are multiple log > folders and log files from each of the folders is desired to be parsed in > Logsearch UI, the lack of a wildcard processing for the previous portions > becomes an issue. Hence, introducing wildcard processing for the log > directory part of the "path" in the Logfeeder's input for the service will > help resolve this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AMBARI-21145) allow wildcard for log directory folder in the path component of Logfeeder input
[ https://issues.apache.org/jira/browse/AMBARI-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1604#comment-1604 ] Keta Patel commented on AMBARI-21145: - Hello Oliver, While processing the *_"path"_* of _*"input"*_ parameter from a -logsearch-conf.xml file, is it possible to select multiple files matching the regex provided in the "path" or just the first match from the many options matching the regex gets selected for Logfeeder input? In my analysis only the 1st match from the "path" regex gets picked up. There is existing logic to check for wildcards, but this code is executed only for those files in the "notReadyList" of the InputManager class. Files in this list are put when no matching file is found for the "path" regex during config loading stage. However, for a regex that matches with at least one of the files in the location, the 1st match gets added to the "inputList" of InputManager class. Please share your input on my understanding of how this logic works. Thank you, Keta > allow wildcard for log directory folder in the path component of Logfeeder > input > > > Key: AMBARI-21145 > URL: https://issues.apache.org/jira/browse/AMBARI-21145 > Project: Ambari > Issue Type: Improvement > Components: logsearch >Affects Versions: 2.5.0 >Reporter: Keta Patel >Assignee: Keta Patel >Priority: Minor > > The wildcard processing in Logfeeder is carried out only for the last file > component in the log path. For a few services, where there are multiple log > folders and log files from each of the folders is desired to be parsed in > Logsearch UI, the lack of a wildcard processing for the previous portions > becomes an issue. Hence, introducing wildcard processing for the log > directory part of the "path" in the Logfeeder's input for the service will > help resolve this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
