---------- Forwarded message ----------
From: rajive kumar <[email protected]>
Date: 25 March 2015 at 00:35
Subject: Re: [heka] Logstreamer Input issue with current log and rolling
file
To: [email protected]
Hi
Thank you.. What you have mention is correct.I have tested that too. But my
problem is when current file rolling Heka is reading all the rolled file
again. For static file its working perfectly.
For example
In server log files line count is : 3015
But heka file reading log side is : 23065
file format which I have used is
[smpp_logs]
type = "LogstreamerInput"
log_directory = "/media/rajive/Office/tmp/test/sender/smpp"
file_match= 'HS_06_SMPP_FULL_CDR(_(?P<Seq>\d*))?'
priority =["Seq"]
[smpp_logs.translation.Seq]
missing = 9999999999999
Please find the attachment for more details. Also I have started freshly
i.e before start i have removed all the files.
On 25 March 2015 at 00:08, Rob Miller <[email protected]> wrote:
> You need to use what we call a "translation map" to map a missing `Seq`
> value to the highest possible value. The following is an example of what
> should work:
>
> [smpp_logs]
> type = "LogstreamerInput"
> log_directory = "/path/to/smpp/logs"
> file_match= 'smpp_cdr(_(?P<Seq>\d*))?'
> priority =["Seq"]
> [smpp_logs.translation.Seq]
> missing = 9999999999999999
>
> Also be aware that you can test out the order that LogstreamerInput will
> load log files before you actually run Heka by pointing the
> heka-logstreamer utility at your configuration:
> hekad.readthedocs.org/en/v0.9.0/pluginconfig/logstreamer.
> html#verifying-settings
>
> -r
>
>
> On 03/24/2015 11:26 AM, rajive kumar wrote:
>
>> Hi Guys
>>
>> Any idea on this
>>
>> I using LogstreamerInput to read my ongoing logs. Currently my logs has
>> been rolling for every 15h min. When using heka its reading duplicate
>> values. I.e its reading same file again and again.
>>
>> Following is the file pattern
>>
>> smpp_cdr (This the current log file)
>> smpp_cdr_2015032416150055
>> smpp_cdr_2015032416300056
>> smpp_cdr_2015032416450057
>>
>> I using following pattern to identify the file pattern
>> file_match= 'smpp_cdr(_(?P<Seq>\d*))?'
>> priority =["Seq"]
>> This is read the rolled files perfectly but its reading the current
>> log file (smpp_cdr) first. Also when current file gets rolled heka
>> reading all the file again. Please advice / help on this.
>>
>> Thank you
>>
>> On 20 March 2015 at 18:44, rajive kumar <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Hi
>>
>> Currently I am using Logstreamer input to process another
>> applications current log file. Its working fine but I have an issue
>> when file rolling -> its reading the file twice. Please help on
>> this. Below the file structure I have in that system
>>
>> *trans_log* <- current log
>> *150320001_trans_log *
>> *150320002_trans_log
>> *
>> *150320003_trans_log
>> *
>> *150320004_trans_log
>> *
>> *150320005_trans_log
>> *
>> *150320006_trans_log*
>>
>> Note my files are rolling in every 15 mins and I want to read the
>> previous logs and current log while its printing(i.e logging).
>>
>>
>> Best Regards,
>>
>>
>>
>>
>> _______________________________________________
>> Heka mailing list
>> [email protected]
>> https://mail.mozilla.org/listinfo/heka
>>
>>
>
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
