I'm able to reproduce the issue, I'll work on getting a fix on the
versions/0.9 branch over the next couple of days.
-r
On 03/27/2015 09:33 PM, rajive kumar wrote:
Hi Guys
Any idea on this..
Best Regards,
On 25 March 2015 at 11:29, rajive kumar <[email protected]
<mailto:[email protected]>> wrote:
Hi
Please find the link for the toml file and sh script which i have
used to test.
http://pastebin.ca/2963610 - toml file
http://pastebin.ca/2963598 - sh script for simulate the log rotation.
Steps I have followed
1. Run the script for some time by increasing the no of records
2. Start the heka while script is running.
3. After some time get the line count of generated files and heka
output files ( Heka out put file have more than generated file
count i.e its containing duplicate values )
Thank you in advance for the help.
Best Regards,
On 25 March 2015 at 05:40, Rob Miller <[email protected]
<mailto:[email protected]>> wrote:
If you're using the translation map as I've specified and you're
still seeing the same file (or files) being read twice, then
there might be a bug. I can't quite tell from what you're saying
whether or not you've applied the change that I recommended. Can
you paste your entire Heka config in a pastebin somewhere so I
can look at it? Or at the very least the entire LogstreamerInput
section, including the translation map subsection if you've
applied it.
Thanks,
-r
On 03/24/2015 12:07 PM, rajive kumar wrote:
---------- Forwarded message ----------
From: *rajive kumar* <[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>>
Date: 25 March 2015 at 00:35
Subject: Re: [heka] Logstreamer Input issue with current log
and rolling
file
To: [email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
Hi
Thank you.. What you have mention is correct.I have tested
that too. But
my problem is when current file rolling Heka is reading all
the rolled
file again. For static file its working perfectly.
For example
In server log files line count is : 3015
But heka file reading log side is : 23065
file format which I have used is
[smpp_logs]
type = "LogstreamerInput"
log_directory =
"/media/rajive/Office/tmp/__test/sender/smpp"
file_match= 'HS_06_SMPP_FULL_CDR(_(?P<Seq>__\d*))?'
priority =["Seq"]
[smpp_logs.translation.Seq]
missing = 9999999999999
Please find the attachment for more details. Also I have
started freshly
i.e before start i have removed all the files.
On 25 March 2015 at 00:08, Rob Miller <[email protected]
<mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>>
wrote:
You need to use what we call a "translation map" to map
a missing
`Seq` value to the highest possible value. The
following is an
example of what should work:
[smpp_logs]
type = "LogstreamerInput"
log_directory = "/path/to/smpp/logs"
file_match= 'smpp_cdr(_(?P<Seq>\d*))?'
priority =["Seq"]
[smpp_logs.translation.Seq]
missing = 9999999999999999
Also be aware that you can test out the order that
LogstreamerInput
will load log files before you actually run Heka by
pointing the
heka-logstreamer utility at your configuration:
hekad.readthedocs.org/en/v0.9.____0/pluginconfig/logstreamer.____html#verifying-settings
<http://hekad.readthedocs.org/en/v0.9.__0/pluginconfig/logstreamer.__html#verifying-settings>
<http://hekad.readthedocs.org/__en/v0.9.0/pluginconfig/__logstreamer.html#verifying-__settings
<http://hekad.readthedocs.org/en/v0.9.0/pluginconfig/logstreamer.html#verifying-settings>>
-r
On 03/24/2015 11:26 AM, rajive kumar wrote:
Hi Guys
Any idea on this
I using LogstreamerInput to read my ongoing logs.
Currently my
logs has
been rolling for every 15h min. When using heka its
reading
duplicate
values. I.e its reading same file again and again.
Following is the file pattern
smpp_cdr (This the current log file)
smpp_cdr_2015032416150055
smpp_cdr_2015032416300056
smpp_cdr_2015032416450057
I using following pattern to identify the file pattern
file_match= 'smpp_cdr(_(?P<Seq>\d*))?'
priority =["Seq"]
This is read the rolled files perfectly but its
reading the
current
log file (smpp_cdr) first. Also when current file
gets rolled heka
reading all the file again. Please advice / help on
this.
Thank you
On 20 March 2015 at 18:44, rajive kumar
<[email protected] <mailto:[email protected]>
<mailto:[email protected]
<mailto:[email protected]>>
<mailto:[email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>>> wrote:
Hi
Currently I am using Logstreamer input to
process another
applications current log file. Its working
fine but I have
an issue
when file rolling -> its reading the file
twice. Please help on
this. Below the file structure I have in that
system
*trans_log* <- current log
*150320001_trans_log *
*150320002_trans_log
*
*150320003_trans_log
*
*150320004_trans_log
*
*150320005_trans_log
*
*150320006_trans_log*
Note my files are rolling in every 15 mins and
I want to
read the
previous logs and current log while its
printing(i.e logging).
Best Regards,
___________________________________________________
Heka mailing list
[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
https://mail.mozilla.org/____listinfo/heka
<https://mail.mozilla.org/__listinfo/heka>
<https://mail.mozilla.org/__listinfo/heka
<https://mail.mozilla.org/listinfo/heka>>
_________________________________________________
Heka mailing list
[email protected] <mailto:[email protected]>
https://mail.mozilla.org/__listinfo/heka
<https://mail.mozilla.org/listinfo/heka>
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
