Hi Thank you...
Best Regards RajiveKumar On 28 March 2015 at 12:14, Rob Miller <[email protected]> wrote: > I'm able to reproduce the issue, I'll work on getting a fix on the > versions/0.9 branch over the next couple of days. > > -r > > > On 03/27/2015 09:33 PM, rajive kumar wrote: > >> Hi Guys >> >> Any idea on this.. >> >> Best Regards, >> >> On 25 March 2015 at 11:29, rajive kumar <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi >> >> Please find the link for the toml file and sh script which i have >> used to test. >> >> http://pastebin.ca/2963610 - toml file >> >> http://pastebin.ca/2963598 - sh script for simulate the log rotation. >> >> Steps I have followed >> >> 1. Run the script for some time by increasing the no of records >> 2. Start the heka while script is running. >> 3. After some time get the line count of generated files and heka >> output files ( Heka out put file have more than generated file >> count i.e its containing duplicate values ) >> >> Thank you in advance for the help. >> >> Best Regards, >> >> >> On 25 March 2015 at 05:40, Rob Miller <[email protected] >> <mailto:[email protected]>> wrote: >> >> If you're using the translation map as I've specified and you're >> still seeing the same file (or files) being read twice, then >> there might be a bug. I can't quite tell from what you're saying >> whether or not you've applied the change that I recommended. Can >> you paste your entire Heka config in a pastebin somewhere so I >> can look at it? Or at the very least the entire LogstreamerInput >> section, including the translation map subsection if you've >> applied it. >> >> Thanks, >> >> -r >> >> >> On 03/24/2015 12:07 PM, rajive kumar wrote: >> >> >> ---------- Forwarded message ---------- >> From: *rajive kumar* <[email protected] >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>>> >> Date: 25 March 2015 at 00:35 >> Subject: Re: [heka] Logstreamer Input issue with current log >> and rolling >> file >> To: [email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> >> >> Hi >> >> Thank you.. What you have mention is correct.I have tested >> that too. But >> my problem is when current file rolling Heka is reading all >> the rolled >> file again. For static file its working perfectly. >> >> For example >> >> In server log files line count is : 3015 >> But heka file reading log side is : 23065 >> >> file format which I have used is >> >> [smpp_logs] >> type = "LogstreamerInput" >> log_directory = >> "/media/rajive/Office/tmp/__test/sender/smpp" >> file_match= 'HS_06_SMPP_FULL_CDR(_(?P<Seq>__\d*))?' >> priority =["Seq"] >> [smpp_logs.translation.Seq] >> missing = 9999999999999 >> >> >> >> Please find the attachment for more details. Also I have >> started freshly >> i.e before start i have removed all the files. >> >> >> On 25 March 2015 at 00:08, Rob Miller <[email protected] >> <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>> >> wrote: >> >> You need to use what we call a "translation map" to map >> a missing >> `Seq` value to the highest possible value. The >> following is an >> example of what should work: >> >> [smpp_logs] >> type = "LogstreamerInput" >> log_directory = "/path/to/smpp/logs" >> file_match= 'smpp_cdr(_(?P<Seq>\d*))?' >> priority =["Seq"] >> [smpp_logs.translation.Seq] >> missing = 9999999999999999 >> >> Also be aware that you can test out the order that >> LogstreamerInput >> will load log files before you actually run Heka by >> pointing the >> heka-logstreamer utility at your configuration: >> hekad.readthedocs.org/en/v0.9.____0/pluginconfig/ >> logstreamer.____html#verifying-settings >> <http://hekad.readthedocs.org/en/v0.9.__0/pluginconfig/ >> logstreamer.__html#verifying-settings> >> >> <http://hekad.readthedocs.org/__en/v0.9.0/pluginconfig/__ >> logstreamer.html#verifying-__settings >> >> <http://hekad.readthedocs.org/en/v0.9.0/pluginconfig/ >> logstreamer.html#verifying-settings>> >> >> -r >> >> >> On 03/24/2015 11:26 AM, rajive kumar wrote: >> >> Hi Guys >> >> Any idea on this >> >> I using LogstreamerInput to read my ongoing logs. >> Currently my >> logs has >> been rolling for every 15h min. When using heka its >> reading >> duplicate >> values. I.e its reading same file again and again. >> >> Following is the file pattern >> >> smpp_cdr (This the current log file) >> smpp_cdr_2015032416150055 >> smpp_cdr_2015032416300056 >> smpp_cdr_2015032416450057 >> >> I using following pattern to identify the file >> pattern >> file_match= 'smpp_cdr(_(?P<Seq>\d*))?' >> priority =["Seq"] >> This is read the rolled files perfectly but its >> reading the >> current >> log file (smpp_cdr) first. Also when current file >> gets rolled heka >> reading all the file again. Please advice / help on >> this. >> >> Thank you >> >> On 20 March 2015 at 18:44, rajive kumar >> <[email protected] <mailto:[email protected]> >> <mailto:[email protected] >> <mailto:[email protected]>> >> <mailto:[email protected] >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>>>> wrote: >> >> Hi >> >> Currently I am using Logstreamer input to >> process another >> applications current log file. Its working >> fine but I have >> an issue >> when file rolling -> its reading the file >> twice. Please help on >> this. Below the file structure I have in that >> system >> >> *trans_log* <- current log >> *150320001_trans_log * >> *150320002_trans_log >> * >> *150320003_trans_log >> * >> *150320004_trans_log >> * >> *150320005_trans_log >> * >> *150320006_trans_log* >> >> Note my files are rolling in every 15 mins and >> I want to >> read the >> previous logs and current log while its >> printing(i.e logging). >> >> >> Best Regards, >> >> >> >> >> ___________________________________________________ >> Heka mailing list >> [email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> https://mail.mozilla.org/____listinfo/heka >> <https://mail.mozilla.org/__listinfo/heka> >> <https://mail.mozilla.org/__listinfo/heka >> <https://mail.mozilla.org/listinfo/heka>> >> >> >> >> >> >> >> _________________________________________________ >> Heka mailing list >> [email protected] <mailto:[email protected]> >> https://mail.mozilla.org/__listinfo/heka >> <https://mail.mozilla.org/listinfo/heka> >> >> >> >> >> >> >> _______________________________________________ >> Heka mailing list >> [email protected] >> https://mail.mozilla.org/listinfo/heka >> >>
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
