Hi Guys Any idea on this..
Best Regards, On 25 March 2015 at 11:29, rajive kumar <[email protected]> wrote: > Hi > > Please find the link for the toml file and sh script which i have used to > test. > > http://pastebin.ca/2963610 - toml file > > http://pastebin.ca/2963598 - sh script for simulate the log rotation. > > Steps I have followed > > 1. Run the script for some time by increasing the no of records > 2. Start the heka while script is running. > 3. After some time get the line count of generated files and heka > output files ( Heka out put file have more than generated file count i.e > its containing duplicate values ) > > Thank you in advance for the help. > > Best Regards, > > > On 25 March 2015 at 05:40, Rob Miller <[email protected]> wrote: > >> If you're using the translation map as I've specified and you're still >> seeing the same file (or files) being read twice, then there might be a >> bug. I can't quite tell from what you're saying whether or not you've >> applied the change that I recommended. Can you paste your entire Heka >> config in a pastebin somewhere so I can look at it? Or at the very least >> the entire LogstreamerInput section, including the translation map >> subsection if you've applied it. >> >> Thanks, >> >> -r >> >> >> On 03/24/2015 12:07 PM, rajive kumar wrote: >> >>> >>> ---------- Forwarded message ---------- >>> From: *rajive kumar* <[email protected] <mailto:[email protected]>> >>> Date: 25 March 2015 at 00:35 >>> Subject: Re: [heka] Logstreamer Input issue with current log and rolling >>> file >>> To: [email protected] <mailto:[email protected]> >>> >>> >>> Hi >>> >>> Thank you.. What you have mention is correct.I have tested that too. But >>> my problem is when current file rolling Heka is reading all the rolled >>> file again. For static file its working perfectly. >>> >>> For example >>> >>> In server log files line count is : 3015 >>> But heka file reading log side is : 23065 >>> >>> file format which I have used is >>> >>> [smpp_logs] >>> type = "LogstreamerInput" >>> log_directory = "/media/rajive/Office/tmp/test/sender/smpp" >>> file_match= 'HS_06_SMPP_FULL_CDR(_(?P<Seq>\d*))?' >>> priority =["Seq"] >>> [smpp_logs.translation.Seq] >>> missing = 9999999999999 >>> >>> >>> >>> Please find the attachment for more details. Also I have started freshly >>> i.e before start i have removed all the files. >>> >>> >>> On 25 March 2015 at 00:08, Rob Miller <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> You need to use what we call a "translation map" to map a missing >>> `Seq` value to the highest possible value. The following is an >>> example of what should work: >>> >>> [smpp_logs] >>> type = "LogstreamerInput" >>> log_directory = "/path/to/smpp/logs" >>> file_match= 'smpp_cdr(_(?P<Seq>\d*))?' >>> priority =["Seq"] >>> [smpp_logs.translation.Seq] >>> missing = 9999999999999999 >>> >>> Also be aware that you can test out the order that LogstreamerInput >>> will load log files before you actually run Heka by pointing the >>> heka-logstreamer utility at your configuration: >>> hekad.readthedocs.org/en/v0.9.__0/pluginconfig/logstreamer._ >>> _html#verifying-settings >>> <http://hekad.readthedocs.org/en/v0.9.0/pluginconfig/ >>> logstreamer.html#verifying-settings> >>> >>> -r >>> >>> >>> On 03/24/2015 11:26 AM, rajive kumar wrote: >>> >>> Hi Guys >>> >>> Any idea on this >>> >>> I using LogstreamerInput to read my ongoing logs. Currently my >>> logs has >>> been rolling for every 15h min. When using heka its reading >>> duplicate >>> values. I.e its reading same file again and again. >>> >>> Following is the file pattern >>> >>> smpp_cdr (This the current log file) >>> smpp_cdr_2015032416150055 >>> smpp_cdr_2015032416300056 >>> smpp_cdr_2015032416450057 >>> >>> I using following pattern to identify the file pattern >>> file_match= 'smpp_cdr(_(?P<Seq>\d*))?' >>> priority =["Seq"] >>> This is read the rolled files perfectly but its reading the >>> current >>> log file (smpp_cdr) first. Also when current file gets rolled >>> heka >>> reading all the file again. Please advice / help on this. >>> >>> Thank you >>> >>> On 20 March 2015 at 18:44, rajive kumar <[email protected] >>> <mailto:[email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>> >>> wrote: >>> >>> Hi >>> >>> Currently I am using Logstreamer input to process another >>> applications current log file. Its working fine but I have >>> an issue >>> when file rolling -> its reading the file twice. Please >>> help on >>> this. Below the file structure I have in that system >>> >>> *trans_log* <- current log >>> *150320001_trans_log * >>> *150320002_trans_log >>> * >>> *150320003_trans_log >>> * >>> *150320004_trans_log >>> * >>> *150320005_trans_log >>> * >>> *150320006_trans_log* >>> >>> Note my files are rolling in every 15 mins and I want to >>> read the >>> previous logs and current log while its printing(i.e >>> logging). >>> >>> >>> Best Regards, >>> >>> >>> >>> >>> _________________________________________________ >>> Heka mailing list >>> [email protected] <mailto:[email protected]> >>> https://mail.mozilla.org/__listinfo/heka >>> <https://mail.mozilla.org/listinfo/heka> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Heka mailing list >>> [email protected] >>> https://mail.mozilla.org/listinfo/heka >>> >>> >> >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
