The easiest way to see what you're actually getting is to set up a
LogOutput or a FileOutput with an RstEncoder. For example, the following
stanzas, added to either Heka config, will send to stdout you the entire
contents of every single message that hits the router for that Heka
instance:
[RstEncoder]
[LogOutput]
message_matcher = "TRUE"
encoder = "RstEncoder"
If you replaced the LogOutput w/ a FileOutput then the message contents
would be written to a file instead of to stdout.
That should help you with debugging in general. Looking at your config,
below, I see the following notes:
* You should be able to combine the two LogstreamerInput stanzas into a
single one, specifying a differentiator to distinguish btn the `gardens`
and `wordpress` streams. More information is available in the full
Logstreamer docs (https://is.gd/5vhFx5). To do this correctly, you'd
need to design a regular expression that accurately matches on and
captures either "garden" or "wordpress" from the file name, then use
that capture group name in your differentiator setting. This is a useful
tool for testing out regular expressions: http://regoio.herokuapp.com/
* Whether you combine them into a single LogstreamerInput or not, you
should see using the RstEncoder trick above that the Logger value is
different for each separate set of log files. Once you've ascertained
what that value is (currently I think it will be the name of the
LogstreamerInput), then you can add that to your matcher.
For instance, I *think* this will work for your matchers to separate the
streams:
message_matcher = "Logger == 'gardens_wellington'"
and
message_matcher = "Logger == 'wordpress_wellington'"
Good luck!
-r
On 05/23/2016 03:47 PM, Watson, Chris (DPIPWE) wrote:
Hi Robert
Point 1 is exactly what I am after but to be honest I have no idea if I am
using LogstreamerInput correctly or not, I am fairly new to Unix as a whole and
even newer yet t any form of coding.
My current issue is how to figure out what each of these unique 'Logger' values
are (or who to set them) so I can flag each of these inputs as different
streams, I work better on working examples but currently I am struggling to
find much information on Heka outside of the heka readme documentation and this
email system.
Here is what I currently have in its entirely, it is not in a working state and
it's because of the message matcher line causing a syntax error, I am hoping
its ok to just show you my current setup in this way.
(This setup is really just what I have made to work from multiple attempts and
retries, so it will be slighty messy I am sorry about that)
---------------------------------------------------------------------
On the client server (sending logs to the central server):
---------------------------------------------------------------------
[hekad]
maxprocs = 2
[TcpInput]
address = "127.0.0.1:5565"
[StatsdInput]
[StatAccumInput]
ticker_interval = 1000
emit_in_fields = true
[gardens_wellington]
type = "LogstreamerInput"
log_directory = "/var/log/apache2"
file_match = 'gardens(?P<FileName>[^/]+).log'
[wordpress_wellington]
type = "LogstreamerInput"
log_directory = "/var/log/apache2"
file_match = 'wordpress(?P<FileName>[^/]+).log'
[PayloadEncoder]
append_newlines = false
[LogOutput]
message_matcher = "TRUE"
encoder = "PayloadEncoder"
[aggregator_output]
type = "TcpOutput"
address = "CentralServer-IP:5565"
local_address = "LocalServer-IP:5565"
message_matcher = "TRUE"
---------------------------------------------------------------------
On the Central Server:
---------------------------------------------------------------------
[hekad]
maxprocs = 2
[LogstreamerInput]
log_directory = "/var/log"
file_match = 'auth\.log'
[StatsdInput]
[StatAccumInput]
ticker_interval = 1000
emit_in_fields = true
[TcpInput]
address = ":5565"
[PayloadEncoder]
append_newlines = false
[LogOutput]
message_matcher = "TRUE"
encoder = "PayloadEncoder"
[gardens_wellington]
type = "FileOutput"
#message_matcher = "Hostname == 'wellington' || Hostname == '147.109.11.247'"
message_matcher = "(Hostname == 'wellington' || Hostname == '147.109.11.247') &&
Logger = 'TRUE'"
path = "/logs/production/apache2/wellington-gardens-%Y%m%d-apache2.log"
perm = "666"
encoder = "PayloadEncoder"
rotation_interval = 24
[wordpress_wellington]
type = "FileOutput"
#message_matcher = "Hostname == 'wellington' || Hostname == '147.109.11.247'"
message_matcher = "(Hostname == 'wellington' || Hostname == '147.109.11.247') &&
Logger = 'TRUE'"
path = "/logs/production/apache2/wellington-wordpress-%Y%m%d-apache2.log"
perm = "666"
encoder = "PayloadEncoder"
rotation_interval = 24
------------------------------------------------------------------------------------------------------------------------
The hashed out section in the above configuration (on the central server) is
the filter I used before I attempted to split the output into multiple files.
As I said above I know this configuration is wrong and it isn't working, but I
have no idea what to set the values of Logger and how to make the FileOutput
tell the difference between the two different outputs.
Thank you again for any help on this, I'm sure this will end up being something
stupid and easy but as I mentioned above my experience with this sort of work
is not great.
-----Original Message-----
From: Rob Miller [mailto:rmil...@mozilla.com]
Sent: Tuesday, 24 May 2016 3:35 AM
To: Watson, Chris (DPIPWE)
Cc: Heka@mozilla.org
Subject: Re: [heka] Heka "TCPOutput" to remote server to 'FileOutput' matching
question
At this point you have two options:
1. Create a different LogOutput on the central server for each output file that
you want to generate, and give each of them a message_matcher such that they
only catch the entries that are specific to the file(s) they should be
catching. LogstreamerInput will set the `Logger` value uniquely for each
separate input plugin (or for each separate Logstream, if a single input plugin
is generating multiple streams) so matching on the Logger value is probably
what you'd want. Note that this assumes that you're using LogstreamerInput
correctly in that you're not interleaving unrelated log files into a single
stream.
2. Alternatively, you could use a SandboxOutput, which would let you write a
bit of Lua code to do whatever you want. This would give you a lot more
flexibility. You'd only need a single output, which would look at each message
coming in and make a decision re: where to write it out to, even generating new
files on the fly if new message types start coming in.
Hope this helps,
-r
On 05/22/2016 07:45 PM, Watson, Chris (DPIPWE) wrote:
HI all
I have been using heka as a proof of concept on remote central logging
at my office, everything is working great so far, but I am a little
confused on the next step.
Currently I have been using TCPOutput to write to a remote server any
and all logs in directories to a single file, for each server we have EG:
On output box:
[Accesslogs_apache]
type = "LogstreamerInput"
log_directory = "/var/log/apache2"
file_match = '(?P<FileName>[^/]+).log'
[LogOutput]
message_matcher = "TRUE"
encoder = "PayloadEncoder"
[server1_output]
type = "TcpOutput"
address = "central-server-IP:5565"
local_address = "Local-IP:5565"
message_matcher = "TRUE"
Then on central-server:
[apache_server1]
type = "FileOutput"
message_matcher = "Hostname == 'server1' || Hostname == 'server1-IP'"
path = "/logs/production/apache2/server1-%Y%m%d.log"
perm = "666"
encoder = "PayloadEncoder"
rotation_interval = 24
that all works fine.
My issue is I now need to produce multiple log files from the same
server and have them output to different files as dumping multiple
logs from server1 into one large file on the central server is no
longer practical, but I am having a really hard time getting this working EG:
Server1 has two different logs files: ‘foo.log’ and ‘bar.log’
I want both these logs to be passed to central-server and both of them
to create different FileOutput’s to separate outputs so they appear
on the Central server:
/logs/production/apache2/foo.log
/logs/production/apache2/bar.log
Currently they all being dumped to a single file
"/logs/production/apache2/server1-%Y%m%d.log"
Where do I need to make the definition that these are two separate
files, on the central server or on server1.
----------------------------------------------------------------------
--
CONFIDENTIALITY NOTICE AND DISCLAIMER
The information in this transmission may be confidential and/or
protected by legal professional privilege, and is intended only for
the person or persons to whom it is addressed. If you are not such a
person, you are warned that any disclosure, copying or dissemination
of the information is unauthorised. If you have received the
transmission in error, please immediately contact this office by
telephone, fax or email, to inform us of the error and to enable
arrangements to be made for the destruction of the transmission, or
its return at our cost. No liability is accepted for any unauthorised
use of the information contained in this transmission.
_______________________________________________
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka
________________________________
CONFIDENTIALITY NOTICE AND DISCLAIMER
The information in this transmission may be confidential and/or protected by
legal professional privilege, and is intended only for the person or persons to
whom it is addressed. If you are not such a person, you are warned that any
disclosure, copying or dissemination of the information is unauthorised. If you
have received the transmission in error, please immediately contact this office
by telephone, fax or email, to inform us of the error and to enable
arrangements to be made for the destruction of the transmission, or its return
at our cost. No liability is accepted for any unauthorised use of the
information contained in this transmission.
_______________________________________________
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka
