Ok thank you Rob.
can i read and print the log message somewhere while running heka? i
mean full mesage(With path, hostname and other fields like that)
Eg:
(Hash message printed by logstash)
My log file: /var/log/nginx/access.log
File content : TEST Log message
Logstash log message:
{"@source":"file://ip-10-142-86-22//var/log/nginx/access.log","@tags":[],"@fields":{"path-pre":["//var/log/"],"source_file_path":["nginx"],"path-post":["/access.log"]},"@timestamp":"2013-09-20T12:31:54.808Z","@source_host":"ip-10-142-86-22","@source_path":"//var/log/nginx/access.log","@message":"TEST
Log message \"-\" 400 0 \"-\" \"-\"","@type":"access-log"}
On Thursday 26 June 2014 10:59 PM, Rob Miller wrote:
Ah, thanks, now I understand.
Unfortunately Heka can't do what you want just yet. The AMQPOutput
only supports static values for the exchange name, one per output. You
can set up multiple AMQPOutputs such that each one only captures the
messages of a particular folder, each sending to a different exchange,
but even then it wouldn't automatically handle new directories. The
LogstreamerInput would find the new directories and start loading them
just fine, but you'd need to add the new AMQPOutput configuration
yourself.
Looking at the code it doesn't seem like it would be terribly
difficult to add support for dynamically declaring exchanges based on
values from the message, w/ a few caveats:
- All of the exchanges would need to have the same type, durability,
and auto-delete settings. Supporting variance here is technically
possible, but adds enough complexity that I'd want to hold off on it
until the basic functionality was worn in.
- If declaration of a new exchange were to fail for any reason you'd
risk losing the entire AMQP channel. The output could restart to
reestablish the connection, but if it were to happen repeatedly this
would cause a lot of bouncing, which would probably slow down the
message processing enough to put back-pressure on the whole Heka
pipeline. Some defensive coding (checking for invalid exchange name
characters, etc.) could reduce risk here, but it couldn't be elimintated.
- There would be a performance impact. How much I don't know, but for
each message we'd need to dynamically compute the exchange name that
should be used, and check to see if the exchange had already been
declared. Happily, we could do it in such a way that you only pay the
performance price if you make use of dynamic exchange names.
If you're still interested, please open a feature request issue in the
github tracker at https://github.com/mozilla-services/heka/issues and
we'll get to it when we can. Thanks!
-r
On Wed 25 Jun 2014 11:41:55 PM PDT, Thomas Alrin wrote:
Hi Rob,
Thank you for liking to help me.
My log files are created dynamically, so that my log directory is
/usr/local/share/megamherk/logs and my file_match will be
DYNAMIC_DIRECTORY_NAME/access.log. I want to output the logs in rabbitmq
exchange with exchange_name as DYNAMIC_DIRECTORY_NAME.
Note : There will be more DYNAMIC_DIRECTORY_NAME created.
Example : If my LogstreamerInput file is
/usr/share/megamherk/logs/TOM/access.log, my AMQPOutput exchange
should be
TOM.
Please help me.
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
--
With Warm Regards,
Thomas Alrin,
Infrastructure Engineer; Megam Systems;
email: [email protected]; cell: +91 9789345999
twitter : @thomasalrin
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
