Hi, Chukwa is a generic framework that some people have adopt to use this framework for data collection and data analysis. Chukwa's strength is to prepare data into map/reduce friendly format and size. The current implementation is good for historical trend analysis, but it is not optimized for log search. Chukwa shines when there are thousands of machines generating logs rapidly. There are 3 tasks to get Chukwa to process new type of log data.
1. Specify which adaptor to stream the log file. 2. Define a demux parser for the new log file. 3. Write some data analytics script in mapreduce or pig script (compute aggregates). Hope this helps. Regards, Eric On 3/6/11 11:29 PM, "msacks" <[email protected]> wrote: Hello, I am considering using Chukwa for an open source method for processing and analyzing application and sys logs (top problems, historical error analysis). Would Chukwa be a good solution for this purpose, or is it overkill? Thanks, msacks
