Thanks for the reply Joel. Please see my answer below On Tue, Oct 4, 2011 at 1:00 AM, Joel Koshy <[email protected]> wrote:
> > I am totally new to Kafka and we are evaluating Kafka like Felix. I'm > > looking for a utility to pipe data from a file (or multiple files) to a > > producer (like command tail -f in linux). This is an our typical > use-case. I > > saw the patch (KAFKA-130) you provided but not sure if this will work > well > > without loosing data if the file (or multiple files) will be rotated. > > Would "tail -F --retry" work for you? Also, if these are log files and > you are using log4j, then kafka has a log4j appender that you can use > to produce messages. > > I believe tail would work but will not be convenient when people have windows box (not always able to use similar utility for windows). Also, log files are produced from variety of services including non-java applications > > > > I would care about plugin that will such as filter message at producer; > > modify message body in order to mark some critical kind of information, > etc. > > It would be good if you provide documentation about how to write a plugin > > for Kafka. > > You can use async producer callback handlers to do this sort of > filtering. I don't see an example in the distribution, but it is > pretty straightforward: see > > https://svn.apache.org/repos/asf/incubator/kafka/trunk/core/src/main/scala/kafka/javaapi/producer/async/CallbackHandler.java > (The kafka log4j producer currently uses the syncproducer, so you > won't be able to use callback handlers with it). > > great ! I will look at this. BTW, do you know is there any documentation like user guide or something similar? Quickstart from Kafka does not help much. > Thanks, > > Joel > > Thanks, ~Thai > > > > On Sun, Oct 2, 2011 at 9:51 PM, Jay Kreps <[email protected]> wrote: > > > >> Hi, > >> > >> Currently all the patch does is take input from stdin. Can you help me > >> understand what you are looking for? > >> > >> What kind of plugin are you trying to write? > >> > >> -Jay > >> > >> > >> On Thu, Sep 29, 2011 at 10:41 PM, Bao Thai Ngo <[email protected] > >wrote: > >> > >>> Jay, > >>> > >>> Does the patch already support log rotation (file rolling)? > >>> > >>> Also, I wonder if there is any documentation about how to write a > plug-in > >>> for Kafka. > >>> > >>> Thanks, > >>> ~Thai > >>> > >>> On Mon, Sep 12, 2011 at 10:17 AM, Jay Kreps <[email protected]> > wrote: > >>> > >>> > I added a patch for this: > >>> > https://issues.apache.org/jira/browse/KAFKA-130 > >>> > > >>> > Felix, if you are still interested in this, let me know if it works > for > >>> > you. > >>> > > >>> > -Jay > >>> > > >>> > On Thu, Sep 1, 2011 at 1:22 PM, Felix Giguere Villegas < > >>> > [email protected]> wrote: > >>> > > >>> > > Hi :) > >>> > > > >>> > > Thanks for the reply :) . I have created > >>> > > KAFKA-130<https://issues.apache.org/jira/browse/KAFKA-130> > >>> > > . > >>> > > > >>> > > I'm not sure I have respected the proper standard for JIRA issues. > >>> Please > >>> > > let me know if I haven't. > >>> > > > >>> > > -- > >>> > > Felix >
