Yes, there is this self loop, but it's not always a 1-to-1 mapping. Since you use async producers, each N messages corresponds to a Kafka produce request. So, if you encounter an error while processing the request, you add one more message (for every N messages). What you have to avoid is per message level error (e.g., serialization) since that will add one more message for each incoming message.
Thanks, Jun On Thu, Aug 1, 2013 at 3:41 AM, David Black <david.bl...@workday.com> wrote: > Hi, > > > I have the following scenario: > > - using the Kafka log4j appender (in fact its a variant on the OOTB > appender) > > - configuring the appender for the root log4j category > > - appender is using the async producer > > I am wondering about what happens when e.g. the producer cannot connect to > any broker and it logs error messages, which (I’m guessing) get passed by > log4j back into the (kafka) appender. I think there is a possibility of a > kind of infinite cycle, where any log message emitted by the kafka producer > framework may result in more messages being emitted by the kafka producer > framework, due to being pushed back through the appender and so on. > > To an extent I can control this with log4j.additivity e.g. by having a > separate category for the kafka root package (direct its output to my other > appenders but not the kafka appender), and setting additivity for kafka > root package to false. But I think messages from other packages (from other > libraries) that the kafka producer uses may still cause this cycling. Would > I have to identify all those packages and also exclude them? I think this > would not work because other parts of my process use some of those same > jars and I need their output to go to kafka via the kafka log4j appender. > > I’m wondering has anyone else encountered / considered / dealt with this > and if so how? > > > Any help much appreciated > > > David Black > > Workday Inc. >
