Hi Faisal, Some observations and next steps. 1. Brian has a great point of using the more appropriate consumer that will leverage the latest API. 2. This still certainly feels like a resource issue -- testing a single Kafka broker first and not growing the cluster seems odd especially in light of using a NiFi cluster. Often these are reversed -- multiple Kafka brokers and a single NiFi node. 3. When you have a concurrency of one, which you are correct -- the right place to start with NiFi, then internally NiFi is constructing a Kafka Consumer pool with 1 thread, which again is fine. It is then asking the Kafka consumer to subscribe to 99 topics and poll on each for changes, the polling is done with a timeout of 10ms so that resources are not tied up -- this is pretty common for Kafka consumption outside of NiFi.
Can we try something to see if we can isolate this a little further? Can you (ideally on a separate node) use the Kafka Console consumer shell script to consume from the 99 topics? If this runs for some period of time without error then we can focus on the NiFi side of things. If you are not sure of the install directory for Kafka, you can find this in Ambari. To use these tools on a different node you would want to take everything in the Kafka bin and lib directories, as the shell scripts will have dependencies. Thanks, Andrew On Wed, Jun 20, 2018 at 10:32 PM Bryan Bende <bbe...@gmail.com> wrote: > I’m not sure if this will help, but you mentioned you are using Kafka > (1.0.0.3.1.0). > > NiFi has processors that correspond to the Kafka version, and generally it > is best to use the client version that matches the broker version. > > So you should be using ConsumeKafka_1_0 which uses the Kafka 1.0.0 client > to go with the 1.0.0 broker. > > The ConsumeKafka processor uses Kafka client 0.9.0. > > > On Jun 20, 2018, at 9:53 PM, Faisal Durrani <te04.0...@gmail.com> wrote: > > Kafka (1.0.0.3.1.0) > > >
