We need to define what "NiFi stops responding" means... Are there tons of flow files queued up before before PublishKafka? Are there back-pressure indicators on any of the queues? Do the kafka related processors show active threads in the top right corners of the processors? Does NiFI crash?
On Thu, Apr 4, 2019 at 11:28 AM Andrew Grande <apere...@gmail.com> wrote: > > What's the concurrency for these processors? What's a global NiFi thread pool > size? > > I wonder if you might be running out of available threads while they are > waiting for external system i/o under load. > > Andrew > > On Thu, Apr 4, 2019, 8:24 AM l vic <lvic4...@gmail.com> wrote: >> >> What's this particular processing group does: writes large dataset to Kafka >> topic, one consumer reads from topic and saves data to Hbase/PQS table, >> another consumer writes to ES index >> >> On Thu, Apr 4, 2019 at 10:58 AM Joe Witt <joe.w...@gmail.com> wrote: >>> >>> Can you share screenshots, logs, and a more detailed description of what >>> you're doing, observing with nifi and the system and what you expect it to >>> be doing. >>> >>> Thanks >>> >>> On Thu, Apr 4, 2019 at 10:56 AM l vic <lvic4...@gmail.com> wrote: >>>> >>>> No, actually what happens is - NiFi stops responding ( if I use it without >>>> rate contol).... >>>> >>>> >>>> On Thu, Apr 4, 2019 at 10:42 AM Joe Witt <joe.w...@gmail.com> wrote: >>>>> >>>>> Hello >>>>> >>>>> There isn't really a feedback mechanism based on load on the Kafka topic. >>>>> When you say overrunning the topic do you mean that you don't want there >>>>> to be a large lag between consumers and their current offset and if that >>>>> grows you want NiFi to slow down? I dont believe there is anything >>>>> inherent to the kafka producer protocol that would inform of us this. We >>>>> could periodically poll for this information and optional back-off. >>>>> >>>>> Is this what you have in mind? >>>>> >>>>> Thanks >>>>> >>>>> On Thu, Apr 4, 2019 at 10:34 AM l vic <lvic4...@gmail.com> wrote: >>>>>> >>>>>> I have to ingest large (200,000 messages) data set into Kafka topic as >>>>>> quickly as possible without overrunning topic... Right now I just use >>>>>> rate limiter to do it but can be there some better "adaptive" way to do >>>>>> it? >>>>>> Thank you... >>>>>> -V
