Each queue has back-pressure settings on it which default to 10k flow
files or 1GB size of flow files. When one of these thresholds is
exceeded, the preceeding processor will not execute until the queue
goes back below the threshold.
Most likely if GenerateFlowFile has a Run Schedule of 0
Actually, it's not Kafka topic but Nifi queue between "generateFlow" and
"PutKafka" gets overflown
On Thu, Apr 4, 2019 at 10:58 AM Joe Witt wrote:
> Can you share screenshots, logs, and a more detailed description of what
> you're doing, observing with nifi and the system and what you expect it
We need to define what "NiFi stops responding" means...
Are there tons of flow files queued up before before PublishKafka?
Are there back-pressure indicators on any of the queues?
Do the kafka related processors show active threads in the top right
corners of the processors?
Does NiFI crash?
On
What's the concurrency for these processors? What's a global NiFi thread
pool size?
I wonder if you might be running out of available threads while they are
waiting for external system i/o under load.
Andrew
On Thu, Apr 4, 2019, 8:24 AM l vic wrote:
> What's this particular processing group
What's this particular processing group does: writes large dataset to Kafka
topic, one consumer reads from topic and saves data to Hbase/PQS table,
another consumer writes to ES index
On Thu, Apr 4, 2019 at 10:58 AM Joe Witt wrote:
> Can you share screenshots, logs, and a more detailed
Can you share screenshots, logs, and a more detailed description of what
you're doing, observing with nifi and the system and what you expect it to
be doing.
Thanks
On Thu, Apr 4, 2019 at 10:56 AM l vic wrote:
> No, actually what happens is - NiFi stops responding ( if I use it without
> rate
No, actually what happens is - NiFi stops responding ( if I use it without
rate contol)
On Thu, Apr 4, 2019 at 10:42 AM Joe Witt wrote:
> Hello
>
> There isn't really a feedback mechanism based on load on the Kafka topic.
> When you say overrunning the topic do you mean that you don't want
Hello
There isn't really a feedback mechanism based on load on the Kafka topic.
When you say overrunning the topic do you mean that you don't want there to
be a large lag between consumers and their current offset and if that grows
you want NiFi to slow down? I dont believe there is anything
I have to ingest large (200,000 messages) data set into Kafka topic as
quickly as possible without overrunning topic... Right now I just use rate
limiter to do it but can be there some better "adaptive" way to do it?
Thank you...
-V
Hello,
Take a look at the DBCP lookup service, it allows you to register one or
more connection pool services and then select one at runtime based on an
incoming flow file having an attribute called database.name.
Thanks,
Bryan
On Thu, Apr 4, 2019 at 8:47 AM Max wrote:
> Hello!
>
> We are
Hello!
We are working on a project that requires importing data from tables across
different database servers (as in, different db connection pools.)
The data flow itself is the same across maybe 40-50 tables and around 10
connections. I tried to create an abstract flow that can be parameterized
Hey fellow NiFi fans,
I was recently loading data into into Solr via HBase (around 700G ~60,000,000
db rows) using NiFi and noticed some inconsistent behaviour with the GetHbase
processor and I'm wondering if anyone else has noticed similar behaviour when
using it.
Here's our environment and
It looks like it will do this if you don’t grant the host access to /buckets
which is a valid resource.
Sent from my iPhone
> On Apr 4, 2019, at 1:45 AM, Koji Kawamura wrote:
>
> Hi Shawn,
>
> The 'No applicable policies could be found.' message can be logged
> when a request is made
Hi Shawn,
The 'No applicable policies could be found.' message can be logged
when a request is made against a resource which doesn't exist.
14 matches
Mail list logo
