How to resolve NiFi controller service conflicts across different processor groups?

[Chandrashekhar Kotekar]

Hi All,

We have developed NiFi flow which has one custom processor (lets call it
CP) which uses custom controller service (lets call it CCS) which in turn
uses NiFi's KeytabCredentialsService (lets call it KCS).

CP ==> CCS ==> KCS

This flow is in processor group called development. We created template of
this flow and gave it testing team. Testing team has imported this template
in processor group called testing. Testing team has configured KCS with
their own keytab.

When testing team executes CP which is in testing processor group,
they get access
denied error. When we checked the processor, it showed the processor has
used KCS of development processor group but in fact KCS configured is with
testing key tab only. So we disabled KCS in testing processor group and
enabled it again and it worked BUT then CP in development processor group
gave same access denied error.

In short, when processor in one processor group runs fine, processor in
other processor group gets access denied error because it uses other keytab
service. So we have to disable and enable processor group to resolve that
error.

Has anyone faced similar issue?

Regards,
Chandra

Can Nifi load balance flowfiles?

[Darren Govoni]

Hi

Let's say I have a splitjson processor. I want to connect 10 processors to it 
such that it will send one output to one processor in an evenly distributed 
manner.

Can Nifi do this?

Darren

Sent from my Verizon, Samsung Galaxy smartphone

Re: Can Nifi load balance flowfiles?

[Joe Witt]

Take a look at DistributeLoad.

thanks

On Fri, Apr 24, 2020 at 7:05 AM Darren Govoni  wrote:

> Hi
>
> Let's say I have a splitjson processor. I want to connect 10 processors to
> it such that it will send one output to one processor in an evenly
> distributed manner.
>
> Can Nifi do this?
>
> Darren
>
> Sent from my Verizon, Samsung Galaxy smartphone
>

Re: Can Nifi load balance flowfiles?

[Darren Govoni]

Hi Joe,
   I've tried to use DistributeLoad, but perhaps I am doing something wrong.

I have 2 flowfiles coming into DistributeLoad and I want that processor to 
distribute them evenly across its outbound connections.

I have 2 connections outbound from DistributeLoad. I expected to see 1 flowfile 
in each outbound queue, but instead I see 2 in each, like a normal processor 
would distribute flowfiles.

Is there a special use of this processor to have to evenly distribute the 
received flowfiles across outbound queues?

thanks!!

From: Joe Witt 
Sent: Friday, April 24, 2020 8:08 AM
To: users@nifi.apache.org 
Subject: Re: Can Nifi load balance flowfiles?

Take a look at DistributeLoad.

thanks

On Fri, Apr 24, 2020 at 7:05 AM Darren Govoni 
mailto:dar...@ontrenet.com>> wrote:
Hi

Let's say I have a splitjson processor. I want to connect 10 processors to it 
such that it will send one output to one processor in an evenly distributed 
manner.

Can Nifi do this?

Darren

Sent from my Verizon, Samsung Galaxy smartphone

Re: Can Nifi load balance flowfiles?

[Joe Witt]

Darren,

It isn't quite clear to me what you want to do.  Is the pattern

FlowFile A and B enter DistributeLoad which has relationships 1 and 2.

[CASE1] A goes to 1, B goes to 2.

OR

[CASE2] is is A goes to 1, B goes to 1 and CopyA goes to 2, CopyB goes to 2?

If it is CASE1 you want distribute load and you set the 'number of
relationships' property to 2 and the strategy you want.  Then you set
relationship 1 to some target processor and relationship 2 to some other
processor.

If it is CASE2 you want then you just use whatever source processor you
have before DistributeLoad and you use the same relationship (for example
success) more than once and we'll make flowfile copies for you.

Hopefully that helps.

Thanks

On Fri, Apr 24, 2020 at 2:19 PM Darren Govoni  wrote:

> Hi Joe,
>I've tried to use DistributeLoad, but perhaps I am doing something
> wrong.
>
> I have 2 flowfiles coming into DistributeLoad and I want that processor to
> distribute them evenly across its outbound connections.
>
> I have 2 connections outbound from DistributeLoad. I expected to see 1
> flowfile in each outbound queue, but instead I see 2 in each, like a normal
> processor would distribute flowfiles.
>
> Is there a special use of this processor to have to evenly distribute the
> received flowfiles across outbound queues?
>
> thanks!!
> --
> *From:* Joe Witt 
> *Sent:* Friday, April 24, 2020 8:08 AM
> *To:* users@nifi.apache.org 
> *Subject:* Re: Can Nifi load balance flowfiles?
>
> Take a look at DistributeLoad.
>
> thanks
>
> On Fri, Apr 24, 2020 at 7:05 AM Darren Govoni  wrote:
>
> Hi
>
> Let's say I have a splitjson processor. I want to connect 10 processors to
> it such that it will send one output to one processor in an evenly
> distributed manner.
>
> Can Nifi do this?
>
> Darren
>
> Sent from my Verizon, Samsung Galaxy smartphone
>
>

Re: Can Nifi load balance flowfiles?

[Darren Govoni]

Thanks Joe.

I'm shooting for [CASE1].

The part I missed was setting the number of relationships. I just dragged 
connections from it thinking the strategy applied.

My regrets and thank you for the clarification!

From: Joe Witt 
Sent: Friday, April 24, 2020 2:31 PM
To: users@nifi.apache.org 
Subject: Re: Can Nifi load balance flowfiles?

Darren,

It isn't quite clear to me what you want to do.  Is the pattern

FlowFile A and B enter DistributeLoad which has relationships 1 and 2.

[CASE1] A goes to 1, B goes to 2.

OR

[CASE2] is is A goes to 1, B goes to 1 and CopyA goes to 2, CopyB goes to 2?

If it is CASE1 you want distribute load and you set the 'number of 
relationships' property to 2 and the strategy you want.  Then you set 
relationship 1 to some target processor and relationship 2 to some other 
processor.

If it is CASE2 you want then you just use whatever source processor you have 
before DistributeLoad and you use the same relationship (for example success) 
more than once and we'll make flowfile copies for you.

Hopefully that helps.

Thanks

On Fri, Apr 24, 2020 at 2:19 PM Darren Govoni 
mailto:dar...@ontrenet.com>> wrote:
Hi Joe,
   I've tried to use DistributeLoad, but perhaps I am doing something wrong.

I have 2 flowfiles coming into DistributeLoad and I want that processor to 
distribute them evenly across its outbound connections.

I have 2 connections outbound from DistributeLoad. I expected to see 1 flowfile 
in each outbound queue, but instead I see 2 in each, like a normal processor 
would distribute flowfiles.

Is there a special use of this processor to have to evenly distribute the 
received flowfiles across outbound queues?

thanks!!

From: Joe Witt mailto:joe.w...@gmail.com>>
Sent: Friday, April 24, 2020 8:08 AM
To: users@nifi.apache.org 
mailto:users@nifi.apache.org>>
Subject: Re: Can Nifi load balance flowfiles?

Take a look at DistributeLoad.

thanks

On Fri, Apr 24, 2020 at 7:05 AM Darren Govoni 
mailto:dar...@ontrenet.com>> wrote:
Hi

Let's say I have a splitjson processor. I want to connect 10 processors to it 
such that it will send one output to one processor in an evenly distributed 
manner.

Can Nifi do this?

Darren

Sent from my Verizon, Samsung Galaxy smartphone

MergeRecord performance

[Robert R. Bruno]

I wanted to see if anyone else has experienced performance issues with the
newest version of nifi and MergeRecord?  We have been running on nifi 1.9.2
for awhile now, and recently upgraded to nifi 1.11.4.  Once upgraded, our
identical flows were no longer able to keep up with our data mainly at
MergeRecord processors.

We ended up downgrading back to nifi 1.9.2.  Once we downgraded, all was
keeping up again.  There were no errors to speak of when we were running
the flow with 1.11.4.  We did see higher load on the OS, but this may have
been caused by the fact there was such a tremendous backlog built up in the
flow.

Another side note, we saw one UpdateRecord processor producing errors when
I tested the flow with nifi 1.11.4 with a small test flow.  I was able to
fix this issue by changing some parameters in my RecordWriter.  So perhaps
some underlying ways records are being handled since 1.9.2 caused the
performance issue we saw?

Any insight anyone has would be greatly appreciated, as we very much would
like to upgrade to nifi 1.11.4.  One thought was switching the MergeRecord
processors to MergeContent since I've been told MergeContent seems to
perform better, but not sure if this is actually true.  We are using the
pattern of chaining a few MergeRecord processors together to help with
performance.

Thanks in advance!

Re: MergeRecord performance

[Joe Witt]

Robert,

Can you please detail the record readers and writers involved and how
schemas are accessed?  There can be very important performance related
changes in the parsers/serializers of the given formats.  And we've added a
lot to make schema caching really capable but you have to opt into it.  It
is of course possible MergeRecord itself is the culprit for performance
reduction but lets get a more full picture here.

Are you able to share a template and sample data which we can use to
replicate?

Thanks

On Fri, Apr 24, 2020 at 4:38 PM Robert R. Bruno  wrote:

> I wanted to see if anyone else has experienced performance issues with the
> newest version of nifi and MergeRecord?  We have been running on nifi 1.9.2
> for awhile now, and recently upgraded to nifi 1.11.4.  Once upgraded, our
> identical flows were no longer able to keep up with our data mainly at
> MergeRecord processors.
>
> We ended up downgrading back to nifi 1.9.2.  Once we downgraded, all was
> keeping up again.  There were no errors to speak of when we were running
> the flow with 1.11.4.  We did see higher load on the OS, but this may have
> been caused by the fact there was such a tremendous backlog built up in the
> flow.
>
> Another side note, we saw one UpdateRecord processor producing errors when
> I tested the flow with nifi 1.11.4 with a small test flow.  I was able to
> fix this issue by changing some parameters in my RecordWriter.  So perhaps
> some underlying ways records are being handled since 1.9.2 caused the
> performance issue we saw?
>
> Any insight anyone has would be greatly appreciated, as we very much would
> like to upgrade to nifi 1.11.4.  One thought was switching the MergeRecord
> processors to MergeContent since I've been told MergeContent seems to
> perform better, but not sure if this is actually true.  We are using the
> pattern of chaining a few MergeRecord processors together to help with
> performance.
>
> Thanks in advance!
>

Re: MergeRecord performance

[Robert R. Bruno]

Joe,

In that part of the flow, we are using avro readers and writers.  We are
using snappy compression (which could be part of the problem).  Since we
are using avro at that point the embedded schema is being used by the
reader and the writer is using the schema name property along with an
internal schema registry in nifi.

I can see what could potentially be shared.

Thanks

On Fri, Apr 24, 2020 at 4:41 PM Joe Witt  wrote:

> Robert,
>
> Can you please detail the record readers and writers involved and how
> schemas are accessed?  There can be very important performance related
> changes in the parsers/serializers of the given formats.  And we've added a
> lot to make schema caching really capable but you have to opt into it.  It
> is of course possible MergeRecord itself is the culprit for performance
> reduction but lets get a more full picture here.
>
> Are you able to share a template and sample data which we can use to
> replicate?
>
> Thanks
>
> On Fri, Apr 24, 2020 at 4:38 PM Robert R. Bruno  wrote:
>
>> I wanted to see if anyone else has experienced performance issues with
>> the newest version of nifi and MergeRecord?  We have been running on nifi
>> 1.9.2 for awhile now, and recently upgraded to nifi 1.11.4.  Once upgraded,
>> our identical flows were no longer able to keep up with our data mainly at
>> MergeRecord processors.
>>
>> We ended up downgrading back to nifi 1.9.2.  Once we downgraded, all was
>> keeping up again.  There were no errors to speak of when we were running
>> the flow with 1.11.4.  We did see higher load on the OS, but this may have
>> been caused by the fact there was such a tremendous backlog built up in the
>> flow.
>>
>> Another side note, we saw one UpdateRecord processor producing errors
>> when I tested the flow with nifi 1.11.4 with a small test flow.  I was able
>> to fix this issue by changing some parameters in my RecordWriter.  So
>> perhaps some underlying ways records are being handled since 1.9.2 caused
>> the performance issue we saw?
>>
>> Any insight anyone has would be greatly appreciated, as we very much
>> would like to upgrade to nifi 1.11.4.  One thought was switching the
>> MergeRecord processors to MergeContent since I've been told MergeContent
>> seems to perform better, but not sure if this is actually true.  We are
>> using the pattern of chaining a few MergeRecord processors together to help
>> with performance.
>>
>> Thanks in advance!
>>
>