Re: Round robin load balancing eventually stops using all nodes

[Ryan Hendrickson]

Joe - We're testing some scenarios.  Andrew captured some confusing
behavior in the UI when enabling and disabling load balancing on a
relationship: "Update UI for Clustered Connections" --
https://issues.apache.org/jira/projects/NIFI/issues/NIFI-9236

Question - When a FlowFile is Load Balanced from one node to another, is
the entire Content Claim load balanced?  Or just the small portion
necessary?

Mike -
We found two tickets that are in the ballpark:

1.  Improve handling of Load Balanced Connections when one node is slow
 --https://issues.apache.org/jira/browse/NIFI-7081
2.  NiFi FlowFiles stuck in queue when using Single Node load balance
strategy   --https://issues.apache.org/jira/browse/NIFI-8970

>From @Simon comment - we know we've seen underperforming nodes in a cluster
before.  We're discussing @Simon's comment is applicable to the issue we're
seeing
  > "The one thing I can think of is the scenario where one (or
more) nodes are significantly slower than the other ones. In these cases it
might happen then the nodes are “running behind” blocks the other nodes
from balancing perspective."

@Simon - I'd like to understand the "blocks other nodes from balancing
perspective" better if you have additional information.  We're trying to
replicate this scenario.

Thanks,
Ryan

On Sat, Sep 18, 2021 at 3:45 PM Mike Thomsen  wrote:

> > there is a ticket to overcome this (there is no ETA),
>
> Do you know what the Jira # is?
>
> On Mon, Sep 6, 2021 at 7:14 AM Simon Bence 
> wrote:
> >
> > Hi Mike,
> >
> > I did a quick check on the round robin balancing and based on what I
> found the reason for the issue must lie somewhere else, not directly within
> it. The one thing I can think of is the scenario where one (or more) nodes
> are significantly slower than the other ones. In these cases it might
> happen then the nodes are “running behind” blocks the other nodes from
> balancing perspective.
> >
> > Based on what you wrote this is a possible reason and there is a ticket
> to overcome this (there is no ETA), but other details might shed light to a
> different root cause.
> >
> > Regards,
> > Bence
> >
> >
> >
> > > On 2021. Sep 3., at 14:13, Mike Thomsen 
> wrote:
> > >
> > > We have a 5 node cluster, and sometimes I've noticed that round robin
> > > load balancing stops sending flowfiles to two of them, and sometimes
> > > toward the end of the data processing can get as low as a single node.
> > > Has anyone seen similar behavior?
> > >
> > > Thanks,
> > >
> > > Mike
> >
>

Re: MiNiFi agent cannot update flow configuration

[Matt Burgess]

Tom,

Which implementation of the Provenance Repository are you using? If
not the VolatileProvenanceRepository, can you try that as a
workaround? Also are you using the 1.14.0 version of the C2 server?

Regards,
Matt

On Tue, Sep 21, 2021 at 3:45 PM Tomislav Novosel
 wrote:
>
> Hi to all,
>
>
> I'm using MiNiFi 1.14.0 with configured change ingestor to pull from HTTP C2 
> server
> whenever there is a change in configuration (change in NiFi flow that suppose 
> to be running
> on MiNiFi).
>
> MiNiFi agent is running on Raspberry Pi 3 with enough disk space.
>
>
> When I make a change and save the new template with the name 
> template_name.v2, C2 pulls it,
> saves it into ./cache folder and sends it to MiNiFi agent.
>
>
> Then in MiNiFi agent log I have this error:
>
>
>
> 2021-09-21 12:54:26,456 ERROR [MiNiFi logging handler] 
> org.apache.nifi.minifi.StdErr Failed to start flow service: Unable to load 
> flow due to: java.lang.RuntimeException: Unable to create Provenance 
> Repository
> 2021-09-21 12:54:26,457 ERROR [MiNiFi logging handler] 
> org.apache.nifi.minifi.StdErr Shutting down...
> 2021-09-21 12:54:27,384 INFO [main] o.apache.nifi.minifi.bootstrap.RunMiNiFi 
> Swap file exists, MiNiFi failed trying to change configuration. Reverting to 
> old configuration.
> 2021-09-21 12:54:27,425 INFO [main] o.apache.nifi.minifi.bootstrap.RunMiNiFi 
> Replacing config file with swap file and deleting swap file
> 2021-09-21 12:54:27,444 INFO [main] o.apache.nifi.minifi.bootstrap.RunMiNiFi 
> Successfully spawned the thread to start Apache MiNiFi with PID 64002
> 2021-09-21 12:54:29,384 INFO [MiNiFi Bootstrap Command Listener] 
> o.apache.nifi.minifi.bootstrap.RunMiNiFi The thread to run Apache MiNiFi is 
> now running and listening for Bootstrap requests on port 38889
>
>
>
> It cannot change the configuration flow because it cannot create Provenance 
> Repository, and then
> reverts to old configuration of the flow.
>
> I tried to delete all the files in ./provenance_repository folder, and start 
> it again, but the same happens.
>
>
>
> Does anybody know why is this?
>
>
>
> Thanks in advance,
>
> Regards,
>
> Tom

MiNiFi agent cannot update flow configuration

[Tomislav Novosel]

Hi to all,

I'm using MiNiFi 1.14.0 with configured change ingestor to pull from HTTP C2 
server
whenever there is a change in configuration (change in NiFi flow that suppose 
to be running
on MiNiFi).
MiNiFi agent is running on Raspberry Pi 3 with enough disk space.

When I make a change and save the new template with the name template_name.v2, 
C2 pulls it,
saves it into ./cache folder and sends it to MiNiFi agent.

Then in MiNiFi agent log I have this error:

2021-09-21 12:54:26,456 ERROR [MiNiFi logging handler] 
org.apache.nifi.minifi.StdErr Failed to start flow service: Unable to load flow 
due to: java.lang.RuntimeException: Unable to create Provenance Repository
2021-09-21 12:54:26,457 ERROR [MiNiFi logging handler] 
org.apache.nifi.minifi.StdErr Shutting down...
2021-09-21 12:54:27,384 INFO [main] o.apache.nifi.minifi.bootstrap.RunMiNiFi 
Swap file exists, MiNiFi failed trying to change configuration. Reverting to 
old configuration.
2021-09-21 12:54:27,425 INFO [main] o.apache.nifi.minifi.bootstrap.RunMiNiFi 
Replacing config file with swap file and deleting swap file
2021-09-21 12:54:27,444 INFO [main] o.apache.nifi.minifi.bootstrap.RunMiNiFi 
Successfully spawned the thread to start Apache MiNiFi with PID 64002
2021-09-21 12:54:29,384 INFO [MiNiFi Bootstrap Command Listener] 
o.apache.nifi.minifi.bootstrap.RunMiNiFi The thread to run Apache MiNiFi is now 
running and listening for Bootstrap requests on port 38889

It cannot change the configuration flow because it cannot create Provenance 
Repository, and then
reverts to old configuration of the flow.
I tried to delete all the files in ./provenance_repository folder, and start it 
again, but the same happens.

Does anybody know why is this?

Thanks in advance,
Regards,
Tom

Re: MiNiFi C2 server 0.5.0 bug or not?

[Jeremy Pemberton-Pigott]

Hi Tom,

I think so, I sent a similar email recently to the list about this.  And
raised the issue here: https://issues.apache.org/jira/browse/NIFI-9075

Jeremy

On Tue, Sep 21, 2021 at 8:10 PM Tomislav Novosel <
tomislav.novo...@clearpeaks.com> wrote:

> Hi to all,
>
>
>
> I am using MiNiFi C2 server version 0.5.0 and MiNiFi version 1.14.0.
>
> C2 server is configured to pull templates from localhost NiFi installation
>
> and MiNiFi is configured with configuration change ingestor to pull config
> from C2 server.
>
>
>
> After I created MiNiFi flow on NiFi canvas including Remote Process Group
> pointed to the same
>
> NiFi installation (IP and port configured and the flow tested on
> localhost, flowfiles are coming
>
> on input port) I saved the flow as template.
>
>
>
> The template has the following ID's for the input port:
>
>
>
> 
>
> 
>
> 
>
>
> 1
>
> true
>
> true
>
> f43fd60b-017b-1000-ad38-01d2ac321927
>
> *b4fbf2a8-46d0-3ac7-b833-21533794f7a7*
>
> DataFromSensors
>
>
> *f43dd13c-017b-1000-df9a-acf01964ae4f*
>
>
>
> When C2 server pulls the template, it apparently sets wrong ID in
> config.yml file, instead of putting input port ,
>
> it sets  into config.yml file, causing MiNiFi agent is unable to
> accept that configuration and denies it, it is simply wrong.
>
>
>
> This is the Exception from MiNiFi log:
>
>
>
> Caused by:
> org.apache.nifi.minifi.bootstrap.exception.InvalidConfigurationException:
> Failed to transform config file due to:[Connection with id
> 8f6c9b69-5b31-313e-- has invalid destination id
> b4fbf2a8-46d0-3ac7-b833-21533794f7a7]
>
>at
> org.apache.nifi.minifi.bootstrap.util.ConfigTransformer.throwIfInvalid(ConfigTransformer.java:131)
>
>at
> org.apache.nifi.minifi.bootstrap.util.ConfigTransformer.transformConfigFile(ConfigTransformer.java:94)
>
>at
> org.apache.nifi.minifi.bootstrap.RunMiNiFi.performTransformation(RunMiNiFi.java:1693)
>
>
>
> To fix it, I need to go  to cached config file in ./cache folder of C2
> server, and set the right ID of the input port manually.
>
>
>
> *NOTE: I tried to convert template file manually using minifi tookit
> 1.14.0 and it sets correct  of the input port.*
>
>
>
> Is this a bug in C2 server or behaviour with purpose? I found the same
> case for minifi toolkit, but apparently it was fixed:
>
> https://stackoverflow.com/questions/59214373/failing-in-minifi-tutorial-toolkit-error-connection-with-id-has-invalid-de
>
>
>
> Thanks,
>
> Tom
>

MiNiFi C2 server 0.5.0 bug or not?

[Tomislav Novosel]

Hi to all,

I am using MiNiFi C2 server version 0.5.0 and MiNiFi version 1.14.0.
C2 server is configured to pull templates from localhost NiFi installation
and MiNiFi is configured with configuration change ingestor to pull config from 
C2 server.

After I created MiNiFi flow on NiFi canvas including Remote Process Group 
pointed to the same
NiFi installation (IP and port configured and the flow tested on localhost, 
flowfiles are coming
on input port) I saved the flow as template.

The template has the following ID's for the input port:





1
true
true
f43fd60b-017b-1000-ad38-01d2ac321927
b4fbf2a8-46d0-3ac7-b833-21533794f7a7
DataFromSensors
f43dd13c-017b-1000-df9a-acf01964ae4f

When C2 server pulls the template, it apparently sets wrong ID in config.yml 
file, instead of putting input port ,
it sets  into config.yml file, causing MiNiFi agent is unable to 
accept that configuration and denies it, it is simply wrong.

This is the Exception from MiNiFi log:

Caused by: 
org.apache.nifi.minifi.bootstrap.exception.InvalidConfigurationException: 
Failed to transform config file due to:[Connection with id 
8f6c9b69-5b31-313e-- has invalid destination id 
b4fbf2a8-46d0-3ac7-b833-21533794f7a7]
   at 
org.apache.nifi.minifi.bootstrap.util.ConfigTransformer.throwIfInvalid(ConfigTransformer.java:131)
   at 
org.apache.nifi.minifi.bootstrap.util.ConfigTransformer.transformConfigFile(ConfigTransformer.java:94)
   at 
org.apache.nifi.minifi.bootstrap.RunMiNiFi.performTransformation(RunMiNiFi.java:1693)

To fix it, I need to go  to cached config file in ./cache folder of C2 server, 
and set the right ID of the input port manually.

NOTE: I tried to convert template file manually using minifi tookit 1.14.0 and 
it sets correct  of the input port.

Is this a bug in C2 server or behaviour with purpose? I found the same case for 
minifi toolkit, but apparently it was fixed:
https://stackoverflow.com/questions/59214373/failing-in-minifi-tutorial-toolkit-error-connection-with-id-has-invalid-de

Thanks,
Tom

Re: MS SQL Server JDBC fail

[Jim Halfpenny]

Hi Stefan,
This looks like a JDBC driver issue with TLS, it could be either that TLS is 
not enabled or that there is an error in establishing the session. The link 
below described a very similar issue with the mssql-jdbc driver, you could try 
their suggested workaround of setting sslProtocol=TLSv1 to see is that helps. 

https://github.com/Microsoft/mssql-jdbc/issues/879 


There’s a link to a Microsoft article in this issue that suggests upgrading you 
JDBC driver to version 4.2 or later, I’d recommend trying that. I’d not 
recommend leaving sslProtocol=TLSv1 in as a long term solution because you’re 
effectively reducing the strength of the TLS implementation used by doing so.

Regards,
Jim


> On 21 Sep 2021, at 09:30, Stefan Kok  wrote:
> 
> Hi Everyone
> 
> I am unable to establish a JDBC connection to a MS SQL 2012 server 
> below I have listed all the details relating to this problem.
> 
> Thank you in advance
> 
> Regards
> Stefan
> 
> 
> 
> 
> NIFI:
> 1.14.0
> 
> uname -a:
> 5.13.14-200.fc34.x86_64 #1 SMP Fri Sep 3 15:33:01 UTC 2021 x86_64
> x86_64 x86_64 GNU/Linux
> 
> Java:
> OpenJDK Runtime Environment 18.9 (build 11.0.12+7)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.12+7, mixed mode, sharing)
> 
> JDBC URL:
> jdbc:sqlserver://;serverName=db.x.xxx.xxx;databaseName=master;trust
> ServerCertificate=true;sslProtocol=TLS
> 
> MS SQL Server:
> Ms SQL server 2010
> 
> OS:
> Windows Server 2012 R2
> 
> ERROR:
> 10:11:42 SASTERRORf312e911-017b-1000-7f31-6ce540797406
> GenerateTableFetch[id=f312e911-017b-1000-7f31-6ce540797406] Failed to
> process session due to java.sql.SQLException: Cannot create
> PoolableConnectionFactory (The driver could not establish a secure
> connection to SQL Server by using Secure Sockets Layer (SSL)
> encryption. Error: "Unexpected rethrowing".): java.io.IOException: SQL
> Server did not return a response. The connection has been closed.
> ClientConnectionId:20991be1-62eb-424a-9df0-5868eb642a12
> ↳ causes: javax.net.ssl.SSLException: Unexpected rethrowing
> ↳ causes: com.microsoft.sqlserver.jdbc.SQLServerException: The driver
> could not establish a secure connection to SQL Server by using Secure
> Sockets Layer (SSL) encryption. Error: "Unexpected rethrowing".
> ↳ causes: java.sql.SQLException: Cannot create
> PoolableConnectionFactory (The driver could not establish a secure
> connection to SQL Server by using Secure Sockets Layer (SSL)
> encryption. Error: "Unexpected rethrowing".)
> ↳ causes: org.apache.nifi.processor.exception.ProcessException:
> java.sql.SQLException: Cannot create PoolableConnectionFactory (The
> driver could not establish a secure connection to SQL Server by using
> Secure Sockets Layer (SSL) encryption. Error: "Unexpected rethrowing".)
> 
> 

MS SQL Server JDBC fail

[Stefan Kok]

Hi Everyone

I am unable to establish a JDBC connection to a MS SQL 2012 server 
below I have listed all the details relating to this problem.

Thank you in advance

Regards
Stefan




NIFI:
1.14.0

uname -a:
5.13.14-200.fc34.x86_64 #1 SMP Fri Sep 3 15:33:01 UTC 2021 x86_64
x86_64 x86_64 GNU/Linux

Java:
OpenJDK Runtime Environment 18.9 (build 11.0.12+7)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.12+7, mixed mode, sharing)

JDBC URL:
jdbc:sqlserver://;serverName=db.x.xxx.xxx;databaseName=master;trust
ServerCertificate=true;sslProtocol=TLS

MS SQL Server:
Ms SQL server 2010

OS:
Windows Server 2012 R2

ERROR:
10:11:42 SASTERRORf312e911-017b-1000-7f31-6ce540797406
GenerateTableFetch[id=f312e911-017b-1000-7f31-6ce540797406] Failed to
process session due to java.sql.SQLException: Cannot create
PoolableConnectionFactory (The driver could not establish a secure
connection to SQL Server by using Secure Sockets Layer (SSL)
encryption. Error: "Unexpected rethrowing".): java.io.IOException: SQL
Server did not return a response. The connection has been closed.
ClientConnectionId:20991be1-62eb-424a-9df0-5868eb642a12
↳ causes: javax.net.ssl.SSLException: Unexpected rethrowing
↳ causes: com.microsoft.sqlserver.jdbc.SQLServerException: The driver
could not establish a secure connection to SQL Server by using Secure
Sockets Layer (SSL) encryption. Error: "Unexpected rethrowing".
↳ causes: java.sql.SQLException: Cannot create
PoolableConnectionFactory (The driver could not establish a secure
connection to SQL Server by using Secure Sockets Layer (SSL)
encryption. Error: "Unexpected rethrowing".)
↳ causes: org.apache.nifi.processor.exception.ProcessException:
java.sql.SQLException: Cannot create PoolableConnectionFactory (The
driver could not establish a secure connection to SQL Server by using
Secure Sockets Layer (SSL) encryption. Error: "Unexpected rethrowing".)