Re: Jira Contributor access to username : naddym

[Mohammed Nadeem]

Thank you so much  :)

On 2020/08/31 20:04:42, Mark Payne  wrote: 
> Hi Nadeem,
> 
> I’ve added you as a contributor to NiFi’s Jira. Welcome!
> 
> Thanks
> -Mark
> 
> 
> > On Aug 31, 2020, at 3:56 PM, Mohammed Nadeem  wrote:
> > 
> > Hi Team,
> > 
> > Can you please provide Apache Jira contributor access to username :- 
> > 'naddym' . I'm working on one of the NiFi issue and want to assign it to 
> > self. Thanks in advance.
> > 
> > Regards,
> > Nadeem
> > 
> 
> 

Jira Contributor access to username : naddym

[Mohammed Nadeem]

Hi Team,

Can you please provide Apache Jira contributor access to username :- 'naddym' . 
I'm working on one of the NiFi issue and want to assign it to self. Thanks in 
advance.

Regards,
Nadeem
 

Re: SSLHandshake Exception from Site-to-Site

[Mohammed Nadeem]

Thanks again Koji for replying and understanding my concern,

I did apply the changes you suggested but still i'm getting same
SSLHandshake error. I believe the Site2Site Remote Listener doesn't run a
server socket with the hostname we specify in *'nifi.remote.input.host'* in
nifi.properties instead it uses wildcard ipaddress *0.0.0.0* to bind
serversocket to all network interfaces to listen for incoming request from
site-to-site client, please check this line of code -  link for code line in
socketremotelisterner class

 
. Since the serversocket is listening on all network interfaces and it waits
for the client to accept the connection, it runs a continuous while loop
waiting to accept the connection from client, please check this segment of
code as well 
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/SocketRemoteSiteListener.java#L123-L129

 
. In kubernetes, some anonymous client is able to get through the connection
to serversocket and while performing handshake its throwing the SSLHandshake
error.

This anonymous client has hostname which as ip-10-200-25-3.compute.internal
with randorm port and thats how the socketremotelisterner tries to creates
SSLsocketchannel with this anonymous client and it throws the error.

Please see the attached image i created explaining the root cause for this
error, Please do suggest if I'm correct or wrong.

 


Thanks again,
Nadeem





--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: SSLHandshake Exception from Site-to-Site

[Mohammed Nadeem]

Thank you so much Koji for replying, 

This issue of SSL Handshake we see is for a single node cluster instance,
where our NiFi application has been deployed in Kubernetes container, Here
is the below configuration we did for site-to-site in nifi.properties file
for a single cluster node. 

# Site to Site properties 
nifi.remote.input.host=
nifi.remote.input.secure=true 
nifi.remote.input.socket.port=9443 
nifi.remote.input.http.enabled=false 

I was trying to understand how site-to-site works internally by going
through source code and also debugging parallel to how it does
communication. I found couple of observations from my analysis 

1. First off, I believe when you give same hostname as nifi application
running in a container for site-to-site in nifi.properties for single
cluster node, the internal site-to-site java code doesn't get the hostname
of the self node when asked for cluster nodes information (NodeInFormant),
instead it gives some other private ip hostname. In the logs we see -* DEBUG
[Site-to-Site Worker Thread-235] o.a.nifi.remote.SocketRemoteSiteListener
org.apache.nifi.remote.SocketRemoteSiteListener$1$1@74dd1923 Connection URL
is nifi://ip-10-200-46-112.us-west-2.compute.internal:22343*
>From above debug log, I see the internal java code is not recognizing that
its a docker container instead its trying to connect with unknown hostname
name with random port. I believe due to incapable of recognizing its a
container instead returning some kubernetes node ip address, its throwing
the ssl handshake error 

Interesting thing is, When the port 'nifi.remote.input.socket.port' (9443)
was reachable at a container level, we see the above SSLHandshake error with
site-to-site worker thread trying to hit different hostname from the point-1
I mentioned above, when we blocked this port from the container, the
SSLHandshake error went away, we no longer see when blocking the port from
'nifi.remote.input.socket.port' in properties. I'm not sure if this make
sense but I want to understand how site-to-site works internally in detail. 

If above of my observations are incorrect or something needs to be done,
please help me in understanding. Please Bryan, Pierre, Marks, Koji or any
NiFi experts pleas help me understanding this. I have gone through almost
all blogs and etc. 

Please suggest the solution, 

Thanks, 
Nadeem



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

SSLHandshake Exception from Site-to-Site

[Mohammed Nadeem]

Hi,

Can someone please help me resolving SSLHandshake issue (Site-to-Site) which
I'm getting in logs. This ERROR doesn't impact us from accessing the NiFi
canvas or any calls we make from Nifi components (like SSL Context Service).
This is something which keeps on throwing every now and then in
nifi-app.logs

Below, is the error we get in the logs

ERROR [Site-to-Site Worker Thread-138]
o.a.n.r.io.socket.ssl.SSLSocketChannel
org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel@938965a Failed to
connect due to {}
javax.net.ssl.SSLHandshakeException: Reached End-of-File marker while
performing handshake
at
org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.performHandshake(SSLSocketChannel.java:248)
at
org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.connect(SSLSocketChannel.java:163)
at
org.apache.nifi.remote.SocketRemoteSiteListener$1$1.run(SocketRemoteSiteListener.java:168)
at java.lang.Thread.run(Thread.java:748)

ERROR [Site-to-Site Worker Thread-138]
o.a.nifi.remote.SocketRemoteSiteListener RemoteSiteListener Unable to accept
connection from Socket[unconnected] due to javax.net.ssl.SSLException:
Inbound closed before receiving peer's close_notify: possible truncation
attack? 

Setup,
CA Server is running on separate host ( eg, ca_server_host ) which generates
self-signed certificates
Each Nifi instance calls CA to get the keystore, trustore etc like the
necessary certs

Please help me understand the issue, I have gone through many resources
online but I wasn't able to resolve,

Thanks,
Nadeem



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: way to gracefully terminate a processor on errors

[Mohammed Nadeem]

Hi Chandrashekaran,

You can use the logic which i generally use to stop/start and monitor
failures from either relationship or bullentins. Use Reporting Tasks from
Global Menu - Controller Settings to monitor bulletins, memory use and other
critical information. These reporting tasks needed to be configured with
certain ports which gets triggered whenever there is a active failures
(bullentins, memory use etc). Connect the configured ports to your dataflow
which handles stopping the processors you would want on failures. Stopping
processors can be achieved through NiFi REST APIs.

Further, there is a nice blog from Pierre on configuring reporting tasks.
https://pierrevillard.com/2017/05/13/monitoring-nifi-site2site-reporting-tasks/

Thanks & Regards,
Nadeem



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Occasional freezing of the processor

[Mohammed Nadeem]

Hello Paresh,

I have a question and I'm trying to understand here, You are running your
processor on primary node?


Regards,
Nadeem



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Code coverage for NiFi processor

[Mohammed Nadeem]

Hi James,

Yes, I had used JaCoCo plugin to get the code coverage for the mocked Junit
testcases written. Copy below details to the nifi-custom-processors pom.xml
file.


jacoco

${project.basedir}/../target/jacoco.exec
java



org.jacoco
jacoco-maven-plugin
0.7.5.201505241946


pre-unit-test

prepare-agent


true
surefireArgLine



post-unit-test
test

report


${sonar.jacoco.reportPath}
true





org.apache.maven.plugins
maven-surefire-plugin
2.15

${surefireArgLine}

${skip.unit.tests}



Since I had used SonarQube for code quality check and code coverage, you
could leverage this or some other platform such as travis-cl build

Regards,
Nadeem



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: simple username+password authentication

[Mohammed Nadeem]

Hi Luke,

To elaborate on Bryan's point about custom login identity provider. There is
a nice example which you can mimic out for implementing simple file-based
authentication which internal stores all of your credentials in
login-credentials.xml file with password encrypted with Brcypt-hashed ( You
can provide more secure way if you would ).

Further link to the file identity provider bundle
https://github.com/BatchIQ/nifi-file-identity-provider-bundle

Regards,
Nadeem



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: How does logging level get set?

[Mohammed Nadeem]

Hi,

To answer to your question, Nifi has a context logger which is mapped to
logback.xml file. Generally INFO level of log is defaulted in nifi-app.logs.
In your code, you can simple use getLogger().debug("your message") or INFO,
ERROR,WARN etc to enable the logging into nifi-app.log with component-id
etc.

As I mentioned, by default INFO level would go to nifi-app.logs, you can
change the level in logback.xml under root.

Thanks & Regards,
Nadeem



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: DBCP Connection Pooling using multiple OJDBC Drivers

[Mohammed Nadeem]

Thanks Koji. But giving a directory of multiple drivers is throwing an error
"can't load database driver " and i verified that with dbcp code and it
specifically checks for ".jar extension".

final ClassLoader classLoader =
ClassLoaderUtils.getCustomClassLoader(
locationString,
this.getClass().getClassLoader(),
(dir, name) -> name != null && name.endsWith(".jar")
);

Your alternative approach is what the problem i wrote earlier creating
multiple instances of dbcp service and making each point to different
version of drivers. The issue was below error for ojdbc7.jar ( oracle 11g)..

*java.sql.SQLException: Object does not wrap anything with requested
interface*
at oracle.jdbc.driver.OracleConnection.unwrap(OracleConnection.java:268)
at
org.apache.commons.dbcp.DelegatingConnection.unwrap(DelegatingConnection.java:553)
at
org.apache.commons.dbcp.DelegatingConnection.unwrap(DelegatingConnection.java:553)

It is not able to unwrap this connection to oracle.jdbc.OracleConnection (
*because im using createOracleArray method of this OracleConnection.class*
). For this i added dependent ojdbc7.jar into my maven build pom.xml . Still
I'm facing this issue
Though it was working with junit testcases where i used dbcp as connection
service.

What could be the reason.. Please help



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: NiFi starts very slowly

[Mohammed Nadeem]

Hi Luby,

If you are using a cluster based setup then Nifi is unstable due to nodes
getting disconnected by the cluster cordinator. Cluster coordinator
disconnects nodes from the cluster when the nodes take a little longer to
respond ( beyond timeout) . The cluster coordinator timeout based on the
settings in the nifi.properties file.

We need to adjust the following properties 

*nifi.cluster.node.connection.timeout * = 5 sec  ( by default )
*nifi.cluster.node.read.timeout *=  5 sec ( by default)

By Increasing the timeout ,from 5 sec to higher value would probably solve a
problem


Thanks,
Nadeem



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: DBCP Connection Pooling using multiple OJDBC Drivers

[Mohammed Nadeem]

Thanks Mark that really helps. But I'm facing below issue 

As you said i loaded drivers externally and it was successful for oracle8i
where i executed simple stored procedure and it is working fine but when i
try to execute stored procedure which has array type for Oracle 11g then im
getting this error ( can't wrap connection to oracle connection )

java.lang.AbstractMethodError: null
at
org.apache.commons.dbcp.DelegatingConnection.unwrap(DelegatingConnection.java:553)
~[na:na]
at
org.apache.commons.dbcp.DelegatingConnection.unwrap(DelegatingConnection.java:553)
~[na:na]
at
ExecuteProcedure.executeStoredProcedure(GE_Scon_ExecuteProcedure.java:584)
~[na:na]
at ExecuteProcedure.onTrigger(GE_Scon_ExecuteProcedure.java:382) 
~[na:na]

It can't unwrap the connection to oracle connection . Earlier i had resolved
this issue by placing ojdbc7.jar in my nifi-lib folder then it started to
work.. But as you said we should not place any jars in the nifi lib folder
.. 
So now its giving this error. Please help here 

Thanks,
Nadeem
Software Engineering Specialist



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: DBCP Connection Pooling using multiple OJDBC Drivers

[Mohammed Nadeem]

Thanks Milan, I have doubt here.

When i execute my custom processor keeping ojdbc14.jar for Oracle 8i in my
nifi-lib folder then custom processor works fine, but when i execute
processor for Oracle 11 g then its throwing me an error "NoSuchMethodError
Oracle.jdbc.OracleConnection.createOracleArray()" though i have ojdbc7.jar
as my dependency in my maven setup. I think it is trying to load only
ojdbc14.jar from where its not able to find that method ( I decompiled the
ojdbc14.jar and it doesn't support that method) only ojdbc7.jar supports. If
do reverse where i place ojdbc7.jar first then oracle 8i is throwing error.

I feel like dbcp connection pool is loading only one driver which will be
enabled first. if ojdbc7.jar is enabled then only that will be used through
out Nifi instance.. Loading other ojdbc driver will be useless i feel. Not
Sure whether im correct or not.

Could you please help me here.



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

DBCP Connection Pooling using multiple OJDBC Drivers

[Mohammed Nadeem]

I'm building a custom processor where i need to execute PL/SQL Procedures
with the help of DBCP Connection Pooling Controller Service. The custom
processor which executes PL/SQL Procedures needs to connect to different
Oracle Databases like 11g and Oracle 8i. 

The Problem i'm facing here is that these oracle databases needs different
ojdbc jars . For example Oracle 11g needs ojdbc7.jar and Oracle 8i needs
ojdbc14.jar . The Custom processor needs ojdbc7.jar as maven dependency to
execute complex Oracle jdbc types such as ARRAY ,STRUCT etc.  When I load
two dbcp controller services which uses different ojdbc.jar's for the same
custom processor it is working for one oracle database but not for other.

Detail Description.

If I connect to Oracle Database 11g where i give diver location as
ojdbc7.jar in dbcp controller service then its throwing an error saying "
java.sql.Exception : can't wrapped connection to requested interface".
To resolve this issue i added ojdbc7.jar in nifi lib folder and the error
went.

Now, when i connect to Oracle 8i with ojdbc14.jar in dbcp controller
service.. It is throwing an error saying " ArrayOutOfBound Exception 7" . I
guess it is trying to use incompatible jar which was given in the lib folder
(ojdbc7.jar) . If I add ojdbc14.jar in the lib then earlier one is not
working giving same error "  java.sql.Exception : can't wrapped connection
to requested interface".

Could you please help me out there.. Not Sure how nifi classloader works ..

Thanks  in advance


Regards,
Nadeem

 



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/