Re: [DISCUSS] First Release of NiFi Registry

[Bryan Bende]

Looks like the work for the initial docs has wrapped up and we are in
a good place to kick out an RC.

I'll start pulling things together and should be able to get something
out tomorrow.

Pierre,

I created a JIRA [1] to capture an improvement for the scenario you
described with the nested process groups.

As you said, I think most of the other stuff has been captured in
JIRA's created by Joe P.

Thanks,

Bryan

[1] https://issues.apache.org/jira/browse/NIFIREG-86

On Mon, Dec 25, 2017 at 5:40 AM, Pierre Villard
 wrote:
> Hey guys,
>
> Not sure that's the best place to give my feedbacks after running some
> tests, let me know if I should open a new thread.
>
> (I believe Joe P. already made some similar comments, but just in case...)
>
> - in an unsecure environment, it's probably better to disable the "Add new
> policy" button (NIFIREG-78)
> - I've seen some logs that could be set to debug? “Access tokens are only
> issued over HTTPS. Returning Conflict response.”, “Registry is not
> configured to internally manage users, groups, or policies. Please contact
> your system administrator.. Returning Conflict response.“
> - general comment for NiFi UI: add tooltips on the icons of the upper
> status bar? We've quite a few new icons coming with the Registry and I
> guess it could help people not very familiar with it yet.
> - is it possible to do a diff between two versions in the Registry UI?
> - when adding a variable to a versioned PG, it does not show changes to
> commit. Is it expected? (it does not to me)
> - how to set a previous version as the new current one? does not seem
> possible unless you stop version control and start again?
> - very very minor comment, in the Registry UI, in the actions list, I'd set
> "Delete" instead of "delete".
>
> Another observation:
>
> I have PG A containing PG B, both versioned. And I have two instances of PG
> A in my NiFi UI PG A1 and PG A2.
> - I deleted PG B tracking in NiFi Registry. I now have 404 errors on the
> PGAx because PG B is not found in registry. All good. Then I disconnect PG
> B in PG A1. PG A1 is shown as OK / up-to-date with nothing to commit.
> - If I try to import a new instance of PG A, it’s not working because “The
> Flow Registry with ID 893e20cc-0160-1000-8ab8-e0507c36aa94 reports that no
> Flow exists with Bucket f66d8eb1-b893-41ad-974b-565bc33c8104, Flow
> 8d2df468-e8e4-4138-aaef-c7eadb71c2c4, Version 4”
> - In PG A2, if I delete PG B, then it shows local change but I cannot
> revert local changes: "Failed to retrieve flow with Flow Registry in order
> to calculate local differences due to Error retrieving flow snapshot:
> Versioned flow does not exist with identifier
> 08e85785-cb41-4cae-a516-6b4d3506960e"
> - In the end I have to delete PG B, commit changes to get everything back
> to normal. I’m wondering if disconnecting PG B shouldn’t be considered as a
> local change to be committed? Because, I could be in a situation where I
> don’t want to delete PG B, I just want to stop version control on it, no?
>
> I'll run some more tests in secured environments.
>
> Pierre
>
>
>
>
> 2017-12-21 18:50 GMT+01:00 Bryan Bende :
>
>> Just wanted to give an update on this...
>>
>> Great progress has been made in the last two weeks in terms of getting
>> ready for an RC. Still a few outstanding items, but I think we could
>> have those wrapped up soon and kick out an RC some time next week.
>> Depending when everything is ready we can adjust the voting period if
>> needed to account for holidays and make sure there is adequate time
>> for review.
>>
>> In the mean time, I encourage anyone who is interested to give it a
>> try. Here is some info about how to get started...
>>
>> 1) Get the code for the registry
>>
>> The Apache repo is here:
>>
>> https://git-wip-us.apache.org/repos/asf/nifi-registry.git
>>
>> The github repo is here if you prefer to fork that:
>>
>> https://github.com/apache/nifi-registry
>>
>> 2) Build the registry code
>>
>> cd nifi-registry
>> mvn clean install
>>
>> 3) Start the registry
>>
>> cd nifi-registry-assembly/target/nifi-registry-0.0.1-SNAPSHOT-
>> bin/nifi-registry-0.0.1-SNAPSHOT/
>> ./bin/nifi-registry.sh start
>>
>> 4) Create a bucket in the registry
>>
>> - Go to the registry UI at http://localhost:18080/nifi-registry
>> - Click the tool icon in the top right corner
>> - Click New Bucket from the bucket table
>> - Enter a name and click create
>>
>> 5) Get the NiFi PR which adds the support for integrating with the registry
>>
>> https://github.com/apache/nifi/pull/2219
>>
>> Build that PR like normal.
>>
>> NOTE: That you must have already built nifi-registry with "mvn clean
>> install" in order to build this PR because it depends on snapshot JARs
>> being in your local Maven repo.
>>
>> 6) Tell NiFi about your local registry instance
>>
>> - Go the controller settings for NiFi from the top-right menu
>> - Select the Registry Clients tab
>> - Add a new Registry Client giving it a name and the url of
>> http://local

Re: DBCP Connection Pooling using multiple OJDBC Drivers

[Koji Kawamura]

Hi Nadeem,

Did you try specifying a external directory instead of an exact jar
location, then put multiple versions of Jar there? This way
DBCPConnectionPool can utilize multiple jars. This would provide the
similar effect with putting one in NiFi lib dir.

If that doesn't work, an alternative approach would be setup different
DBCPConnectionPool instance per required JDBC driver version then
switching those by some condition.

I haven't tested above myself, so no guarantee, but just wanted to
share my thoughts.

Thanks,
Koji

On Thu, Dec 28, 2017 at 12:14 AM, Mohammed Nadeem  wrote:
> Thanks Mark that really helps. But I'm facing below issue
>
> As you said i loaded drivers externally and it was successful for oracle8i
> where i executed simple stored procedure and it is working fine but when i
> try to execute stored procedure which has array type for Oracle 11g then im
> getting this error ( can't wrap connection to oracle connection )
>
> java.lang.AbstractMethodError: null
> at
> org.apache.commons.dbcp.DelegatingConnection.unwrap(DelegatingConnection.java:553)
> ~[na:na]
> at
> org.apache.commons.dbcp.DelegatingConnection.unwrap(DelegatingConnection.java:553)
> ~[na:na]
> at
> ExecuteProcedure.executeStoredProcedure(GE_Scon_ExecuteProcedure.java:584)
> ~[na:na]
> at ExecuteProcedure.onTrigger(GE_Scon_ExecuteProcedure.java:382) 
> ~[na:na]
>
> It can't unwrap the connection to oracle connection . Earlier i had resolved
> this issue by placing ojdbc7.jar in my nifi-lib folder then it started to
> work.. But as you said we should not place any jars in the nifi lib folder
> ..
> So now its giving this error. Please help here
>
> Thanks,
> Nadeem
> Software Engineering Specialist
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Compile avro to RecordSchema

[Koji Kawamura]

Hi Mike,

You might already have found it, but AvroTypeUtil.createSchema is
probably what you are looking for.
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java#L341

Thanks,
Koji

On Wed, Dec 27, 2017 at 9:15 PM, Mike Thomsen  wrote:
> Is there an API for compiling an avro schema to a RecordSchema object, or
> is it more involved than that?
>
> Thanks,
>
> Mike

Re: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0

[Michael Moser]

Steven,

Joe and Bryan's suggestions are best, but there is another way to try to
resolve your issue,  It comes with a large "Beware of class loaders" sign,
but it can work.

Your custom nar can be built so that it depends on the classes found in
nifi-standard-nar, by using a nar dependency.  A nar dependency essentially
tells NiFi that your custom nar's class loader should use the nar
dependency's class loader as its parent  To setup a nar dependency, you
would add the below snippet to the pom.xml that builds your custom nar.


  org.apache.nifi
  nifi-standard-nar
  1.4.0
  nar


The nar is very important and is recognized by the
nifi-nar-maven-plugin.

Then you must mark the jars that exist inside the nifi-standard-nar as
provided so that they do not also get copied into your
custom nar.  When code in your custom nar needs the inherited code, your
nar's class loader will load the inherited classes using its parent class
loader.

Regards,
-- Mike


On Tue, Dec 26, 2017 at 2:04 PM, Joe Witt  wrote:

> Steven
>
> Understood.  That it now shows you this is a good thing in 1.4.0
> because what was happening in the older versions is one of the sets of
> processors was being loaded.  You are probably seeing a warning in the
> app log at startup.  So depending on which nar is loaded first
> (standard or your custom one) the code will actually run out of there.
> So, in short the behavior in 1.4.0 even in your current setup is
> probably better than the behavior prior.
>
> Hope that helps
>
> Thanks
> Joe
>
> On Tue, Dec 26, 2017 at 2:00 PM, Byers, Steven K (Steve) CTR USARMY
> MEDCOM JMLFDC (US)  wrote:
> > Yes, I agree. Unfortunately, our schedule may not allow for that at this
> time so we will probably have to put off migrating for now.
> >
> > Thanks to you and Bryan for your replies.
> >
> > Thank you,
> >
> > Steven K. Byers
> > DXC Technology - Contractor
> > Software Developer - Joint Medical Logistics Functional Development
> Center (JMLFDC)
> > Defense Health Agency (DHA)/ Health Information Technology (HIT)
> Directorate/ Solution Delivery Division (SDD)/Clinical Support Branch/JMLFDC
> > 1681 Nelson Street, Fort Detrick, MD  21702
> > (443) 538-7575 | (410) 872-4923
> > Email: steven.k.byers@mail.mil
> >
> >
> >
> > -Original Message-
> > From: Joe Witt [mailto:joe.w...@gmail.com]
> > Sent: Tuesday, December 26, 2017 1:56 PM
> > To: dev@nifi.apache.org
> > Subject: Re: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0
> >
> > Steven
> >
> > To Bryan's point if it will be best if you simply copy the code/base
> classes over to your Nar rather than extending from existing concrete
> implemented processors.
> >
> > Thanks
> >
> > On Tue, Dec 26, 2017 at 1:50 PM, Byers, Steven K (Steve) CTR USARMY
> MEDCOM JMLFDC (US)  wrote:
> >> Bryan,
> >>
> >> We have extended a few processors from standard processors.  For
> >> example, one of the processors we are extending is:
> >> org.apache.nifi.processors.standard.JmsConsumer
> >>
> >> In another processor we are importing org.apache.nifi.processors.
> standard.util.JdbcCommon.
> >>
> >> I tried making the dependency on standard processors optional
> (optional>true) which excludes the standard processors but then
> NiFi won't start because those custom processors that depend on it cannot
> be instantiated.
> >>
> >>
> >>
> >> Thank you,
> >>
> >> Steven K. Byers
> >> DXC Technology - Contractor
> >> Software Developer - Joint Medical Logistics Functional Development
> >> Center (JMLFDC) Defense Health Agency (DHA)/ Health Information
> >> Technology (HIT) Directorate/ Solution Delivery Division
> >> (SDD)/Clinical Support Branch/JMLFDC
> >> 1681 Nelson Street, Fort Detrick, MD  21702
> >> (443) 538-7575 | (410) 872-4923
> >> Email: steven.k.byers@mail.mil
> >>
> >>
> >>
> >> -Original Message-
> >> From: Bryan Bende [mailto:bbe...@gmail.com]
> >> Sent: Tuesday, December 26, 2017 1:34 PM
> >> To: dev@nifi.apache.org
> >> Subject: Re: [Non-DoD Source] Re: Moving from version 1.1.2 to 1.4.0
> >>
> >> Can you give some more details about what the dependency is for?
> >>
> >> If it is some utility code that exists in standard processors then we
> >> should be looking to move that to other reusable modules so that you
> >> can depend on it without depending on the processors.
> >>
> >> If it is because you extended a processor from standard processors,
> >> you would probably want to just copy the processor code into your own
> >> NAR and modify/extend it.
> >>
> >> On Tue, Dec 26, 2017 at 1:26 PM Byers, Steven K (Steve) CTR USARMY
> >> MEDCOM JMLFDC (US)  wrote:
> >>
> >>> Thanks for the reply, Bryan,
> >>>
> >>> Yes, two of our custom processors have a dependency on the standard
> >>> processors.  The dependency cannot be removed or those processors
> >>> will not compile.
> >>>
> >>> Thank you,
> >>>
> >>> Steven K. Byers
> >>> DXC Technology - Contractor
> >>> Software Developer - Joint Medical Logistics Functional Developme

Re: Provenance events id disorder bug

[נתנאל ביטאן]

Thanks for your quick comment.
I've attached the script to this jira:
https://issues.apache.org/jira/plugins/servlet/mobile#issue/NIFI-4638
And uploaded it as a gist in this address:
https://gist.github.com/bitanetanel/808cb3a0204aad5830f31d6d7ff89299

‫בתאריך יום ד׳, 27 בדצמ׳ 2017 ב-21:06 מאת ‪Joe Witt‬‏ <‪joe.w...@gmail.com
‬‏>:‬

> hello
>
> it is best to attach the script to a jira.
>
> thanks
>
> On Dec 27, 2017 1:52 PM, "נתנאל ביטאן"  wrote:
>
> > Sorry, the script is attached to this mail.
> >
> > בתאריך יום ד׳, 27 בדצמ׳ 2017, 20:04, מאת נתנאל ביטאן ‏<
> > bitaneta...@gmail.com>:
> >
> >> Hi!
> >>
> >> In my attempts to try to get to the bottom of the events disorder bug (
> >> https://issues.apache.org/jira/plugins/servlet/mobile#issue/NIFI-4638)
> I
> >> have written a small script that finds the disorders. I hope you might
> find
> >> this useful in trying to recreate the bug, and find out if you are
> >> experiencing it in any of your own nifi flows.
> >>
> >> To run the script simply configure the name of the nifi host and the
> >> amount of time back you want the script to check at the top of the
> document.
> >>
> >> Its interesting to note that we have not been able to reproduce this bug
> >> in any of our more simple flows, and it seems to happen only in a very
> >> complicated flow consisting of 75+ processors.
> >>
> >> I will be happy to receive updates regarding your progress.
> >>
> >> Best regards,
> >> Netanel Bitan
> >>
> >
>

Re: Nifi "event driven" scheduling strategy Custom Processor

[Michael Moser]

Hello!

An "event" in the context of the NiFi @EventDriven annotation is the
arrival of a flow file in the queue that is an input to a processor.  The
only purpose of @EventDriven is to save system resources when compared to
the default Timer Driven strategy, in which a processor polls its input
queues for flow files to work on.

If this matches your expectation for the Event Driven scheduling strategy,
then you must generate an "event" with another processor on the NiFi
graph.  This could be as simple as a GenerateFlowFile processor that runs
on a predictable Timer Driven period or Cron Driven schedule.  You could
also use another ingress [1], or data ingestion processor [2], to create a
new flow file containing an event.

[1] -
http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#ingress
[2] -
http://nifi.apache.org/docs/nifi-docs/html/getting-started.html#data-ingestion

I hope this helps,
-- Mike



On Wed, Dec 27, 2017 at 10:34 AM, Milan Das  wrote:

> I am trying to build a custom processor with “Event Driven”. I added
>  “@EventDriven” annotation. It is showing EventDriven in the scheduling
> strategy.
>
> Question: how to send an event to this processor to intiate the flow? What
> are my options?
>
>
>
> Regards,
>
>
>
> [image: graph]
>
> *Milan Das*
> Sr. System Architect
>
> email: m...@interset.com
>
> [image: dIn icon] 
>
> www.interset.com
>
>
>
>
>
>
>

Re: Provenance events id disorder bug

[Joe Witt]

hello

it is best to attach the script to a jira.

thanks

On Dec 27, 2017 1:52 PM, "נתנאל ביטאן"  wrote:

> Sorry, the script is attached to this mail.
>
> בתאריך יום ד׳, 27 בדצמ׳ 2017, 20:04, מאת נתנאל ביטאן ‏<
> bitaneta...@gmail.com>:
>
>> Hi!
>>
>> In my attempts to try to get to the bottom of the events disorder bug (
>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/NIFI-4638) I
>> have written a small script that finds the disorders. I hope you might find
>> this useful in trying to recreate the bug, and find out if you are
>> experiencing it in any of your own nifi flows.
>>
>> To run the script simply configure the name of the nifi host and the
>> amount of time back you want the script to check at the top of the document.
>>
>> Its interesting to note that we have not been able to reproduce this bug
>> in any of our more simple flows, and it seems to happen only in a very
>> complicated flow consisting of 75+ processors.
>>
>> I will be happy to receive updates regarding your progress.
>>
>> Best regards,
>> Netanel Bitan
>>
>

Re: Provenance events id disorder bug

[נתנאל ביטאן]

Sorry, the script is attached to this mail.

בתאריך יום ד׳, 27 בדצמ׳ 2017, 20:04, מאת נתנאל ביטאן ‏:

> Hi!
>
> In my attempts to try to get to the bottom of the events disorder bug (
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/NIFI-4638) I
> have written a small script that finds the disorders. I hope you might find
> this useful in trying to recreate the bug, and find out if you are
> experiencing it in any of your own nifi flows.
>
> To run the script simply configure the name of the nifi host and the
> amount of time back you want the script to check at the top of the document.
>
> Its interesting to note that we have not been able to reproduce this bug
> in any of our more simple flows, and it seems to happen only in a very
> complicated flow consisting of 75+ processors.
>
> I will be happy to receive updates regarding your progress.
>
> Best regards,
> Netanel Bitan
>

Provenance events id disorder bug

[נתנאל ביטאן]

Hi!

In my attempts to try to get to the bottom of the events disorder bug (
https://issues.apache.org/jira/plugins/servlet/mobile#issue/NIFI-4638) I
have written a small script that finds the disorders. I hope you might find
this useful in trying to recreate the bug, and find out if you are
experiencing it in any of your own nifi flows.

To run the script simply configure the name of the nifi host and the amount
of time back you want the script to check at the top of the document.

Its interesting to note that we have not been able to reproduce this bug in
any of our more simple flows, and it seems to happen only in a very
complicated flow consisting of 75+ processors.

I will be happy to receive updates regarding your progress.

Best regards,
Netanel Bitan

Nifi "event driven" scheduling strategy Custom Processor

[Milan Das]

I am trying to build a custom processor with “Event Driven”. I added  
“@EventDriven” annotation. It is showing EventDriven in the scheduling strategy.

Question: how to send an event to this processor to intiate the flow? What are 
my options?

 

Regards,

 

Milan Das
Sr. System Architect
email: m...@interset.com

www.interset.com

 

 

 

Re: DBCP Connection Pooling using multiple OJDBC Drivers

[Mohammed Nadeem]

Thanks Mark that really helps. But I'm facing below issue 

As you said i loaded drivers externally and it was successful for oracle8i
where i executed simple stored procedure and it is working fine but when i
try to execute stored procedure which has array type for Oracle 11g then im
getting this error ( can't wrap connection to oracle connection )

java.lang.AbstractMethodError: null
at
org.apache.commons.dbcp.DelegatingConnection.unwrap(DelegatingConnection.java:553)
~[na:na]
at
org.apache.commons.dbcp.DelegatingConnection.unwrap(DelegatingConnection.java:553)
~[na:na]
at
ExecuteProcedure.executeStoredProcedure(GE_Scon_ExecuteProcedure.java:584)
~[na:na]
at ExecuteProcedure.onTrigger(GE_Scon_ExecuteProcedure.java:382) 
~[na:na]

It can't unwrap the connection to oracle connection . Earlier i had resolved
this issue by placing ojdbc7.jar in my nifi-lib folder then it started to
work.. But as you said we should not place any jars in the nifi lib folder
.. 
So now its giving this error. Please help here 

Thanks,
Nadeem
Software Engineering Specialist



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: DBCP Connection Pooling using multiple OJDBC Drivers

[Mark Payne]

Nadeem,

You should not be placing any jar file into nifi’s lib directory. Doing so will 
load the jar into all classloaders in the system. Instead, you should place the 
jar outside of the lib/ directory and then configure the controller service to 
point to the jar file, wherever it is placed. 

Thanks
-Mark

Sent from my iPhone

> On Dec 27, 2017, at 9:33 AM, Mohammed Nadeem  wrote:
> 
> Thanks Milan, I have doubt here.
> 
> When i execute my custom processor keeping ojdbc14.jar for Oracle 8i in my
> nifi-lib folder then custom processor works fine, but when i execute
> processor for Oracle 11 g then its throwing me an error "NoSuchMethodError
> Oracle.jdbc.OracleConnection.createOracleArray()" though i have ojdbc7.jar
> as my dependency in my maven setup. I think it is trying to load only
> ojdbc14.jar from where its not able to find that method ( I decompiled the
> ojdbc14.jar and it doesn't support that method) only ojdbc7.jar supports. If
> do reverse where i place ojdbc7.jar first then oracle 8i is throwing error.
> 
> I feel like dbcp connection pool is loading only one driver which will be
> enabled first. if ojdbc7.jar is enabled then only that will be used through
> out Nifi instance.. Loading other ojdbc driver will be useless i feel. Not
> Sure whether im correct or not.
> 
> Could you please help me here.
> 
> 
> 
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: DBCP Connection Pooling using multiple OJDBC Drivers

[Mohammed Nadeem]

Thanks Milan, I have doubt here.

When i execute my custom processor keeping ojdbc14.jar for Oracle 8i in my
nifi-lib folder then custom processor works fine, but when i execute
processor for Oracle 11 g then its throwing me an error "NoSuchMethodError
Oracle.jdbc.OracleConnection.createOracleArray()" though i have ojdbc7.jar
as my dependency in my maven setup. I think it is trying to load only
ojdbc14.jar from where its not able to find that method ( I decompiled the
ojdbc14.jar and it doesn't support that method) only ojdbc7.jar supports. If
do reverse where i place ojdbc7.jar first then oracle 8i is throwing error.

I feel like dbcp connection pool is loading only one driver which will be
enabled first. if ojdbc7.jar is enabled then only that will be used through
out Nifi instance.. Loading other ojdbc driver will be useless i feel. Not
Sure whether im correct or not.

Could you please help me here.



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: DBCP Connection Pooling using multiple OJDBC Drivers

[Milan Das]

Hi Nadeem,
I think you can do very similar to Kafka implementation. For different version 
NIFI have two different processor for Kafka 10 and 11  “PublishKafka_0_10” and 
“PublishKafka_0_11”

Regards,


Milan Das
Sr. System Architect
email: m...@interset.com

www.interset.com 
 


On 12/27/17, 7:08 AM, "Mohammed Nadeem"  wrote:

I'm building a custom processor where i need to execute PL/SQL Procedures
with the help of DBCP Connection Pooling Controller Service. The custom
processor which executes PL/SQL Procedures needs to connect to different
Oracle Databases like 11g and Oracle 8i. 

The Problem i'm facing here is that these oracle databases needs different
ojdbc jars . For example Oracle 11g needs ojdbc7.jar and Oracle 8i needs
ojdbc14.jar . The Custom processor needs ojdbc7.jar as maven dependency to
execute complex Oracle jdbc types such as ARRAY ,STRUCT etc.  When I load
two dbcp controller services which uses different ojdbc.jar's for the same
custom processor it is working for one oracle database but not for other.

Detail Description.

If I connect to Oracle Database 11g where i give diver location as
ojdbc7.jar in dbcp controller service then its throwing an error saying "
java.sql.Exception : can't wrapped connection to requested interface".
To resolve this issue i added ojdbc7.jar in nifi lib folder and the error
went.

Now, when i connect to Oracle 8i with ojdbc14.jar in dbcp controller
service.. It is throwing an error saying " ArrayOutOfBound Exception 7" . I
guess it is trying to use incompatible jar which was given in the lib folder
(ojdbc7.jar) . If I add ojdbc14.jar in the lib then earlier one is not
working giving same error "  java.sql.Exception : can't wrapped connection
to requested interface".

Could you please help me out there.. Not Sure how nifi classloader works ..

Thanks  in advance


Regards,
Nadeem

 



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Compile avro to RecordSchema

[Mike Thomsen]

Is there an API for compiling an avro schema to a RecordSchema object, or
is it more involved than that?

Thanks,

Mike

DBCP Connection Pooling using multiple OJDBC Drivers

[Mohammed Nadeem]

I'm building a custom processor where i need to execute PL/SQL Procedures
with the help of DBCP Connection Pooling Controller Service. The custom
processor which executes PL/SQL Procedures needs to connect to different
Oracle Databases like 11g and Oracle 8i. 

The Problem i'm facing here is that these oracle databases needs different
ojdbc jars . For example Oracle 11g needs ojdbc7.jar and Oracle 8i needs
ojdbc14.jar . The Custom processor needs ojdbc7.jar as maven dependency to
execute complex Oracle jdbc types such as ARRAY ,STRUCT etc.  When I load
two dbcp controller services which uses different ojdbc.jar's for the same
custom processor it is working for one oracle database but not for other.

Detail Description.

If I connect to Oracle Database 11g where i give diver location as
ojdbc7.jar in dbcp controller service then its throwing an error saying "
java.sql.Exception : can't wrapped connection to requested interface".
To resolve this issue i added ojdbc7.jar in nifi lib folder and the error
went.

Now, when i connect to Oracle 8i with ojdbc14.jar in dbcp controller
service.. It is throwing an error saying " ArrayOutOfBound Exception 7" . I
guess it is trying to use incompatible jar which was given in the lib folder
(ojdbc7.jar) . If I add ojdbc14.jar in the lib then earlier one is not
working giving same error "  java.sql.Exception : can't wrapped connection
to requested interface".

Could you please help me out there.. Not Sure how nifi classloader works ..

Thanks  in advance


Regards,
Nadeem

 



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: proper way in nifi to sync status between custom processors

[Koji Kawamura]

Hi Ben, you can filter events by timestamp as well.
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#searching-for-events

On Wed, Dec 27, 2017 at 6:28 PM, 尹文才  wrote:
> Hi Koji, I saw it was only showing the 1000 events so I couldn't see the
> event when the FlowFile was created.
>
> Regards,
> Ben
>
> 2017-12-27 17:21 GMT+08:00 Koji Kawamura :
>
>> I see, thanks. The easiest way to look at provenance events would be
>> by right clicking a processor instance you are interested in, then
>> select 'View data provenance' context menu. This way, NiFi displays
>> provenance events for the selected processor.
>>
>> Koji
>>
>> On Wed, Dec 27, 2017 at 6:17 PM, 尹文才  wrote:
>> > Hi Koji, sorry about the provenance exception, it was because there's no
>> > space left on the machine(filled up with logs)
>> >
>> > Regards,
>> > Ben
>> >
>> > 2017-12-27 17:11 GMT+08:00 尹文才 :
>> >
>> >> Hi Koji, thanks, the names of the temp tables are created with format
>> >> "MMddHHmmssSSS-", the first time indicates the time and the
>> second
>> >> part is a random number with length of 4.
>> >> So I think it's not possible to have 2 duplicate table names, the only
>> >> possibly I could think is the flowfile is passed into the processor
>> twice.
>> >>
>> >> About the provenance, I had updated to use the
>> >> WriteAheadProvenanceRepository implementation, but when I tried to check
>> >> the data provenance, it showed me the following exception message:
>> >> HTTP ERROR 500
>> >>
>> >> Problem accessing /nifi/provenance. Reason:
>> >>
>> >> Server Error
>> >>
>> >> Caused by:
>> >>
>> >> javax.servlet.ServletException: org.eclipse.jetty.servlet.ServletHolder$1:
>> java.lang.NullPointerException
>> >>   at org.eclipse.jetty.server.handler.HandlerCollection.
>> handle(HandlerCollection.java:138)
>> >>   at org.eclipse.jetty.server.handler.gzip.GzipHandler.
>> handle(GzipHandler.java:561)
>> >>   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>> HandlerWrapper.java:132)
>> >>   at org.eclipse.jetty.server.Server.handle(Server.java:564)
>> >>   at org.eclipse.jetty.server.HttpChannel.handle(
>> HttpChannel.java:320)
>> >>   at org.eclipse.jetty.server.HttpConnection.onFillable(
>> HttpConnection.java:251)
>> >>   at org.eclipse.jetty.io.AbstractConnection$
>> ReadCallback.succeeded(AbstractConnection.java:279)
>> >>   at org.eclipse.jetty.io.FillInterest.fillable(
>> FillInterest.java:110)
>> >>   at org.eclipse.jetty.io.ssl.SslConnection.onFillable(
>> SslConnection.java:258)
>> >>   at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(
>> SslConnection.java:147)
>> >>   at org.eclipse.jetty.io.FillInterest.fillable(
>> FillInterest.java:110)
>> >>   at org.eclipse.jetty.io.ChannelEndPoint$2.run(
>> ChannelEndPoint.java:124)
>> >>   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
>> QueuedThreadPool.java:672)
>> >>   at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
>> QueuedThreadPool.java:590)
>> >>   at java.lang.Thread.run(Thread.java:745)
>> >> Caused by: org.eclipse.jetty.servlet.ServletHolder$1:
>> java.lang.NullPointerException
>> >>   at org.eclipse.jetty.servlet.ServletHolder.makeUnavailable(
>> ServletHolder.java:596)
>> >>   at org.eclipse.jetty.servlet.ServletHolder.initServlet(
>> ServletHolder.java:655)
>> >>   at org.eclipse.jetty.servlet.ServletHolder.getServlet(
>> ServletHolder.java:498)
>> >>   at org.eclipse.jetty.servlet.ServletHolder.ensureInstance(
>> ServletHolder.java:785)
>> >>   at org.eclipse.jetty.servlet.ServletHolder.prepare(
>> ServletHolder.java:770)
>> >>   at org.eclipse.jetty.servlet.ServletHandler.doHandle(
>> ServletHandler.java:538)
>> >>   at org.eclipse.jetty.server.handler.ScopedHandler.handle(
>> ScopedHandler.java:143)
>> >>   at org.eclipse.jetty.security.SecurityHandler.handle(
>> SecurityHandler.java:548)
>> >>   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
>> HandlerWrapper.java:132)
>> >>   at org.eclipse.jetty.server.handler.ScopedHandler.
>> nextHandle(ScopedHandler.java:190)
>> >>   at org.eclipse.jetty.server.session.SessionHandler.
>> doHandle(SessionHandler.java:1593)
>> >>   at org.eclipse.jetty.server.handler.ScopedHandler.
>> nextHandle(ScopedHandler.java:188)
>> >>   at org.eclipse.jetty.server.handler.ContextHandler.
>> doHandle(ContextHandler.java:1239)
>> >>   at org.eclipse.jetty.server.handler.ScopedHandler.
>> nextScope(ScopedHandler.java:168)
>> >>   at org.eclipse.jetty.servlet.ServletHandler.doScope(
>> ServletHandler.java:481)
>> >>   at org.eclipse.jetty.server.session.SessionHandler.
>> doScope(SessionHandler.java:1562)
>> >>   at org.eclipse.jetty.server.handler.ScopedHandler.
>> nextScope(ScopedHandler.java:166)
>> >>   at org.eclipse.jetty.server.handler.ContextHandler.
>> doScope(ContextHandler.java:1141)
>> >>   at org.eclipse.jetty.server.handler.ScopedHandle

Re: proper way in nifi to sync status between custom processors

[尹文才]

Hi Koji, I saw it was only showing the 1000 events so I couldn't see the
event when the FlowFile was created.

Regards,
Ben

2017-12-27 17:21 GMT+08:00 Koji Kawamura :

> I see, thanks. The easiest way to look at provenance events would be
> by right clicking a processor instance you are interested in, then
> select 'View data provenance' context menu. This way, NiFi displays
> provenance events for the selected processor.
>
> Koji
>
> On Wed, Dec 27, 2017 at 6:17 PM, 尹文才  wrote:
> > Hi Koji, sorry about the provenance exception, it was because there's no
> > space left on the machine(filled up with logs)
> >
> > Regards,
> > Ben
> >
> > 2017-12-27 17:11 GMT+08:00 尹文才 :
> >
> >> Hi Koji, thanks, the names of the temp tables are created with format
> >> "MMddHHmmssSSS-", the first time indicates the time and the
> second
> >> part is a random number with length of 4.
> >> So I think it's not possible to have 2 duplicate table names, the only
> >> possibly I could think is the flowfile is passed into the processor
> twice.
> >>
> >> About the provenance, I had updated to use the
> >> WriteAheadProvenanceRepository implementation, but when I tried to check
> >> the data provenance, it showed me the following exception message:
> >> HTTP ERROR 500
> >>
> >> Problem accessing /nifi/provenance. Reason:
> >>
> >> Server Error
> >>
> >> Caused by:
> >>
> >> javax.servlet.ServletException: org.eclipse.jetty.servlet.ServletHolder$1:
> java.lang.NullPointerException
> >>   at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:138)
> >>   at org.eclipse.jetty.server.handler.gzip.GzipHandler.
> handle(GzipHandler.java:561)
> >>   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:132)
> >>   at org.eclipse.jetty.server.Server.handle(Server.java:564)
> >>   at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:320)
> >>   at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:251)
> >>   at org.eclipse.jetty.io.AbstractConnection$
> ReadCallback.succeeded(AbstractConnection.java:279)
> >>   at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:110)
> >>   at org.eclipse.jetty.io.ssl.SslConnection.onFillable(
> SslConnection.java:258)
> >>   at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(
> SslConnection.java:147)
> >>   at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:110)
> >>   at org.eclipse.jetty.io.ChannelEndPoint$2.run(
> ChannelEndPoint.java:124)
> >>   at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:672)
> >>   at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(
> QueuedThreadPool.java:590)
> >>   at java.lang.Thread.run(Thread.java:745)
> >> Caused by: org.eclipse.jetty.servlet.ServletHolder$1:
> java.lang.NullPointerException
> >>   at org.eclipse.jetty.servlet.ServletHolder.makeUnavailable(
> ServletHolder.java:596)
> >>   at org.eclipse.jetty.servlet.ServletHolder.initServlet(
> ServletHolder.java:655)
> >>   at org.eclipse.jetty.servlet.ServletHolder.getServlet(
> ServletHolder.java:498)
> >>   at org.eclipse.jetty.servlet.ServletHolder.ensureInstance(
> ServletHolder.java:785)
> >>   at org.eclipse.jetty.servlet.ServletHolder.prepare(
> ServletHolder.java:770)
> >>   at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:538)
> >>   at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
> >>   at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
> >>   at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:132)
> >>   at org.eclipse.jetty.server.handler.ScopedHandler.
> nextHandle(ScopedHandler.java:190)
> >>   at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:1593)
> >>   at org.eclipse.jetty.server.handler.ScopedHandler.
> nextHandle(ScopedHandler.java:188)
> >>   at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1239)
> >>   at org.eclipse.jetty.server.handler.ScopedHandler.
> nextScope(ScopedHandler.java:168)
> >>   at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:481)
> >>   at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:1562)
> >>   at org.eclipse.jetty.server.handler.ScopedHandler.
> nextScope(ScopedHandler.java:166)
> >>   at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1141)
> >>   at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
> >>   at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:118)
> >>   at org.eclipse.jetty.server.handler.gzip.GzipHandler.
> handle(GzipHandler.java:561)
> >>   at org.eclipse.jetty.server.handler.HandlerWrapper

Re: proper way in nifi to sync status between custom processors

[Koji Kawamura]

I see, thanks. The easiest way to look at provenance events would be
by right clicking a processor instance you are interested in, then
select 'View data provenance' context menu. This way, NiFi displays
provenance events for the selected processor.

Koji

On Wed, Dec 27, 2017 at 6:17 PM, 尹文才  wrote:
> Hi Koji, sorry about the provenance exception, it was because there's no
> space left on the machine(filled up with logs)
>
> Regards,
> Ben
>
> 2017-12-27 17:11 GMT+08:00 尹文才 :
>
>> Hi Koji, thanks, the names of the temp tables are created with format
>> "MMddHHmmssSSS-", the first time indicates the time and the second
>> part is a random number with length of 4.
>> So I think it's not possible to have 2 duplicate table names, the only
>> possibly I could think is the flowfile is passed into the processor twice.
>>
>> About the provenance, I had updated to use the
>> WriteAheadProvenanceRepository implementation, but when I tried to check
>> the data provenance, it showed me the following exception message:
>> HTTP ERROR 500
>>
>> Problem accessing /nifi/provenance. Reason:
>>
>> Server Error
>>
>> Caused by:
>>
>> javax.servlet.ServletException: org.eclipse.jetty.servlet.ServletHolder$1: 
>> java.lang.NullPointerException
>>   at 
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:138)
>>   at 
>> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:561)
>>   at 
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>   at org.eclipse.jetty.server.Server.handle(Server.java:564)
>>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
>>   at 
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
>>   at 
>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
>>   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
>>   at 
>> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
>>   at 
>> org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
>>   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
>>   at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
>>   at 
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
>>   at 
>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
>>   at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.eclipse.jetty.servlet.ServletHolder$1: 
>> java.lang.NullPointerException
>>   at 
>> org.eclipse.jetty.servlet.ServletHolder.makeUnavailable(ServletHolder.java:596)
>>   at 
>> org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:655)
>>   at 
>> org.eclipse.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:498)
>>   at 
>> org.eclipse.jetty.servlet.ServletHolder.ensureInstance(ServletHolder.java:785)
>>   at 
>> org.eclipse.jetty.servlet.ServletHolder.prepare(ServletHolder.java:770)
>>   at 
>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:538)
>>   at 
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>>   at 
>> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>>   at 
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>   at 
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
>>   at 
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1593)
>>   at 
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
>>   at 
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1239)
>>   at 
>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
>>   at 
>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481)
>>   at 
>> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1562)
>>   at 
>> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
>>   at 
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1141)
>>   at 
>> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>>   at 
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:118)
>>   at 
>> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:561)
>>   at 
>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>>   at org.eclipse.jetty.server.Server.handle(Server.java:564)
>>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
>>   at 
>> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
>>   at 
>> or

Re: proper way in nifi to sync status between custom processors

[尹文才]

Hi Koji, sorry about the provenance exception, it was because there's no
space left on the machine(filled up with logs)

Regards,
Ben

2017-12-27 17:11 GMT+08:00 尹文才 :

> Hi Koji, thanks, the names of the temp tables are created with format
> "MMddHHmmssSSS-", the first time indicates the time and the second
> part is a random number with length of 4.
> So I think it's not possible to have 2 duplicate table names, the only
> possibly I could think is the flowfile is passed into the processor twice.
>
> About the provenance, I had updated to use the
> WriteAheadProvenanceRepository implementation, but when I tried to check
> the data provenance, it showed me the following exception message:
> HTTP ERROR 500
>
> Problem accessing /nifi/provenance. Reason:
>
> Server Error
>
> Caused by:
>
> javax.servlet.ServletException: org.eclipse.jetty.servlet.ServletHolder$1: 
> java.lang.NullPointerException
>   at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:138)
>   at 
> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:561)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>   at org.eclipse.jetty.server.Server.handle(Server.java:564)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
>   at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
>   at 
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
>   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
>   at 
> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
>   at 
> org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
>   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
>   at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
>   at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.eclipse.jetty.servlet.ServletHolder$1: 
> java.lang.NullPointerException
>   at 
> org.eclipse.jetty.servlet.ServletHolder.makeUnavailable(ServletHolder.java:596)
>   at 
> org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:655)
>   at 
> org.eclipse.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:498)
>   at 
> org.eclipse.jetty.servlet.ServletHolder.ensureInstance(ServletHolder.java:785)
>   at 
> org.eclipse.jetty.servlet.ServletHolder.prepare(ServletHolder.java:770)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:538)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>   at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1593)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1239)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
>   at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481)
>   at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1562)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
>   at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1141)
>   at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>   at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:118)
>   at 
> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:561)
>   at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>   at org.eclipse.jetty.server.Server.handle(Server.java:564)
>   at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
>   at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
>   at 
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
>   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
>   at 
> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
>   at 
> org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
>   at org.eclipse.jetty.io.FillInterest.fillable(FillInterest

Re: proper way in nifi to sync status between custom processors

[尹文才]

Hi Koji, thanks, the names of the temp tables are created with format
"MMddHHmmssSSS-", the first time indicates the time and the second
part is a random number with length of 4.
So I think it's not possible to have 2 duplicate table names, the only
possibly I could think is the flowfile is passed into the processor twice.

About the provenance, I had updated to use the
WriteAheadProvenanceRepository implementation, but when I tried to check
the data provenance, it showed me the following exception message:
HTTP ERROR 500

Problem accessing /nifi/provenance. Reason:

Server Error

Caused by:

javax.servlet.ServletException:
org.eclipse.jetty.servlet.ServletHolder$1:
java.lang.NullPointerException
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:138)
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:561)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at 
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
at 
org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.eclipse.jetty.servlet.ServletHolder$1:
java.lang.NullPointerException
at 
org.eclipse.jetty.servlet.ServletHolder.makeUnavailable(ServletHolder.java:596)
at 
org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:655)
at 
org.eclipse.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:498)
at 
org.eclipse.jetty.servlet.ServletHolder.ensureInstance(ServletHolder.java:785)
at 
org.eclipse.jetty.servlet.ServletHolder.prepare(ServletHolder.java:770)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:538)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1593)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1239)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481)
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1562)
at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1141)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:118)
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:561)
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:564)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at 
org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:258)
at 
org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:147)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at 
org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:122)
at 
org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(ExecutingExecu

Re: proper way in nifi to sync status between custom processors

[Koji Kawamura]

Hi Ben,

The ExecuteSqlCommand retry logic does not execute the same query
multiple times if it succeeds.
So, there must be input FlowFiles containing the same query had been
passed more than once.
It could be the same FlowFile, or different FlowFiles generated by the
first processor for some reason.
To investigate those kind of FlowFile level information, NiFi
provenance data and FlowFile lineage will be very useful.
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#viewing-flowfile-lineage

I didn't mention about it earlier because you were having Provenance
repository performance issue, but I hope you can use it now with the
WriteAheadProvenanceRepository.

Thanks,
Koji

On Wed, Dec 27, 2017 at 5:44 PM, 尹文才  wrote:
> Thanks Koji, for the ExecuteSqlCommand issue, I was trying to re-execute
> the sql query if the connection is lost(connection could be unstable), my
> idea is to only transfer the FlowFile to the success relationship
> after successfully executing the sql query. You could see the do while loop
> in the code, the transaction will be rollbacked if the execution failed; if
> the connection is lost, it will retry to execute the sql.
> Will this logic cause my sql to be executed twice?
>
> For the WaitBatch processor, I will take your approach to test individually
> to see if the WaitBatch processor could cause the FlowFile repository
> checkpointing failure.
>
> Regards,
> Ben
>
> 2017-12-27 16:10 GMT+08:00 Koji Kawamura :
>
>> Hi Ben,
>>
>> Excuse me, I'm trying, but probably I don't fully understand what you
>> want to achieve with the flow.
>>
>> It looks weird that WaitBatch is failing with such FlowFile repository
>> error, while other processor such as ReplaceText succeeds.
>> I recommend to test WaitBatch alone first without combining the
>> database related processors, by feeding a test FlowFile having
>> expected FlowFile attributes.
>> Such input FlowFiles can be created by GenerateFlowFile processor.
>> If the same error happens with only WaitBatch processor, then it
>> should be easier to debug.
>>
>> Thanks,
>> Koji
>>
>> On Wed, Dec 27, 2017 at 4:49 PM, Koji Kawamura 
>> wrote:
>> > Hi Ben,
>> >
>> > The one thing that looks strange in the screenshot is the
>> > ExecuteSqlCommand having FlowFiles queued in its incoming connection.
>> > Those should be transferred to 'failure' relationship.
>> >
>> > Following executeSql() method, shouldn't it re-throw the caught
>> exception?
>> >
>> >
>> > try (Connection con = dbcpService.getConnection()) {
>> > logger.debug("设置autoCommit为false");
>> > con.setAutoCommit(false);
>> >
>> > try (Statement stmt = con.createStatement()) {
>> > logger.info("执行sql语句: {}", new Object[]{sql});
>> > stmt.execute(sql);
>> >
>> > // 所有sql语句执行在一个transaction内
>> > logger.debug("提交transaction");
>> > con.commit();
>> > } catch (Exception ex) {
>> > logger.error("执行sql语句失败:{}", new Object[]{sql, ex});
>> > con.rollback();
>> > //将exception抛到外层处理
>> > throw ex;
>> > } finally {
>> > logger.debug("重新设置autoCommit为true");
>> > con.setAutoCommit(true);
>> > }
>> > } catch (Exception ex) {
>> > // HERE, the exception is swallowed, that's why the FlowFiles stay in
>> > the incoming connection.
>> > logger.error("重试执行sql语句:{}", new Object[]{sql, ex});
>> > retryOnFail = true;
>> > }
>> >
>> > Thanks,
>> > Koji
>> >
>> > On Wed, Dec 27, 2017 at 2:38 PM, 尹文才  wrote:
>> >> Hi Koji, no problem. You could check the code of processor WaitBatch at
>> the
>> >> link:
>> >> https://drive.google.com/open?id=1DMpW5GMiXpyZQdui989Rr3D9rlchQfWQ
>> >>
>> >> I also uploaded a snapshot of part of NiFi flow which includes the
>> >> ExecuteSqlCommand and WaitBatch, you could check the picture at the
>> link:
>> >> https://drive.google.com/file/d/1vdxlWj8ANHQH0CMrXnydLni5o-3IVi2h/view
>> >>
>> >> You mentioned above that FlowFile repository fails checkpointing will
>> >> affect other processors to process same FlowFile again, but as you could
>> >> see from my snapshot image, the ExecuteSqlCommand is the second
>> processor
>> >> and before the WaitBatch processor, even if the FlowFile repository
>> >> checkpointing failure is caused by WaitBatch, could it lead to the
>> >> processors before it to process a FlowFile multiple times? Thanks.
>> >>
>> >> Regards,
>> >> Ben
>> >>
>> >> 2017-12-27 12:36 GMT+08:00 Koji Kawamura :
>> >>
>> >>> Hi Ben,
>> >>>
>> >>> I was referring these two log messages in your previous email.
>> >>> These two messages are both written by ExecuteSqlCommand, it does not
>> >>> mean 'it was executed again'.
>> >>>
>> >>> ```
>> >>> 2017-12-26 07:00:01,312 INFO [Timer-Driven Pro

Re: proper way in nifi to sync status between custom processors

[尹文才]

Thanks Koji, for the ExecuteSqlCommand issue, I was trying to re-execute
the sql query if the connection is lost(connection could be unstable), my
idea is to only transfer the FlowFile to the success relationship
after successfully executing the sql query. You could see the do while loop
in the code, the transaction will be rollbacked if the execution failed; if
the connection is lost, it will retry to execute the sql.
Will this logic cause my sql to be executed twice?

For the WaitBatch processor, I will take your approach to test individually
to see if the WaitBatch processor could cause the FlowFile repository
checkpointing failure.

Regards,
Ben

2017-12-27 16:10 GMT+08:00 Koji Kawamura :

> Hi Ben,
>
> Excuse me, I'm trying, but probably I don't fully understand what you
> want to achieve with the flow.
>
> It looks weird that WaitBatch is failing with such FlowFile repository
> error, while other processor such as ReplaceText succeeds.
> I recommend to test WaitBatch alone first without combining the
> database related processors, by feeding a test FlowFile having
> expected FlowFile attributes.
> Such input FlowFiles can be created by GenerateFlowFile processor.
> If the same error happens with only WaitBatch processor, then it
> should be easier to debug.
>
> Thanks,
> Koji
>
> On Wed, Dec 27, 2017 at 4:49 PM, Koji Kawamura 
> wrote:
> > Hi Ben,
> >
> > The one thing that looks strange in the screenshot is the
> > ExecuteSqlCommand having FlowFiles queued in its incoming connection.
> > Those should be transferred to 'failure' relationship.
> >
> > Following executeSql() method, shouldn't it re-throw the caught
> exception?
> >
> >
> > try (Connection con = dbcpService.getConnection()) {
> > logger.debug("设置autoCommit为false");
> > con.setAutoCommit(false);
> >
> > try (Statement stmt = con.createStatement()) {
> > logger.info("执行sql语句: {}", new Object[]{sql});
> > stmt.execute(sql);
> >
> > // 所有sql语句执行在一个transaction内
> > logger.debug("提交transaction");
> > con.commit();
> > } catch (Exception ex) {
> > logger.error("执行sql语句失败:{}", new Object[]{sql, ex});
> > con.rollback();
> > //将exception抛到外层处理
> > throw ex;
> > } finally {
> > logger.debug("重新设置autoCommit为true");
> > con.setAutoCommit(true);
> > }
> > } catch (Exception ex) {
> > // HERE, the exception is swallowed, that's why the FlowFiles stay in
> > the incoming connection.
> > logger.error("重试执行sql语句:{}", new Object[]{sql, ex});
> > retryOnFail = true;
> > }
> >
> > Thanks,
> > Koji
> >
> > On Wed, Dec 27, 2017 at 2:38 PM, 尹文才  wrote:
> >> Hi Koji, no problem. You could check the code of processor WaitBatch at
> the
> >> link:
> >> https://drive.google.com/open?id=1DMpW5GMiXpyZQdui989Rr3D9rlchQfWQ
> >>
> >> I also uploaded a snapshot of part of NiFi flow which includes the
> >> ExecuteSqlCommand and WaitBatch, you could check the picture at the
> link:
> >> https://drive.google.com/file/d/1vdxlWj8ANHQH0CMrXnydLni5o-3IVi2h/view
> >>
> >> You mentioned above that FlowFile repository fails checkpointing will
> >> affect other processors to process same FlowFile again, but as you could
> >> see from my snapshot image, the ExecuteSqlCommand is the second
> processor
> >> and before the WaitBatch processor, even if the FlowFile repository
> >> checkpointing failure is caused by WaitBatch, could it lead to the
> >> processors before it to process a FlowFile multiple times? Thanks.
> >>
> >> Regards,
> >> Ben
> >>
> >> 2017-12-27 12:36 GMT+08:00 Koji Kawamura :
> >>
> >>> Hi Ben,
> >>>
> >>> I was referring these two log messages in your previous email.
> >>> These two messages are both written by ExecuteSqlCommand, it does not
> >>> mean 'it was executed again'.
> >>>
> >>> ```
> >>> 2017-12-26 07:00:01,312 INFO [Timer-Driven Process Thread-1]
> >>> c.z.nifi.processors.ExecuteSqlCommand
> >>> ExecuteSqlCommand[id=3c97dfd8-aaa4-3a37-626e-fed5a4822d14] 执行sql语句:
> SELECT
> >>> TOP 0 * INTO tmp.ods_extractDataDebug_20171226031801926_9195 FROM
> >>> dbo.ods_extractDataDebug;
> >>> alter table tmp.ods_extractDataDebug_20171226031801926_9195 drop
> column
> >>> _id;
> >>>
> >>> and it was executed again later:
> >>>
> >>> 2017-12-26 07:00:01,315 ERROR [Timer-Driven Process Thread-1]
> >>> c.z.nifi.processors.ExecuteSqlCommand
> >>> ExecuteSqlCommand[id=3c97dfd8-aaa4-3a37-626e-fed5a4822d14]
> >>> 执行sql语句失败:SELECT
> >>> ```
> >>>
> >>> As you written, the case where FlowFile repository fails checkpointing
> >>> will affect other processors to process same FlowFiles again. However
> >>> there won't be a simple solution to every processor to rollback its
> >>> job as different proce

Re: proper way in nifi to sync status between custom processors

[Koji Kawamura]

Hi Ben,

Excuse me, I'm trying, but probably I don't fully understand what you
want to achieve with the flow.

It looks weird that WaitBatch is failing with such FlowFile repository
error, while other processor such as ReplaceText succeeds.
I recommend to test WaitBatch alone first without combining the
database related processors, by feeding a test FlowFile having
expected FlowFile attributes.
Such input FlowFiles can be created by GenerateFlowFile processor.
If the same error happens with only WaitBatch processor, then it
should be easier to debug.

Thanks,
Koji

On Wed, Dec 27, 2017 at 4:49 PM, Koji Kawamura  wrote:
> Hi Ben,
>
> The one thing that looks strange in the screenshot is the
> ExecuteSqlCommand having FlowFiles queued in its incoming connection.
> Those should be transferred to 'failure' relationship.
>
> Following executeSql() method, shouldn't it re-throw the caught exception?
>
>
> try (Connection con = dbcpService.getConnection()) {
> logger.debug("设置autoCommit为false");
> con.setAutoCommit(false);
>
> try (Statement stmt = con.createStatement()) {
> logger.info("执行sql语句: {}", new Object[]{sql});
> stmt.execute(sql);
>
> // 所有sql语句执行在一个transaction内
> logger.debug("提交transaction");
> con.commit();
> } catch (Exception ex) {
> logger.error("执行sql语句失败:{}", new Object[]{sql, ex});
> con.rollback();
> //将exception抛到外层处理
> throw ex;
> } finally {
> logger.debug("重新设置autoCommit为true");
> con.setAutoCommit(true);
> }
> } catch (Exception ex) {
> // HERE, the exception is swallowed, that's why the FlowFiles stay in
> the incoming connection.
> logger.error("重试执行sql语句:{}", new Object[]{sql, ex});
> retryOnFail = true;
> }
>
> Thanks,
> Koji
>
> On Wed, Dec 27, 2017 at 2:38 PM, 尹文才  wrote:
>> Hi Koji, no problem. You could check the code of processor WaitBatch at the
>> link:
>> https://drive.google.com/open?id=1DMpW5GMiXpyZQdui989Rr3D9rlchQfWQ
>>
>> I also uploaded a snapshot of part of NiFi flow which includes the
>> ExecuteSqlCommand and WaitBatch, you could check the picture at the link:
>> https://drive.google.com/file/d/1vdxlWj8ANHQH0CMrXnydLni5o-3IVi2h/view
>>
>> You mentioned above that FlowFile repository fails checkpointing will
>> affect other processors to process same FlowFile again, but as you could
>> see from my snapshot image, the ExecuteSqlCommand is the second processor
>> and before the WaitBatch processor, even if the FlowFile repository
>> checkpointing failure is caused by WaitBatch, could it lead to the
>> processors before it to process a FlowFile multiple times? Thanks.
>>
>> Regards,
>> Ben
>>
>> 2017-12-27 12:36 GMT+08:00 Koji Kawamura :
>>
>>> Hi Ben,
>>>
>>> I was referring these two log messages in your previous email.
>>> These two messages are both written by ExecuteSqlCommand, it does not
>>> mean 'it was executed again'.
>>>
>>> ```
>>> 2017-12-26 07:00:01,312 INFO [Timer-Driven Process Thread-1]
>>> c.z.nifi.processors.ExecuteSqlCommand
>>> ExecuteSqlCommand[id=3c97dfd8-aaa4-3a37-626e-fed5a4822d14] 执行sql语句: SELECT
>>> TOP 0 * INTO tmp.ods_extractDataDebug_20171226031801926_9195 FROM
>>> dbo.ods_extractDataDebug;
>>> alter table tmp.ods_extractDataDebug_20171226031801926_9195 drop column
>>> _id;
>>>
>>> and it was executed again later:
>>>
>>> 2017-12-26 07:00:01,315 ERROR [Timer-Driven Process Thread-1]
>>> c.z.nifi.processors.ExecuteSqlCommand
>>> ExecuteSqlCommand[id=3c97dfd8-aaa4-3a37-626e-fed5a4822d14]
>>> 执行sql语句失败:SELECT
>>> ```
>>>
>>> As you written, the case where FlowFile repository fails checkpointing
>>> will affect other processors to process same FlowFiles again. However
>>> there won't be a simple solution to every processor to rollback its
>>> job as different processors do different things. Creating a temp table
>>> if not exist seems right approach to me.
>>>
>>> At the same time, the route cause of getting FlowFile repository
>>> failed should be investigated. Is it possible to share WaitBatch code?
>>> The reason why ask this is all 'FlowFile Repository failed to update'
>>> is related to WaitBatch processor in the log that you shared earlier.
>>>
>>> Thanks,
>>> Koji
>>>
>>> On Wed, Dec 27, 2017 at 1:19 PM, 尹文才  wrote:
>>> > Hi Koji, I will print the sql before actually executing it, but I checked
>>> > the error log line you mentioned in your reply, this error was thrown by
>>> > NiFi from within another processor called WaitBatch.
>>> > I didn't find similar errors as the one from the ExecuteSqlCommand
>>> > processor, I think it's because only the ExecuteSqlCommand is used to
>>> > create temp database tables.
>>> > You could check my ExecuteSqlCommand code via t