Oracle ODBCConnectionPool driver

[Mark Bean]

What driver is required to connect to an Oracle database using the
DBCPConnectionPool Controller Service? Will the same driver work for both
11g and 12c databases?

Thanks,
Mark

Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?

[Jeff]

A quick update on RC1 for NiFi 1.8.0.  There are some L issues that are
currently being worked, so I'd like to hold off on creating the RC until
they are resolved.  Sorry for the delay!

On Tue, Oct 16, 2018 at 3:46 PM Nathan Gough  wrote:

> So, taking a closer look at this PR and doing some further testing I've
> found that upgrading Guava is going to require some more careful
> assessment, as the changes will affect core NiFi functionality around
> clustering. I will hold off on this change until after the release to have
> more time to upgrade and test.
>
> Nathan
>
> On 10/16/18, 10:58 AM, "Nathan Gough"  wrote:
>
> Hi Mike,
>
> Sure I can look at fixing up PR-2977 today.
>
> Nathan
>
> On 10/16/18, 6:13 AM, "Mike Thomsen"  wrote:
>
> Does 5562 need to be addressed in 1.8?
>
> https://github.com/apache/nifi/pull/2977
>
> On Mon, Oct 15, 2018 at 6:33 PM Jeff  wrote:
>
> > NiFi Devs,
> >
> > The Release page [1] for 1.8.0 now reports that all issues are
> done!  I'd
> > like to start the release candidate preparation tomorrow, around
> 1200 EST.
> >
> > Thanks to everyone for all the great work that's been done!  196
> issues
> > resolved in this version with some great new features!
> >
> > [1]
> https://issues.apache.org/jira/projects/NIFI/versions/12343482
> >
> > On Mon, Oct 15, 2018 at 7:30 AM Sivaprasanna <
> sivaprasanna...@gmail.com>
> > wrote:
> >
> > > Great. Thanks. :)
> > >
> > > -
> > > Sivaprasanna
> > >
> > > On Mon, Oct 15, 2018 at 7:09 AM Koji Kawamura <
> ijokaruma...@gmail.com>
> > > wrote:
> > >
> > > > Jeff, Sivasprasanna,
> > > >
> > > > NIFI-5698 (PR3073) Fixing DeleteAzureBlob bug is merged.
> > > >
> > > > Thanks,
> > > > Koji
> > > > On Mon, Oct 15, 2018 at 10:18 AM Koji Kawamura <
> ijokaruma...@gmail.com
> > >
> > > > wrote:
> > > > >
> > > > > Thank you for the fix Sivaprasanna,
> > > > > I have Azure account. Reviewing it now.
> > > > >
> > > > > Koji
> > > > > On Sun, Oct 14, 2018 at 11:21 PM Jeff 
> wrote:
> > > > > >
> > > > > > Sivaprasanna,
> > > > > >
> > > > > > Thanks for submitting a pull request for that issue!
> Later today
> > or
> > > > > > tomorrow I'll have to check to see if I've already used
> up my
> > > free-tier
> > > > > > access to Azure.  If I still have access, I can review
> your PR and
> > > > we'll
> > > > > > get it into 1.8.0.
> > > > > >
> > > > > > On Sun, Oct 14, 2018 at 4:30 AM Sivaprasanna <
> > > > sivaprasanna...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > All - Just found one bug with DeleteAzureBlobStorage
> processor.
> > It
> > > > was
> > > > > > > shared by one user on StackOverflow [1] and I later
> confirmed it.
> > > It
> > > > looks
> > > > > > > to be introduced by NIFI-4199. I have created a Jira
> [2] and made
> > > the
> > > > > > > necessary changes (not huge, just few lines) and
> raised a PR
> > [3]. I
> > > > think,
> > > > > > > if we can spend a little time in getting it reviewed,
> we can mark
> > > it
> > > > for
> > > > > > > 1.8.0. Thoughts?
> > > > > > >
> > > > > > > [1] -
> > > > > > >
> > > > > > >
> > > >
> > >
> >
> https://stackoverflow.com/questions/52766991/apache-nifi-deleteazureblobstorage-processor-is-throwing-an-error
> > > > > > > [2] - https://issues.apache.org/jira/browse/NIFI-5698
> > > > > > > [3] - https://github.com/apache/nifi/pull/3073
> > > > > > >
> > > > > > > -
> > > > > > > Sivaprasanna
> > > > > > >
> > > > > > > On Fri, Oct 12, 2018 at 9:05 PM Mike Thomsen <
> > > mikerthom...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > 4811 should be ready for review now. Rebased and
> cleaned it up
> > > > with a
> > > > > > > full
> > > > > > > > listing of the Spring dependencies.
> > > > > > > >
> > > > > > > > On Fri, Oct 12, 2018 at 11:23 AM Joe Witt <
> joe.w...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > > > Jeff,
> > > > > > > > >
> > > > > > > > > I think for anything not tagged to 1.8.0 we just
> keep
> > rolling.
> > > > For
> > > > > > > > > anything tagged 1.8.0 that should not be we should
> remove it
> > > > until
> > > > > > > > > ready.  For things tagged to 1.8.0 that cannot be
> moved we
> > > 

Re: Compiling custom processor

[Bryan Bende]

Any reason why the commons-lang3 dependency is provided and not normal
compile scope?

Generally the only cases where a dependency should be provided are
when a parent NAR provides it (the API scenario earlier), or when it
is a jar directly in lib which means it is on the classpath of
everything (like nifi-api or slf4j).
On Tue, Oct 16, 2018 at 3:47 PM John McGinn  wrote:
>
> I have a directory /home/user/nifi-nars, which is listed in the 
> nifi.properties file located under the nifi-assembly directory from the 
> nifi-master code. That's where I redropped my NAR file at.
>
> The error was "due to uncaught Exception: java.lang.NoClassDefFoundError: 
> org/apache/commons/lang3/StringUtils". I added a provided dependency of 
> version 3.4 for commons-lang3 in my processors pom.xml. That worked 
> initially, but with the redropping of the NAR file, the error occurs. The 
> stacktrace points to the session.get() lambda function trying to use 
> StringUtils.isBlank(). (Again, this is a copy and renamed class of the Wait 
> Processor. Trying to turn it into a Yield, as it were, hold unless it's free 
> to move forward.)
> 
> On Tue, 10/16/18, Bryan Bende  wrote:
>
>  Subject: Re: Compiling custom processor
>  To: dev@nifi.apache.org
>  Date: Tuesday, October 16, 2018, 3:34 PM
>
>  There shouldn't be any issue
>  doing that, generally you just copy the
>  new
>  NAR into the lib directory and restart.
>
>  What were the specifics of the class not found
>  error?
>
>  On Tue, Oct 16, 2018 at
>  3:24 PM John McGinn 
>  wrote:
>  >
>  > And I'm
>  back.
>  >
>  > So, I had my
>  processor running, no errors, so things were good.
>  >
>  > I then add 2 if
>  statements with logger messages, run the same mvn command,
>  copied the new NAR on top of the old one, same versionId and
>  all, and now I get a class not found error, that didn't
>  occur before I stopped, redeployed and started. And I
>  can't delete the event in the queue even though source
>  and destination are stopped.
>  >
>  > Is there problems when redeploying a
>  custom processor NAR on top of itself? Even if the NiFi
>  instance is stopped? Serialization issues?
>  >
>  > Frustratedly
>  yours,
>  > John McGinn
>  >
>  >
>  
>  > On Tue, 10/16/18, John McGinn 
>  wrote:
>  >
>  >  Subject:
>  Re: Compiling custom processor
>  >  To: dev@nifi.apache.org
>  >  Date: Tuesday, October 16, 2018, 1:21
>  PM
>  >
>  >  Thanks
>  Bryan,
>  >
>  >  I
>  restarted with a fresh maven
>  >
>  generate, and only added the API to the processor pom,
>  and
>  >  then the API-NAR to the NAR pom.
>  I then figured out a
>  >  subsequent
>  dependency (lang3), and got that in, and now my
>  >  processor shows up in local NiFi
>  instance.
>  >
>  >
>  Regarding the copying of the standard
>  >
>  processor, when I copied Wait over, I renamed it from
>  Wait
>  >  to MyProcessorWait to be unique,
>  and not copied. I'm just
>  >
>  attempting changes to Wait without modifying the actual
>  one.
>  >  Also, thanks on the
>  META-INF/services file information.
>  >
>  >  Now I can debug and test.
>  >
>  >  Sincerely,
>  >  John McGinn
>  >
>  >
>  
>  >  On Tue, 10/16/18, Bryan Bende 
>  >  wrote:
>  >
>  >   Subject: Re: Compiling custom
>  >  processor
>  >   To: dev@nifi.apache.org
>  >   Date: Tuesday, October 16, 2018,
>  9:52
>  >  AM
>  >
>  >   In general, if your processor
>  >   uses a controller service then the
>  >   processors pom file needs a provided
>  >  dependency
>  >   on
>  the API of the CS,
>  >   and your NAR pom
>  needs
>  >   a NAR dependency on the NAR
>  where the
>  >  CS API
>  >   is.
>  >
>  >   Example is
>  >
>  shown here in the section linking
>  >
>  processors and
>  >   controller
>  >   services - 
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
>  >
>  >   Also, you do not
>  want to
>  >   include NiFI's standard
>  processors in
>  >  your
>  >   own NAR, this will result in two
>  >  copies of
>  >   every
>  standard processor.
>  >   You will want
>  to
>  >   copy whatever code you need into
>  your
>  >  own NAR.
>  >
>  >   In you last statement, if you
>  >   made a new processor in
>  >   nifi-standard-processors and it
>  didn't
>  >  show
>  >   up, it is most likely
>  >   because you didn't
>  >   update META-INF/services file to
>  >  include the new
>  >
>  processor.
>  >   On Tue, Oct 16, 2018
>  at
>  >   9:36 AM John McGinn
>  >   
>  >   wrote:
>  >   >
>  >   > Ok, I am
>  >
>  far from a maven expert, and am
>  >
>  struggling on this
>  >   problem.
>  >   >
>  >   > I
>  created
>  >   a new project using the
>  maven generate
>  >  process, and
>  compiled
>  >   that sample processor and
>  everything
>  >  was fine, and I could
>  >   see it in my local NiFi instance. I
>  >  then copied over the
>  >   Wait processor on top of the
>  

Re: Compiling custom processor

[John McGinn]

I have a directory /home/user/nifi-nars, which is listed in the nifi.properties 
file located under the nifi-assembly directory from the nifi-master code. 
That's where I redropped my NAR file at.

The error was "due to uncaught Exception: java.lang.NoClassDefFoundError: 
org/apache/commons/lang3/StringUtils". I added a provided dependency of version 
3.4 for commons-lang3 in my processors pom.xml. That worked initially, but with 
the redropping of the NAR file, the error occurs. The stacktrace points to the 
session.get() lambda function trying to use StringUtils.isBlank(). (Again, this 
is a copy and renamed class of the Wait Processor. Trying to turn it into a 
Yield, as it were, hold unless it's free to move forward.)

On Tue, 10/16/18, Bryan Bende  wrote:

 Subject: Re: Compiling custom processor
 To: dev@nifi.apache.org
 Date: Tuesday, October 16, 2018, 3:34 PM
 
 There shouldn't be any issue
 doing that, generally you just copy the
 new
 NAR into the lib directory and restart.
 
 What were the specifics of the class not found
 error?
 
 On Tue, Oct 16, 2018 at
 3:24 PM John McGinn 
 wrote:
 >
 > And I'm
 back.
 >
 > So, I had my
 processor running, no errors, so things were good.
 >
 > I then add 2 if
 statements with logger messages, run the same mvn command,
 copied the new NAR on top of the old one, same versionId and
 all, and now I get a class not found error, that didn't
 occur before I stopped, redeployed and started. And I
 can't delete the event in the queue even though source
 and destination are stopped.
 >
 > Is there problems when redeploying a
 custom processor NAR on top of itself? Even if the NiFi
 instance is stopped? Serialization issues?
 >
 > Frustratedly
 yours,
 > John McGinn
 >
 >
 
 > On Tue, 10/16/18, John McGinn 
 wrote:
 >
 >  Subject:
 Re: Compiling custom processor
 >  To: dev@nifi.apache.org
 >  Date: Tuesday, October 16, 2018, 1:21
 PM
 >
 >  Thanks
 Bryan,
 >
 >  I
 restarted with a fresh maven
 > 
 generate, and only added the API to the processor pom,
 and
 >  then the API-NAR to the NAR pom.
 I then figured out a
 >  subsequent
 dependency (lang3), and got that in, and now my
 >  processor shows up in local NiFi
 instance.
 >
 > 
 Regarding the copying of the standard
 > 
 processor, when I copied Wait over, I renamed it from
 Wait
 >  to MyProcessorWait to be unique,
 and not copied. I'm just
 > 
 attempting changes to Wait without modifying the actual
 one.
 >  Also, thanks on the
 META-INF/services file information.
 >
 >  Now I can debug and test.
 >
 >  Sincerely,
 >  John McGinn
 >
 > 
 
 >  On Tue, 10/16/18, Bryan Bende 
 >  wrote:
 >
 >   Subject: Re: Compiling custom
 >  processor
 >   To: dev@nifi.apache.org
 >   Date: Tuesday, October 16, 2018,
 9:52
 >  AM
 >
 >   In general, if your processor
 >   uses a controller service then the
 >   processors pom file needs a provided
 >  dependency
 >   on
 the API of the CS,
 >   and your NAR pom
 needs
 >   a NAR dependency on the NAR
 where the
 >  CS API
 >   is.
 >
 >   Example is
 >  
 shown here in the section linking
 > 
 processors and
 >   controller
 >   services - 
 >https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
 >
 >   Also, you do not
 want to
 >   include NiFI's standard
 processors in
 >  your
 >   own NAR, this will result in two
 >  copies of
 >   every
 standard processor.
 >   You will want
 to
 >   copy whatever code you need into
 your
 >  own NAR.
 >
 >   In you last statement, if you
 >   made a new processor in
 >   nifi-standard-processors and it
 didn't
 >  show
 >   up, it is most likely
 >   because you didn't
 >   update META-INF/services file to
 >  include the new
 >  
 processor.
 >   On Tue, Oct 16, 2018
 at
 >   9:36 AM John McGinn
 >   
 >   wrote:
 >   >
 >   > Ok, I am
 >  
 far from a maven expert, and am
 > 
 struggling on this
 >   problem.
 >   >
 >   > I
 created
 >   a new project using the
 maven generate
 >  process, and
 compiled
 >   that sample processor and
 everything
 >  was fine, and I could
 >   see it in my local NiFi instance. I
 >  then copied over the
 >   Wait processor on top of the
 >  MyProcessor.java class, changed
 >   the package name, and the class
 name,
 >  and attempted to
 >   compile. Got errors due to
 dependency
 >  issues. This is where
 >   I get confused.
 >  
 >
 >   > In
 >  
 the processor directory, I modify the
 > 
 pom.xml with a
 >   dependency line for
 nifi-api,
 >  nifi-utils,
 >
 > 
 nifi-distribute-cache-client-service-api and
 >   nifi-standard-processors. This
 seemed
 >  to make things work,
 >   and I get a 33 meg NAR file, which
 >  contains lots of JARs. I
 >   load that up, and I get a problem
 with
 >  CalculateStats, or
 >   similar, because of
 >  RecordReaderFactory. I go back in and
 >   include
 > 
 nifi-record-serialization-services, and it compiles,

Re: Compiling custom processor

[Bryan Bende]

There shouldn't be any issue doing that, generally you just copy the
new NAR into the lib directory and restart.

What were the specifics of the class not found error?

On Tue, Oct 16, 2018 at 3:24 PM John McGinn  wrote:
>
> And I'm back.
>
> So, I had my processor running, no errors, so things were good.
>
> I then add 2 if statements with logger messages, run the same mvn command, 
> copied the new NAR on top of the old one, same versionId and all, and now I 
> get a class not found error, that didn't occur before I stopped, redeployed 
> and started. And I can't delete the event in the queue even though source and 
> destination are stopped.
>
> Is there problems when redeploying a custom processor NAR on top of itself? 
> Even if the NiFi instance is stopped? Serialization issues?
>
> Frustratedly yours,
> John McGinn
>
> 
> On Tue, 10/16/18, John McGinn  wrote:
>
>  Subject: Re: Compiling custom processor
>  To: dev@nifi.apache.org
>  Date: Tuesday, October 16, 2018, 1:21 PM
>
>  Thanks Bryan,
>
>  I restarted with a fresh maven
>  generate, and only added the API to the processor pom, and
>  then the API-NAR to the NAR pom. I then figured out a
>  subsequent dependency (lang3), and got that in, and now my
>  processor shows up in local NiFi instance.
>
>  Regarding the copying of the standard
>  processor, when I copied Wait over, I renamed it from Wait
>  to MyProcessorWait to be unique, and not copied. I'm just
>  attempting changes to Wait without modifying the actual one.
>  Also, thanks on the META-INF/services file information.
>
>  Now I can debug and test.
>
>  Sincerely,
>  John McGinn
>
>  
>  On Tue, 10/16/18, Bryan Bende 
>  wrote:
>
>   Subject: Re: Compiling custom
>  processor
>   To: dev@nifi.apache.org
>   Date: Tuesday, October 16, 2018, 9:52
>  AM
>
>   In general, if your processor
>   uses a controller service then the
>   processors pom file needs a provided
>  dependency
>   on the API of the CS,
>   and your NAR pom needs
>   a NAR dependency on the NAR where the
>  CS API
>   is.
>
>   Example is
>   shown here in the section linking
>  processors and
>   controller
>   services - 
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
>
>   Also, you do not want to
>   include NiFI's standard processors in
>  your
>   own NAR, this will result in two
>  copies of
>   every standard processor.
>   You will want to
>   copy whatever code you need into your
>  own NAR.
>
>   In you last statement, if you
>   made a new processor in
>   nifi-standard-processors and it didn't
>  show
>   up, it is most likely
>   because you didn't
>   update META-INF/services file to
>  include the new
>   processor.
>   On Tue, Oct 16, 2018 at
>   9:36 AM John McGinn
>   
>   wrote:
>   >
>   > Ok, I am
>   far from a maven expert, and am
>  struggling on this
>   problem.
>   >
>   > I created
>   a new project using the maven generate
>  process, and compiled
>   that sample processor and everything
>  was fine, and I could
>   see it in my local NiFi instance. I
>  then copied over the
>   Wait processor on top of the
>  MyProcessor.java class, changed
>   the package name, and the class name,
>  and attempted to
>   compile. Got errors due to dependency
>  issues. This is where
>   I get confused.
>   >
>   > In
>   the processor directory, I modify the
>  pom.xml with a
>   dependency line for nifi-api,
>  nifi-utils,
>
>  nifi-distribute-cache-client-service-api and
>   nifi-standard-processors. This seemed
>  to make things work,
>   and I get a 33 meg NAR file, which
>  contains lots of JARs. I
>   load that up, and I get a problem with
>  CalculateStats, or
>   similar, because of
>  RecordReaderFactory. I go back in and
>   include
>  nifi-record-serialization-services, and it compiles,
>   and the JAR is included in the NAR
>  file now, and I still get
>   class not found.
>   >
>   >
>   I've also tried to change the
>  dependencies to a status
>   of provided, and therefore get a
>  minimal 25k or so NAR file
>   with no JARs included. I get the same
>  issue with the class
>   not found.
>   >
>   > I had
>   also attempted to copy Wait.java to
>  AnotherWait.java within
>   the actually NiFi source code, and it
>  compiles, and shows up
>   in the JARs and NARs as I'd expect,
>  but I cannot get to
>   it within my NiFi instance.
>   >
>   > Can someone let me know what I'm
>  doing
>   wrong.
>   >
>   > Thanks,
>   > John McGinn
>

Re: License of NLKBufferedReader

[Joe Witt]

Yeah that looks super sketchy frankly and we need to address that immediately.

JeffStorck: This is an RC blocker.  I'll file a JIRA shortly.  I am
shocked this hasn't been caught before.
On Tue, Oct 16, 2018 at 3:11 PM Peter Wicks (pwicks)  wrote:
>
> While digging into ReplaceText today, I noticed that 
> org.apache.nifi.processors.standard.util.NLKBufferedReader is actually a 
> direct copy of the JDK version of BufferedReader, with a few minor code edits 
> made on top of it so that new line characters are kept. This code was in the 
> original code commit from way back.
>
> It feels like there is a licensing issue here, as this is Oracle source code. 
> Joe?
>
> --Peter
>

Re: Compiling custom processor

[John McGinn]

And I'm back.

So, I had my processor running, no errors, so things were good.

I then add 2 if statements with logger messages, run the same mvn command, 
copied the new NAR on top of the old one, same versionId and all, and now I get 
a class not found error, that didn't occur before I stopped, redeployed and 
started. And I can't delete the event in the queue even though source and 
destination are stopped.

Is there problems when redeploying a custom processor NAR on top of itself? 
Even if the NiFi instance is stopped? Serialization issues?

Frustratedly yours,
John McGinn


On Tue, 10/16/18, John McGinn  wrote:

 Subject: Re: Compiling custom processor
 To: dev@nifi.apache.org
 Date: Tuesday, October 16, 2018, 1:21 PM
 
 Thanks Bryan,
 
 I restarted with a fresh maven
 generate, and only added the API to the processor pom, and
 then the API-NAR to the NAR pom. I then figured out a
 subsequent dependency (lang3), and got that in, and now my
 processor shows up in local NiFi instance.
 
 Regarding the copying of the standard
 processor, when I copied Wait over, I renamed it from Wait
 to MyProcessorWait to be unique, and not copied. I'm just
 attempting changes to Wait without modifying the actual one.
 Also, thanks on the META-INF/services file information.
 
 Now I can debug and test.
 
 Sincerely,
 John McGinn
 
 
 On Tue, 10/16/18, Bryan Bende 
 wrote:
 
  Subject: Re: Compiling custom
 processor
  To: dev@nifi.apache.org
  Date: Tuesday, October 16, 2018, 9:52
 AM
  
  In general, if your processor
  uses a controller service then the
  processors pom file needs a provided
 dependency
  on the API of the CS,
  and your NAR pom needs
  a NAR dependency on the NAR where the
 CS API
  is.
  
  Example is
  shown here in the section linking
 processors and
  controller
  services - 
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
  
  Also, you do not want to
  include NiFI's standard processors in
 your
  own NAR, this will result in two
 copies of
  every standard processor.
  You will want to
  copy whatever code you need into your
 own NAR.
  
  In you last statement, if you
  made a new processor in
  nifi-standard-processors and it didn't
 show
  up, it is most likely
  because you didn't
  update META-INF/services file to
 include the new
  processor.
  On Tue, Oct 16, 2018 at
  9:36 AM John McGinn
  
  wrote:
  >
  > Ok, I am
  far from a maven expert, and am
 struggling on this
  problem.
  >
  > I created
  a new project using the maven generate
 process, and compiled
  that sample processor and everything
 was fine, and I could
  see it in my local NiFi instance. I
 then copied over the
  Wait processor on top of the
 MyProcessor.java class, changed
  the package name, and the class name,
 and attempted to
  compile. Got errors due to dependency
 issues. This is where
  I get confused.
  >
  > In
  the processor directory, I modify the
 pom.xml with a
  dependency line for nifi-api,
 nifi-utils,
 
 nifi-distribute-cache-client-service-api and
  nifi-standard-processors. This seemed
 to make things work,
  and I get a 33 meg NAR file, which
 contains lots of JARs. I
  load that up, and I get a problem with
 CalculateStats, or
  similar, because of
 RecordReaderFactory. I go back in and
  include
 nifi-record-serialization-services, and it compiles,
  and the JAR is included in the NAR
 file now, and I still get
  class not found.
  >
  >
  I've also tried to change the
 dependencies to a status
  of provided, and therefore get a
 minimal 25k or so NAR file
  with no JARs included. I get the same
 issue with the class
  not found.
  >
  > I had
  also attempted to copy Wait.java to
 AnotherWait.java within
  the actually NiFi source code, and it
 compiles, and shows up
  in the JARs and NARs as I'd expect,
 but I cannot get to
  it within my NiFi instance.
  >
  > Can someone let me know what I'm
 doing
  wrong.
  >
  > Thanks,
  > John McGinn
 

License of NLKBufferedReader

[Peter Wicks (pwicks)]

While digging into ReplaceText today, I noticed that 
org.apache.nifi.processors.standard.util.NLKBufferedReader is actually a direct 
copy of the JDK version of BufferedReader, with a few minor code edits made on 
top of it so that new line characters are kept. This code was in the original 
code commit from way back.

It feels like there is a licensing issue here, as this is Oracle source code. 
Joe?

--Peter

RE: [EXT] Calcite version used in Nifi 1.7+

[Ruurd Schoonheim]

Thnx Peter!

Then, somehow CROSS APPLY should work in Nifi 1.7.x+... 樂
On 16 Oct 2018, 15:38 +0200, Peter Wicks (pwicks) , wrote:
> In NiFi 1.7.0 it was Calcite v1.12.0. In NiFi 1.8 it will be Calcite v1.17.0
>
> -Original Message-
> From: ruurd.schoonh...@dbiq.nl [mailto:ruurd.schoonh...@dbiq.nl]
> Sent: Tuesday, October 16, 2018 6:58 AM
> To: dev@nifi.apache.org
> Subject: [EXT] Calcite version used in Nifi 1.7+
>
> Hi guys,
>
> Which version of calcite is used by QueryRecord processor in Nifi 1.7 or 
> above?
>
> I’m looking for CROSS APPLY suport in my query as introduced in Calcite 1.11 
> (january 2017)
>
> Or has anyone a solution to perform select on self joined dataset (recursive)
>
> Kind regards
> Ruurd Schoonheim

Re: Compiling custom processor

[John McGinn]

Thanks Bryan,

I restarted with a fresh maven generate, and only added the API to the 
processor pom, and then the API-NAR to the NAR pom. I then figured out a 
subsequent dependency (lang3), and got that in, and now my processor shows up 
in local NiFi instance.

Regarding the copying of the standard processor, when I copied Wait over, I 
renamed it from Wait to MyProcessorWait to be unique, and not copied. I'm just 
attempting changes to Wait without modifying the actual one. Also, thanks on 
the META-INF/services file information.

Now I can debug and test.

Sincerely,
John McGinn


On Tue, 10/16/18, Bryan Bende  wrote:

 Subject: Re: Compiling custom processor
 To: dev@nifi.apache.org
 Date: Tuesday, October 16, 2018, 9:52 AM
 
 In general, if your processor
 uses a controller service then the
 processors pom file needs a provided dependency
 on the API of the CS,
 and your NAR pom needs
 a NAR dependency on the NAR where the CS API
 is.
 
 Example is
 shown here in the section linking processors and
 controller
 services - 
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
 
 Also, you do not want to
 include NiFI's standard processors in your
 own NAR, this will result in two copies of
 every standard processor.
 You will want to
 copy whatever code you need into your own NAR.
 
 In you last statement, if you
 made a new processor in
 nifi-standard-processors and it didn't show
 up, it is most likely
 because you didn't
 update META-INF/services file to include the new
 processor.
 On Tue, Oct 16, 2018 at
 9:36 AM John McGinn
 
 wrote:
 >
 > Ok, I am
 far from a maven expert, and am struggling on this
 problem.
 >
 > I created
 a new project using the maven generate process, and compiled
 that sample processor and everything was fine, and I could
 see it in my local NiFi instance. I then copied over the
 Wait processor on top of the MyProcessor.java class, changed
 the package name, and the class name, and attempted to
 compile. Got errors due to dependency issues. This is where
 I get confused.
 >
 > In
 the processor directory, I modify the pom.xml with a
 dependency line for nifi-api, nifi-utils,
 nifi-distribute-cache-client-service-api and
 nifi-standard-processors. This seemed to make things work,
 and I get a 33 meg NAR file, which contains lots of JARs. I
 load that up, and I get a problem with CalculateStats, or
 similar, because of RecordReaderFactory. I go back in and
 include nifi-record-serialization-services, and it compiles,
 and the JAR is included in the NAR file now, and I still get
 class not found.
 >
 >
 I've also tried to change the dependencies to a status
 of provided, and therefore get a minimal 25k or so NAR file
 with no JARs included. I get the same issue with the class
 not found.
 >
 > I had
 also attempted to copy Wait.java to AnotherWait.java within
 the actually NiFi source code, and it compiles, and shows up
 in the JARs and NARs as I'd expect, but I cannot get to
 it within my NiFi instance.
 >
 > Can someone let me know what I'm doing
 wrong.
 >
 > Thanks,
 > John McGinn

Re: Fixing unstable nifi cluster.

[Jeff]

Hello,

Pierre's suggestions should be helpful for you, since you are running
Zookeeper on the same nodes as NiFi.  If it's possible for you to run
Zookeeper on separate hosts from NiFi so that ZK and NiFi are not
colocated, you should see better results.

- Jeff

On Tue, Oct 16, 2018 at 8:03 AM Pierre Villard 
wrote:

> Hi,
>
> Can you try increasing the below parameters? That's usually what I
> recommend, our default values being probably a bit too aggressively low.
>
> nifi.zookeeper.connect.timeout=15 secs
> nifi.zookeeper.session.timeout=15 secs
> nifi.cluster.node.read.timeout=30 sec
>
> Pierre
>
> Le mar. 16 oct. 2018 à 13:02, ashwin konale  a
> écrit :
>
> > Hi,
> > We have a 3 node nifi cluster (With separate zookeper instances running
> in
> > the same machines) which pulls the data from mysql and write to hdfs. I
> am
> > frequently running into problems with cluster. Nodes keeps disconnecting
> > from each other, primary nodes keeps switching and sometimes it just goes
> > into zombie state when I just cannot access the ui. I have followed best
> > practices guide and tweaked params in nifi.properties, have switched
> > provenanceRepositoryImplementation to volatile because cluster was not
> able
> > to keep up with incoming traffic. Data traffic is not high at all
> (4Mbps).
> > This is the message I frequently get from the logs.
> >
> > *INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State
> change:
> > LOST*
> > *INFO [Curator-ConnectionStateManager-0]
> > o.a.n.c.l.e.CuratorLeaderElectionManager
> >
> >
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@56ebedec
> > Connection State changed to LOST*
> > *INFO [Curator-ConnectionStateManager-0]
> > o.a.n.c.l.e.CuratorLeaderElectionManager
> >
> >
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1b0e2055
> > Connection State changed to LOST*
> > *INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State
> change:
> > RECONNECTED*
> >
> > Am I doing something wrong with cluster setup ? Can someone give me some
> > guidance on how to go about debugging this issue ? What kind of system
> > metrics to look at etc.
> >
> > Ashwin
> >
>

Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?

[Andrew Lim]

I created a Jira to document the new load balancing functionality [1] and just 
submitted a PR for it [2].

I think it would be beneficial for users if it could be reviewed and included 
in the release.

[1] https://issues.apache.org/jira/browse/NIFI-5701
[2] https://github.com/apache/nifi/pull/3080

Thanks!

-Drew

> On Oct 16, 2018, at 11:13 AM, Jeff  wrote:
> 
> Since NIFI-5562 is an improvement JIRA, and we're cutting it very close to
> the release candidate, it may not make it into the release.  However, since
> Mark Payne is investigating some issues that may have been introduced in
> 1.8.0, I can delay creating RC1.
> 
> On Tue, Oct 16, 2018 at 10:24 AM Mark Payne  wrote:
> 
>> Jeff / all,
>> 
>> I ran into an issue with NIFI-375 and re-opened the ticket. If a processor
>> is stopped or started in a cluster,
>> the stats that come back in the response are incorrect because the
>> response is not being properly merged
>> from all nodes in the cluster.
>> 
>> I also have run into a couple of other issues that appear to have been
>> introduced in 1.8. I have not yet created JIRA's
>> for them because I want to understand the issues better before trying to
>> write it up. I am seeing, for example, some errors
>> in the logs indicating that provenance events don't have a FlowFile UUID
>> assigned to them when being read from the
>> repository. This seems to have been introduced in 1.8 so would like to get
>> a resolution before releasing.
>> 
>> Thanks
>> -Mark
>> 
>> 
>>> On Oct 15, 2018, at 6:33 PM, Jeff  wrote:
>>> 
>>> NiFi Devs,
>>> 
>>> The Release page [1] for 1.8.0 now reports that all issues are done!  I'd
>>> like to start the release candidate preparation tomorrow, around 1200
>> EST.
>>> 
>>> Thanks to everyone for all the great work that's been done!  196 issues
>>> resolved in this version with some great new features!
>>> 
>>> [1] https://issues.apache.org/jira/projects/NIFI/versions/12343482
>>> 
>>> On Mon, Oct 15, 2018 at 7:30 AM Sivaprasanna 
>>> wrote:
>>> 
 Great. Thanks. :)
 
 -
 Sivaprasanna
 
 On Mon, Oct 15, 2018 at 7:09 AM Koji Kawamura 
 wrote:
 
> Jeff, Sivasprasanna,
> 
> NIFI-5698 (PR3073) Fixing DeleteAzureBlob bug is merged.
> 
> Thanks,
> Koji
> On Mon, Oct 15, 2018 at 10:18 AM Koji Kawamura >> 
> wrote:
>> 
>> Thank you for the fix Sivaprasanna,
>> I have Azure account. Reviewing it now.
>> 
>> Koji
>> On Sun, Oct 14, 2018 at 11:21 PM Jeff  wrote:
>>> 
>>> Sivaprasanna,
>>> 
>>> Thanks for submitting a pull request for that issue!  Later today or
>>> tomorrow I'll have to check to see if I've already used up my
 free-tier
>>> access to Azure.  If I still have access, I can review your PR and
> we'll
>>> get it into 1.8.0.
>>> 
>>> On Sun, Oct 14, 2018 at 4:30 AM Sivaprasanna <
> sivaprasanna...@gmail.com>
>>> wrote:
>>> 
 All - Just found one bug with DeleteAzureBlobStorage processor. It
> was
 shared by one user on StackOverflow [1] and I later confirmed it.
 It
> looks
 to be introduced by NIFI-4199. I have created a Jira [2] and made
 the
 necessary changes (not huge, just few lines) and raised a PR [3]. I
> think,
 if we can spend a little time in getting it reviewed, we can mark
 it
> for
 1.8.0. Thoughts?
 
 [1] -
 
 
> 
 
>> https://stackoverflow.com/questions/52766991/apache-nifi-deleteazureblobstorage-processor-is-throwing-an-error
 [2] - https://issues.apache.org/jira/browse/NIFI-5698
 [3] - https://github.com/apache/nifi/pull/3073
 
 -
 Sivaprasanna
 
 On Fri, Oct 12, 2018 at 9:05 PM Mike Thomsen <
 mikerthom...@gmail.com
>> 
 wrote:
 
> 4811 should be ready for review now. Rebased and cleaned it up
> with a
 full
> listing of the Spring dependencies.
> 
> On Fri, Oct 12, 2018 at 11:23 AM Joe Witt 
> wrote:
> 
>> Jeff,
>> 
>> I think for anything not tagged to 1.8.0 we just keep rolling.
> For
>> anything tagged 1.8.0 that should not be we should remove it
> until
>> ready.  For things tagged to 1.8.0 that cannot be moved we
 should
>> resolve.  For the tagged 1.8.0 section you had.
>> 
>>  - NIFI-4811 <
 https://issues.apache.org/jira/browse/NIFI-4811>
> -
 Use a
>>  newer version of spring-data-redis
>>  - PR 2856 
>> *This needs to be resolved by either reverting the commit or
> ensuring
>> L accurately reflects all.  We have to do this always and for
> every
>> nar.  The process isnt easy or fun but it is necessary to
 produce
>> valid ASF releases.  Landing commits which 

Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?

[Jeff]

Since NIFI-5562 is an improvement JIRA, and we're cutting it very close to
the release candidate, it may not make it into the release.  However, since
Mark Payne is investigating some issues that may have been introduced in
1.8.0, I can delay creating RC1.

On Tue, Oct 16, 2018 at 10:24 AM Mark Payne  wrote:

> Jeff / all,
>
> I ran into an issue with NIFI-375 and re-opened the ticket. If a processor
> is stopped or started in a cluster,
> the stats that come back in the response are incorrect because the
> response is not being properly merged
> from all nodes in the cluster.
>
> I also have run into a couple of other issues that appear to have been
> introduced in 1.8. I have not yet created JIRA's
> for them because I want to understand the issues better before trying to
> write it up. I am seeing, for example, some errors
> in the logs indicating that provenance events don't have a FlowFile UUID
> assigned to them when being read from the
> repository. This seems to have been introduced in 1.8 so would like to get
> a resolution before releasing.
>
> Thanks
> -Mark
>
>
> > On Oct 15, 2018, at 6:33 PM, Jeff  wrote:
> >
> > NiFi Devs,
> >
> > The Release page [1] for 1.8.0 now reports that all issues are done!  I'd
> > like to start the release candidate preparation tomorrow, around 1200
> EST.
> >
> > Thanks to everyone for all the great work that's been done!  196 issues
> > resolved in this version with some great new features!
> >
> > [1] https://issues.apache.org/jira/projects/NIFI/versions/12343482
> >
> > On Mon, Oct 15, 2018 at 7:30 AM Sivaprasanna 
> > wrote:
> >
> >> Great. Thanks. :)
> >>
> >> -
> >> Sivaprasanna
> >>
> >> On Mon, Oct 15, 2018 at 7:09 AM Koji Kawamura 
> >> wrote:
> >>
> >>> Jeff, Sivasprasanna,
> >>>
> >>> NIFI-5698 (PR3073) Fixing DeleteAzureBlob bug is merged.
> >>>
> >>> Thanks,
> >>> Koji
> >>> On Mon, Oct 15, 2018 at 10:18 AM Koji Kawamura  >
> >>> wrote:
> 
>  Thank you for the fix Sivaprasanna,
>  I have Azure account. Reviewing it now.
> 
>  Koji
>  On Sun, Oct 14, 2018 at 11:21 PM Jeff  wrote:
> >
> > Sivaprasanna,
> >
> > Thanks for submitting a pull request for that issue!  Later today or
> > tomorrow I'll have to check to see if I've already used up my
> >> free-tier
> > access to Azure.  If I still have access, I can review your PR and
> >>> we'll
> > get it into 1.8.0.
> >
> > On Sun, Oct 14, 2018 at 4:30 AM Sivaprasanna <
> >>> sivaprasanna...@gmail.com>
> > wrote:
> >
> >> All - Just found one bug with DeleteAzureBlobStorage processor. It
> >>> was
> >> shared by one user on StackOverflow [1] and I later confirmed it.
> >> It
> >>> looks
> >> to be introduced by NIFI-4199. I have created a Jira [2] and made
> >> the
> >> necessary changes (not huge, just few lines) and raised a PR [3]. I
> >>> think,
> >> if we can spend a little time in getting it reviewed, we can mark
> >> it
> >>> for
> >> 1.8.0. Thoughts?
> >>
> >> [1] -
> >>
> >>
> >>>
> >>
> https://stackoverflow.com/questions/52766991/apache-nifi-deleteazureblobstorage-processor-is-throwing-an-error
> >> [2] - https://issues.apache.org/jira/browse/NIFI-5698
> >> [3] - https://github.com/apache/nifi/pull/3073
> >>
> >> -
> >> Sivaprasanna
> >>
> >> On Fri, Oct 12, 2018 at 9:05 PM Mike Thomsen <
> >> mikerthom...@gmail.com
> 
> >> wrote:
> >>
> >>> 4811 should be ready for review now. Rebased and cleaned it up
> >>> with a
> >> full
> >>> listing of the Spring dependencies.
> >>>
> >>> On Fri, Oct 12, 2018 at 11:23 AM Joe Witt 
> >>> wrote:
> >>>
>  Jeff,
> 
>  I think for anything not tagged to 1.8.0 we just keep rolling.
> >>> For
>  anything tagged 1.8.0 that should not be we should remove it
> >>> until
>  ready.  For things tagged to 1.8.0 that cannot be moved we
> >> should
>  resolve.  For the tagged 1.8.0 section you had.
> 
>    - NIFI-4811 <
> >> https://issues.apache.org/jira/browse/NIFI-4811>
> >>> -
> >> Use a
>    newer version of spring-data-redis
>    - PR 2856 
>  *This needs to be resolved by either reverting the commit or
> >>> ensuring
>  L accurately reflects all.  We have to do this always and for
> >>> every
>  nar.  The process isnt easy or fun but it is necessary to
> >> produce
>  valid ASF releases.  Landing commits which change dependencies
>  requires this due diligence.  Now, we've put a lot of energy
> >> into
>  updating Spring dependencies because some older Spring libs had
>  vulnerabilities which while we likely aren't exposed to them we
> >>> want
>  to fix in due course.  So reverting may require more analysis
> >>> than if
>  we were just get L fixed with this new change.  I commented
> >> on
> >>> the
> 

Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?

[Nathan Gough]

Hi Mike,

Sure I can look at fixing up PR-2977 today. 

Nathan

On 10/16/18, 6:13 AM, "Mike Thomsen"  wrote:

Does 5562 need to be addressed in 1.8?

https://github.com/apache/nifi/pull/2977

On Mon, Oct 15, 2018 at 6:33 PM Jeff  wrote:

> NiFi Devs,
>
> The Release page [1] for 1.8.0 now reports that all issues are done!  I'd
> like to start the release candidate preparation tomorrow, around 1200 EST.
>
> Thanks to everyone for all the great work that's been done!  196 issues
> resolved in this version with some great new features!
>
> [1] https://issues.apache.org/jira/projects/NIFI/versions/12343482
>
> On Mon, Oct 15, 2018 at 7:30 AM Sivaprasanna 
> wrote:
>
> > Great. Thanks. :)
> >
> > -
> > Sivaprasanna
> >
> > On Mon, Oct 15, 2018 at 7:09 AM Koji Kawamura 
> > wrote:
> >
> > > Jeff, Sivasprasanna,
> > >
> > > NIFI-5698 (PR3073) Fixing DeleteAzureBlob bug is merged.
> > >
> > > Thanks,
> > > Koji
> > > On Mon, Oct 15, 2018 at 10:18 AM Koji Kawamura  >
> > > wrote:
> > > >
> > > > Thank you for the fix Sivaprasanna,
> > > > I have Azure account. Reviewing it now.
> > > >
> > > > Koji
> > > > On Sun, Oct 14, 2018 at 11:21 PM Jeff  wrote:
> > > > >
> > > > > Sivaprasanna,
> > > > >
> > > > > Thanks for submitting a pull request for that issue!  Later today
> or
> > > > > tomorrow I'll have to check to see if I've already used up my
> > free-tier
> > > > > access to Azure.  If I still have access, I can review your PR and
> > > we'll
> > > > > get it into 1.8.0.
> > > > >
> > > > > On Sun, Oct 14, 2018 at 4:30 AM Sivaprasanna <
> > > sivaprasanna...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > All - Just found one bug with DeleteAzureBlobStorage processor.
> It
> > > was
> > > > > > shared by one user on StackOverflow [1] and I later confirmed 
it.
> > It
> > > looks
> > > > > > to be introduced by NIFI-4199. I have created a Jira [2] and 
made
> > the
> > > > > > necessary changes (not huge, just few lines) and raised a PR
> [3]. I
> > > think,
> > > > > > if we can spend a little time in getting it reviewed, we can 
mark
> > it
> > > for
> > > > > > 1.8.0. Thoughts?
> > > > > >
> > > > > > [1] -
> > > > > >
> > > > > >
> > >
> >
> 
https://stackoverflow.com/questions/52766991/apache-nifi-deleteazureblobstorage-processor-is-throwing-an-error
> > > > > > [2] - https://issues.apache.org/jira/browse/NIFI-5698
> > > > > > [3] - https://github.com/apache/nifi/pull/3073
> > > > > >
> > > > > > -
> > > > > > Sivaprasanna
> > > > > >
> > > > > > On Fri, Oct 12, 2018 at 9:05 PM Mike Thomsen <
> > mikerthom...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > 4811 should be ready for review now. Rebased and cleaned it up
> > > with a
> > > > > > full
> > > > > > > listing of the Spring dependencies.
> > > > > > >
> > > > > > > On Fri, Oct 12, 2018 at 11:23 AM Joe Witt 
> > > wrote:
> > > > > > >
> > > > > > > > Jeff,
> > > > > > > >
> > > > > > > > I think for anything not tagged to 1.8.0 we just keep
> rolling.
> > > For
> > > > > > > > anything tagged 1.8.0 that should not be we should remove it
> > > until
> > > > > > > > ready.  For things tagged to 1.8.0 that cannot be moved we
> > should
> > > > > > > > resolve.  For the tagged 1.8.0 section you had.
> > > > > > > >
> > > > > > > >- NIFI-4811 <
> > https://issues.apache.org/jira/browse/NIFI-4811>
> > > -
> > > > > > Use a
> > > > > > > >newer version of spring-data-redis
> > > > > > > >- PR 2856 
> > > > > > > > *This needs to be resolved by either reverting the commit or
> > > ensuring
> > > > > > > > L accurately reflects all.  We have to do this always and
> for
> > > every
> > > > > > > > nar.  The process isnt easy or fun but it is necessary to
> > produce
> > > > > > > > valid ASF releases.  Landing commits which change
> dependencies
> > > > > > > > requires this due diligence.  Now, we've put a lot of energy
> > into
> > > > > > > > updating Spring dependencies because some older Spring libs
> had
> > > > > > > > vulnerabilities which while we likely aren't exposed to them
> we
> > > want
> > > > > > > > to fix in due course.  So reverting may require more 
analysis
> > > than if
> > > > > > > > we were just get L fixed with this new change.  I 
commented
> > on
> > > the
> > > > > > > > JIRA.  But this needs to be resolved.
> > > > > > > >
> > > > > > > >
> > > > > > > >- NIFI-5426 <
> > 

Re: Compiling custom processor

[Mike Thomsen]

If you want to look at a large Maven project that builds several custom
NARs from one unified build system, take a look at my data generation
bundle here:

https://github.com/MikeThomsen/nifi-datageneration-bundle

On Tue, Oct 16, 2018 at 9:53 AM Bryan Bende  wrote:

> In general, if your processor uses a controller service then the
> processors pom file needs a provided dependency on the API of the CS,
> and your NAR pom needs a NAR dependency on the NAR where the CS API
> is.
>
> Example is shown here in the section linking processors and controller
> services -
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
>
> Also, you do not want to include NiFI's standard processors in your
> own NAR, this will result in two copies of every standard processor.
> You will want to copy whatever code you need into your own NAR.
>
> In you last statement, if you made a new processor in
> nifi-standard-processors and it didn't show up, it is most likely
> because you didn't update META-INF/services file to include the new
> processor.
> On Tue, Oct 16, 2018 at 9:36 AM John McGinn
>  wrote:
> >
> > Ok, I am far from a maven expert, and am struggling on this problem.
> >
> > I created a new project using the maven generate process, and compiled
> that sample processor and everything was fine, and I could see it in my
> local NiFi instance. I then copied over the Wait processor on top of the
> MyProcessor.java class, changed the package name, and the class name, and
> attempted to compile. Got errors due to dependency issues. This is where I
> get confused.
> >
> > In the processor directory, I modify the pom.xml with a dependency line
> for nifi-api, nifi-utils, nifi-distribute-cache-client-service-api and
> nifi-standard-processors. This seemed to make things work, and I get a 33
> meg NAR file, which contains lots of JARs. I load that up, and I get a
> problem with CalculateStats, or similar, because of RecordReaderFactory. I
> go back in and include nifi-record-serialization-services, and it compiles,
> and the JAR is included in the NAR file now, and I still get class not
> found.
> >
> > I've also tried to change the dependencies to a status of provided, and
> therefore get a minimal 25k or so NAR file with no JARs included. I get the
> same issue with the class not found.
> >
> > I had also attempted to copy Wait.java to AnotherWait.java within the
> actually NiFi source code, and it compiles, and shows up in the JARs and
> NARs as I'd expect, but I cannot get to it within my NiFi instance.
> >
> > Can someone let me know what I'm doing wrong.
> >
> > Thanks,
> > John McGinn
>

Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?

[Mark Payne]

Jeff / all,

I ran into an issue with NIFI-375 and re-opened the ticket. If a processor is 
stopped or started in a cluster,
the stats that come back in the response are incorrect because the response is 
not being properly merged
from all nodes in the cluster.

I also have run into a couple of other issues that appear to have been 
introduced in 1.8. I have not yet created JIRA's
for them because I want to understand the issues better before trying to write 
it up. I am seeing, for example, some errors
in the logs indicating that provenance events don't have a FlowFile UUID 
assigned to them when being read from the
repository. This seems to have been introduced in 1.8 so would like to get a 
resolution before releasing.

Thanks
-Mark


> On Oct 15, 2018, at 6:33 PM, Jeff  wrote:
> 
> NiFi Devs,
> 
> The Release page [1] for 1.8.0 now reports that all issues are done!  I'd
> like to start the release candidate preparation tomorrow, around 1200 EST.
> 
> Thanks to everyone for all the great work that's been done!  196 issues
> resolved in this version with some great new features!
> 
> [1] https://issues.apache.org/jira/projects/NIFI/versions/12343482
> 
> On Mon, Oct 15, 2018 at 7:30 AM Sivaprasanna 
> wrote:
> 
>> Great. Thanks. :)
>> 
>> -
>> Sivaprasanna
>> 
>> On Mon, Oct 15, 2018 at 7:09 AM Koji Kawamura 
>> wrote:
>> 
>>> Jeff, Sivasprasanna,
>>> 
>>> NIFI-5698 (PR3073) Fixing DeleteAzureBlob bug is merged.
>>> 
>>> Thanks,
>>> Koji
>>> On Mon, Oct 15, 2018 at 10:18 AM Koji Kawamura 
>>> wrote:
 
 Thank you for the fix Sivaprasanna,
 I have Azure account. Reviewing it now.
 
 Koji
 On Sun, Oct 14, 2018 at 11:21 PM Jeff  wrote:
> 
> Sivaprasanna,
> 
> Thanks for submitting a pull request for that issue!  Later today or
> tomorrow I'll have to check to see if I've already used up my
>> free-tier
> access to Azure.  If I still have access, I can review your PR and
>>> we'll
> get it into 1.8.0.
> 
> On Sun, Oct 14, 2018 at 4:30 AM Sivaprasanna <
>>> sivaprasanna...@gmail.com>
> wrote:
> 
>> All - Just found one bug with DeleteAzureBlobStorage processor. It
>>> was
>> shared by one user on StackOverflow [1] and I later confirmed it.
>> It
>>> looks
>> to be introduced by NIFI-4199. I have created a Jira [2] and made
>> the
>> necessary changes (not huge, just few lines) and raised a PR [3]. I
>>> think,
>> if we can spend a little time in getting it reviewed, we can mark
>> it
>>> for
>> 1.8.0. Thoughts?
>> 
>> [1] -
>> 
>> 
>>> 
>> https://stackoverflow.com/questions/52766991/apache-nifi-deleteazureblobstorage-processor-is-throwing-an-error
>> [2] - https://issues.apache.org/jira/browse/NIFI-5698
>> [3] - https://github.com/apache/nifi/pull/3073
>> 
>> -
>> Sivaprasanna
>> 
>> On Fri, Oct 12, 2018 at 9:05 PM Mike Thomsen <
>> mikerthom...@gmail.com
 
>> wrote:
>> 
>>> 4811 should be ready for review now. Rebased and cleaned it up
>>> with a
>> full
>>> listing of the Spring dependencies.
>>> 
>>> On Fri, Oct 12, 2018 at 11:23 AM Joe Witt 
>>> wrote:
>>> 
 Jeff,
 
 I think for anything not tagged to 1.8.0 we just keep rolling.
>>> For
 anything tagged 1.8.0 that should not be we should remove it
>>> until
 ready.  For things tagged to 1.8.0 that cannot be moved we
>> should
 resolve.  For the tagged 1.8.0 section you had.
 
   - NIFI-4811 <
>> https://issues.apache.org/jira/browse/NIFI-4811>
>>> -
>> Use a
   newer version of spring-data-redis
   - PR 2856 
 *This needs to be resolved by either reverting the commit or
>>> ensuring
 L accurately reflects all.  We have to do this always and for
>>> every
 nar.  The process isnt easy or fun but it is necessary to
>> produce
 valid ASF releases.  Landing commits which change dependencies
 requires this due diligence.  Now, we've put a lot of energy
>> into
 updating Spring dependencies because some older Spring libs had
 vulnerabilities which while we likely aren't exposed to them we
>>> want
 to fix in due course.  So reverting may require more analysis
>>> than if
 we were just get L fixed with this new change.  I commented
>> on
>>> the
 JIRA.  But this needs to be resolved.
 
 
   - NIFI-5426 <
>> https://issues.apache.org/jira/browse/NIFI-5426>
>>> - Use
   NIO.2 API for ListFile to avoid multiple disk reads
  - PR 2889 
 *This just needed to be marked resolved.  The commit went in
>> the
>>> day
 after we cut 1.7.1.  So this one is sorted.
 
   - NIFI-5448 <
>> https://issues.apache.org/jira/browse/NIFI-5448>
>>> -
>>> Failed
   EL date parsing 

Re: Cluster Warnings

[Joe Witt]

please keep this thread on users.

On Sun, Oct 14, 2018, 11:53 PM Karthik Kothareddy (karthikk) [CONT - Type
2]  wrote:

> Hello,
>
> We're running a 4-node cluster on NiFi 1.7.1. The fourth node was added
> recently and as soon as we added the 4th node, we started seeing below
> warnings
>
> Response time from NODE2 was slow for each of the last 3 requests made. To
> see more information about timing, enable DEBUG logging for
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator
>
> Initially we though the problem was with the recent node added and cross
> checked all the configs on the box and everything seemed to be just fine.
> After enabling the DEBUG mode for cluster logging we noticed that the
> warning is not specific to any node and every-time we see a warning like
> above there is one slow node which takes forever to send a response like
> below (in this case the slow node is NIFI04). Sometimes these will lead to
> node-disconnects needing a manual intervention.
>
> DEBUG [Replicate Request Thread-50]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Node Responses for GET
> /nifi-api/site-to-site (Request ID b2c6e983-5233-4007-bd54-13d21b7068d5):
> NIFI04:8443: 1386 millis
> NIFI02:8443: 3 millis
> NIFI01:8443: 5 millis
> NIFI03:8443: 3 millis
> DEBUG [Replicate Request Thread-41]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Node Responses for GET
> /nifi-api/site-to-site (Request ID d182fdab-f1d4-4ac9-97fd-e24c41dc4622):
> NIFI04:8443: 1143 millis
> NIFI02:8443: 22 millis
> NIFI01:8443: 3 millis
> NIFI03:8443: 2 millis
> DEBUG [Replicate Request Thread-31]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Node Responses for GET
> /nifi-api/site-to-site (Request ID e4726027-27c7-4bbb-8ab6-d02bb41f1920):
> NIFI04:8443: 1053 millis
> NIFI02:8443: 3 millis
> NIFI01:8443: 3 millis
> NIFI03:8443: 2 millis
>
> We tried changing the configurations in nifi.properties like bumping up
> the "nifi.cluster.node.protocol.max.threads" but none of them seems to be
> working and we're still stuck with the slow communication between the
> nodes. We use an external zookeeper as this is our production server.
> Below are some of our configs
>
> # cluster node properties (only configure for cluster nodes) #
> nifi.cluster.is.node=true
> nifi.cluster.node.address=fslhdppnifi01.imfs.micron.com
> nifi.cluster.node.protocol.port=11443
> nifi.cluster.node.protocol.threads=100
> nifi.cluster.node.protocol.max.threads=120
> nifi.cluster.node.event.history.size=25
> nifi.cluster.node.connection.timeout=90 sec
> nifi.cluster.node.read.timeout=90 sec
> nifi.cluster.node.max.concurrent.requests=1000
> nifi.cluster.firewall.file=
> nifi.cluster.flow.election.max.wait.time=30 sec
> nifi.cluster.flow.election.max.candidates=
>
> Any thoughts on why this is happening?
>
>
> -Karthik
>

Re: Compiling custom processor

[Bryan Bende]

In general, if your processor uses a controller service then the
processors pom file needs a provided dependency on the API of the CS,
and your NAR pom needs a NAR dependency on the NAR where the CS API
is.

Example is shown here in the section linking processors and controller
services - 
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions

Also, you do not want to include NiFI's standard processors in your
own NAR, this will result in two copies of every standard processor.
You will want to copy whatever code you need into your own NAR.

In you last statement, if you made a new processor in
nifi-standard-processors and it didn't show up, it is most likely
because you didn't update META-INF/services file to include the new
processor.
On Tue, Oct 16, 2018 at 9:36 AM John McGinn
 wrote:
>
> Ok, I am far from a maven expert, and am struggling on this problem.
>
> I created a new project using the maven generate process, and compiled that 
> sample processor and everything was fine, and I could see it in my local NiFi 
> instance. I then copied over the Wait processor on top of the 
> MyProcessor.java class, changed the package name, and the class name, and 
> attempted to compile. Got errors due to dependency issues. This is where I 
> get confused.
>
> In the processor directory, I modify the pom.xml with a dependency line for 
> nifi-api, nifi-utils, nifi-distribute-cache-client-service-api and 
> nifi-standard-processors. This seemed to make things work, and I get a 33 meg 
> NAR file, which contains lots of JARs. I load that up, and I get a problem 
> with CalculateStats, or similar, because of RecordReaderFactory. I go back in 
> and include nifi-record-serialization-services, and it compiles, and the JAR 
> is included in the NAR file now, and I still get class not found.
>
> I've also tried to change the dependencies to a status of provided, and 
> therefore get a minimal 25k or so NAR file with no JARs included. I get the 
> same issue with the class not found.
>
> I had also attempted to copy Wait.java to AnotherWait.java within the 
> actually NiFi source code, and it compiles, and shows up in the JARs and NARs 
> as I'd expect, but I cannot get to it within my NiFi instance.
>
> Can someone let me know what I'm doing wrong.
>
> Thanks,
> John McGinn

RE: [EXT] Calcite version used in Nifi 1.7+

[Peter Wicks (pwicks)]

In NiFi 1.7.0 it was Calcite v1.12.0. In NiFi 1.8 it will be Calcite v1.17.0

-Original Message-
From: ruurd.schoonh...@dbiq.nl [mailto:ruurd.schoonh...@dbiq.nl] 
Sent: Tuesday, October 16, 2018 6:58 AM
To: dev@nifi.apache.org
Subject: [EXT] Calcite version used in Nifi 1.7+

Hi guys,

Which version of calcite is used by QueryRecord processor in Nifi 1.7 or above?

I’m looking for CROSS APPLY suport in my query as introduced in Calcite 1.11 
(january 2017)

Or has anyone a solution to perform select on self joined dataset (recursive)

Kind regards
Ruurd Schoonheim

Compiling custom processor

[John McGinn]

Ok, I am far from a maven expert, and am struggling on this problem.

I created a new project using the maven generate process, and compiled that 
sample processor and everything was fine, and I could see it in my local NiFi 
instance. I then copied over the Wait processor on top of the MyProcessor.java 
class, changed the package name, and the class name, and attempted to compile. 
Got errors due to dependency issues. This is where I get confused.

In the processor directory, I modify the pom.xml with a dependency line for 
nifi-api, nifi-utils, nifi-distribute-cache-client-service-api and 
nifi-standard-processors. This seemed to make things work, and I get a 33 meg 
NAR file, which contains lots of JARs. I load that up, and I get a problem with 
CalculateStats, or similar, because of RecordReaderFactory. I go back in and 
include nifi-record-serialization-services, and it compiles, and the JAR is 
included in the NAR file now, and I still get class not found.

I've also tried to change the dependencies to a status of provided, and 
therefore get a minimal 25k or so NAR file with no JARs included. I get the 
same issue with the class not found.

I had also attempted to copy Wait.java to AnotherWait.java within the actually 
NiFi source code, and it compiles, and shows up in the JARs and NARs as I'd 
expect, but I cannot get to it within my NiFi instance.

Can someone let me know what I'm doing wrong.

Thanks,
John McGinn

Calcite version used in Nifi 1.7+

[ruurd . schoonheim]

Hi guys,

Which version of calcite is used by QueryRecord processor in Nifi 1.7 or above?

I’m looking for CROSS APPLY suport in my query as introduced in Calcite 1.11 
(january 2017)

Or has anyone a solution to perform select on self joined dataset (recursive)

Kind regards
Ruurd Schoonheim

Re: Fixing unstable nifi cluster.

[Pierre Villard]

Hi,

Can you try increasing the below parameters? That's usually what I
recommend, our default values being probably a bit too aggressively low.

nifi.zookeeper.connect.timeout=15 secs
nifi.zookeeper.session.timeout=15 secs
nifi.cluster.node.read.timeout=30 sec

Pierre

Le mar. 16 oct. 2018 à 13:02, ashwin konale  a
écrit :

> Hi,
> We have a 3 node nifi cluster (With separate zookeper instances running in
> the same machines) which pulls the data from mysql and write to hdfs. I am
> frequently running into problems with cluster. Nodes keeps disconnecting
> from each other, primary nodes keeps switching and sometimes it just goes
> into zombie state when I just cannot access the ui. I have followed best
> practices guide and tweaked params in nifi.properties, have switched
> provenanceRepositoryImplementation to volatile because cluster was not able
> to keep up with incoming traffic. Data traffic is not high at all (4Mbps).
> This is the message I frequently get from the logs.
>
> *INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change:
> LOST*
> *INFO [Curator-ConnectionStateManager-0]
> o.a.n.c.l.e.CuratorLeaderElectionManager
>
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@56ebedec
> Connection State changed to LOST*
> *INFO [Curator-ConnectionStateManager-0]
> o.a.n.c.l.e.CuratorLeaderElectionManager
>
> org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1b0e2055
> Connection State changed to LOST*
> *INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change:
> RECONNECTED*
>
> Am I doing something wrong with cluster setup ? Can someone give me some
> guidance on how to go about debugging this issue ? What kind of system
> metrics to look at etc.
>
> Ashwin
>

Fixing unstable nifi cluster.

[ashwin konale]

Hi,
We have a 3 node nifi cluster (With separate zookeper instances running in
the same machines) which pulls the data from mysql and write to hdfs. I am
frequently running into problems with cluster. Nodes keeps disconnecting
from each other, primary nodes keeps switching and sometimes it just goes
into zombie state when I just cannot access the ui. I have followed best
practices guide and tweaked params in nifi.properties, have switched
provenanceRepositoryImplementation to volatile because cluster was not able
to keep up with incoming traffic. Data traffic is not high at all (4Mbps).
This is the message I frequently get from the logs.

*INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change:
LOST*
*INFO [Curator-ConnectionStateManager-0]
o.a.n.c.l.e.CuratorLeaderElectionManager
org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@56ebedec
Connection State changed to LOST*
*INFO [Curator-ConnectionStateManager-0]
o.a.n.c.l.e.CuratorLeaderElectionManager
org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@1b0e2055
Connection State changed to LOST*
*INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change:
RECONNECTED*

Am I doing something wrong with cluster setup ? Can someone give me some
guidance on how to go about debugging this issue ? What kind of system
metrics to look at etc.

Ashwin

Re: [DISCUSS] Closing in on a release of NiFi 1.8.0?

[Mike Thomsen]

Does 5562 need to be addressed in 1.8?

https://github.com/apache/nifi/pull/2977

On Mon, Oct 15, 2018 at 6:33 PM Jeff  wrote:

> NiFi Devs,
>
> The Release page [1] for 1.8.0 now reports that all issues are done!  I'd
> like to start the release candidate preparation tomorrow, around 1200 EST.
>
> Thanks to everyone for all the great work that's been done!  196 issues
> resolved in this version with some great new features!
>
> [1] https://issues.apache.org/jira/projects/NIFI/versions/12343482
>
> On Mon, Oct 15, 2018 at 7:30 AM Sivaprasanna 
> wrote:
>
> > Great. Thanks. :)
> >
> > -
> > Sivaprasanna
> >
> > On Mon, Oct 15, 2018 at 7:09 AM Koji Kawamura 
> > wrote:
> >
> > > Jeff, Sivasprasanna,
> > >
> > > NIFI-5698 (PR3073) Fixing DeleteAzureBlob bug is merged.
> > >
> > > Thanks,
> > > Koji
> > > On Mon, Oct 15, 2018 at 10:18 AM Koji Kawamura  >
> > > wrote:
> > > >
> > > > Thank you for the fix Sivaprasanna,
> > > > I have Azure account. Reviewing it now.
> > > >
> > > > Koji
> > > > On Sun, Oct 14, 2018 at 11:21 PM Jeff  wrote:
> > > > >
> > > > > Sivaprasanna,
> > > > >
> > > > > Thanks for submitting a pull request for that issue!  Later today
> or
> > > > > tomorrow I'll have to check to see if I've already used up my
> > free-tier
> > > > > access to Azure.  If I still have access, I can review your PR and
> > > we'll
> > > > > get it into 1.8.0.
> > > > >
> > > > > On Sun, Oct 14, 2018 at 4:30 AM Sivaprasanna <
> > > sivaprasanna...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > All - Just found one bug with DeleteAzureBlobStorage processor.
> It
> > > was
> > > > > > shared by one user on StackOverflow [1] and I later confirmed it.
> > It
> > > looks
> > > > > > to be introduced by NIFI-4199. I have created a Jira [2] and made
> > the
> > > > > > necessary changes (not huge, just few lines) and raised a PR
> [3]. I
> > > think,
> > > > > > if we can spend a little time in getting it reviewed, we can mark
> > it
> > > for
> > > > > > 1.8.0. Thoughts?
> > > > > >
> > > > > > [1] -
> > > > > >
> > > > > >
> > >
> >
> https://stackoverflow.com/questions/52766991/apache-nifi-deleteazureblobstorage-processor-is-throwing-an-error
> > > > > > [2] - https://issues.apache.org/jira/browse/NIFI-5698
> > > > > > [3] - https://github.com/apache/nifi/pull/3073
> > > > > >
> > > > > > -
> > > > > > Sivaprasanna
> > > > > >
> > > > > > On Fri, Oct 12, 2018 at 9:05 PM Mike Thomsen <
> > mikerthom...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > 4811 should be ready for review now. Rebased and cleaned it up
> > > with a
> > > > > > full
> > > > > > > listing of the Spring dependencies.
> > > > > > >
> > > > > > > On Fri, Oct 12, 2018 at 11:23 AM Joe Witt 
> > > wrote:
> > > > > > >
> > > > > > > > Jeff,
> > > > > > > >
> > > > > > > > I think for anything not tagged to 1.8.0 we just keep
> rolling.
> > > For
> > > > > > > > anything tagged 1.8.0 that should not be we should remove it
> > > until
> > > > > > > > ready.  For things tagged to 1.8.0 that cannot be moved we
> > should
> > > > > > > > resolve.  For the tagged 1.8.0 section you had.
> > > > > > > >
> > > > > > > >- NIFI-4811 <
> > https://issues.apache.org/jira/browse/NIFI-4811>
> > > -
> > > > > > Use a
> > > > > > > >newer version of spring-data-redis
> > > > > > > >- PR 2856 
> > > > > > > > *This needs to be resolved by either reverting the commit or
> > > ensuring
> > > > > > > > L accurately reflects all.  We have to do this always and
> for
> > > every
> > > > > > > > nar.  The process isnt easy or fun but it is necessary to
> > produce
> > > > > > > > valid ASF releases.  Landing commits which change
> dependencies
> > > > > > > > requires this due diligence.  Now, we've put a lot of energy
> > into
> > > > > > > > updating Spring dependencies because some older Spring libs
> had
> > > > > > > > vulnerabilities which while we likely aren't exposed to them
> we
> > > want
> > > > > > > > to fix in due course.  So reverting may require more analysis
> > > than if
> > > > > > > > we were just get L fixed with this new change.  I commented
> > on
> > > the
> > > > > > > > JIRA.  But this needs to be resolved.
> > > > > > > >
> > > > > > > >
> > > > > > > >- NIFI-5426 <
> > https://issues.apache.org/jira/browse/NIFI-5426>
> > > - Use
> > > > > > > >NIO.2 API for ListFile to avoid multiple disk reads
> > > > > > > >   - PR 2889 
> > > > > > > > *This just needed to be marked resolved.  The commit went in
> > the
> > > day
> > > > > > > > after we cut 1.7.1.  So this one is sorted.
> > > > > > > >
> > > > > > > >- NIFI-5448 <
> > https://issues.apache.org/jira/browse/NIFI-5448>
> > > -
> > > > > > > Failed
> > > > > > > >EL date parsing live-locks processors without a failure
> > > relationship
> > > > > > > > * The commit needs to be reverted.  I'm working on that now.
> > > Once the
> > > > > > > >