Who or what sets up NIFI_HOME as a system environment variable?

[Russell Bateman]

It's hard for me to determine this for customers who may already have 
installed Apache NiFi and just drop my NAR in or count on my Docker 
container. I'd like a clearer idea of who or what is responsible for that.

Re: ...not the most recent version of this FlowFile within this session...

[Russell Bateman]

Yes, what you described is what was happening, Mark. I didn't display 
all of the code to the session methods, and I did re-read the in-coming 
flowfile for different purposes than I had already read and written it. 
So, I wasn't helpful enough. In the end, however, I had forgotten, 
immediately after the call to session.putAllAttributes(), to update the 
resulting flowfile for passing to session.transfer(). That solved it for 
1.1.2 which wasn't necessary for 1.13.2 or later versions. Being 
helpful, the newer versions made me a spoiled, entitled child and I will 
repent immediately.


Thanks, guys! DevOps are happy they don't have to upgrade the customers 
to NiFi 1.13.2. (In a way, I'm unhappy about that, but...).


Best regards,

Russ

On 5/7/24 11:53, Mark Payne wrote:

The call to session.putAttribute would throw an Exception because you
provided an outdated version of the flowFile (did not capturing the
result of calling session.write)

Now, as NiFi matured, we found that:
(a) for more complex processors that aren’t just a series of sequential
steps it becomes difficult to manage all of that bookkeeping.
(b) it was not intuitive to require this
(c) the ProcessSession already had more or less what it needed in order
to determine what the most up-to-date version of the FlowFile was.

So we updated the ProcessSession to automatically grab the latest
version of the FlowFile for these methods. But since you’re trying to
run an old version, you’ll need to make sure that you capture all of
those outputs and always keep track of the most recent version of a
FlowFile.

...not the most recent version of this FlowFile within this session...

[Russell Bateman]

In /pom.xml/ I specify using NiFi framework libraries from 1.13.2.

There are peculiar reasons (that we are trying to fix) that inhibit us 
from moving forward as all of our customer machines are running 1.1.2. 
(Don't shoot me, I'm not DevOps, but just the guy who writes custom 
processors.)


I have custom processor code that, built as noted using 1.13.2, runs 
_perfectly_ on any version of NiFi from 1.13.2 to 1.25.0. When loaded on 
1.1.2, one processor fails to process a very simple flowfile with the 
error message below.


I'm wondering if there's a dance I can do to tweak one or more of what I 
suspect are the calls where this is happening (in onTrigger()):


   FlowFile result = session.*write*( flowfile, new StreamCallback()
  session.*read*( flowfile, new InputStreamCallback()
  session.*removeAllAttributes*( result, keys );
  session.*putAllAttributes*( result, attributes );
  session.*transfer*( result, SUCCESS );


Error message:

10:32:47 MDTERRORc78b2d48-79a6-39ae-1454-4ee57bbf8928
ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928] \
failed to process session due to
org.apache.nifi.processor.exception.FlowFileHandlingException: \
StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim
 \
[resourceClaim=StandardResourceClaim[id=1715094869203-1, container=default, 
section=1], offset=0, 
length=1077766],offset=0,name=30575811037248611,size=1077766] \
is not the most recent version of this FlowFile within this session 
(StandardProcessSession[id=3923]): \
org.apache.nifi.processor.exception.FlowFileHandlingException: 
StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim
 \
[resourceClaim=StandardResourceClaim[id=1715094869203-1, container=default, 
section=1], offset=0, 
length=1077766],offset=0,name=30575811037248611,size=1077766] \
is not the most recent version of this FlowFile within this session 
(StandardProcessSession[id=3923])

10:32:47 MDTWARNINGc78b2d48-79a6-39ae-1454-4ee57bbf8928
ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928] Processor 
Administratively Yielded for 1 sec due to processing failure


I know this is old stuff and I'm prepared be told that I'm out of luck 
or that the solution is unfeasible or would take too much effort to dig 
through. I'm just hoping that someone remembers something quick off the 
top of the head and can give advice.


Thanks,

Russ

Re: SELinux and NiFi

[Russell Bateman]

That's what lies in those "SELinux policies."

I think it's simple: Use SELinux to lock the filesystem (and other 
stuff) up so no one can get in or go around. Then create specific 
policies that allow, in this case, NiFi, access to its filesystem (like 
//opt/nifi/current-nifi//) so that it can do work. Obviously, when you 
install NiFi, things can get complicated like where do you want each 
repository to live--you'll have to provide NiFi access to each place, no 
longer a single filesystem.


This is handled by DevOps guys and not me (I just write custom 
processors), but if you get real pointed, I can ask them better 
questions they can answer.


Russ


On 3/8/24 15:04, Mike Thomsen wrote:
I think the admin told me that even a simple nifi.sh start won’t work. 
Just won’t even start the script and it is marked executable. I was 
wondering if there were any gotchas to getting a basic setup running.



Sent from my iPhone

On Mar 8, 2024, at 4:29 PM, Russell Bateman  
wrote:


 We have run on CentOS with SELinux set to enforcing and have run 
NiFi in that environment for probably 8 or 9 years now. We do install 
some SELinux policies that allow NiFi to access the filesystem 
underneath itself and not outside that filesystem.


What specifically are you asking?

On 3/8/24 14:04, Mike Thomsen wrote:
Does anyone have experience setting up NiFi w/ SELinux set to 
"enforcing?"

Re: SELinux and NiFi

[Russell Bateman]

We have run on CentOS with SELinux set to enforcing and have run NiFi in 
that environment for probably 8 or 9 years now. We do install some 
SELinux policies that allow NiFi to access the filesystem underneath 
itself and not outside that filesystem.


What specifically are you asking?

On 3/8/24 14:04, Mike Thomsen wrote:

Does anyone have experience setting up NiFi w/ SELinux set to "enforcing?"

Re: NiFi 2.0.0-M2

[Russell Bateman]

Don't know if this is relevant let along helpful, but we were on CentOS 
for many years and, when it died, we moved to Alma.


Best of luck to you,

Russ

On 3/5/24 07:56, Marton Szasz wrote:
CentOS 7 is a very old distribution, so you may run into issues, but 
in theory, if you can install Java 21 on it, and start NiFi using 
that, then it should work.


I'd be surprised if Java 21 was included in the CentOS 7 package 
repositories. You'll most likely need to install it manually, and set 
JAVA_HOME appropriately in the environment you're using to start NiFi.


Marton

On 3/5/24 14:10, Pierre Villard wrote:

Hi,

NiFi is OS-agnostic for linux based distributions. You can download the
tar.gz from the download page of the website and follow the 
instructions to

get started on CentOS 7.

HTH,
Pierre

Le mar. 5 mars 2024 à 14:07, Gleb Efimov  a 
écrit :



Good afternoon I'm trying to deploy NiFi 2.0.0-M2 on Centos 7, but,
unfortunately, I can't find a tar archive with the distribution I need.
Could you tell me where I can get it?
And is it even possible to deploy NiFi 2.0.0-M2 on Centos 7?
Thank you very much.

Sincerely, Efimov Gleb.

Re: Block start/stop processors

[Russell Bateman]

Isha,

Wait, by "lefthand toolbox" are you referring to the process group's 
toolbar start button or to the start button in the pallet at the upper 
left of the page?


Please clarify.

Thanks


On 2/22/24 04:39, Isha Lamboo wrote:

always use the right-click context menu on a process group you want to
stop/start and never use the button in the lefthand toolbox.
Right-clicking the PG also selects it, so you don't have the risk of
clicking stop with no selection.

Re: Issue with Nifi pipeline

[Russell Bateman]

Sonia,

It sounds like you may prefer the Users Mailing List 
(us...@nifi.apache.org)rather than this one which is more for custom 
processors and other development-related activities.


Best regards,

Russ


On 11/27/23 22:49, Sonia Soleimani wrote:

Hello,
I am working for Telus and there has been a legacy pipeline that I am
trying to fix but get some error which could be because I am new to Nifi. I
would appreciate it if I could get someone from Nifi team to resolve this
issue . It is mostly related to some tables not loading completely in GCP
Bigquery (destination). This could be because the "fetch" step doesn't
fully finish .

Thanks,
Sonia Soleimani

Internationalization and localization of the UI and processors

[Russell Bateman]

Looking around, I see there have been statements of intent by folk to 
localize NiFi [1], but few statements as to how far they got. I saw a 
question on stackoverflow [2] on how to hack relevant Java-annotated 
references (@CapabilityDescription) which isn't exactly 
internationalization, but it's a start.


Most of what our down-streamers use are custom processors, but they use 
some standard NiFi ones too, but the UI would be a big job and a pretty 
important piece.


Has this sort of thing gone pretty far and I haven't found the right 
Google string to find it? I'd happily write a guide for custom processor 
coding if it's worth walking the path because the UI is taken care of 
(assuming also that my company thinks it's worth the effort and expense, 
but I might happily crack the nut and do notes or a tutorial within the 
context of writing custom processors).


[1 https://cwiki.apache.org/confluence/display/NIFI/Localize+NiFi ]
[2 
https://stackoverflow.com/questions/69766648/how-to-internationalization-in-java-annotations 
]

Custom-processor configuration suggestions

[Russell Bateman]

I'm posting this plea for suggestions as I'm short on imagination here.

We have some custom processors that need extraordinary amounts of 
configuration of the sort a flow writer would have to copy and paste 
in--huge amounts of Yaml, regular expressions, etc. This is what our 
flow writers are already doing. It would be easier to insert a filename 
or -path, but...


...asking a custom processor to perform filesystem I/O is icky because 
of unpredictable filesystem access post installation. Thinking about how 
installation is beyond my control, I don't want to make installation 
messy, etc. Containers, Kubernetes deployment, etc. complicate this.


I thought of wiring /GetFile/ to a subdirectory (problematic, but less 
so?) and accepting files as input to pass on to needy processors who 
would recognize, adopt and incorporate configuration based on 
higher-level and simpler cues posted by flow writers as property values.


Assuming you both grok and are interested in what I'm asking, do you 
have thoughts, cautionary statements or even cat-calls to offer? Maybe 
there are obvious answers I'm just not thinking of.


Profuse thanks,

Russ

Re: new PackageFlowFile processor

[Russell Bateman]

Uh, sorry, "Version 3" refers to what exactly?

On 9/8/23 12:48, David Handermann wrote:

I agree that this would be a useful general feature. I also agree with
Joe that format support should be limited to*Version 3*  due to the
limitations of the earlier versions.

This is definitely something that would be useful on the 1.x support
branch to provide a smooth upgrade path for NiFi 2.

This general topic also came up on the dev channel on the Apache NiFi
Slack group:

https://apachenifi.slack.com/archives/C0L9S92JY/p1692115270146369

One key thing to note from that discussion is supporting
interoperability with services outside of NiFi. That may be too much
of a stretch for an initial implementation, but it is something I am
planning to evaluate as time allows.

For now, something focused narrowly on FlowFile Version 3 encoding
seems like the best approach.

I recommend referencing this discussion in a new Jira issue and
outlining the general design goals.

Regards,
David Handermann


On Fri, Sep 8, 2023 at 1:11 PM Adam Taft  wrote:

And also ... if we can land this in a 1.x release, this would help
tremendously to those who are going to need a replacement for PostHTTP and
don't want to "go dark" when they make the transition.

That is, without this processor in 1.x, when a user upgrades from 1.x to
2.x, they will either have to have a MergeContent/InvokeHTTP solution in
place already to replace PostHTTP, or they will have to take a (hopefully
short) outage when they bring their canvas back up (removing PostHTTP and
replacing with PackageFlowFile + InvokeHTTP).

With this processor in 1.x, they can make that transition while PostHTTP is
still available on their canvas. Wishful thinking that we can make the
entire journey from 1.x to 2.x as smooth as possible, but this could
potentially help some.


On Fri, Sep 8, 2023 at 10:55 AM Adam Taft  wrote:


+1 on this as well. It's something I've kind of griped about before (with
the loss of PostHTTP).

I don't think it would be horrible (as per Joe's concern) to offer a N:1
"bundling" property. It would just have to be stupid simple. No "groups",
timeouts, correlation attributes, minimum entries, etc. It should just
basically call the ProcessSession#get(int maxResults) where "maxResults" is
a configurable property. Whatever number of flowfiles returned in the list
is what is "bundled" into FFv3 format for output.

/Adam


On Fri, Sep 8, 2023 at 7:19 AM Phillip Lord
wrote:


+1 from me.
I’ve experimented with both methods.  The simplicity of a PackageFlowfile
straight up 1:1 is convenient and straightforward.
MergeContent on the other hand can be difficult to understand and tweak
appropriately to gain desired results/throughput.
On Sep 8, 2023 at 10:14 AM -0400, Joe Witt, wrote:

Ok. Certainly simplifies it but likely makes it applicable to larger
flowfiles only. The format is meant to allow appending and result in

large

sets of flowfiles for io efficiency and specifically for storage as the
small files/tons of files thing can cause poor performance pretty

quickly

(10s of thousands of files in a single directory).

But maybe that simplicity is fine and we just link to the MergeContent
packaging option if users need more.

On Fri, Sep 8, 2023 at 7:06 AM Michael Moser

wrote:

I was thinking 1 file in -> 1 flowfile-v3 file out. No merging of

multiple

files at all. Probably change the mime.type attribute. It might not

even

have any config properties at all if we only support flowfile-v3 and

not v1

or v2.

-- Mike


On Fri, Sep 8, 2023 at 9:56 AM Joe Witt  wrote:


Mike

In user terms this makes sense to me. Id only bother with v3 or

whatever

is

latest. We want to dump the old code. And if there are seriously

older

versions v1,v2 then nifi 1.x can be used.

The challenge is that you end up needing some of the same

complexity in

implementation and config of merge content i think. What did you

have in

mind for that?

Thanks

On Fri, Sep 8, 2023 at 6:53 AM Michael Moser

wrote:

Devs,

I can't find if this was suggested before, so here goes. With the

demise

of PostHTTP in NiFi 2.0, the recommended alternative is to

MergeContent 1

file into FlowFile-v3 format then InvokeHTTP. What does the

community

think about supporting a new PackageFlowFile processor that is

simple

to

configure (compared to MergeContent!) and simply packages flowfile
attributes + content into a FlowFile-v[1,2,3] format? This would

also

offer a simple way to export flowfiles from NiFi that could later

be

re-ingested and recovered using UnpackContent. I don't want to

submit

a

PR

for such a processor without first asking the community whether

this

would

be acceptable.

Thanks,
-- Mike

Re: Refreshing HTML displayed for View usage

[Russell Bateman]

Thanks, Matt. This is happening locally during development where I'm not 
using DNS (just localhost). I find this problem less acute on older 
versions of NiFi (1.1.3.2) than more recent ones (1.23.0, 1.19.1).


Thanks,
Russ

On 8/20/23 19:03, Matthew Hawkins wrote:

Hi Russell,

Something I've noticed myself, and it's easily reproducible on the Usage
pages for processors, is that Nifi is doing a reverse DNS lookup when
loading the page. If your DNS is broken, it can take 10-30 seconds for the
page content to appear.

Kr,

On Fri, 18 Aug 2023, 07:29 Russell Bateman,  wrote:


Matt,

I experimented with the Open in New Window button, but the new window,
which probably isn't an iframe, nevertheless doesn't respond to attempts
to get around the cache.

You're probably right about the version not changing being at the root
of the browser's inability to respond to a change. Sadly, the
edit-display-debug-edit cycle doesn't excite me into changing versions
just so I can see the result.

I'm certainly not going to clamor for work to be done to get around
this. It's annoying, but not crippling.

Thanks,

Russ


On 8/17/23 13:54, Matt Gilman wrote:

Russell,

Thanks for the response. The suggestion to open a new tab was for the
generated documentation, not the NiFi canvas itself. The generated
documentation is shown in an iframe which will offer you the menu item in
the context menu for opening in a new tab. IIRC correctly, the path to

the

generated documentation contains the version of the component. For folks
simply using a released version of NiFi this works great since versions
will differ and any browser caching will only optimize page loading for

the

user. If you're a developer, however, you're likely making changes to

your

component and the version is not changing (x.y.z-SNAPSHOT). This is where
the browser cache could result in the behavior your seeing.

Matt

On Thu, Aug 17, 2023 at 3:21 PM Russell Bateman
wrote:


Thanks, Matt,

(Since I use Chrome 99% of the time, I'm using Chrome here.)

Clicking on the reload control next to the address bar, while holding
down Shift, reloads the canvas, but the NiFi Documentation page
disappears and I have to reload it using the View usage context menu of
my processor. Once reloaded, nothing has changed.

This is also the behavior of the Configure Processor dialog. As I reload
View usage, ...

In Chrome, there's no effective content-menu option to right-click in
the page contents and open in a new tab. I can...

  Back
  Forward
  Reload
  ---
  Save As...
  Print
  Cast
  Search images with Google
  -
  Send to your devices
  Create QR code for this page
  
  Translate
  -
  View page source
  View frame source
  Reload frame
  Inspect


If I right-click the current tab itself, I'm able to choose to
Duplicate., but that doesn't seem to lead to anything more helpful.

By the way, I'm doing this using NiFi 1.23.0. I have also verified the
date of my NAR to ensure that I'm not repeatedly working using a NAR
with old content. Also, the version displayed for the custom processor
is that of the various /pom.xml/ files I'm building with.

I usually develop on the latest NiFi version. However, my company has
NARs that won't load beginning 1.14.0. I just brought up 1.13.2, what
most of our customers run, and tried my latest NAR which *does* display
my changes.Same with 1.1.2, which represents the oldest platform
executing at any custom site. Exceptionally, I set up 1.19.1 where it
*also works*.

This is unexpected; maybe it does point, as unlikely as it seems, to
something changed in 1.23.0 (instead of being caused by anything between
the dossier of my chair and the space bar of my keyboard as I have been
expecting to learn. ;-)  )

It could still be me and I'm just not seeing the obvious yet.


On 8/17/23 12:44, Matt Gilman wrote:

Russell,

Assuming this is a browser cache issue, can you try right-clicking

anywhere

in the page contents of the generated documentation and open it in a

new

tab. Once that is open, can you try doing a hard refresh by holding

Shift

while clicking Reload next to the address bar? This should clear the

cache

and fetch the updated generated documentation. At this point, you

should

be

able to close that tab and retry from the NiFi UI.

Let us know if this doesn't help and we can see if something isn't

getting

generated and updated correctly.

Matt

On Thu, Aug 17, 2023 at 2:38 PM Russell Bateman
wrote:


Seems like a really stupid user/browser question, but I cannot seem to
get changes I've made to properties, relationships, attributes read or
written, etc. *for custom processors*. Also, from the Properties tab

in

Configuring Processor, the cartoon blurbs obtained by hovering over

(?)

aren't updated either.

This is despite that changes I make to my /additionalDetails.html/

files

come

Re: Refreshing HTML displayed for View usage

[Russell Bateman]

Matt,

I experimented with the Open in New Window button, but the new window, 
which probably isn't an iframe, nevertheless doesn't respond to attempts 
to get around the cache.


You're probably right about the version not changing being at the root 
of the browser's inability to respond to a change. Sadly, the 
edit-display-debug-edit cycle doesn't excite me into changing versions 
just so I can see the result.


I'm certainly not going to clamor for work to be done to get around 
this. It's annoying, but not crippling.


Thanks,

Russ


On 8/17/23 13:54, Matt Gilman wrote:

Russell,

Thanks for the response. The suggestion to open a new tab was for the
generated documentation, not the NiFi canvas itself. The generated
documentation is shown in an iframe which will offer you the menu item in
the context menu for opening in a new tab. IIRC correctly, the path to the
generated documentation contains the version of the component. For folks
simply using a released version of NiFi this works great since versions
will differ and any browser caching will only optimize page loading for the
user. If you're a developer, however, you're likely making changes to your
component and the version is not changing (x.y.z-SNAPSHOT). This is where
the browser cache could result in the behavior your seeing.

Matt

On Thu, Aug 17, 2023 at 3:21 PM Russell Bateman
wrote:


Thanks, Matt,

(Since I use Chrome 99% of the time, I'm using Chrome here.)

Clicking on the reload control next to the address bar, while holding
down Shift, reloads the canvas, but the NiFi Documentation page
disappears and I have to reload it using the View usage context menu of
my processor. Once reloaded, nothing has changed.

This is also the behavior of the Configure Processor dialog. As I reload
View usage, ...

In Chrome, there's no effective content-menu option to right-click in
the page contents and open in a new tab. I can...

 Back
 Forward
 Reload
 ---
 Save As...
 Print
 Cast
 Search images with Google
 -
 Send to your devices
 Create QR code for this page
 
 Translate
 -
 View page source
 View frame source
 Reload frame
 Inspect


If I right-click the current tab itself, I'm able to choose to
Duplicate., but that doesn't seem to lead to anything more helpful.

By the way, I'm doing this using NiFi 1.23.0. I have also verified the
date of my NAR to ensure that I'm not repeatedly working using a NAR
with old content. Also, the version displayed for the custom processor
is that of the various /pom.xml/ files I'm building with.

I usually develop on the latest NiFi version. However, my company has
NARs that won't load beginning 1.14.0. I just brought up 1.13.2, what
most of our customers run, and tried my latest NAR which *does* display
my changes.Same with 1.1.2, which represents the oldest platform
executing at any custom site. Exceptionally, I set up 1.19.1 where it
*also works*.

This is unexpected; maybe it does point, as unlikely as it seems, to
something changed in 1.23.0 (instead of being caused by anything between
the dossier of my chair and the space bar of my keyboard as I have been
expecting to learn. ;-)  )

It could still be me and I'm just not seeing the obvious yet.


On 8/17/23 12:44, Matt Gilman wrote:

Russell,

Assuming this is a browser cache issue, can you try right-clicking

anywhere

in the page contents of the generated documentation and open it in a new
tab. Once that is open, can you try doing a hard refresh by holding Shift
while clicking Reload next to the address bar? This should clear the

cache

and fetch the updated generated documentation. At this point, you should

be

able to close that tab and retry from the NiFi UI.

Let us know if this doesn't help and we can see if something isn't

getting

generated and updated correctly.

Matt

On Thu, Aug 17, 2023 at 2:38 PM Russell Bateman
wrote:


Seems like a really stupid user/browser question, but I cannot seem to
get changes I've made to properties, relationships, attributes read or
written, etc. *for custom processors*. Also, from the Properties tab in
Configuring Processor, the cartoon blurbs obtained by hovering over (?)
aren't updated either.

This is despite that changes I make to my /additionalDetails.html/ files
come through with no problem.

I have tried Chrome, Opera, Brave and Firefox. All [mis]behave

identically.

I have tried killing the browser running the NiFi UI, stopping also the
NiFi instance, relaunching NiFi, quickly asking the browser to display
the canvas, then holding down the Ctrl key as I click Reload.

In past times, I have observed that eventually the changes are
recognized and displayed, so this is not permanent, however, it's very
annoying especially when I'm polishing my processor documentation
wording in the class that extends AbstractProcessor. All the while, as I
say, the processor's

Re: Refreshing HTML displayed for View usage

[Russell Bateman]

Thanks, Matt,

(Since I use Chrome 99% of the time, I'm using Chrome here.)

Clicking on the reload control next to the address bar, while holding 
down Shift, reloads the canvas, but the NiFi Documentation page 
disappears and I have to reload it using the View usage context menu of 
my processor. Once reloaded, nothing has changed.


This is also the behavior of the Configure Processor dialog. As I reload 
View usage, ...


In Chrome, there's no effective content-menu option to right-click in 
the page contents and open in a new tab. I can...


   Back
   Forward
   Reload
   ---
   Save As...
   Print
   Cast
   Search images with Google
   -
   Send to your devices
   Create QR code for this page
   
   Translate
   -
   View page source
   View frame source
   Reload frame
   Inspect


If I right-click the current tab itself, I'm able to choose to 
Duplicate., but that doesn't seem to lead to anything more helpful.


By the way, I'm doing this using NiFi 1.23.0. I have also verified the 
date of my NAR to ensure that I'm not repeatedly working using a NAR 
with old content. Also, the version displayed for the custom processor 
is that of the various /pom.xml/ files I'm building with.


I usually develop on the latest NiFi version. However, my company has 
NARs that won't load beginning 1.14.0. I just brought up 1.13.2, what 
most of our customers run, and tried my latest NAR which *does* display 
my changes.Same with 1.1.2, which represents the oldest platform 
executing at any custom site. Exceptionally, I set up 1.19.1 where it 
*also works*.


This is unexpected; maybe it does point, as unlikely as it seems, to 
something changed in 1.23.0 (instead of being caused by anything between 
the dossier of my chair and the space bar of my keyboard as I have been 
expecting to learn. ;-)  )


It could still be me and I'm just not seeing the obvious yet.


On 8/17/23 12:44, Matt Gilman wrote:

Russell,

Assuming this is a browser cache issue, can you try right-clicking anywhere
in the page contents of the generated documentation and open it in a new
tab. Once that is open, can you try doing a hard refresh by holding Shift
while clicking Reload next to the address bar? This should clear the cache
and fetch the updated generated documentation. At this point, you should be
able to close that tab and retry from the NiFi UI.

Let us know if this doesn't help and we can see if something isn't getting
generated and updated correctly.

Matt

On Thu, Aug 17, 2023 at 2:38 PM Russell Bateman
wrote:


Seems like a really stupid user/browser question, but I cannot seem to
get changes I've made to properties, relationships, attributes read or
written, etc. *for custom processors*. Also, from the Properties tab in
Configuring Processor, the cartoon blurbs obtained by hovering over (?)
aren't updated either.

This is despite that changes I make to my /additionalDetails.html/ files
come through with no problem.

I have tried Chrome, Opera, Brave and Firefox. All [mis]behave identically.

I have tried killing the browser running the NiFi UI, stopping also the
NiFi instance, relaunching NiFi, quickly asking the browser to display
the canvas, then holding down the Ctrl key as I click Reload.

In past times, I have observed that eventually the changes are
recognized and displayed, so this is not permanent, however, it's very
annoying especially when I'm polishing my processor documentation
wording in the class that extends AbstractProcessor. All the while, as I
say, the processor's corresponding /additionalDetails.html/ displays
changes I make to it in a timely and accurate fashion.

Suggestions?

Refreshing HTML displayed for View usage

[Russell Bateman]

Seems like a really stupid user/browser question, but I cannot seem to 
get changes I've made to properties, relationships, attributes read or 
written, etc. *for custom processors*. Also, from the Properties tab in 
Configuring Processor, the cartoon blurbs obtained by hovering over (?) 
aren't updated either.


This is despite that changes I make to my /additionalDetails.html/ files 
come through with no problem.


I have tried Chrome, Opera, Brave and Firefox. All [mis]behave identically.

I have tried killing the browser running the NiFi UI, stopping also the 
NiFi instance, relaunching NiFi, quickly asking the browser to display 
the canvas, then holding down the Ctrl key as I click Reload.


In past times, I have observed that eventually the changes are 
recognized and displayed, so this is not permanent, however, it's very 
annoying especially when I'm polishing my processor documentation 
wording in the class that extends AbstractProcessor. All the while, as I 
say, the processor's corresponding /additionalDetails.html/ displays 
changes I make to it in a timely and accurate fashion.


Suggestions?

Custom processor once had properties, has no more now, but they still show up

[Russell Bateman]

I have a custom processor I modified losing a few properties because a 
specification change made them useless. I removed them. The processor 
works, but in configuration for this processor (in the NiFi UI), the 
processor appears to have kept them, i.e.: they're not disappearing. I 
would have expected trash cans on each, but they're just still there.


Yes, the processor bears the latest NAR version number and there is no 
"duplicate" or older NAR in the /extensions/ subdirectory.


Is my best recourse to remove the processor from the flow and re-add it? 
This is not a problem, but I'll have down-streamers with their own flows 
I'm not in contact with who will find this behavior unsettling. I could 
issue documentation, but I try to keep extra work to a minimum for them.


Thanks for any thoughts.

Russ

Re: Use of attribute uuid and other "native" attributes

[Russell Bateman]

Of course, a custom processor can create any attribute, including an 
"external id field." I don't think it can "lose" the original uuid 
since, if it attempts to reset it, the action will be quietly ignored 
(Mark).


Note that uuid figures prominently in the display of provenance--in my 
mind the crucial nature of my question. [1]


My question was about the "sanctified" state (or not) of uuid and Matt 
and Mark gave succinct and useful answers that I will explore. I was 
unaware of the suggested "best practice" of considering losing any and 
all previously established attributes before sending flowfiles on. I 
have long done this explicitly in the case of attributes I create, but 
will now contemplate doing it for other attributes I did not create and 
therefore have respected "religiously."


Russ

[1] 
https://www.tutorialspoint.com/apache_nifi/apache_nifi_data_provenance.htm


On 7/18/23 14:07, Edward Armes wrote:

Hmm,

I've seen this come up a few times now I wonder is there need for a rename
of the uuid field and a creation of an external id field?

Edward

On Tue, 18 Jul 2023, 20:53 Lucas Ottersbach,
wrote:


Hey Matt,

you wrote that both `Session.create` and `Session.clone` set a new FlowFile
UUID to the resulting FlowFile. This somewhat sounds like there is an
alternative way where the UUID is not controlled by the framework itself?

I've got a different use case than Russell, but was wondering whether it is
even possible to control the FlowFile UUID as a Processor developer? I've
got a processor pair for inter-cluster transfer of FlowFiles (where
Site-to-Site is not applicable). As of now, the UUID on the receiving side
differs from the original on the origin cluster, because I'm using
`Session.create`.
Is there a way to control the UUID of new FlowFiles?


Best regards,

Lucas

Matt Burgess  schrieb am Di., 18. Juli 2023, 20:23:


In general I recommend only sending on those attributes that will be
used at some point downstream (unless you have an "original"
relationship that should maintain the original state with respect to
provenance). If you don't know that ahead of time you'll probably need
to send all/most of the attributes just in case.

Are you using session.create() or session.clone()? They both set a new
"uuid" attribute on the created FlowFile, with at least the latter
setting some other attributes as well (see the Developer Guide [1] for
more details).

Regards,
Matt

[1]https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html

On Tue, Jul 18, 2023 at 12:25 PM Russell Bateman
wrote:

I have a custom processor, /SplitHl7v4Resources/, that splits out
individual FHIR resources (Patients, Observations, Encounters, etc.)
from great Bundle flowfiles. So, for a given flowfile, it's split into
hundreds of smaller ones.

When I do this, I leave the existing NiFi attributes as they were on

the

original flowfile.

As I contemplate the uuid attribute, it occurs to me that I should find
out what its *significance is for provenance and other potential
debugging/tracing concerns*. I never really look at it, but, if there
were some kind of melt-down in a production environment, would I care
that it multiplied across hundreds of flowfiles besided the original

one?

Also these two other NiFi attributes remain unchanged:

 filename
 path


I do garnish each flowfile with many pointed/significant new attributes
like resource.type that are my own. In my processing, I don't care

about

NiFi's original attributes, but should I?

Thanks,
Russ

Use of attribute uuid and other "native" attributes

[Russell Bateman]

I have a custom processor, /SplitHl7v4Resources/, that splits out 
individual FHIR resources (Patients, Observations, Encounters, etc.) 
from great Bundle flowfiles. So, for a given flowfile, it's split into 
hundreds of smaller ones.


When I do this, I leave the existing NiFi attributes as they were on the 
original flowfile.


As I contemplate the uuid attribute, it occurs to me that I should find 
out what its *significance is for provenance and other potential 
debugging/tracing concerns*. I never really look at it, but, if there 
were some kind of melt-down in a production environment, would I care 
that it multiplied across hundreds of flowfiles besided the original one?


Also these two other NiFi attributes remain unchanged:

   filename
   path


I do garnish each flowfile with many pointed/significant new attributes 
like resource.type that are my own. In my processing, I don't care about 
NiFi's original attributes, but should I?


Thanks,
Russ

Re: [HAPI-devel] [External] Unable to inspect gender and birthdate in Patient

[Russell Bateman]

Yeah, sorry. It's been a very long time since I posted a question on 
HAPI FHIR and I couldn't find the forum. Then, there's the fact that I 
discovered that I had put totally bogus values into those fields which 
HAPI FHIR didn't catch in its parse, but refused to give them back when 
I asked for them. So, I fixed that and I'm working.


Again, sorry for the multiple screw-ups.

On 6/13/23 15:49, McClendon, Craig wrote:


If I’m not mistaken, this is the mailing list for the HAPI HL7v2 library.

This code below looks to be using HAPI FHIR – different library.

See the get help section here for more info: https://hapifhir.io/

Incidentally, you are creating a new (empty) Patient object, then 
trying to fetch the name. It doesn’t have one because you haven’t set 
one.


*From:* Russell Bateman 
*Sent:* Tuesday, June 13, 2023 1:54 PM
*To:* hl7api-devel@lists.sourceforge.net
*Subject:* [External] [HAPI-devel] Unable to inspect gender and 
birthdate in Patient


This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly 
with links and attachments.




In the code below, I have an incoming Patient which has all the fields 
that interest me filled in (see debugger image, but if it doesn't come 
through, it shows that patient is filled out for name, gender, 
birthdate and address).


However, when I attempt to inspect *gender* and *birthdate*, these are 
*null* despite the API telling me they exist (which, of course, they 
do--see attached image). Note that I create a new Patient, named 
worthy, and experimentation demonstrates that my code below copies the 
name and the address successfully to worthy. It's just trying to copy 
gender and birthdate that result in null. If I stop at the lines with 
***, patient.getGender() and patient.getBirthDate() return null.


There must be some very fundamental bit of understanding of which I'm 
completely ignorant.


  {
    worthy = new Patient();

    // name
    if( !patient.hasName() )
  patientTooThin( "name" );
    HumanName  humanName = patient.getNameFirstRep();
    List< StringType > givens    = humanName.getGiven();
    String family    = humanName.getFamily();
    String first = null, middle = null;
    for( StringType given : givens )
    {
  if( isNull( first ) )
    first = given.toString();
  else if( isNull( middle ) )
    middle = given.toString();
  else
    break;
    }

    if( isNull( family ) || isNull( first ) )
  patientTooThin( "first or last name" );

    worthy.setName( Collections.singletonList( humanName ) );

    // gender  *
    if( !patient.hasGender() )
  patientTooThin( "gender" );

    worthy.setGender( patient.getGender() );        // ***

    // birthDate
    if( !patient.hasBirthDate() )
  patientTooThin( "birth date" );

    worthy.setBirthDate( patient.getBirthDate() );  // ***

    // address
    if( !patient.hasAddress() )
  patientTooThin( "address" );

    worthy.setAddress( Collections.singletonList( 
patient.getAddressFirstRep() ) );

  }





This message is for the designated recipient only and may contain 
privileged, proprietary, or otherwise confidential information. If you 
have received it in error, please notify the sender immediately and 
delete the original. Any other use of the e-mail by you is prohibited. 
Where allowed by local law, electronic communications with Accenture 
and its affiliates, including e-mail and instant messaging (including 
content), may be scanned by our systems for the purposes of 
information security, AI-powered support capabilities, and assessment 
of internal compliance with Accenture policy. Your privacy is 
important to us. Accenture uses your personal data only in compliance 
with data protection laws. For further information on how Accenture 
processes your personal data, please see our privacy statement at 
https://www.accenture.com/us-en/privacy-policy.

__

www.accenture.com
___
Hl7api-devel mailing list
Hl7api-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/hl7api-devel

[HAPI-devel] Unable to inspect gender and birthdate in Patient

[Russell Bateman]

In the code below, I have an incoming Patient which has all the fields 
that interest me filled in (see debugger image, but if it doesn't come 
through, it shows that patient is filled out for name, gender, birthdate 
and address).


However, when I attempt to inspect *gender* and *birthdate*, these are 
*null* despite the API telling me they exist (which, of course, they 
do--see attached image). Note that I create a new Patient, named worthy, 
and experimentation demonstrates that my code below copies the name and 
the address successfully to worthy. It's just trying to copy gender and 
birthdate that result in null. If I stop at the lines with ***, 
patient.getGender() and patient.getBirthDate() return null.


There must be some very fundamental bit of understanding of which I'm 
completely ignorant.


  {
    worthy = new Patient();

    // name
    if( !patient.hasName() )
  patientTooThin( "name" );
    HumanName  humanName = patient.getNameFirstRep();
    List< StringType > givens    = humanName.getGiven();
    String family    = humanName.getFamily();
    String first = null, middle = null;
    for( StringType given : givens )
    {
  if( isNull( first ) )
    first = given.toString();
  else if( isNull( middle ) )
    middle = given.toString();
  else
    break;
    }

    if( isNull( family ) || isNull( first ) )
  patientTooThin( "first or last name" );

    worthy.setName( Collections.singletonList( humanName ) );

    // gender  *
    if( !patient.hasGender() )
  patientTooThin( "gender" );

    worthy.setGender( patient.getGender() );   // ***

    // birthDate
    if( !patient.hasBirthDate() )
  patientTooThin( "birth date" );

    worthy.setBirthDate( patient.getBirthDate() );  // ***

    // address
    if( !patient.hasAddress() )
  patientTooThin( "address" );

    worthy.setAddress( Collections.singletonList( 
patient.getAddressFirstRep() ) );

  }


___
Hl7api-devel mailing list
Hl7api-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/hl7api-devel

Re: Possible Docker bug in 1.13.2

[Russell Bateman]

Thanks, Chris.

In fact, I merely hand-corrected the instances of the version in 
/Dockerfile/ and in /DockerImage.txt/, then ran /DockerBuild.sh/. I got 
exactly the image I wanted. It looked like a potential if irrelevant bug 
and I thought I'd report it.


Best,
Russ

On 6/7/23 19:17, Chris Sampson wrote:

The DockerImage.txt file isn't always updated in the repo, a bit like the
README for Docker Hub - it probably should be, but is often forgotten (same
for all of the convenience images built after a release). Indeed, this is
currently set to 1.15.1 on `main`, but the images in Docker Hub since then
have certainly not used an incorrect version.

Nifi 1.16.0 [1] updated the DockerBuild.sh script to allow easier
specification of the NIFI_IMAGE_VERSION from the command line. A small
issue has just been spotted with this script's use of the text file to
obtain the version's default value too [4].

Before then, I think the process was for someone (with access to push
images to Docker Hub) to change the text file locally, build & push the
image, but not always then raise a PR to update the text file.

To manually build the Docker Image locally containing the correct version
of the downloaded convenience binary from the Apache servers, simply
recreate the `docker build` command from within the DockerBuild.sh [2]
file, substituting the correct values for the `--build-arg`s.

Alternatively, run the dockermaven [3] build for the repo (having checked
out the correct tag from GitHub), enabling the `-P docker` profile as you
do so in your `mvn` command. Note, however, that there were differences
between the dockermaven and dockerhub builds (both the included script
files and the Dockerfile structure) that weren't rationalised until NiFi
1.16.0+.


[1]:
https://github.com/apache/nifi/blob/rel/nifi-1.16.0/nifi-docker/dockerhub/DockerBuild.sh

[2]:
https://github.com/apache/nifi/blob/rel/nifi-1.13.2/nifi-docker/dockerhub/DockerBuild.sh#L36

[3]:
https://github.com/apache/nifi/tree/rel/nifi-1.16.0/nifi-docker/dockermaven

[4]:
https://apachenifi.slack.com/archives/CDGMCSDJT/p1686135561932289?thread_ts=1686135561.932289=CDGMCSDJT

On Thu, 8 Jun 2023, 00:10 Russell Bateman,  wrote:


I'm re-rolling in order to update the Java inside to 11 in order to
permit using the new Java HTTP client. This seems to work well; I fixed
the bug locally.

Maybe too old to be important, but NiFi 1.14.0 is a quantum step up for
several aspects of processor writing requiring refactoring. So, until we
can shed the old NAR we cannot rebuild, we're stuck at NiFi 1.13.2.


On 6/7/23 15:31, Russell Bateman wrote:

I downloaded sources to 1.13.2 in order to hand-spin my own container
image. When I got down to
/nifi-1.13.2/nifi-docker/dockerhub/Dockerfile/, I found:

 ...
 ARG NIFI_VERSION=1.13.1
 ...

and the version is also wrong in /DockerImage.txt/ which
/DockerBuild.sh/ consumes.

Indeed, the image that is built appears to be versioned 1.13.1 and not
1.13.2 when listing local Docker repository images:

 *REPOSITORY TAG IMAGE ID   CREATED  SIZE*
 apache/nifi 1.13.1  8c18038f152a   30 minutes ago 2.06GB

Why am I juggling so ancient a version? Because I have custom
processors that cannot be rebuilt (source-code gone) and will not run
on 1.14.0 and later.

Russ

Re: Possible Docker bug in 1.13.2

[Russell Bateman]

I'm re-rolling in order to update the Java inside to 11 in order to 
permit using the new Java HTTP client. This seems to work well; I fixed 
the bug locally.


Maybe too old to be important, but NiFi 1.14.0 is a quantum step up for 
several aspects of processor writing requiring refactoring. So, until we 
can shed the old NAR we cannot rebuild, we're stuck at NiFi 1.13.2.



On 6/7/23 15:31, Russell Bateman wrote:
I downloaded sources to 1.13.2 in order to hand-spin my own container 
image. When I got down to 
/nifi-1.13.2/nifi-docker/dockerhub/Dockerfile/, I found:


...
ARG NIFI_VERSION=1.13.1
...

and the version is also wrong in /DockerImage.txt/ which 
/DockerBuild.sh/ consumes.


Indeed, the image that is built appears to be versioned 1.13.1 and not 
1.13.2 when listing local Docker repository images:


*REPOSITORY TAG IMAGE ID   CREATED  SIZE*
apache/nifi 1.13.1  8c18038f152a   30 minutes ago 2.06GB

Why am I juggling so ancient a version? Because I have custom 
processors that cannot be rebuilt (source-code gone) and will not run 
on 1.14.0 and later.


Russ

Possible Docker bug in 1.13.2

[Russell Bateman]

I downloaded sources to 1.13.2 in order to hand-spin my own container 
image. When I got down to 
/nifi-1.13.2/nifi-docker/dockerhub/Dockerfile/, I found:


   ...
   ARG NIFI_VERSION=1.13.1
   ...

and the version is also wrong in /DockerImage.txt/ which 
/DockerBuild.sh/ consumes.


Indeed, the image that is built appears to be versioned 1.13.1 and not 
1.13.2 when listing local Docker repository images:


   *REPOSITORY TAG IMAGE ID   CREATED  SIZE*
   apache/nifi 1.13.1  8c18038f152a   30 minutes ago   2.06GB

Why am I juggling so ancient a version? Because I have custom processors 
that cannot be rebuilt (source-code gone) and will not run on 1.14.0 and 
later.


Russ

Re: Usage Documentation for Custom Processors

[Russell Bateman]

Matthew,

If you feel that the documentation generated from the annotations at the 
top of your custom processor class (@CapabilityDescription, etc., of 
which Bryan spoke) is insufficient, it's also possible to supplement it with


   
/src/main/resources/docs/.CustomProcessorClass/additionalDetails.html/

You write it in simple HTML with embedded CSS. By your user, it's 
reached via a hyperlink on the (standard) processor usage page put there 
when the framework notices that you've supplied it (directory name 
including package path, filesystem location, etc. are crucial).


I do this for almost every last custom processor I write as a favor to 
my downstream flow writers.


Cheers,

Russ


On 4/4/23 08:54, Matthew Baine wrote:

Hi Bryan,

Sorry, on a separate note, what would be the best way to set up Usage 
Documentation for a custom processor?


image.png

We can't seem to get this right with the information online and on the 
Nifi developer guide 
(https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html). 
Our custom processors seem to only publish documentation of the native 
processors.



Kind Regards,
Matthew

On Tue, 4 Apr 2023 at 13:54, Matthew Baine  
wrote:


Hi Bryan,

Sorry for the delayed response, and thank you so much for the
feedback!

We will attempt the advised approach and revert if we run into any
trouble.

Thanks again!

Regards,

On Thu, 30 Mar 2023 at 16:49, Bryan Bende  wrote:

Hello,

This might not give you exactly what you want, but the Minifi
Toolkit
already has the ability to transform the JSON snapshot from
registry,
there are actually two commands:

"transform" - for XML templates
"transform-vfs" - for versioned flow snapshot (JSON from
registry) [1]

It doesn't pull the snapshot from registry directly, so you
would have
to script something to download the snapshot and then run
transform-vfs.

Thanks,

Bryan

[1]

https://github.com/apache/nifi/blob/main/minifi/minifi-toolkit/minifi-toolkit-configuration/src/main/java/org/apache/nifi/minifi/toolkit/configuration/ConfigMain.java#L62

On Thu, Mar 30, 2023 at 10:22 AM Simeon Wentzel
 wrote:
>
> Dear Nifi dev team
>
> Can you add extended functionality to the MiNiFi toolkit to
extract a flow
> from the NiFi Registry software and convert it to the
appropriate conf.yml
> file?
>
> We have found a limitation regarding the conversion in the
minifi toolkit
> that it can only convert the .xml file template extracted
from a Nifi
> canvas on Java version 8, it can not do the conversion on
java 11 that we
> have migrated to.
>
> Although extracting the flow as a template out of nifi and
then converting
> it to the conf.yaml file works we find it a bit cumbersome
because we can
> not implement it in our pipeline to automate the process.
>
> By allowing the minifi toolkit to pull a flow from the Nifi
registry and
> then convert it will give us the functionality to add this
in our Jenkins
> pipeline to build individual docker containers for each of
our flows.
>
> Regards
> Simeon
> DevOps Engineer



-- 





*Matthew Baine | *DevOps Engineer

*Johannesburg Head Office*

E: matt...@airvantage.co.za  | M:
+27 (0) 71053 9012 

T: +27 (0) 11 100 1880  | W:
www.airvantage.co.za 

*Skype: matthew.baine57*



--




*Matthew Baine | *DevOps Engineer

*Johannesburg Head Office*

E: matt...@airvantage.co.za  | M: +27 
(0) 71053 9012 


T: +27 (0) 11 100 1880  | W: www.airvantage.co.za 



*Skype: matthew.baine57*

Re: Size of CLOB...

[Russell Bateman]

Late yesterday afternoon, I hit upon the idea of inserting a '?' where 
the CLOB would be, then using PreparedStatement to satisfy it. *This 
worked and I'm able to insert even 20Mb-long rows*:


final String INSERT_CLOB = "INSERT INTO ... ? ...)"; Clob  clob  =  
connection.createClob();

clob.setString(1,MsgContent  );

PreparedStatement  preparedStatement  =  
connection.prepareStatement(INSERT_CLOB  );

preparedStatement.setClob(1,clob  );


Your answers, received just this morning, would have led me to this same 
solution. Thank you, very much!


Love Derby, I do!

Russ

On 3/23/23 14:54, Russell Bateman wrote:
In fact, experimentation seems to suggest that the breaking point is a 
length of 32K+. Whether or not the length of the rest of the INSERT 
INTO statement counts I do not know for certain, but the delta between 
MsgContent and the rest is negligeable, so I don't care.


On 3/23/23 13:02, Russell Bateman wrote:
According to what I'm reading, a CLOB should easily hold more than 
the 128K I'm trying to put into it.


My table schema:
private static final StringTABLE_SCHEMA ="\n" +" MsgId BIGINT NOT NULL GENERATED ALWAYS AS IDENTITY,\n" +" MsgContent CLOB DEFAULT NULL,\n" +" MsgReceived TIMESTAMP NOT NULL WITH DEFAULT 
CURRENT_TIMESTAMP,\n" +" MsgSource VARCHAR(256) DEFAULT NULL,\n" +" MsgFacility VARCHAR(256) DEFAULT NULL,\n" +" MsgType1 VARCHAR(256) DEFAULT NULL,\n" +" MsgType2 VARCHAR(256) DEFAULT NULL,\n" 
+" MsgAltId1 VARCHAR(256) DEFAULT NULL,\n" +" MsgAltId2 VARCHAR(256) DEFAULT NULL,\n" +" MsgMisc1 VARCHAR(256) DEFAULT NULL,\n" +" MsgMisc2 VARCHAR(256) DEFAULT NULL,\n" +" MsgMisc3 VARCHAR(256) 
DEFAULT NULL,\n" +" MsgStatus CHAR(3) DEFAULT NULL,\n" +" MsgStatusDate TIMESTAMP DEFAULT NULL,\n" +" MsgStatusInfo VARCHAR(256) DEFAULT NULL\n";
My insert string:

*INSERT INTO* ssdi_small_tbl( MsgContent, MsgReceived, MsgSource,
MsgFacility, MsgType1, MsgType2, MsgAltId1, MsgAltId2, MsgMisc1,
MsgStatus, MsgStatusDate, MsgStatusInfo )
*VALUES*(
    '

',
'2023-02-23 12:49:13',
'MIRTH_0ed9b98c-a94f-4c38-964800743a00c0061',
    'MOCKH',
    'ADT-A28',
    'EPIC',
    '1',
'228750038',
    'ZZZ TEST NEW',
    'REJ',
'2023-03-23 12:49:13',
    'Test Messages'
 )


My Java code performing the insertion:
try {
   PreparedStatement preparedStatement = connection.prepareStatement( INSERT );
   return( preparedStatement.executeUpdate() ==1 );
}
catch( SQLException throwables )
{
   throwables.printStackTrace();
   return false;
}
The SQLException I'm getting:

A string constant starting with "
Shorter insert string. For this string, Derby's happy and I am able 
to perform the insertion as well as select out the row later 
successfully (same code, etc.):
static final StringCONTENTS ="'MSH|^~&|LAB|HCA_OGDR|LAB||201112051038||ORU^R01^ORU_R01|4257313..LAB.COCCBM|P|2.1\\r\n" 
+" 
PID|1|665892|23422234234^^^HCA_OGDR||HERMAN^MUNSTER||18501031|M||W|1313 
MOCKINGBIRD LANE^^MOCKINGBIRD 
HEIGHTS^CA^90210||^626^2959626||English|M^Married^HL73202|Catholic|Q02690008175|529609893||202005061342|\\r\n" 
+" 
PV1|1|I|Q.IMC^Q.109^A|EM|||VICTOR^FRANKENSTEIN^IGOR^HCA||VICTOR^FRANKENSTEIN^IGOR^HCA|IN||01|||COCCBM||ADM|||202002040238\\r\n" 
+" NK1|0222555|MUNSTER^LILY^DRACULA|FA|1313 MOCKINGBIRD 
LANE^^MOCKINGBIRD HEIGHTS^CA^90210|(626)111-Mockingbird 
Heights Clinic\\r\n" +" OBR|1|L3266930^LAB|^LAB|BMP^BASIC METABOLIC 
PANEL^L|||201312050507|||QLB.XXP|||DT699 
STROKE|202002050514||VICTOR^FRANKENSTEIN^^HCA||1205:CBM:C03008R201112051038|||F\\r\n" 
+" NTE|1\\r\n" +" NTE|2||Site Legend:.brML:MOCKINGBIRD HEIGHTS CLINIC 15475 
NORTH HOLLYWOOD BOULEVARD MOCKINGBIRD HEIGHTS CA 90210.br\\r\n" +" TQ1|R^Routine^HL70485\\r\n" +" 
OBX|1|NM|NA^SODIUM^L^2951-2^SODIUM^LN|1|144|mmol/L|136-145|N||AS|F|201108241525|ML:MOCKINGBIRD 
HEIGHTS CLINIC|||ML:MOCKINGBIRD HEIGHTS CLINIC^15475 NORTH 
HOLLYWOOD BOULEVARD MOCKINGBIRD HEIGHTS CA 90210\\r\n" +" NTE|1|AD|Performed At: ML\\r\n" +" OBX|25|ST|GFR^GLOMERULAR FILTRATION 
RATE^L|1|>60||60-130|N||AS|F|201104071132|ML:MOCKINGBIRD HEIGHTS 
CLINIC|||ML:MOCKINGBIRD HEIGHTS CLINIC^15475 NORTH HOLLYWOOD 
BOULEVARD MOCKINGBIRD HEIGHTS CA 90210\\r\n" +" NTE|1|AD|Performed At: ML\\r\n" +" NTE|1||If the patient is African American, multiply by 1.210\\r\n" +" NTE|2\\r\n" +" NTE|3||eGFR Reference Range for adults =60 ml/min/1.73 m2\\r\n" +" 
OBR|2|L3261930^LAB|^LAB|PHOS^PHOSPHORUS^L|||202002050507|||QLB.XXP|||DT699 
STROKE|201112050514||FRANKENSTEIN^Victor^I

Re: Size of CLOB...

[Russell Bateman]

In fact, experimentation seems to suggest that the breaking point is a 
length of 32K+. Whether or not the length of the rest of the INSERT INTO 
statement counts I do not know for certain, but the delta between 
MsgContent and the rest is negligeable, so I don't care.


On 3/23/23 13:02, Russell Bateman wrote:
According to what I'm reading, a CLOB should easily hold more than the 
128K I'm trying to put into it.


My table schema:
private static final StringTABLE_SCHEMA ="\n" +" MsgId BIGINT NOT NULL GENERATED ALWAYS AS IDENTITY,\n" +" MsgContent CLOB DEFAULT NULL,\n" +" MsgReceived TIMESTAMP NOT NULL WITH DEFAULT 
CURRENT_TIMESTAMP,\n" +" MsgSource VARCHAR(256) DEFAULT NULL,\n" +" MsgFacility VARCHAR(256) DEFAULT NULL,\n" +" MsgType1 VARCHAR(256) DEFAULT NULL,\n" +" MsgType2 VARCHAR(256) DEFAULT NULL,\n" 
+" MsgAltId1 VARCHAR(256) DEFAULT NULL,\n" +" MsgAltId2 VARCHAR(256) DEFAULT NULL,\n" +" MsgMisc1 VARCHAR(256) DEFAULT NULL,\n" +" MsgMisc2 VARCHAR(256) DEFAULT NULL,\n" +" MsgMisc3 VARCHAR(256) 
DEFAULT NULL,\n" +" MsgStatus CHAR(3) DEFAULT NULL,\n" +" MsgStatusDate TIMESTAMP DEFAULT NULL,\n" +" MsgStatusInfo VARCHAR(256) DEFAULT NULL\n";
My insert string:

*INSERT INTO* ssdi_small_tbl( MsgContent, MsgReceived, MsgSource,
MsgFacility, MsgType1, MsgType2, MsgAltId1, MsgAltId2, MsgMisc1,
MsgStatus, MsgStatusDate, MsgStatusInfo )
*VALUES*(
    '

',
    '2023-02-23 12:49:13',
'MIRTH_0ed9b98c-a94f-4c38-964800743a00c0061',
    'MOCKH',
    'ADT-A28',
    'EPIC',
    '1',
    '228750038',
    'ZZZ TEST NEW',
    'REJ',
    '2023-03-23 12:49:13',
    'Test Messages'
 )


My Java code performing the insertion:
try {
   PreparedStatement preparedStatement = connection.prepareStatement( INSERT );
   return( preparedStatement.executeUpdate() ==1 );
}
catch( SQLException throwables )
{
   throwables.printStackTrace();
   return false;
}
The SQLException I'm getting:

A string constant starting with "
Shorter insert string. For this string, Derby's happy and I am able to 
perform the insertion as well as select out the row later successfully 
(same code, etc.):
static final StringCONTENTS ="'MSH|^~&|LAB|HCA_OGDR|LAB||201112051038||ORU^R01^ORU_R01|4257313..LAB.COCCBM|P|2.1\\r\n" 
+" 
PID|1|665892|23422234234^^^HCA_OGDR||HERMAN^MUNSTER||18501031|M||W|1313 
MOCKINGBIRD LANE^^MOCKINGBIRD 
HEIGHTS^CA^90210||^626^2959626||English|M^Married^HL73202|Catholic|Q02690008175|529609893||202005061342|\\r\n" 
+" 
PV1|1|I|Q.IMC^Q.109^A|EM|||VICTOR^FRANKENSTEIN^IGOR^HCA||VICTOR^FRANKENSTEIN^IGOR^HCA|IN||01|||COCCBM||ADM|||202002040238\\r\n" 
+" NK1|0222555|MUNSTER^LILY^DRACULA|FA|1313 MOCKINGBIRD 
LANE^^MOCKINGBIRD HEIGHTS^CA^90210|(626)111-Mockingbird 
Heights Clinic\\r\n" +" OBR|1|L3266930^LAB|^LAB|BMP^BASIC METABOLIC 
PANEL^L|||201312050507|||QLB.XXP|||DT699 
STROKE|202002050514||VICTOR^FRANKENSTEIN^^HCA||1205:CBM:C03008R201112051038|||F\\r\n" 
+" NTE|1\\r\n" +" NTE|2||Site Legend:.brML:MOCKINGBIRD HEIGHTS CLINIC 15475 
NORTH HOLLYWOOD BOULEVARD MOCKINGBIRD HEIGHTS CA 90210.br\\r\n" +" TQ1|R^Routine^HL70485\\r\n" +" 
OBX|1|NM|NA^SODIUM^L^2951-2^SODIUM^LN|1|144|mmol/L|136-145|N||AS|F|201108241525|ML:MOCKINGBIRD 
HEIGHTS CLINIC|||ML:MOCKINGBIRD HEIGHTS CLINIC^15475 NORTH 
HOLLYWOOD BOULEVARD MOCKINGBIRD HEIGHTS CA 90210\\r\n" +" NTE|1|AD|Performed At: ML\\r\n" +" OBX|25|ST|GFR^GLOMERULAR FILTRATION 
RATE^L|1|>60||60-130|N||AS|F|201104071132|ML:MOCKINGBIRD HEIGHTS 
CLINIC|||ML:MOCKINGBIRD HEIGHTS CLINIC^15475 NORTH HOLLYWOOD 
BOULEVARD MOCKINGBIRD HEIGHTS CA 90210\\r\n" +" NTE|1|AD|Performed At: ML\\r\n" +" NTE|1||If the patient is African American, multiply by 1.210\\r\n" +" NTE|2\\r\n" +" NTE|3||eGFR Reference Range for adults =60 ml/min/1.73 m2\\r\n" +" 
OBR|2|L3261930^LAB|^LAB|PHOS^PHOSPHORUS^L|||202002050507|||QLB.XXP|||DT699 
STROKE|201112050514||FRANKENSTEIN^Victor^IGOR^HCA||1205:CBM:C8R202002051038|||F\\r 
NTE|1\\r\n" +" NTE|2||Site Legend:.brML:MOCKINGBIRD HEIGHTS CLINIC15475 
NORTH HOLLYWOOD BOULEVARD MOCKINGBIRD HEIGHTS CA 90210.br\\r\n" +" TQ1|R^Routine^HL70485\\r\n" +" 
OBX|1|NM|PHOS^PHOSPHORUS^L^2777-1^PHOSPHATE^LN|1|2.4|mg/dL|2.5-4.9|L||AS|F|202002241521|ML:MOCKINGBIRD 
HEIGHTS CLINIC|||ML:MOCKINGBIRD HEIGHTS CLINIC^15475 NORTH 
HOLLYWOOD BOULEVARD MOCKINGBIRD HEIGHTS CA 90210\\r\n" +" NTE|1|AD|Performed At: ML\\r'";


Any thoughts?

Russ

Size of CLOB...

[Russell Bateman]

According to what I'm reading, a CLOB should easily hold more than the 
128K I'm trying to put into it.


My table schema:

private static final StringTABLE_SCHEMA ="\n" +" MsgId BIGINT NOT NULL GENERATED ALWAYS AS IDENTITY,\n" +" MsgContent CLOB DEFAULT NULL,\n" +" MsgReceived TIMESTAMP NOT NULL WITH DEFAULT 
CURRENT_TIMESTAMP,\n" +" MsgSource VARCHAR(256) DEFAULT NULL,\n" +" MsgFacility VARCHAR(256) DEFAULT NULL,\n" +" MsgType1 VARCHAR(256) DEFAULT NULL,\n" +" MsgType2 VARCHAR(256) DEFAULT NULL,\n" 
+" MsgAltId1 VARCHAR(256) DEFAULT NULL,\n" +" MsgAltId2 VARCHAR(256) DEFAULT NULL,\n" +" MsgMisc1 VARCHAR(256) DEFAULT NULL,\n" +" MsgMisc2 VARCHAR(256) DEFAULT NULL,\n" +" MsgMisc3 VARCHAR(256) 
DEFAULT NULL,\n" +" MsgStatus CHAR(3) DEFAULT NULL,\n" +" MsgStatusDate TIMESTAMP DEFAULT NULL,\n" +" MsgStatusInfo VARCHAR(256) DEFAULT NULL\n";

My insert string:

   *INSERT INTO* ssdi_small_tbl( MsgContent, MsgReceived, MsgSource,
   MsgFacility, MsgType1, MsgType2, MsgAltId1, MsgAltId2, MsgMisc1,
   MsgStatus, MsgStatusDate, MsgStatusInfo )
   *VALUES*(
    '
   
   ',
    '2023-02-23 12:49:13',
   'MIRTH_0ed9b98c-a94f-4c38-964800743a00c0061',
    'MOCKH',
    'ADT-A28',
    'EPIC',
    '1',
    '228750038',
    'ZZZ TEST NEW',
    'REJ',
    '2023-03-23 12:49:13',
    'Test Messages'
 )


My Java code performing the insertion:

try {
  PreparedStatement preparedStatement = connection.prepareStatement( INSERT );
  return( preparedStatement.executeUpdate() ==1 );
}
catch( SQLException throwables )
{
  throwables.printStackTrace();
  return false;
}

The SQLException I'm getting:

   A string constant starting with "
   Shorter insert string. For this string, Derby's happy and I am able to 
perform the insertion as well as select out the row later successfully 
(same code, etc.):


static final StringCONTENTS ="'MSH|^~&|LAB|HCA_OGDR|LAB||201112051038||ORU^R01^ORU_R01|4257313..LAB.COCCBM|P|2.1\\r\n" 
+" 
PID|1|665892|23422234234^^^HCA_OGDR||HERMAN^MUNSTER||18501031|M||W|1313 
MOCKINGBIRD LANE^^MOCKINGBIRD 
HEIGHTS^CA^90210||^626^2959626||English|M^Married^HL73202|Catholic|Q02690008175|529609893||202005061342|\\r\n" 
+" 
PV1|1|I|Q.IMC^Q.109^A|EM|||VICTOR^FRANKENSTEIN^IGOR^HCA||VICTOR^FRANKENSTEIN^IGOR^HCA|IN||01|||COCCBM||ADM|||202002040238\\r\n" 
+" NK1|0222555|MUNSTER^LILY^DRACULA|FA|1313 MOCKINGBIRD LANE^^MOCKINGBIRD 
HEIGHTS^CA^90210|(626)111-Mockingbird Heights Clinic\\r\n" +" OBR|1|L3266930^LAB|^LAB|BMP^BASIC METABOLIC 
PANEL^L|||201312050507|||QLB.XXP|||DT699 
STROKE|202002050514||VICTOR^FRANKENSTEIN^^HCA||1205:CBM:C03008R201112051038|||F\\r\n" 
+" NTE|1\\r\n" +" NTE|2||Site Legend:.brML:MOCKINGBIRD HEIGHTS CLINIC 15475 
NORTH HOLLYWOOD BOULEVARD MOCKINGBIRD HEIGHTS CA 90210.br\\r\n" +" TQ1|R^Routine^HL70485\\r\n" +" 
OBX|1|NM|NA^SODIUM^L^2951-2^SODIUM^LN|1|144|mmol/L|136-145|N||AS|F|201108241525|ML:MOCKINGBIRD 
HEIGHTS CLINIC|||ML:MOCKINGBIRD HEIGHTS CLINIC^15475 NORTH HOLLYWOOD 
BOULEVARD MOCKINGBIRD HEIGHTS CA 90210\\r\n" +" NTE|1|AD|Performed At: ML\\r\n" +" OBX|25|ST|GFR^GLOMERULAR FILTRATION 
RATE^L|1|>60||60-130|N||AS|F|201104071132|ML:MOCKINGBIRD HEIGHTS 
CLINIC|||ML:MOCKINGBIRD HEIGHTS CLINIC^15475 NORTH HOLLYWOOD 
BOULEVARD MOCKINGBIRD HEIGHTS CA 90210\\r\n" +" NTE|1|AD|Performed At: ML\\r\n" +" NTE|1||If the patient is African American, multiply by 1.210\\r\n" +" NTE|2\\r\n" +" NTE|3||eGFR Reference Range for adults =60 ml/min/1.73 m2\\r\n" +" 
OBR|2|L3261930^LAB|^LAB|PHOS^PHOSPHORUS^L|||202002050507|||QLB.XXP|||DT699 
STROKE|201112050514||FRANKENSTEIN^Victor^IGOR^HCA||1205:CBM:C8R202002051038|||F\\r 
NTE|1\\r\n" +" NTE|2||Site Legend:.brML:MOCKINGBIRD HEIGHTS CLINIC15475 NORTH 
HOLLYWOOD BOULEVARD MOCKINGBIRD HEIGHTS CA 90210.br\\r\n" +" TQ1|R^Routine^HL70485\\r\n" +" 
OBX|1|NM|PHOS^PHOSPHORUS^L^2777-1^PHOSPHATE^LN|1|2.4|mg/dL|2.5-4.9|L||AS|F|202002241521|ML:MOCKINGBIRD 
HEIGHTS CLINIC|||ML:MOCKINGBIRD HEIGHTS CLINIC^15475 NORTH HOLLYWOOD 
BOULEVARD MOCKINGBIRD HEIGHTS CA 90210\\r\n" +" NTE|1|AD|Performed At: ML\\r'";



Any thoughts?

Russ

Re: NiFi failing to start

[Russell Bateman]

In case you or someone else wishes only to run, develop, start, stop, 
start over, etc., and doesn't care to authenticate a (non-production) 
installation, I have followed this since NiFi 1.14 and last used it for 
1.19:


https://www.javahotchocolate.com/notes/nifi.html#20210716

If this doesn't work it's usually because the properties file has become 
too modified. Start over with a fresh download.


Russ


On 12/28/22 12:03, Chris Sampson wrote:
I think you will need to remove/comment out the references to 
single-user-provider in authorisers.xml and login-providers.xml as 
well as removing it from nifi.properties (see the comments in these 
files as they're provided in the nifi distributions).


If you are using 2-way TLS authentication then I don't think you need 
to configure anything else, but remember that all of your nifi 
instances in your cluster (if applicable) will need to trust one 
another's certificates along with all user certificates - the easiest 
way of doing this is typically to trust a common CA that issues all 
the nifi instance and user certs. This could be nifi-toolkit, but 
beware that the CA used by toolkit is auto-generated on startup, so 
you need to retain and configure the same CA for toolkit of you plan 
to use it to issue new certs in future.


On Wed, 28 Dec 2022, 17:32 James McMahon,  wrote:

I continue to experience errors when I try to start my nifi 1.16.3
instance. I have followed this guide in an effort to use the
toolkit to generate self-0signed certs for user admin, signed by a
nifi truststore:

Apache NiFi Walkthroughs


I seem to be having issues with this in my nifi.properties:
nifi.security.user.authorizer=single-user-authorizer

When I set it to nothing, it tells me this is required. When I set
it to single-user-authorizer, this error results in the log:
 Error creating bean with name 'authorizer': FactoryBean threw
exception on object creation; nested exception is
java.lang.Exception: The specified authorizer
'single-user-authorizer' could not be found.

I suspect my authorizers.xml and/or my
login-identity-providers.xml files are misconfigured. How should
those two config files be structured if I wish to run a secure
nifi instance where mith my self-signed certs, generated using the
nifi toolkit?

Re: ReplaceText 1.19 not processing Flowfile

[Russell Bateman]

Also, this strikes me as a NiFi Users List (us...@nifi.apache.org) 
question though many of us haunt both forums.



On 12/11/22 07:55, Mark Payne wrote:

Hello,

It looks like the attachment didn’t come through. The mailing list often strips 
out attachments from emails. Perhaps put them in pastebin or google drive or a 
GitHub gist and send a link?

Thanks
Mark

Sent from my iPhone


On Dec 11, 2022, at 7:33 AM,develo...@alucio.dev  wrote:

Hello, I am new to Nifi (right now on 1.19.0) and currently having trouble 
with ReplaceText processor.
Did my best to find resources on the net to find out what's going on , but 
nothing seems to address my issue.
I built a simple Flow...generating flowfile -> ReplaceText.

I can see that the Read counter in ReplaceText increased , but somehow the 
Queue is stuck (Data is being penalized).
On the ReplaceText right top corner, I can see some info window saying that the flowfile 
as transferred to "Success" but I don't see anything going on in the Success 
route.

I don't see anything in the log file that would indicate any errors relating to 
ReplaceText.
Attached are my flow and the ReplaceText Config.
Could anyone point me to the right direction please?

Seems like this issue is similar to NIFI-6426

Thanks

Re: Grammar error in error message

[Russell Bateman]

Yes it should be the infinitive */receive/* instead of the past 
participle /received/.


On 11/16/22 08:18, Paul Schou wrote:

This error message does not look like it is grammatically correct:

https://github.com/gkatta4113/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListenHTTP.java#L332

getLogger().warn("failed to received acknowledgment for HOLD with ID {}
sent by {}; rolling back session", new Object[] {id,
wrapper.getClientIP()});

- Paul Schou

Re: On configuring SSLContextService...

[Russell Bateman]

If you'll permit, I want to cap this thread I started off a bit by a) 
thanking the many who contributed to it and b) summing up the solution I 
am using based on that help.


Here are the command lines and germane instructions. For localhost 
below, substitute the DNS name (or, at least, //etc/hosts/ name) of the 
VM/hardware running Tomcat. Substitute your own password for "changeit" 
and modify any other details according to need.


*1. Generate Tomcat a keystore with certificate and key inside plus a 
/subject alternative name/ (SAN)--crucial for the client's use.*
keytool -genkeypair -keyalg RSA -keysize 2048 -validity 365 -dname 
"CN=tomcat" -ext san=dns:localhost -alias tomcat -keystore tomcat.jks 
-storepass changeit -keypass changeit


**2. Inspect**Tomcat's new keystore. You're looking to see the SAN.*
*keytool -list -v -keystore tomcat.jks -storepass changeit
*
3. Configure this keystore in Tomcat's **/conf/server.xml/**via a 
definition.*



  
    
  


*4. Get Tomcat's certificate "live." (Tomcat must be running with the 
new certificate.) In addition to getting the certificate, this should 
preserve the crucial SAN from step #1.*

openssl s_client -connect localhost:8443 -showcerts > client.cer

*5. Import that certificate into a keystore (that will be used in the 
client's trust store).*
keytool -importcert -file client.cer -alias tomcat -keystore 
client-truststore.jks -keypass changeit -storepass changeit -noprompt


*6. Verify the client's trust store. Again, you're looking to see the SAN.*
keytool -list -v -keystore client-truststore.jks -storepass changeit


The two artifacts to take away are /tomcat.jks/, for Tomcat's use, and 
/client-truststore.jks/, for the client's use.


In the case of this thread, the "client" in question was Apache NiFi's 
/InvokeHTTP/ and the configuration was done partly in that processor and 
partly in the accompanying /SSLContextService/ (I used 
/StandardRestrictedSSLContextService/). Those configurations looked like 
this:


/InvokeHTTP/:
HTTP URL: https://localhost:8443//servicename/

/SSLContextService/:
Truststore Filename: /client-truststore.jks/  (this must be a 
full path in your filesystem)

Truststore Password: changeit
Truststore Type: JKS
TLS Protocol: TLS

Re: On configuring SSLContextService...

[Russell Bateman]

That is an important goal to me as well (giving back as I have been 
given to). Last week, I had obligations outside of work that took up 
some time.


I have made progress in that (either of the) /SSLContextService/ options 
now validates my artifacts, however, I'm not out of the woods yet and 
have slowed my attack on this problem to deepen my personal 
understanding and knowledge of TLS, key, certificate, keystore, trust 
store, etc. theory.


At present, with a working /SSLContextService/, I'm investigating the 
following (failure) error back from /InvokeHTTP/ when I expected a response:


   Request Processing failed:
   java.net.ConnectException: Failed to connect to localhost/127.0.0.1:8443
   - Caused by java.net.ConnectException: Connection refused

I believe this is because Tomcat as now configured isn't accepting HTTPS.

I'm engaged in an attempt to understand the whole--generating of the 
self-signed certificate for use by Tomcat, transformation of same into a 
trust store certificate (what you have helped with). I fear that what I 
generated for Tomcat's use, which resides in a keystore referenced from 
/server.xml/, is not correct.


Russ

On 8/9/22 07:28, Paul Grey wrote:
To aid future searches, I wanted to follow up on the mailing list and 
make sure you had worked through your problem.



On Tue, Aug 2, 2022 at 3:41 PM Paul Grey  wrote:

Just tried out these (command line) steps; hope they work for you,
too.

(1) openssl s_client -connect localhost:8443 -showcerts
This will dump a bunch of text to stdout.  You want to grab the
text between these two text lines:
-BEGIN CERTIFICATE-
...
-END CERTIFICATE-
... and save that text to a file "trust.cer".

(2) openssl x509 -in trust.cer -text
This will verify that you got the right text.

(3) keytool -importcert -keystore trust.jks -file trust.cer -alias 1
This will create a JKS keystore that contains your certificate. 
The command will ask for a password (twice to confirm); pick
something easy.

(4) Enter "Truststore Filename" (/full/path/to/trust.jks) ,
"Truststore Password", and "Truststore Type" (JKS) into your
StandardSSLContextService properties.

Cheers!


On Tue, Aug 2, 2022 at 11:33 AM Nathan Gough 
wrote:

That's right Russ, if you purchased a commercially signed
certificate, InvokeHTTP should work by default. With your self
signed certificate, you will need an SSL context service, and
I believe you will need to insert the self signed certificate
into the trust store. I suggest using KeystoreExplorer to
create a trust store, import the self signed certificate, save
that trust store file and then point to that file in the SSL
context service. Let us know if you have issues with that part.

The screenshot error message you sent is showing that
InvokeHTTP does not trust the remote server when it says
'unable to find valid certfiication path to requested target'
(a pretty confusing error in my opinion).

X509 key/certs and key/trust store stuff is pretty tricky to
understand the first time around.

Nathan


    On Tue, Aug 2, 2022, 11:00 AM Russell Bateman
 wrote:

Yes, of course. And, therefore, /SSLContextService/ is
required was my conclusion. I see the point more clearly
now, but my conclusion seems inescapable. To forego
/SSLContextService/ we would have to purchase a
commercially signed certificate for use by Tomcat, right?

I will experiment with just somehow injecting the
self-signed certificate we created into the trust store
certificate? --which I thought I had done already, but
/SSLContextService/ has steadfastly refused to accept
anything I throw at it. (I reiterate that this is my first
TLS rodeo; I have never had to do this kind of work
before. I greatly appreciate your patience.)

Russ

On 8/1/22 19:03, Paul Grey wrote:

1.  You've instructed your browser to accept the
(untrusted) certificate that is being presented by Tomcat.
untrusted.cert.png

2. You've supplied the "--insecure" flag to curl.
https://curl.se/docs/manpage.html#-k

3.  The NiFi equivalent to these two instructions is to
provide a truststore, which contains a record specifying
the certificate being served by your Tomcat server.


    On Mon, Aug 1, 2022 at 6:27 PM Russell Bateman
 wrote:


Okay. I'm hoping this is true, but it's not what I'm
seeing. It's not dawning on me yet what I'm doing wrong.

1. If I hit my service from a browser,
htt

Re: On configuring SSLContextService...

[Russell Bateman]

Just a note (for later readers of this thread)...

My experience now with this trick seems to say that, as long as "https" 
is in the URL, a /SSLContextService/ must be supplied. As a URL with 
"https" and port number 8443 is the only way I have to engage TLS at the 
far end, I must live with /SSLContextService/'s requirements.


On 7/26/22 19:17, Paul Grey wrote:

leave the InvokeHTTP property SSLContextService blank.

On configuring SSLContextService...

[Russell Bateman]

I have hesitated between providing some huge tl;dr exposé and something 
shorter. I'll do shorter here.


0. For now, I'm using "changeit" below as password rolling a self-signed 
certificate for key, key store and trust store.
1. I have a service running in Tomcat that I hit via HTTPs because the 
content always involves personal health information.
2. I use a key store containing my certificate. No trust store is needed 
or involved in Tomcat.

3. I need to hit my Tomcat service using /InvokeHTTP/ in my flow.
4. This means configuring an instance of 
/Standard[Restricted]SSLContextService/.
5. The SSL context service insists on a defined key store with key 
password and key store password.
6. The SSL context service insists on a defined trust store. The best I 
have been able to do is to roll the key store certificate into a trust 
store.
7. When either key- or trust store file is missing, the SSL context 
service complains that a resource is missing (for key store or trust store).

8. Once both files/resources exist, all three passwords appear crucial.
9. Despite password used to create key and certificates, it is always 
wrong according to SSL context service validator which consistently issues:


   /Keystore Properties is invalid because invalid keystore password or
   type specified for file __.//
   //Truststore Properties is invalid because invalid truststore
   password or type specified for file __./

It would be nice to see a step-by-step illustration of creating the key, 
key store and trust store artifacts required by SSL context service and 
perhaps the full configuration of the SSL context service.


Other notes:

1. I seem to get pretty far toward a solution using Java's keytool.
2. I don't get very far using openssl.
3. I get even less traction trying to use NiFi's TLS toolkit to solve this.
4. I guess I could simply write my own SSL context service that doesn't 
require a trust store?


Huge thanks for any help or comments.

Russ

P.S. I have a scratch sheet that reveals how I created artifacts and 
thought through the problem at:


https://www.javahotchocolate.com/notes/keytool-experience.html

Re: Placement and specification of certificates for StandardRestrictedSSLContextService

[Russell Bateman]

David,

Sadly, this is my experience. "changeit" works for me. And I tried 
reconfiguring the three passwords in 
/StandardRestrictedSSLContextService/ to no avail.


   ~/dev/nifi/nifi-1.15.0/conf $ *keytool -list -v -keystore
   mdmi-keystore.jks*
   Enter keystore password: *changeit*
   Keystore type: PKCS12
   Keystore provider: SUN

   Your keystore contains 1 entry

   Alias name: mdmi
   Creation date: Jul 20, 2022
   Entry type: PrivateKeyEntry
   Certificate chain length: 1
   Certificate[1]:
   Owner: CN=windofkeltia.com, OU=Unknown, O=Wind of Keltia, L=Provo,
   ST=UT, C=US
   Issuer: CN=windofkeltia.com, OU=Unknown, O=Wind of Keltia, L=Provo,
   ST=UT, C=US
   Serial number: 1e7288f7
   Valid from: Wed Jul 20 15:39:23 MDT 2022 until: Thu Jul 20 15:39:23
   MDT 2023
   Certificate fingerprints:
     SHA1: B9:58:6E:C1:0D:DA:1D:CF:7D:02:16:54:F2:FA:1F:C4:19:01:F5:1B
     SHA256:
   
FF:0B:3B:4A:59:69:9B:B8:B3:23:1F:4E:72:03:C7:24:11:A9:DF:11:C6:76:89:32:44:F7:12:A4:26:F5:9D:4B
   Signature algorithm name: SHA256withRSA
   Subject Public Key Algorithm: 2048-bit RSA key
   Version: 3

   Extensions:

   #1: ObjectId: 2.5.29.14 Criticality=false
   SubjectKeyIdentifier [
   KeyIdentifier [
   : 69 63 BD 7E 67 A1 EC 0A   54 3C 61 2F 51 D7 64 46 
   ic..g...T
Hi Russell,

Thanks for describing the steps used to generate the keystore and 
truststore files.


The validation warnings on StandardRestrictedSSLContextService appear 
to indicate that the configured password properties do not match the 
keystore and truststore passwords.


It would be helpful to enter the password properties again and confirm 
that there are no trailing spaces.


The following keytool commands can also be used to verify the passwords:

keytool -list -v -keystore mdmi-keystore.jks
keytool -list -v -keystore mdmi-truststore.jks

The configuration appears to be correct, so confirming the password on 
both files is a good next step.

Placement and specification of certificates for StandardRestrictedSSLContextService

[Russell Bateman]

I'm trying to set up TLS for a service using /InvokeHTTP/ against an 
external-to-NiFi Tomcat-based service and I have configured 
/StandardRestrictedSSLContextService/ thus:


https://www.javahotchocolate.com/notes/nifi-images/mdmi-standard-ssl-context-service.png

...which results in the errors shown here:

https://www.javahotchocolate.com/notes/nifi-images/s-sslcontextservice.png

Do the NiFi errors mean that "changeit" can't be used as a password?

At the risk of over-simplifying their placement, I dropped them into 
/${NIFI_ROOT}/conf/.


   ~/dev/nifi/nifi-1.15.0/conf $ *ll mdmi**
   -rw-rw-r-- 1 russ russ  899 Jul 20 15:40 mdmi-keystore.crt
   -rw-rw-r-- 1 russ russ 2725 Jul 20 15:39 *mdmi-keystore.jks*
   -rw-rw-r-- 1 russ russ 1255 Jul 20 15:53 *mdmi-truststore.jks*

/mdmi-keystore.crt/ is self-signed for now and (for now) I have used 
"changeit":


   ~/dev/nifi/nifi-1.15.0/conf $ *keytool -genkey -keyalg RSA -alias
   mdmi -keystore mdmi-keystore.jks -validity 365 -keysize 2048*
   Enter keystore password:  changeit
   Re-enter new password:  changeit
   What is your first and last name?
   ...

   ~/dev/nifi/nifi-1.15.0/conf $ *keytool -export -alias mdmi -file
   mdmi-**keystore.crt -keystore mdmi-keystore.jks -storepass changeit*
   Certificate stored in file 
   ~/dev/nifi/nifi-1.15.0/conf $ *keytool -import -noprompt
   -trustcacerts**-alias mdmi -file mdmi-keystore.crt -keystore
   mdmi-truststore.jks**-storepass changeit*
   Certificate was added to keystore

This all works fine via curl or Postman outside of NiFi for hitting the 
service (I put the keytool artifacts into /${CATALINA_BASE}/conf/and 
note this in /${CATALINA_BASE}/conf/server.xml/).


When it comes to TLS in NiFi, this is my first rodeo. I'm open to 
suggestions on any other this. Thanks.

Re: How to manage security artifacts from a custom processor

[Russell Bateman]

I appreciate the responses. I will try out the canonical 
/StandardSSLContextService/ first (since that's what I am using with 
Kafka), then imitate the other sample depending.


However, where/how do I install the certificates I'll be given for use? 
I would expect something for certain representing the third-party 
service in a truststore and maybe another (a private key) in a keystore.



On 7/5/22 16:30, Russell Bateman wrote:
From a custom processor, I intend to interface with a third-party 
service (via simple HTTP client), however, I would need as I 
understand it to


a) maintain a private key by which I can identify myself to that
third-party service and
b) maintain a trusted-store certificate by which I can guarantee
the identity of the service.

This is pretty far outside my own experience. I have been reading on 
how this is achieved in Java, but in my mind a complication arises 
from the fact that a custom NiFi processor lives within NiFi's JVM. My 
question is therefore, how can I control the certificates and 
authorities for my use in or associated with NiFi's JVM. Clearly, I 
don't grok this well enough even to ask the question; I'm hoping 
someone can see through what I'm asking and point me in a good 
direction to study.


I've written a pile of successful and useful custom NiFi processors to 
cover proprietary needs, so custom-processor writing isn't a mystery. 
Certificates, keys, trusts and security in general still is.


Profuse thanks,

Russ

How to manage security artifacts from a custom processor

[Russell Bateman]

From a custom processor, I intend to interface with a third-party 
service (via simple HTTP client), however, I would need as I understand 
it to


   a) maintain a private key by which I can identify myself to that
   third-party service and
   b) maintain a trusted-store certificate by which I can guarantee the
   identity of the service.

This is pretty far outside my own experience. I have been reading on how 
this is achieved in Java, but in my mind a complication arises from the 
fact that a custom NiFi processor lives within NiFi's JVM. My question 
is therefore, how can I control the certificates and authorities for my 
use in or associated with NiFi's JVM. Clearly, I don't grok this well 
enough even to ask the question; I'm hoping someone can see through what 
I'm asking and point me in a good direction to study.


I've written a pile of successful and useful custom NiFi processors to 
cover proprietary needs, so custom-processor writing isn't a mystery. 
Certificates, keys, trusts and security in general still is.


Profuse thanks,

Russ

Re: Reg Nifi java code generation

[Russell Bateman]

You don't have to write Java code to benefit from NiFi which is an 
insanely useful framework all by itself with jillions of super-useful 
processors ready for use. However, if you plan to code your own, 
proprietary processor to do something that hasn't been covered, here's a 
likely place to start:


https://nifi.apache.org/developer-guide.html

In addition, googling will help you find many samples to imitate.

Best luck to you.

On 5/2/22 06:32, AKHILESH PATIDAR wrote:

Hi,
great work is done by you on nifi java code generation. I'm working on my
college project which needs nifi java code can you please help me in
understanding that.


Thank You
Akhilesh


Re: how to setup nifi.content.repository.archive.max.retention.period in a Nifi docker image ?

[Russell Bateman]

Breno,

While we're on this topic, what's best practice for changing something 
like "the default nifi files on a clean docker image, ..."? Use sed or 
awk from a RUN command? (This is really a Docker question, but you 
raised it. Anything you suggest would be helpful.)


Russ

On 2/22/22 07:07, Breno Cesar wrote:

Hi Nicolas,
As far as i know, there is no variable for this config.The 
documentation has a lack about this topic and does not explain about it.

Doing some "googling", i found that someone already maped this variables.

https://github.com/dprophet/nifi/blob/master/nifi-docker/dockerhub/CONFIGURATION.md

Assuming this configuration you need is a tunnig, and will not be done 
frequenly, as far as I know, you can change it in the default nifi 
files on a clean docker image, and export it for later use.


*Breno *

Em ter., 22 de fev. de 2022 às 10:01, Nicolas Belot 
 escreveu:


Hello everyone.

I use a docker image of nifi and I need to tune
nifi.content.repository.archive.max.retention.period.

Is there a way to simply  set it up through an env variable ?

I read the start.sh script but
nifi.content.repository.archive.max.retention.period  does not
appear in this script.

Regards

N.

Re:

[Russell Bateman]

Ismaël,

Please send an e-mail to users-unsubscr...@nifi.apache.org.

Thanks.

On 12/20/21 8:45 AM, ismaelmartin.ri...@post.ch wrote:


unsubscribe

Re: How to restrict custom processor execution time

[Russell Bateman]

Sanjeet,

It occurred to me that you may not be getting replies because you report 
"building a custom processor" and that's the subject of the Apache NiFi 
Developers' forum. Try posting there.


Best regards,

Russ

On 8/10/21 7:33 AM, sanjeet rath wrote:

Hi ,

I am building a custom processor and there is restriction i want to 
put for the  processor that it should not schedule to run 2 times in 1 
hour time period.


I can acheive this by passing "run schedule" 60 mins.

Is there any other way i can do in my custom processor code, So that 
it won't allow the user to select "run schedule " time less than 60 
mins.basically similar to we can restrict the procesor to execute on 
prinary node.


Any other thought is really helpfull.

Thanks,
Sanjeet

Re: odd performance behavior 1.14

[Russell Bateman]

(I'm sorry, I meant to say that I read that the Java Flight Recorder is 
now available for unlicensed--in the original Oracle sense--for use.)


On 8/2/21 12:44 PM, Russell Bateman wrote:

Scott,

I believe I read somewhere in the last year. I found this:

https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u

And, I used it to look into a problem I had with Apache NiFi a few 
years ago. For what it's worth, my experience is recorded here:


https://www.javahotchocolate.com/notes/jfr.html

Hope some of this helps.

Russ

On 8/2/21 12:20 PM, scott wrote:
I'm using openjdk-11.0.7.10-4 as I was on the previous version of 
NiFi. I'll look around for a free Java profiler to use to dig deeper.

Thanks,
Scott

On Sat, Jul 31, 2021 at 7:56 PM Joe Witt <mailto:joe.w...@gmail.com>> wrote:


Scott

Nope this sounds pretty dang unique

What JVM?   May need to attach a profiler.

I have seen buried exceptions happening at massive rates causing
horrid performance among a few other scenarios but nothing
specific to 1.14

thanks

On Sat, Jul 31, 2021 at 4:01 PM scott mailto:tcots8...@gmail.com>> wrote:

Hi All,
I upgraded to 1.14 last week and within a few days I started
to see some pretty odd behavior, I'm hoping someone has
either seen it before or could point me to a deeper level of
troubleshooting.

Here are the symptoms I observed.
* Performance issues:
    Processor performance very poor. Even simple processors
like router and updateattribute went from being able to
process 100,000recs/min to 100recs/min or stop all together,
but not consistently.
    Processors needing to be force killed, even simple ones
like updateattribute.

* Weirdness. One of my routers lost its mind and
didn't recognize the routes configured anymore. It changed
all the arrows to dotted lines except for the default. I
ended up copying it and replacing it with the copy, no
changes mind you, but it worked fine.

* Errors: I have not found any obvious errors in the nifi
logs that could explain this, but one error keeps repeating
in the logs: SQLServerDriver is not found . I have dozens of
processors that use SQL Server, all seem to be working fine.
This is not tied to a particular processor's configuration. I
don't think this is related.

* Server resources fine. I use htop and sar to troubleshoot
hardware issues usually, all looks normal. I added 1/3 more
memory to the JVM, now at 24GB, just for good measure, but
that had no effect.

Is it possible there are some hidden performance issues going
on within the JVM I need a special tool to see?

Any help would be greatly appreciated.

Thanks,
Scott

Re: odd performance behavior 1.14

[Russell Bateman]

Scott,

I believe I read somewhere in the last year. I found this:

https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u

And, I used it to look into a problem I had with Apache NiFi a few years 
ago. For what it's worth, my experience is recorded here:


    https://www.javahotchocolate.com/notes/jfr.html

Hope some of this helps.

Russ

On 8/2/21 12:20 PM, scott wrote:
I'm using openjdk-11.0.7.10-4 as I was on the previous version of 
NiFi. I'll look around for a free Java profiler to use to dig deeper.

Thanks,
Scott

On Sat, Jul 31, 2021 at 7:56 PM Joe Witt > wrote:


Scott

Nope this sounds pretty dang unique

What JVM?   May need to attach a profiler.

I have seen buried exceptions happening at massive rates causing
horrid performance among a few other scenarios but nothing
specific to 1.14

thanks

On Sat, Jul 31, 2021 at 4:01 PM scott mailto:tcots8...@gmail.com>> wrote:

Hi All,
I upgraded to 1.14 last week and within a few days I started
to see some pretty odd behavior, I'm hoping someone has either
seen it before or could point me to a deeper level of
troubleshooting.

Here are the symptoms I observed.
* Performance issues:
    Processor performance very poor. Even simple processors
like router and updateattribute went from being able to
process 100,000recs/min to 100recs/min or stop all together,
but not consistently.
    Processors needing to be force killed, even simple ones
like updateattribute.

* Weirdness. One of my routers lost its mind and
didn't recognize the routes configured anymore. It changed all
the arrows to dotted lines except for the default. I ended up
copying it and replacing it with the copy, no changes
mind you, but it worked fine.

* Errors: I have not found any obvious errors in the nifi logs
that could explain this, but one error keeps repeating in the
logs: SQLServerDriver is not found . I have dozens of
processors that use SQL Server, all seem to be working fine.
This is not tied to a particular processor's configuration. I
don't think this is related.

* Server resources fine. I use htop and sar to troubleshoot
hardware issues usually, all looks normal. I added 1/3 more
memory to the JVM, now at 24GB, just for good measure, but
that had no effect.

Is it possible there are some hidden performance issues going
on within the JVM I need a special tool to see?

Any help would be greatly appreciated.

Thanks,
Scott

Re: [DISCUSS] NiFi 2.0 Release Goals

[Russell Bateman]

Bringing up Elastic also reminds me that the Elastic framework has just 
recently transitioned out of Open Source, so to acknowledge that, maybe 
some effort toward OpenSearch--I say this not understanding exactly how 
this sort of thing is considered in a large-scale, world-class software 
project like Apache NiFi. (I'm not a contributor, just a grateful consumer.)


Russ

On 7/23/21 10:28 AM, Matt Burgess wrote:

Along with the itemized list for ancient components we should look at
updating versions of drivers, SDKs, etc. for external systems such as
Elasticsearch, Cassandra, etc. There may be breaking changes but 2.0
is probably the right time to get things up to date to make them more
useful to more people.

On Fri, Jul 23, 2021 at 12:21 PM Nathan Gough  wrote:

I'm a +1 for removing pretty much all of this stuff. There are security
implications to keeping old dependencies around, so the more old code we
can remove the better. I agree that eventually we need to move to
supporting only Java 11+, and as our next release will probably be about 4
- 6 months from now that doesn't seem too soon. We could potentially break
this in two and remove the deprecated processors and leave 1.x on Java 8,
and finally start on 2.x which would support only Java 11. I'm unsure of
what implications changing the date and time handling would have - for
running systems that use long term historical logs, unexpected impacts to
time logging could be a problem.

As Joe says I think feature work will have to be dedicated to 2.x and we
could support 1.x for security fixes for some period of time. 2.x seems
like a gargantuan task but it's probably time to get started. Not sure how
we handle all open PRs and the transition between 1.x and 2.x.

On Fri, Jul 23, 2021 at 10:57 AM Joe Witt  wrote:


Jon

You're right we have to be careful and you're right there are still
significant Java 8 users out there.  But we also have to be careful
about security and sustainability of the codebase.  If we had talked
about this last year when that article came out I'd have agreed it is
too early.  Interestingly that link seems to get updated and I tried
[1] and found more recent data (not sure how recent).  Anyway it
suggests Java 8 is still the top dog but we see good growth on 11.  In
my $dayjob this aligns to what I'm seeing too.  Customers didn't seem
to care about Java 11 until later half last year and now suddenly it
is all over the place.

I think once we put out a NiFi 2.0 release we'd see rapid decrease in
work on the 1.x line just being blunt.  We did this many years ago
with 0.x to 1.x and we stood behind 0.x for a while (maybe a year or
so) but it was purely bug fix/security related bits.  We would need to
do something similar.  But feature work would almost certainly go to
the 2.x line.  Maybe there are other workable models but my instinct
suggests this is likely to follow a similar path.

...anyway I agree it isn't that easy of a call to dump Java 8.  We
need to make the call in both the interests of the user base and the
contributor base of the community.

[1] https://www.jetbrains.com/lp/devecosystem-2021/java/


Thanks
Joe

On Fri, Jul 23, 2021 at 7:46 AM Joe Witt  wrote:

Russ

Yeah the flow registry is a key part of it.  But also now you can
download the flow definition in JSON (upload i think is there now
too).  Templates offered a series of challenges such as we store them
in the flow definition which has made flows massive in an unintended
way which isn't fun for cluster behavior.

We have a couple cases where we headed down a particular concept and
came up with better approaches later.  We need to reconcile these with
the benefit of hindsight, and while being careful to be not overly
disruptive to existing users, to reduce the codebase/maintenance
burden and allow continued evolution of the project.

Thanks

On Fri, Jul 23, 2021 at 7:43 AM Russell Bateman 

wrote:

Joe,

I apologize for the off-topic intrusion, but what replaces templates?
The Registry? Templates rocked and we have used them since 0.5.x.

Russ

On 7/23/21 8:31 AM, Joe Witt wrote:

David,

I think this is a highly reasonable approach and such a focus will
greatly help make a 2.0 release far more approachable to knock out.
Not only that but tech debt reduction would help make work towards
major features we'd think about in a 'major release' sense more
approachable.

We should remove all deprecated things (as well as verify we have the
right list).  We should remove/consider removal of deprecated

concepts

like templates.  We should consider whether we can resolve the

various

ways we've handled what are now parameters down to one clean

approach.

We should remove options in the nifi.properties which turn out to
never be used quite right (if there are).  There is quite a bit we

can

do purely in the name of tech debt reduction.

Lots to consider here but I think this is the right discussion.

Than ks

On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende 

wrote

Re: [DISCUSS] NiFi 2.0 Release Goals

[Russell Bateman]

Joe,

I apologize for the off-topic intrusion, but what replaces templates? 
The Registry? Templates rocked and we have used them since 0.5.x.


Russ

On 7/23/21 8:31 AM, Joe Witt wrote:

David,

I think this is a highly reasonable approach and such a focus will
greatly help make a 2.0 release far more approachable to knock out.
Not only that but tech debt reduction would help make work towards
major features we'd think about in a 'major release' sense more
approachable.

We should remove all deprecated things (as well as verify we have the
right list).  We should remove/consider removal of deprecated concepts
like templates.  We should consider whether we can resolve the various
ways we've handled what are now parameters down to one clean approach.
We should remove options in the nifi.properties which turn out to
never be used quite right (if there are).  There is quite a bit we can
do purely in the name of tech debt reduction.

Lots to consider here but I think this is the right discussion.

Than ks

On Fri, Jul 23, 2021 at 7:26 AM Bryan Bende  wrote:

I'm a +1 for this... Not sure if this falls under "Removing Deprecated
Components", but I think we should also look at anything that has been
marked as deprecated throughout the code base as a candidate for
removal. There are quite a few classes, methods, properties, etc that
have been waiting for a chance to be removed.

On Fri, Jul 23, 2021 at 10:13 AM David Handermann
 wrote:

Team,

With all of the excellent work that many have contributed to NiFi over the
years, the code base has also accumulated some amount of technical debt. A
handful of components have been marked as deprecated, and some components
remain in the code base to support integration with old versions of various
products. Following the principles of semantic versioning, introducing a
major release would provide the opportunity to remove these deprecated and
unsupported components.

Rather than focusing the next major release on new features, what do you
think about focusing on technical debt removal? This approach would not
make for the most interesting release, but it provides the opportunity to
clean up elements that involve breaking changes.

Focusing on technical debt, at least three primary goals come to mind for
the next major release:

1. Removal of deprecated and unmaintained components
2. Require Java 11 as the minimum supported version
3. Transition internal date and time handling to JSR 310 java.time
components

*Removing Deprecated Components*

Removing support for older and deprecated components provides a great
opportunity to improve the overall security posture when it comes to
maintaining dependencies. The OWASP dependency plugin report currently
generates 50 MB of HTML for questionable dependencies, many of which are
related to old versions of various libraries.

As a starting point, here are a handful of components and extension modules
that could be targeted for removal in a major version:

- PostHTTP and GetHTTP
- ListenLumberjack and the entire nifi-lumberjack-bundle
- ListenBeats and the entire nifi-beats-bundle
- Elasticsearch 5 components
- Hive 1 and 2 components

*Requiring Java 11*

Java 8 is now over seven years old, and NiFi has supported general
compatibility with Java 11 for several years. NiFi 1.14.0 incorporated
internal improvements specifically related to TLS 1.3, which allowed
closing out the long-running Java 11 compatibility epic NIFI-5174. Making
Java 11 the minimum required version provides the opportunity to address
any lingering edge cases and put NiFi in a better position to support
current Java versions.

*JSR 310 for Date and Time Handling*

Without making the scope too broad, transitioning internal date and time
handling to use DateTimeFormatter instead of SimpleDateFormat would provide
a number of advantages. The Java Time components provide much better
clarity when it comes to handling localized date and time representations,
and also avoid the inherent confusion of java.sql.Date extending
java.util.Date. Many internal components, specifically Record-oriented
processors and services, rely on date parsing, leading to confusion and
various workarounds. The pattern formats of SimpleDateFormat and
DateTimeFormatter are very similar, but there are a few subtle differences.
Making this transition would provide a much better foundation going forward.

*Conclusion*

Thanks for giving this proposal some consideration. Many of you have been
developing NiFi for years and I look forward to your feedback. I would be
glad to put together a more formalized recommendation on Confluence and
write up Jira epics if this general approach sounds agreeable to the
community.

Regards,
David Handermann

Re: Problem with the GetFile processor deleting my entire installation

[Russell Bateman]

Oh, sorry, to finish the answer, yes, you do need to be *very* careful 
how you specify the Input Directory and File Filter properties; the last 
one is a regular expression. It's true that the documentation is less 
than flag-waving or hair-lighting-on-fire as it presents its help on 
filling those in.


Russ

On 6/4/21 11:16 AM, Russell Bateman wrote:
Sorry for this behavior of /GetFile/ which is purposeful. If you 
configure to keep the files instead of removing them, you'll keep 
getting the same files ingested over and over again as flow files. 
It's just how it is.


The secret was to read the help blurb when configuring this processor.

Hope this helps,

Russ

On 6/4/21 10:44 AM, Ruth, Thomas wrote:


Hello,

I recently built a 3-node NiFi cluster in my organization as a 
proof-of-concept for some work we are doing. I used version 1.13.2 
and installed it onto 3 CentOS 7.9 systems. In my organization, I 
don’t have root access to the system, so I used a different user 
called “nfadm” to install and run the product. I don’t remember 
seeing anything in the documentation that stated that this would be 
an issue.


I am also new to NiFi, and was relying heavily on the Admin 
documentation on the web site for instructions to set up the OS and 
NiFi installations. I configured certificate-based security and 
distributed them to my users. I also configured policies for groups 
that I thought were OK for them from a development standpoint.


I had an incident occur yesterday in which a user, who is also new to 
NiFi, ran a component called “GetFile” for the filesystem “/” with 
the default settings (Recurse=true, KeepSourceFile=false). Well, this 
essentially ran “rm -rf /” as the user that owns all the installation 
files and files in the various repositories, nfadm, the same user 
running the NiFi processes. This deleted all the installation and 
configuration files for the entire cluster, making it completely 
useless now.


I am surprised to find out that NiFi allowed a user to basically wipe 
out all the files the user running the NiFi server had access to. I 
would expect much higher security to be present in a default system. 
I have some questions that hopefully you can help me with:


Is this a known issue in NiFi?

Am I doing something wrong when configuring or installing NiFi?

Is there a section in the documentation that warns me of this type of 
scenario?


Thanks in advance for your help with this,

Tom Ruth

Sr. Data Engineer

Optum, Inc.

E: thomas.r...@optum.com


This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the 
intended

recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.

Re: Problem with the GetFile processor deleting my entire installation

[Russell Bateman]

Sorry for this behavior of /GetFile/ which is purposeful. If you 
configure to keep the files instead of removing them, you'll keep 
getting the same files ingested over and over again as flow files. It's 
just how it is.


The secret was to read the help blurb when configuring this processor.

Hope this helps,

Russ

On 6/4/21 10:44 AM, Ruth, Thomas wrote:


Hello,

I recently built a 3-node NiFi cluster in my organization as a 
proof-of-concept for some work we are doing. I used version 1.13.2 and 
installed it onto 3 CentOS 7.9 systems. In my organization, I don’t 
have root access to the system, so I used a different user called 
“nfadm” to install and run the product. I don’t remember seeing 
anything in the documentation that stated that this would be an issue.


I am also new to NiFi, and was relying heavily on the Admin 
documentation on the web site for instructions to set up the OS and 
NiFi installations. I configured certificate-based security and 
distributed them to my users. I also configured policies for groups 
that I thought were OK for them from a development standpoint.


I had an incident occur yesterday in which a user, who is also new to 
NiFi, ran a component called “GetFile” for the filesystem “/” with the 
default settings (Recurse=true, KeepSourceFile=false). Well, this 
essentially ran “rm -rf /” as the user that owns all the installation 
files and files in the various repositories, nfadm, the same user 
running the NiFi processes. This deleted all the installation and 
configuration files for the entire cluster, making it completely 
useless now.


I am surprised to find out that NiFi allowed a user to basically wipe 
out all the files the user running the NiFi server had access to. I 
would expect much higher security to be present in a default system. I 
have some questions that hopefully you can help me with:


Is this a known issue in NiFi?

Am I doing something wrong when configuring or installing NiFi?

Is there a section in the documentation that warns me of this type of 
scenario?


Thanks in advance for your help with this,

Tom Ruth

Sr. Data Engineer

Optum, Inc.

E: thomas.r...@optum.com


This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.

Re: Penalizing one part of a flow over another

[Russell Bateman]

Thanks, Mark, both comments are very helpful.

Cheers,

Russ

On 4/22/21 11:19 AM, Mark Payne wrote:

Russell,

You can’t really set a “priority” of one flow of the other. A couple of options 
that may make sense for you though:

- You can set the Run Schedule to something other than “0 sec” for processors 
in the sub-flow. Perhaps set them to “100 millis” or something like that. This 
will leave to more latency in that flow but schedule the processors less 
frequently so they won’t interfere with your main flow as much. Here, though, 
if there’s a bunch of data coming in, it could result in backpressure all the 
way back to the main flow. So you’d want to consider if FlowFile Expiration is 
appropriate. That way you’d say if data sits in this first queue for more than 
3 seconds, for instance, expire it, so that it doesn’t cause back flow. You 
could schedule just the first processor in the sub flow to run at a slower pace 
or all of them, depending on if you’re just trying to slow down the ingestion 
into the flow or all of the processing.

- Similarly, rather than mess with the Run Schedule, you could use a Control 
Rate and say that you’re only going to allow a throughput of maybe 10 MB/sec 
into the sub-flow. Again, that could cause backpressure so you’d want to 
consider FlowFile Expiration if you’d rather lose the FlowFiles than allow them 
to affect the main flow.

Hope that’s helpful!

Thanks
-Mark


On Apr 22, 2021, at 9:44 AM, Russell Bateman  wrote:

I have a flow performing ETL of HL7v4 (FHIR) document on their way to indexing and 
storage. Custom processors perform the important transformations. Performance of this 
flow is at a premium for us. At some point along the way I want to gate off copies of raw 
or of transformed FHIR records (the flow writer's choice) to a new flow (a 
"subflow" of the total flow) for the purpose of validating those FHIR records 
as an option.

The main ETL flow will thus not be interrupted. Also, its performance should 
not be too hugely impacted by this new subflow. I have looked at priority 
techniques discussed, but usually the discussion is geared more toward a 
resulting order. I want to deprecate the performance of this new subflow to 
avoid handicapping the main flow, ideally from almost shutting down the subflow 
to allowing it equal performance with the main ETL flow.

Are there recommendations for such a thing? As I author many custom processors, 
is there something I could be doing in my code to aid this? I want rather to 
put the amount of crippling into the hands of my flow writers  a) by natural, 
existing configuration that's a feature of most NiFi processors and/or b) 
surfacing programming choices as configuration in my custom processor's 
configuration. Etc.

Any comments on this are hoped for and very welcome.

(Because I wrote so many custom processors that are crucial to my flows, I 
chose the NiFi developer- instead of the users list.)

Penalizing one part of a flow over another

[Russell Bateman]

I have a flow performing ETL of HL7v4 (FHIR) document on their way to 
indexing and storage. Custom processors perform the important 
transformations. Performance of this flow is at a premium for us. At 
some point along the way I want to gate off copies of raw or of 
transformed FHIR records (the flow writer's choice) to a new flow (a 
"subflow" of the total flow) for the purpose of validating those FHIR 
records as an option.


The main ETL flow will thus not be interrupted. Also, its performance 
should not be too hugely impacted by this new subflow. I have looked at 
priority techniques discussed, but usually the discussion is geared more 
toward a resulting order. I want to deprecate the performance of this 
new subflow to avoid handicapping the main flow, ideally from almost 
shutting down the subflow to allowing it equal performance with the main 
ETL flow.


Are there recommendations for such a thing? As I author many custom 
processors, is there something I could be doing in my code to aid this? 
I want rather to put the amount of crippling into the hands of my flow 
writers  a) by natural, existing configuration that's a feature of most 
NiFi processors and/or b) surfacing programming choices as configuration 
in my custom processor's configuration. Etc.


Any comments on this are hoped for and very welcome.

(Because I wrote so many custom processors that are crucial to my flows, 
I chose the NiFi developer- instead of the users list.)

Re: NiFi 1.11.4 Custom Processor Development

[Russell Bateman]

I'm sorry. I mistyped (numbers soup). I meant to say NiFi 1.2.0, a 
version several years ago already.


On 4/13/21 9:04 AM, Russell Bateman wrote:
There shouldn't be any problem. I have many custom processors in a NAR 
I haven't rebuilt since 1.12 and we use them successfully. They're 
just missing versions because we had not yet upgraded the Maven we 
used to build the NAR (not exactly relevant, I know, but I thought I'd 
I would point this out).



On 4/13/21 8:55 AM, nathan.engl...@bt.com wrote:

Hi Joe,

Thanks for that information.

Will there be any issue in running a Custom NAR file built against 
1.9.2 on a different versioned cluster. We haven’t experienced 
anything yet, but would hate to spend ages in debugging an issue that 
is the result of this!


Kind Regards,

Nathan

*From:* Joe Witt 
*Sent:* Tuesday, April 13, 2021 5:48 pm
*To:* users@nifi.apache.org
*Subject:* Re: NiFi 1.11.4 Custom Processor Development
Hello

In moving to support maven 3.8 we found only https based repos are 
allowed by default.  In addressing this we updated the build to move 
away from bintray and another location as well.


This is now the case when you build master. You could try to cherry 
pick and edit the key commits or hang tight for us to kick out a 1.14 
release.  No time table for that yet though.


Thanks

On Tue, Apr 13, 2021 at 7:45 AM <mailto:nathan.engl...@bt.com>> wrote:


Hi There,

I am in the process of upgrading the version of the
nifi-nar-bundle from 1.9.2 to 1.12.1 however I’ve hit a
unexpected issue.

We develop on an offline platform, with Maven Central Mirrored
using a Nexus Repository Manager, with all dependencies delivered
through this mechanism. Today when I have changed the version
from 1.9.2 to 1.12.1 I have found that the parent (I assume for
the groovy processor) has a dependency on groovy-eclipse-batch
version 2.5.4-01 which seems to be only available from a Groovy
Repository (https://dl.bintray.com/groovy/maven/

<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdl.bintray.com%2Fgroovy%2Fmaven%2F=04%7C01%7Cnathan.english%40bt.com%7Cedb498f9213242aee05108d8fe8b3c57%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637539221253446565%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=9T1Ii386gRcclHLZw6Dw9%2FPEal1hrqaNGDjWAgJ6Hz8%3D=0>)
and not maven central.

I also noticed that bintray will be closing shortly
 
(https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/

<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fjfrog.com%2Fblog%2Finto-the-sunset-bintray-jcenter-gocenter-and-chartcenter%2F=04%7C01%7Cnathan.english%40bt.com%7Cedb498f9213242aee05108d8fe8b3c57%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637539221253456560%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=8nphGAZUMxc0QX19bvZMrhm6JdLBGpmdxktTSjsiccs%3D=0>),
so I guess the dependency will need correcting before then?

Kind Regards,

Nathan

Nathan English
Applications Specialist - Cyber Delivery & DevOps

Re: NiFi 1.11.4 Custom Processor Development

[Russell Bateman]

There shouldn't be any problem. I have many custom processors in a NAR I 
haven't rebuilt since 1.12 and we use them successfully. They're just 
missing versions because we had not yet upgraded the Maven we used to 
build the NAR (not exactly relevant, I know, but I thought I'd I would 
point this out).



On 4/13/21 8:55 AM, nathan.engl...@bt.com wrote:

Hi Joe,

Thanks for that information.

Will there be any issue in running a Custom NAR file built against 
1.9.2 on a different versioned cluster. We haven’t experienced 
anything yet, but would hate to spend ages in debugging an issue that 
is the result of this!


Kind Regards,

Nathan

*From:* Joe Witt 
*Sent:* Tuesday, April 13, 2021 5:48 pm
*To:* users@nifi.apache.org
*Subject:* Re: NiFi 1.11.4 Custom Processor Development
Hello

In moving to support maven 3.8 we found only https based repos are 
allowed by default.  In addressing this we updated the build to move 
away from bintray and another location as well.


This is now the case when you build master. You could try to cherry 
pick and edit the key commits or hang tight for us to kick out a 1.14 
release.  No time table for that yet though.


Thanks

On Tue, Apr 13, 2021 at 7:45 AM > wrote:


Hi There,

I am in the process of upgrading the version of the
nifi-nar-bundle from 1.9.2 to 1.12.1 however I’ve hit a unexpected
issue.

We develop on an offline platform, with Maven Central Mirrored
using a Nexus Repository Manager, with all dependencies delivered
through this mechanism. Today when I have changed the version from
1.9.2 to 1.12.1 I have found that the parent (I assume for the
groovy processor) has a dependency on groovy-eclipse-batch version
2.5.4-01 which seems to be only available from a Groovy Repository
(https://dl.bintray.com/groovy/maven/

)
and not maven central.

I also noticed that bintray will be closing shortly
 
(https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/

),
so I guess the dependency will need correcting before then?

Kind Regards,

Nathan

Nathan English
Applications Specialist - Cyber Delivery & DevOps

Re: [DISCUSS] Processors Market

[Russell Bateman]

Javi,

Don't despair. Could just be that folk are busy and haven't had time to 
reflect upon it.



On 3/23/21 11:55 PM, Javi Roman wrote:

I see that it has not been well received, I thought it would be a good idea
:-(

--
Javi Roman

Twitter: @javiromanrh
GitHub: github.com/javiroman
Linkedin: es.linkedin.com/in/javiroman
Big Data Blog: dataintensive.info


On Tue, Mar 23, 2021 at 6:44 AM Javi Roman  wrote:


Hi!

I'm not sure whether this topic has been discussed in the past. I would
like to open a thread talking about the possibility of creating a kind of
market of NiFi Processors, for third-party processors.

I'm thinking about something similar to Kubernetes Helm hub [1], or
Kubernetes Operators Hub [2], but for NiFi processors (NAR bundles).

This repository could be managed by NiFi Registry in order to use
processors in different stages of development (preview, stable, official
... and so forth).

This is a way of slimming down the NiFi image and opening the development
to new users.

It would also be interesting to discuss if this "processors market" could
be hosted in ASF resources.

What do you think?

[1] https://artifacthub.io/
[2] https://operatorhub.io/
--
Javi Roman

Twitter: @javiromanrh
GitHub: github.com/javiroman
Linkedin: es.linkedin.com/in/javiroman
Big Data Blog: dataintensive.info

Re: Preconfiguring dynamic properties

[Russell Bateman]

Thanks, that probably gets me all or most of what I wanted. I'll try it out.

On 2/24/21 11:19 AM, Bryan Bende wrote:

I don't think it was the intent to pre-add a dynamic property. You
should be able to set a default value though, the user still has to
click the + icon to add the property though.

On Wed, Feb 24, 2021 at 12:02 PM Russell Bateman  wrote:

I have a dynamic property in a custom processor that my down-streamers
struggle a little bit to configure (requires newlines and a peculiar
format). I would like to "preconfigure" a dynamic property as an example
that they can either modify or erase to add their own. Most of them
would probably just use what I preconfigure.

The point is that I don't wish it to be a full-raging, static property.
I want to see a trash can to the right of it. The trash can is not a
feature of static properties.

Is this possible, wrong-headed, what?

Thanks,
Russ

Preconfiguring dynamic properties

[Russell Bateman]

I have a dynamic property in a custom processor that my down-streamers 
struggle a little bit to configure (requires newlines and a peculiar 
format). I would like to "preconfigure" a dynamic property as an example 
that they can either modify or erase to add their own. Most of them 
would probably just use what I preconfigure.


The point is that I don't wish it to be a full-raging, static property. 
I want to see a trash can to the right of it. The trash can is not a 
feature of static properties.


Is this possible, wrong-headed, what?

Thanks,
Russ

Re: [discuss] we need to enable secure by default...

[Russell Bateman]

I second the concerns expressed, but second especially Bryan's pointing 
out that requiring LDAP/AD to be set up in order even to begin to use 
our framework would be a bit onerous for developers just interested in 
getting work done and a barrier to considering the framework should it 
be erected a little too high. Should we at least glance at how this is 
solved by the likes of other projects, Kafka and Cassandra come to mind, 
even if it means resorting to a store of a name or two? I didn't find 
getting into developing with them a pain, but making me jump through the 
hoop of setting up LDAP may very well have changed that.


These unsecure instances of NiFi out there are not our community's 
fault. I suppose we're worried about getting splattered by bad press?


On 2/10/21 5:47 AM, Bryan Bende wrote:

I agree with the overall idea, although I would think it requires a
major release to make this kind of change to the default behavior.

Also, we have always avoided NiFi being a store of usernames and
passwords, so we don't have a login provider that uses a local file or
a database, we've always said you connect to LDAP/AD for that.

Obviously it can be implemented, but just pointing out that we'd have
to change our stance here if we want to provide a default username and
password to authenticate with.

On Tue, Feb 9, 2021 at 11:25 PM Andrew Grande  wrote:

Mysql has been generating an admin password on default installs for, like,
forever. This workflow should be familiar for many users.

I'd suggest taking the automation tooling into account and how a production
rollout (user-provided password) would fit into the workflow.

Andrew

On Tue, Feb 9, 2021, 8:15 PM Tony Kurc  wrote:


Joe,
In addition to your suggestions, were you thinking of making this processor
disabled by default as well?

Tony


On Tue, Feb 9, 2021, 11:04 PM Joe Witt  wrote:


Team

While secure by default may not be practical perhaps ‘not blatantly wide
open’ by default should be adopted.

I think we should consider killing support for http entirely and support
only https.  We should consider auto generating a user and password and
possibly server cert if nothing is configured and log the generated user
and password.   Sure it could still be configured to be non secure but

that

would truly be an admins fault.  Now its just ‘on’

This tweet is a great example of why

https://twitter.com/_escctrl_/status/1359280656174510081?s=21


Who agrees?  Who disagrees?   Please share ideas.

Thanks

Re: java api for changing parameter context

[Russell Bateman]

Wait! Can't this be done using the ReST APIs?

On 1/27/21 3:24 AM, u...@moosheimer.com wrote:

Hello NiFi-Core-Team,

Are you planning to create a high-level Java API for setting (and
clearing) individual parameters in the parameter context, so we can use
this API in processor development?

Example:
setParameter(string contextName, string parameterName, string
parameterValue, boolean sensitive);
deleteParameter(string contextName, string parameterName);

Some of our customers have systems with weekly changing parameter values
and/or access passphrase.
Apart from these nothing changes in the system and the changes can be
automated with self written processor.

Best regards,
Kay-Uwe Moosheimer

Re: Static processor design

[Russell Bateman]

Mark,

Thanks for responding. I think my question is a little more naive than 
that on my part.


I want to get those files through there as fast as possible. If I ask 
for /n/ files, how many would contribute to them getting them through 
the quickest? After all, I will do nothing at all to any except transfer 
them on and I don't care how many.


I write a lot of custom processors that do specific things to flowfiles 
one at a time. This isn't one of those. I don't care what's coming 
through, I just want to get every flowfile straight through with no changes.


Thanks.

Russ

On 1/9/21 9:09 AM, Mark Bean wrote:

Russell,

You can use "session.get(N)" where N is an integer. This will get up to N
flowfiles per OnTrigger() call.

-Mark


On Fri, Jan 8, 2021 at 5:07 PM Russell Bateman 
wrote:


Very well, I have decided to force customer flowfiles through this
processor (I did check out the /Listen/* processors, but chose this
easier solution). This now works. However,

It brings up another question: is this the most efficient way to pass
flowfiles straight through this processor (we're not processing them in
any way), or is there a batching technique that's faster, etc. I want
this to be straight-through, no back-pressure, throttling or influencing
their passage whatsoever (because I didn't want them coming through in
the first place). It should be ask if this processor weren't there.

Thanks for any and all thoughts on this.

public class HumanReadables extends AbstractProcessor{private boolean
propertyModified = false;@Override public void onTrigger( final
ProcessContext context, final ProcessSession session ) throws
ProcessException{FlowFile flowfile = session.get();if( propertyModified
){propertyModified = false;// record effects of changed
properties...}if( nonNull( flowfile ) )session.transfer( flowfile,
SUCCESS );}...@Override public void onPropertyModified( final
PropertyDescriptor descriptor, final String oldValue, final String
newValue ){propertyModified = true;}

Re: Static processor design

[Russell Bateman]

Very well, I have decided to force customer flowfiles through this 
processor (I did check out the /Listen/* processors, but chose this 
easier solution). This now works. However,


It brings up another question: is this the most efficient way to pass 
flowfiles straight through this processor (we're not processing them in 
any way), or is there a batching technique that's faster, etc. I want 
this to be straight-through, no back-pressure, throttling or influencing 
their passage whatsoever (because I didn't want them coming through in 
the first place). It should be ask if this processor weren't there.


Thanks for any and all thoughts on this.

public class HumanReadables extends AbstractProcessor{private boolean 
propertyModified = false;@Override public void onTrigger( final 
ProcessContext context, final ProcessSession session ) throws 
ProcessException{FlowFile flowfile = session.get();if( propertyModified 
){propertyModified = false;// record effects of changed 
properties...}if( nonNull( flowfile ) )session.transfer( flowfile, 
SUCCESS );}...@Override public void onPropertyModified( final 
PropertyDescriptor descriptor, final String oldValue, final String 
newValue ){propertyModified = true;}

Re: Static processor design

[Russell Bateman]

I only put the code I want to execute in onTrigger(), I suspected it 
would not fire there. I know that this isn't what processors do. 
Configuration is a messy problem to solve when your downstreamers want 
it made easy. This is supposed to be a solution that allows them to 
remain in the NiFi UI and not have to run off doing harder things to 
configure. I could put what I'm doing in /HumanReadables/ into a real, 
running processor, but then, I would kind of have to add them to several 
processors and I wanted to avoid the confusion that created.


Here's the code. Thanks.

import com.windofkeltia.constants.HumanReadableMappings;

...

@TriggerWhenEmpty
@SideEffectFree
@CapabilityDescription( "Dynamic properties can be created to specify 
(or add to) static configuration"
  + " of key-value substitution pairs. See 
additional details." )

public class HumanReadables extends AbstractProcessor
{
  @Override
  public void onTrigger( final ProcessContext context, final 
ProcessSession session ) throws ProcessException

  {
    getLogger().info( "inside onTrigger()..." );

    for( Map.Entry< PropertyDescriptor, String > entry : 
context.getProperties().entrySet() )

    {
  PropertyDescriptor property = entry.getKey();

  // do work here--maybe pre-wipe all key-value pairs if we're just 
going to recreate them?

  final String PROPERTY_NAME  = property.getName();
  final String PROPERTY_VALUE = entry.getValue();

  logger.trace( "Processing configurable mappings titled \"" + 
PROPERTY_NAME + "\"" );


  try
  {
    harvestDynamicPropertyMappings( PROPERTY_VALUE );
  }
  catch( Exception e )
  {
getLogger().debug( e.getMessage() );
  }
    }
  }

  protected static void harvestDynamicPropertyMappings( final String 
PROPERTY_VALUE )

  {
    final String[] LINES  = PROPERTY_VALUE.split( "\n" );
    int    lineNumber = 0;

    if( LINES.length < 1 )
  return;

    final String WHICH_LIST = LINES[ 0 ];

    for( final String VALUE_LINE : LINES )
    {
  char delimiter = VALUE_LINE.charAt( 0 );
  int  position = VALUE_LINE.indexOf( delimiter, 1 );
  String key, value;

  key   = ( position < 0 ) ? VALUE_LINE.substring( 1 ) : 
VALUE_LINE.substring( 1, position ).trim();
  value = ( position > 0 ) ? VALUE_LINE.substring( position + 1 
).trim() : "";


  HumanReadableMappings.add( key, value );
    }
  }

  @Override
  protected PropertyDescriptor getSupportedDynamicPropertyDescriptor( 
final String propertyDescriptorName )

  {
    return new PropertyDescriptor.Builder()
 .required( false )
 .name( propertyDescriptorName )
 .addValidator( 
StandardValidators.NON_EMPTY_VALIDATOR )
 // or .addValidator( Validator.VALID ) 
if you do not wish it validated!

 .dynamic( true )
 .build();
  }

  private volatile Set< String > dynamicPropertyNames = new HashSet<>();

  @Override
  public void onPropertyModified( final PropertyDescriptor descriptor, 
final String oldValue, final String newValue )

  {
    getLogger().info( oldValue + " -> " + newValue );

    final Set< String > newDynamicPropertyNames = new HashSet<>( 
dynamicPropertyNames );


    if( isNull( newValue ) )
  newDynamicPropertyNames.remove( descriptor.getName() );
    else if( isNull( oldValue ) && descriptor.isDynamic() )
  newDynamicPropertyNames.add( descriptor.getName() );

    dynamicPropertyNames = Collections.unmodifiableSet( 
newDynamicPropertyNames );


    final Set< String > allDynamicProperties = dynamicPropertyNames;
  }

  @OnScheduled public void processProperties( final ProcessContext 
context )

  {
    for( Map.Entry< PropertyDescriptor, String > entry : 
context.getProperties().entrySet() )

    {
  PropertyDescriptor descriptor = entry.getKey();

  if( descriptor.isDynamic() )
    getLogger().debug( "Dynamic property named:\n    " + 
descriptor.getName()
    + ", value: " + 
entry.getValue().replaceAll( "\n", " + " ) );

    }
  }

  protected static final String DEFAULT_MAPPING_VALUE = ""
    + "|http://loinc.org |LOINC\n"
    + "|http://snomed.info/sct |SNOMED\n"
    + "|http://www.ama-assn.org/go/cpt  |CPT\n"
    + "|http://aapc.org |CPT\n"
    + "|http://www.nlm.nih.gov/research/umls/rxnorm |RxNorm\n"
    + "|http://hl7.org/fhir/sid/ndc |NDC\n"
    + "|http://hl7.org/fhir/sid/icd-9-cm |ICD-9\n"
    + "|http://hl7.org/fhir/sid/icd-10 |ICD-10\n";

  public static final PropertyDescriptor DEFAULT_MAPPINGS = new 
PropertyDescriptor.Builder()

  .name( "default mappings" )
  .displayName( "Default mappings" )
  .required( false )
  .expressionLanguageSupported( ExpressionLanguageScope.NONE )
  .defaultValue( 

Re: Static processor design

[Russell Bateman]

The code I really want to run is sitting in onTrigger(), though I could 
move it elsewhere.


Yes, I have tried

*Scheduling Strategy*of Timer driven
*Run Schedule*of 10 sec

...but the getLogger().info( "called from onTrigger()" )never reaches 
/logs/nifi-app.log/ (while the logging statement from 
onPropertyModified()does reach the log every time I change properties to 
remove old or introduce new properties).



On 1/7/21 6:38 PM, Russell Bateman wrote:

(Inadequate title; didn't know what to call it.)

I have written a processor that doesn't feature any relationships.

It accepts dynamically properties that, in theory, when created (or 
removed, or values added or changed), and sets data into a class 
inside my NAR.


I wonder, however, at what I expect of it because, while it works in 
unit testing, it does not in practice. I can sort of guess why, but 
I'm not sure what to do about it. Given that I can create methods to 
be called at various opportunities by annotating thus:


@OnAdded
@OnEnabled
@OnRemoved
@OnScheduled
@OnUnscheduled
@OnStopped
@OnShutdown

There isn't one of these annotations that says to my brain, "When a 
dynamic property is added, changed or removed, wake up and run this 
method." Except, of course, for onPropertyModified(). A new property 
is duly added when created in configuration; my call to 
getLogger.info()from onPropertyModified()shows


2021-01-07 18:32:51,923 INFO [NiFi Web Server-78] 
c.windofkeltia.processor.HumanReadables 
HumanReadables[id=afa5b637-0176-1000-78bd-a74904054649] null -> 
|http://hospital.smarthealthit.org|Smart Health IT


But, how do I incite some code after the fact to awaken and analyze 
the newly added configuration then affect the 
HumanReadableMappingsclass instance?


(Hope I haven't explained this too badly. I am willing to attach 
code--it's a very tiny processor.)


Thanks

Static processor design

[Russell Bateman]

(Inadequate title; didn't know what to call it.)

I have written a processor that doesn't feature any relationships.

It accepts dynamically properties that, in theory, when created (or 
removed, or values added or changed), and sets data into a class inside 
my NAR.


I wonder, however, at what I expect of it because, while it works in 
unit testing, it does not in practice. I can sort of guess why, but I'm 
not sure what to do about it. Given that I can create methods to be 
called at various opportunities by annotating thus:


   @OnAdded
   @OnEnabled
   @OnRemoved
   @OnScheduled
   @OnUnscheduled
   @OnStopped
   @OnShutdown

There isn't one of these annotations that says to my brain, "When a 
dynamic property is added, changed or removed, wake up and run this 
method." Except, of course, for onPropertyModified(). A new property is 
duly added when created in configuration; my call to 
getLogger.info()from onPropertyModified()shows


2021-01-07 18:32:51,923 INFO [NiFi Web Server-78] 
c.windofkeltia.processor.HumanReadables 
HumanReadables[id=afa5b637-0176-1000-78bd-a74904054649] null -> 
|http://hospital.smarthealthit.org|Smart Health IT


But, how do I incite some code after the fact to awaken and analyze the 
newly added configuration then affect the HumanReadableMappingsclass 
instance?


(Hope I haven't explained this too badly. I am willing to attach 
code--it's a very tiny processor.)


Thanks

Re: Safely updating custom processors in existing flows...

[Russell Bateman]

I will be very happy to read what others contribute. Chris' suggestion, 
which has huge implications, I think, for my IDE and project structure, 
will be something to look into.


However, I was hoping someone would articulate the difference between 
*property names* and *property display names*, which of the two can be 
changed without invalidating the processor when simply replaced with the 
new one, etc. I have read a lot about this topic, but I'm still 
confused. Is display name supposed to be what shows up in the UI? Can it 
be changed without screwing up the processor? Is it the name that can be 
fixed? Which is the safe thing to modify?


I was hoping for some advice on taking quantum leaps (such as Chris 
discussed), but also on baby steps that must have been more common back 
in the days before processor versioning was supported.


Thanks

Safely updating custom processors in existing flows...

[Russell Bateman]

I have a custom processor actively used in customer flows. I need to 
enhance it, but avoid invalidating it when I update custom's existing 
NiFi installations.


   I know that the processor properties, in particular, the conflict
   between the property .name()and .displayName(), is a good way to get
   myself into trouble by changing the wording, correcting spelling, etc.

   Also, adding to or subtracting properties and/or relationships is a
   good way to blow it all up.

   How does versioning the processor solve any of this?

   Etc.

Can anyone share canonical advice on this?

Thanks,
Russ

Re: ETL to Nifi Migration

[Russell Bateman]

Unless you're certain you will need to write custom processors for NiFi, 
the forum you really want to subscribe to and post in is


NiFi Users 

Best regards!

On 12/26/20 7:37 AM, Sumith Karthikeyan wrote:

Hi Team,

Hope you all doing well !!!

This regards a Middle East public sector requirement to move their current ETL 
to Apache Nifi in AWS cloud. We have prepared a questionnaire in-align with our 
expectation and request your technical support to collect the details against 
each.

Please help me to connect right team/support members to get this done. 
Questionnaire is in excel format and not included here as I am not sure it's 
the right forum to do so.

Thanks in advance !!!

Merry Christmas and Happy New Year !!!

Thanks & Regards,
Sumith K


Disclaimer: This transmittal and/or attachments have been issued by LMRA. The 
information contained here within may be privileged or confidential. If you are 
not the intended recipient, you are hereby notified that you have received this 
transmittal in error; any review, dissemination, distribution or copying of 
this transmittal is strictly prohibited. If you have received this transmittal 
and/or attachments in error, please notify us immediately by reply to the 
sender or delete this message and all its attachments immediately.

Re: Okay to manage ComponentLog via ThreadLocal?

[Russell Bateman]

Yes, well, I would consider that I had out-foxed myself, but it's 
probably due to not quite having all my fingers around the scope of the 
processor. I guess there's only one component logger for the currently 
executing NiFi instance? Yeah, I was thinking really hard about 
per-thread stuff, which is how I used this before.


You're right. I swallowed the whole camel.

Thanks, Mark!


On 12/8/20 2:58 PM, Mark Payne wrote:

Russ,

Why not just use:

Public class Foo {
public void bar() {
ComponentLogger logger = CustomProcessor.getLogger();
logger.warn( “This is a warning!” );
}
}


Perhaps I’m missing something - or perhaps you made things simpler than they 
really are for demonstration purposes?

Thanks
-Mark



On Dec 8, 2020, at 4:54 PM, Russell Bateman  wrote:

Because it's so onerous to pass a reference to the logger down through 
parameters lists, I thought I might try using Java's thread-local store. I 
haven't been using it for anything else either, but I thought I'd start. For 
now, the logger is the only thing that tempts me. In past lives as a 
web-application writer, I used it quite a bit.

My question is, "Does this offend anyone who cares to give an opinion?" If 
there's a reason not to do this, I'll go to the effort (and muddy my parameter lists). 
Otherwise, I'll give it a try.


public class CustomProcessor extends AbstractProcessor
{
   private static ThreadLocal< ComponentLog > tls = new ThreadLocal<>();

   public static ThreadLocal< ComponentLog > getTls() { return tls; }

   @Override
   public void onTrigger( final ProcessContext context, final ProcessSession 
session ) throws ProcessException
   {
 // grab the logger and store it on the thread...
 tls.set( getLogger() );

 ...

 // we've finished using it--dump the logger...
 tls.remove();
   }
}

public class Foo
{
   public void bar()
   {
 ComponentLog logger = CustomProcessor.getTls().get();
 logger.warn( "This is a warning!" );
   }
}

Thanks for any opinions, statements of best practice, cat calls, sneers, etc.

;-)

Russ

Okay to manage ComponentLog via ThreadLocal?

[Russell Bateman]

Because it's so onerous to pass a reference to the logger down through 
parameters lists, I thought I might try using Java's thread-local store. 
I haven't been using it for anything else either, but I thought I'd 
start. For now, the logger is the only thing that tempts me. In past 
lives as a web-application writer, I used it quite a bit.


My question is, "Does this offend anyone who cares to give an opinion?" 
If there's a reason not to do this, I'll go to the effort (and muddy my 
parameter lists). Otherwise, I'll give it a try.



public class CustomProcessor extends AbstractProcessor
{
  private static ThreadLocal< ComponentLog > tls = new ThreadLocal<>();

  public static ThreadLocal< ComponentLog > getTls() { return tls; }

  @Override
  public void onTrigger( final ProcessContext context, final 
ProcessSession session ) throws ProcessException

  {
    // grab the logger and store it on the thread...
    tls.set( getLogger() );

    ...

    // we've finished using it--dump the logger...
    tls.remove();
  }
}

public class Foo
{
  public void bar()
  {
    ComponentLog logger = CustomProcessor.getTls().get();
    logger.warn( "This is a warning!" );
  }
}

Thanks for any opinions, statements of best practice, cat calls, sneers, 
etc.


;-)

Russ

Re: Tuning for flow with lots of processors

[Russell Bateman]

Eric, Mark,

This is an example of an exchange in this forum that is of immeasurable 
help to to on-lookers. Thanks for discussing it so thoroughly!


Russ

On 11/23/20 5:55 PM, Eric Secules wrote:

Hello everyone,

I was wondering if there was a metric for the amount of time 
tImer-driven processors spend in a queue ready and waiting to be run. 
I use NiFi in an atypical way and my flow has over 2000 processors 
running on a single node, but there are usually less than 10 
connections that have one or more flowfiles in them at any given time.


I have a theory that the number of processors in use is slowing down 
the system overall. But I would need to see some more metrics to know 
whether that's the case and tell whether anything I am doing is 
helping. Are there some logs that I could look for or internal stats I 
could poke at with a debugger?


Should I be able to see increased throughput by increasing the number 
of timer-driven threads, or is there a different mechanism responsible 
for going through all the runnable processors to see whether they have 
input to process. I also noticed "nifi.bored.yield.duration" would it 
be good to increase the yield duration in this setting?


Thanks,
Eric

Re: Run Nifi in IntelliJ to debug?

[Russell Bateman]

Hmmm... It's rare that I debug NiFi code. And it's also rare that I 
debug my own in that context since the NiFi test runner allows me to 
fend off most surprises via my JUnit tests.


I think back in 2016, I was debugging a start-up problem involving NiFi 
start-up and incompatibility with the Java Flight Recorder. As I recall, 
I downloaded the relevant NiFi code sources matching the version of NiFi 
I was debugging remotely. I remember ultimately making a slight (and 
only temporary) change to NiFi start-up that fixed the problem. At that 
point I must have been building my own copy to have seen it fixed.. It 
had to do with the order in which NiFi was getting command-line 
arguments making it so the JFR wasn't running. I'd have to dig back to 
figure out what I was doing, but it's probably not too relevant to what 
you need to do.


What do you need to see in this?

Russ

On 10/26/20 5:38 PM, Darren Govoni wrote:
Correct. Primarily the nifi-web-api module and AccessResource class. 
For starters.


Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android <https://aka.ms/ghei36>


*From:* Russell Bateman 
*Sent:* Monday, October 26, 2020 7:37:13 PM
*To:* Darren Govoni ; users@nifi.apache.org 


*Subject:* Re: Run Nifi in IntelliJ to debug?
Darren,

This is just Apache NiFi code out of NARs you want to step through or 
is it yours? You haven't stripped debug information or anything, right?


Russ

On 10/26/20 5:30 PM, Darren Govoni wrote:

Kevin/Russel

Thanks for the info. I did set things up this way.

IntelliJ does connect to the nifi jvm and nifi runs and works but 
intellij isnt breaking on code it should.


I did set the module where the code/classes are located (in the 
remote connection dialog) and i see the exception im tracking print 
on the console output but intellij never breaks.


Is there an extra step needed? Generate sources?

For future it would be nice if there was a maven goal for debug.

Much appreciated!
Darren

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android <https://aka.ms/ghei36>
----
*From:* Russell Bateman  
<mailto:r...@windofkeltia.com>

*Sent:* Monday, October 26, 2020 4:09:50 PM
*To:* users@nifi.apache.org <mailto:users@nifi.apache.org> 
 <mailto:users@nifi.apache.org>; Darren Govoni 
 <mailto:dar...@ontrenet.com>

*Subject:* Re: Run Nifi in IntelliJ to debug?
Darren,

I was out this morning and didn't see your plea until I got in just 
now. Here's a step by step I wrote up for both IntelliJ IDEA and 
Eclipse (I'm more an IntelliJ guy). It also covers using an IP tunnel.


https://www.javahotchocolate.com/notes/nifi.html#20160323

On 10/26/20 9:52 AM, Darren Govoni wrote:

Hi
   Is it possible to run Nifi from inside IntelliJ with debugging 
such that I can hit the app from my browser and trigger breakpoints?


If anyone has done this can you please share any info?

Thanks in advance!
Darren

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android <https://aka.ms/ghei36>

Re: Run Nifi in IntelliJ to debug?

[Russell Bateman]

Darren,

This is just Apache NiFi code out of NARs you want to step through or is 
it yours? You haven't stripped debug information or anything, right?


Russ

On 10/26/20 5:30 PM, Darren Govoni wrote:

Kevin/Russel

Thanks for the info. I did set things up this way.

IntelliJ does connect to the nifi jvm and nifi runs and works but 
intellij isnt breaking on code it should.


I did set the module where the code/classes are located (in the remote 
connection dialog) and i see the exception im tracking print on the 
console output but intellij never breaks.


Is there an extra step needed? Generate sources?

For future it would be nice if there was a maven goal for debug.

Much appreciated!
Darren

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android <https://aka.ms/ghei36>

*From:* Russell Bateman 
*Sent:* Monday, October 26, 2020 4:09:50 PM
*To:* users@nifi.apache.org ; Darren Govoni 


*Subject:* Re: Run Nifi in IntelliJ to debug?
Darren,

I was out this morning and didn't see your plea until I got in just 
now. Here's a step by step I wrote up for both IntelliJ IDEA and 
Eclipse (I'm more an IntelliJ guy). It also covers using an IP tunnel.


https://www.javahotchocolate.com/notes/nifi.html#20160323

On 10/26/20 9:52 AM, Darren Govoni wrote:

Hi
   Is it possible to run Nifi from inside IntelliJ with debugging 
such that I can hit the app from my browser and trigger breakpoints?


If anyone has done this can you please share any info?

Thanks in advance!
Darren

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android <https://aka.ms/ghei36>

Re: Run Nifi in IntelliJ to debug?

[Russell Bateman]

Darren,

I was out this morning and didn't see your plea until I got in just now. 
Here's a step by step I wrote up for both IntelliJ IDEA and Eclipse (I'm 
more an IntelliJ guy). It also covers using an IP tunnel.


https://www.javahotchocolate.com/notes/nifi.html#20160323

On 10/26/20 9:52 AM, Darren Govoni wrote:

Hi
   Is it possible to run Nifi from inside IntelliJ with debugging such 
that I can hit the app from my browser and trigger breakpoints?


If anyone has done this can you please share any info?

Thanks in advance!
Darren

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android 

Re: Hive NAR not loading because of snappy?

[Russell Bateman]

Our current installations are running on NiFi 1.1.2, I think. Recently, 
the company wanted to move up in the world. So, we don't have that 
continuous experience. We started on 0.7.1.


We're running Java 8 on CentOS 8, I think.

Personally, I run Linux Mint (Ubuntu, corrected) for development. I have 
never before had any reason to think that NiFi on CentOS was any 
different than on Fedora, Ubuntu or Mint (among the Linux boxes I have 
run on over the last 4-5 years).


On 10/13/20 6:35 PM, Matt Burgess wrote:
Understood. But outside of a permissions issue or usage of 
CompressContent or something else that would try to load the same 
version of Snappy, I’m not sure what’s going on there. AFAIK other 
folks install this on centos without issue. The related snappy issues 
were supposedly resolved well before 1.11.4. Did it used to work on 
previous versions? Have you tried the latest release to see if it 
works? Has the version of Java changed?


Sent from my iPhone

On Oct 13, 2020, at 8:13 PM, Russell Bateman  
wrote:


 We didn't want to make deletion or modification of a shipped 
component to be a required part of its installation since we don't 
produce the NiFi download. We'd rather install it as it comes.


On 10/13/20 4:45 PM, Matt Burgess wrote:
Ouch! It does happen on the loading of the NAR to ensure the native 
library gets loaded. If you are not using Hive I’d think you could 
safely delete the nifi-hive-nar and it shouldn’t happen. Hard to 
tell why the native library couldn’t be installed though.



On Oct 13, 2020, at 6:26 PM, Russell Bateman 
 wrote:


 No, we don't even use (nor have we ever used) Hive in our flows. 
It's just there and we didn't want to modify the NiFi download. 
Should this not even happen if we're not using it?


On 10/13/20 4:24 PM, Matt Burgess wrote:
Ugh now I remember, that version of Hive uses a version of Snappy 
that doesn’t create a unique path under /tmp, do you have multiple 
PutHiveStreaming processors in the flow? I don’t think that works 
because we can’t load a single native library into multiple 
classloaders.



On Oct 13, 2020, at 6:15 PM, Russell Bateman 
 wrote:


 I see

-rwxr-xr-x. 1 nifi nifi  48432 Oct 13 13:48 
snappy-1.0.5-libsnappyjava.so


in //tmp/. Therefore, not a permissions issue? Launching this way 
works:


$ ( export 
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so" 
&& /opt/nifi/bin/nifi.sh start )


but it's not something we want to do (in case that shared object 
disappears from //tmp/).



On 10/13/20 3:42 PM, Matt Burgess wrote:
IIRC this is likely a permissions issue, Xerial Snappy tries to 
unzip the native library to the location pointed to by 
“java.io.tempdir” which on *nix defaults to /tmp. Does the NiFi 
user have write access to that directory? If not you can change 
the Java temp dir or set it specifically for Snappy (I don’t 
have the property on hand but a quick Google should find it)


Regards,
Matt

Sent from my iPhone

On Oct 13, 2020, at 5:36 PM, Russell Bateman 
 wrote:


 Should I be seeing this in the log of a vanilla NiFi 
installation on CentOS?


ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to 
java.util.ServiceConfigurationError: 
org.apache.nifi.processor.Processor: Provider 
org.apache.nifi.processors.hive.PutHiveStreaming could not be 
initialized

.
.
.
Caused by: org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null


"Snappy" makes me think Ubuntu. This is CentOS. Did I pull down 
the wrong /nifi-1.11.4-bin.zip/?

Re: Hive NAR not loading because of snappy?

[Russell Bateman]

We didn't want to make deletion or modification of a shipped component 
to be a required part of its installation since we don't produce the 
NiFi download. We'd rather install it as it comes.


On 10/13/20 4:45 PM, Matt Burgess wrote:
Ouch! It does happen on the loading of the NAR to ensure the native 
library gets loaded. If you are not using Hive I’d think you could 
safely delete the nifi-hive-nar and it shouldn’t happen. Hard to tell 
why the native library couldn’t be installed though.



On Oct 13, 2020, at 6:26 PM, Russell Bateman  
wrote:


 No, we don't even use (nor have we ever used) Hive in our flows. 
It's just there and we didn't want to modify the NiFi download. 
Should this not even happen if we're not using it?


On 10/13/20 4:24 PM, Matt Burgess wrote:
Ugh now I remember, that version of Hive uses a version of Snappy 
that doesn’t create a unique path under /tmp, do you have multiple 
PutHiveStreaming processors in the flow? I don’t think that works 
because we can’t load a single native library into multiple 
classloaders.



On Oct 13, 2020, at 6:15 PM, Russell Bateman 
 wrote:


 I see

-rwxr-xr-x. 1 nifi    nifi 48432 Oct 13 13:48 
snappy-1.0.5-libsnappyjava.so


in //tmp/. Therefore, not a permissions issue? Launching this way 
works:


$ ( export 
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so" 
&& /opt/nifi/bin/nifi.sh start )


but it's not something we want to do (in case that shared object 
disappears from //tmp/).



On 10/13/20 3:42 PM, Matt Burgess wrote:
IIRC this is likely a permissions issue, Xerial Snappy tries to 
unzip the native library to the location pointed to by 
“java.io.tempdir” which on *nix defaults to /tmp. Does the NiFi 
user have write access to that directory? If not you can change 
the Java temp dir or set it specifically for Snappy (I don’t have 
the property on hand but a quick Google should find it)


Regards,
Matt

Sent from my iPhone

On Oct 13, 2020, at 5:36 PM, Russell Bateman 
 wrote:


 Should I be seeing this in the log of a vanilla NiFi 
installation on CentOS?


ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to 
java.util.ServiceConfigurationError: 
org.apache.nifi.processor.Processor: Provider 
org.apache.nifi.processors.hive.PutHiveStreaming could not be 
initialized

.
.
.
Caused by: org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null


"Snappy" makes me think Ubuntu. This is CentOS. Did I pull down 
the wrong /nifi-1.11.4-bin.zip/?

Re: Hive NAR not loading because of snappy?

[Russell Bateman]

No, we don't even use (nor have we ever used) Hive in our flows. It's 
just there and we didn't want to modify the NiFi download. Should this 
not even happen if we're not using it?


On 10/13/20 4:24 PM, Matt Burgess wrote:
Ugh now I remember, that version of Hive uses a version of Snappy that 
doesn’t create a unique path under /tmp, do you have multiple 
PutHiveStreaming processors in the flow? I don’t think that works 
because we can’t load a single native library into multiple classloaders.



On Oct 13, 2020, at 6:15 PM, Russell Bateman  
wrote:


 I see

-rwxr-xr-x. 1 nifi    nifi  48432 Oct 13 13:48 
snappy-1.0.5-libsnappyjava.so


in //tmp/. Therefore, not a permissions issue? Launching this way works:

$ ( export 
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so" 
&& /opt/nifi/bin/nifi.sh start )


but it's not something we want to do (in case that shared object 
disappears from //tmp/).



On 10/13/20 3:42 PM, Matt Burgess wrote:
IIRC this is likely a permissions issue, Xerial Snappy tries to 
unzip the native library to the location pointed to by 
“java.io.tempdir” which on *nix defaults to /tmp. Does the NiFi user 
have write access to that directory? If not you can change the Java 
temp dir or set it specifically for Snappy (I don’t have the 
property on hand but a quick Google should find it)


Regards,
Matt

Sent from my iPhone

On Oct 13, 2020, at 5:36 PM, Russell Bateman 
 wrote:


 Should I be seeing this in the log of a vanilla NiFi installation 
on CentOS?


ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to 
java.util.ServiceConfigurationError: 
org.apache.nifi.processor.Processor: Provider 
org.apache.nifi.processors.hive.PutHiveStreaming could not be 
initialized

.
.
.
Caused by: org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null


"Snappy" makes me think Ubuntu. This is CentOS. Did I pull down the 
wrong /nifi-1.11.4-bin.zip/?

Re: Hive NAR not loading because of snappy?

[Russell Bateman]

I see

-rwxr-xr-x. 1 nifi    nifi  48432 Oct 13 13:48 
snappy-1.0.5-libsnappyjava.so


in //tmp/. Therefore, not a permissions issue? Launching this way works:

$ ( export 
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so" && 
/opt/nifi/bin/nifi.sh start )


but it's not something we want to do (in case that shared object 
disappears from //tmp/).



On 10/13/20 3:42 PM, Matt Burgess wrote:
IIRC this is likely a permissions issue, Xerial Snappy tries to unzip 
the native library to the location pointed to by “java.io.tempdir” 
which on *nix defaults to /tmp. Does the NiFi user have write access 
to that directory? If not you can change the Java temp dir or set it 
specifically for Snappy (I don’t have the property on hand but a quick 
Google should find it)


Regards,
Matt

Sent from my iPhone

On Oct 13, 2020, at 5:36 PM, Russell Bateman  
wrote:


 Should I be seeing this in the log of a vanilla NiFi installation 
on CentOS?


ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to 
java.util.ServiceConfigurationError: 
org.apache.nifi.processor.Processor: Provider 
org.apache.nifi.processors.hive.PutHiveStreaming could not be initialized

.
.
.
Caused by: org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null


"Snappy" makes me think Ubuntu. This is CentOS. Did I pull down the 
wrong /nifi-1.11.4-bin.zip/?

Hive NAR not loading because of snappy?

[Russell Bateman]

Should I be seeing this in the log of a vanilla NiFi installation on CentOS?

ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to 
java.util.ServiceConfigurationError: 
org.apache.nifi.processor.Processor: Provider 
org.apache.nifi.processors.hive.PutHiveStreaming could not be initialized

.
.
.
Caused by: org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null


"Snappy" makes me think Ubuntu. This is CentOS. Did I pull down the 
wrong /nifi-1.11.4-bin.zip/?

Re: APACHE NIFI CLUSTER INSTALLATION WALKTHROUGHS FOR 2NODE ON LINUX OS

[Russell Bateman]

You have treated Google as your friend and worked through the dozen or 
so examples by other folk doing this, right?


--just a suggestion.

On 9/23/20 1:21 PM, Abiodun Adegbile wrote:

*/Hello Team,/*

Still looking forward to your reply.
/
/

/I got this though after i tried setting up the 2-node cluster on two 
separate physical servers vms/


*/

/*

*/
/*

Your prompt response will be highly appreciated.

*/
/*

*/KEEP SAFE, STAY SAFE & STAY STRONG/*

/
/

/I celebrate you.../


/*/ADEGBILE ABIODUN A.(OCA, OCP, OCIFA, OCIAA, OCIAP)/*/

*/Database Administrator/Infrastructure Services/*

*/Bluechip Technologies Limited/*

*/Plot 9B, Onikoyi Lane/*

*/Ikoyi. Lagos State./*

*/Mobile : +234 806 206 9970/*

*/Website : www.bluechiptech.biz /*

*/
/*

.../*2016 Winner Oracle Transformational Deal of the Year - Nigeria*./



*From:* Abiodun Adegbile
*Sent:* Sunday, September 20, 2020 2:33 AM
*To:* dev@nifi.apache.org 
*Subject:* APACHE NIFI CLUSTER INSTALLATION WALKTHROUGHS FOR 2NODE ON 
LINUX OS

*/Hello Team,/*

Good to write to you.

I am planning to implement a NIFI cluster installation for two 
separate physical linux servers(to-be-clustered) for my test lab. I 
was hoping you could send a walkthrough documentation for this.


Many thanks for your help. Will appreciate your prompt response.

*/
/*

*/
/*

*/KEEP SAFE, STAY SAFE & STAY STRONG/*

/
/

/I celebrate you.../


/*/ADEGBILE ABIODUN A.(OCA, OCP, OCIFA, OCIAA, OCIAP)/*/

*/Database Administrator/Infrastructure Services/*

*/Bluechip Technologies Limited/*

*/Plot 9B, Onikoyi Lane/*

*/Ikoyi. Lagos State./*

*/Mobile : +234 806 206 9970/*

*/Website : www.bluechiptech.biz /*

*/
/*

.../*2016 Winner Oracle Transformational Deal of the Year - Nigeria*./

Re: APACHE NIFI CLUSTER INSTALLATION WALKTHROUGHS FOR 2NODE ON LINUX OS

[Russell Bateman]

Google is your friend.

https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/nifi-configuration-best-practices/content/basic-cluster-setup.html
https://bryanbende.com/development/2018/10/23/apache-nifi-secure-cluster-setup
https://mintopsblog.com/2017/11/12/apache-nifi-cluster-configuration/
https://www.nifi.rocks/apache-nifi-docker-compose-cluster/

etc.

On 9/19/20 7:33 PM, Abiodun Adegbile wrote:

*/Hello Team,/*

Good to write to you.

I am planning to implement a NIFI cluster installation for two 
separate physical linux servers(to-be-clustered) for my test lab. I 
was hoping you could send a walkthrough documentation for this.


Many thanks for your help. Will appreciate your prompt response.

*/
/*

*/
/*

*/KEEP SAFE, STAY SAFE & STAY STRONG/*

/
/

/I celebrate you.../


/*/ADEGBILE ABIODUN A.(OCA, OCP, OCIFA, OCIAA, OCIAP)/*/

*/Database Administrator/Infrastructure Services/*

*/Bluechip Technologies Limited/*

*/Plot 9B, Onikoyi Lane/*

*/Ikoyi. Lagos State./*

*/Mobile : +234 806 206 9970/*

*/Website : www.bluechiptech.biz /*

*/
/*

.../*2016 Winner Oracle Transformational Deal of the Year - Nigeria*./

Re: TestRunner: enqueueing multiple flowfiles

[Russell Bateman]

Oh, no, Brian, calling runner.enqueue()multiple times is a perfect 
solution. It wasn't clear that this was an option. I guess I missed that 
semantic in the Javadoc, but, certainly, the name, "enqueue" should have 
been a huge hint to me.


Thanks!

On 8/31/20 7:26 AM, Bryan Bende wrote:

I think you could call any of the enqueue methods multiple times to queue
multiple flow files.

If you really want to use the one that takes var args of FlowFile, then you
would need to create the MockFlowFiles yourself doing something like this...

https://github.com/apache/nifi/blob/main/nifi-mock/src/main/java/org/apache/nifi/util/StandardProcessorTestRunner.java#L443-L448

Instead of creating a new MockProcessSession, you would get the
ProcessSessionFactory from the TestRunner and then call
createProcessSession().


On Mon, Aug 31, 2020 at 9:01 AM Russell Bateman 
wrote:


In my JUnit testing of a custom processor, I need to queue up at least
two flowfiles. I see that there is an implementation of
TestRunner.enqueue()that takes*a list of flowfiles*, but I'm used to
using the implementation of this method that creates me a flowfile from
bytes or a stream. I do not know how to create a flowfile from scratch
inside test code. MockFlowFile's two constructors are no help. Getting
there via interface ProcessorSessionseems a long road to travel just for
this.

Examples using TestRunner.enqueue( FlowFile ... flowfiles ) do not
abound out there in Googleland. Has someone else done this?

Thanks.

TestRunner: enqueueing multiple flowfiles

[Russell Bateman]

In my JUnit testing of a custom processor, I need to queue up at least 
two flowfiles. I see that there is an implementation of 
TestRunner.enqueue()that takes*a list of flowfiles*, but I'm used to 
using the implementation of this method that creates me a flowfile from 
bytes or a stream. I do not know how to create a flowfile from scratch 
inside test code. MockFlowFile's two constructors are no help. Getting 
there via interface ProcessorSessionseems a long road to travel just for 
this.


Examples using TestRunner.enqueue( FlowFile ... flowfiles ) do not 
abound out there in Googleland. Has someone else done this?


Thanks.

Re: From one flowfile to two...

[Russell Bateman]

In case anyone cares,
https://www.javahotchocolate.com/notes/nifi-custom.html#two-split-from-one

On 8/27/20 11:15 AM, Andy LoPresto wrote:

Russell,

Glad you found a working solution. Maybe it would be better for you to write up 
your findings and share them with a broader audience. I have often seen the 
best explanations are written by people who were recently in the “how do I do 
X?” state, as they are closest to the problem and can walk through their 
process of gathering understanding. Someone who works on these methods day in 
and day out may not write for the appropriate audience or explain the 
experience as well.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


On Aug 27, 2020, at 10:10 AM, Russell Bateman  wrote:

I needed to get back here...

I took this advice to heart and finished my processor. Thanks to Matt and Mark 
for all their suggestions! They cleared up a few things. There was one bug in 
the code that was mine, small, but significant in its effect on the rest. That 
mistake also explained why I thought the uuidwas identical between at least two 
of the cloned flowfiles. What I would wish for, and am probably not strong 
enough to write, would be a synthesis of the session methods read() and write() 
and how best to use them (one-to-one, one-to-many, etc.). Javadoc is too 
paratactic by nature, the NiFi Developer's Guide almost silent on these 
methods. If it were not for the many existing examples using these methods, it 
would be hard to learn to do even simple things. I did look for something 
closer to what I needed to do, but unsuccessfully.

Thanks again. If anything, the NiFi mailing lists are a place both for great 
information and being treated well.

Russ

On 8/25/20 12:24 PM, Mark Payne wrote:

Russ,

Several comments here. I’ve included them inline, below.

Hope it’s helpful.

Thanks
-Mark



On Aug 25, 2020, at 2:09 PM, Russell Bateman  wrote:

Thanks for your suggestions, Matt.

I decided to keep the original flowfile only upon failure. So, I have the 
embedded-document file and the serialized POJOs created from processing the non 
embedded-document part as the result if successful. (Condensed code at end...)

Now I have three questions...

1. I seem not to have placated NiFi with the assurance that I have transferred 
or disposed of all three flowfiles suitably. I get:

java.lang.AssertionError: 
org.apache.nifi.processor.exception.FlowFileHandlingException: Cannot commit 
session because the following FlowFiles have not been removed or transferred: 
[2]

This is probably because at the end of the block, you catch Exception and then 
route the original FlowFile to failure. But you’ve already cloned it and didn’t 
deal with the clone.

*Which of the three flowfiles does [2] refer to? Or does it just mean I botched 
two flowfiles. *

2. session.clone()generates a new flowfile with the identical uuid. I don't 
think I want the result to be two flowfiles with the same uuid. I am binding 
them together so I can associate them later using attribute embedded-document. 
*Should I/How do I force cloning to acquire new **uuid**s?*

This appears to actually be a bug in the mock framework. It *should* have a 
unique uuid, and would in a running NiFi instance. Feel free to file a Jira for 
that.

3. A question on theory... *Wouldn't all of this cloning be expensive* and I 
should just clone for one of the new files and then mangle the original 
flowfile to become the other?

session.clone() is not particularly expensive. It’s just creating a new 
FlowFile object. It doesn’t clone the FlowFile’s contents.

That said, it is probably more appropriate to call session.create(flowFile), 
rather than session.clone(flowFile). It makes little difference in practice but 
what you’re really doing is forking a child, and that will come across more 
cleanly in the Provenance lineage that is generated if using 
session.create(flowFile).

Additional comments in code below.



Thanks,
Russ


@Override
public void onTrigger( final ProcessContext context, final ProcessSession 
session ) throws ProcessException
{
   FlowFile flowfile = session.get();

   if( flowfile == null )
   {
 context.yield();

No need to yield here. Let the framework handle the scheduling. 
ProcessContext.yield() is meant for cases where you’re communicating with some 
external service, for instance, and you know the service is unavailable or rate 
limiting you or something like that. You can’t make any progress, so tell NiFi 
to not bother wasting CPU cycles with this Processor.

 return;
   }

   try
   {
 final String UUID = flowfile.getAttribute( NiFiUtilities.UUID );

 FlowFile document = session.clone( flowfile );

*// excerpt and write the embedded document to a new flowfile...*
 session.write( document, new OutputStreamCallback()
 {
   @Override public void process( OutputStream outputStream

Re: From one flowfile to two...

[Russell Bateman]

I will be sure to do that. I keep several pages of NiFi notes on my 
website (javahotchocolate.com). The notes are mostly for me to 
re-consult, but I have tutorials about writing custom processors and the 
like. I'll be putting out a skeletal copy of my recent code soon.


Thanks!

On 8/27/20 11:15 AM, Andy LoPresto wrote:

Russell,

Glad you found a working solution. Maybe it would be better for you to write up 
your findings and share them with a broader audience. I have often seen the 
best explanations are written by people who were recently in the “how do I do 
X?” state, as they are closest to the problem and can walk through their 
process of gathering understanding. Someone who works on these methods day in 
and day out may not write for the appropriate audience or explain the 
experience as well.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


On Aug 27, 2020, at 10:10 AM, Russell Bateman  wrote:

I needed to get back here...

I took this advice to heart and finished my processor. Thanks to Matt and Mark 
for all their suggestions! They cleared up a few things. There was one bug in 
the code that was mine, small, but significant in its effect on the rest. That 
mistake also explained why I thought the uuidwas identical between at least two 
of the cloned flowfiles. What I would wish for, and am probably not strong 
enough to write, would be a synthesis of the session methods read() and write() 
and how best to use them (one-to-one, one-to-many, etc.). Javadoc is too 
paratactic by nature, the NiFi Developer's Guide almost silent on these 
methods. If it were not for the many existing examples using these methods, it 
would be hard to learn to do even simple things. I did look for something 
closer to what I needed to do, but unsuccessfully.

Thanks again. If anything, the NiFi mailing lists are a place both for great 
information and being treated well.

Russ

On 8/25/20 12:24 PM, Mark Payne wrote:

Russ,

Several comments here. I’ve included them inline, below.

Hope it’s helpful.

Thanks
-Mark



On Aug 25, 2020, at 2:09 PM, Russell Bateman  wrote:

Thanks for your suggestions, Matt.

I decided to keep the original flowfile only upon failure. So, I have the 
embedded-document file and the serialized POJOs created from processing the non 
embedded-document part as the result if successful. (Condensed code at end...)

Now I have three questions...

1. I seem not to have placated NiFi with the assurance that I have transferred 
or disposed of all three flowfiles suitably. I get:

java.lang.AssertionError: 
org.apache.nifi.processor.exception.FlowFileHandlingException: Cannot commit 
session because the following FlowFiles have not been removed or transferred: 
[2]

This is probably because at the end of the block, you catch Exception and then 
route the original FlowFile to failure. But you’ve already cloned it and didn’t 
deal with the clone.

*Which of the three flowfiles does [2] refer to? Or does it just mean I botched 
two flowfiles. *

2. session.clone()generates a new flowfile with the identical uuid. I don't 
think I want the result to be two flowfiles with the same uuid. I am binding 
them together so I can associate them later using attribute embedded-document. 
*Should I/How do I force cloning to acquire new **uuid**s?*

This appears to actually be a bug in the mock framework. It *should* have a 
unique uuid, and would in a running NiFi instance. Feel free to file a Jira for 
that.

3. A question on theory... *Wouldn't all of this cloning be expensive* and I 
should just clone for one of the new files and then mangle the original 
flowfile to become the other?

session.clone() is not particularly expensive. It’s just creating a new 
FlowFile object. It doesn’t clone the FlowFile’s contents.

That said, it is probably more appropriate to call session.create(flowFile), 
rather than session.clone(flowFile). It makes little difference in practice but 
what you’re really doing is forking a child, and that will come across more 
cleanly in the Provenance lineage that is generated if using 
session.create(flowFile).

Additional comments in code below.



Thanks,
Russ


@Override
public void onTrigger( final ProcessContext context, final ProcessSession 
session ) throws ProcessException
{
   FlowFile flowfile = session.get();

   if( flowfile == null )
   {
 context.yield();

No need to yield here. Let the framework handle the scheduling. 
ProcessContext.yield() is meant for cases where you’re communicating with some 
external service, for instance, and you know the service is unavailable or rate 
limiting you or something like that. You can’t make any progress, so tell NiFi 
to not bother wasting CPU cycles with this Processor.

 return;
   }

   try
   {
 final String UUID = flowfile.getAttribute( NiFiUtilities.UUID );

 FlowFile document = session.clone( flowfile );

*// excerpt

Re: From one flowfile to two...

[Russell Bateman]

I needed to get back here...

I took this advice to heart and finished my processor. Thanks to Matt 
and Mark for all their suggestions! They cleared up a few things. There 
was one bug in the code that was mine, small, but significant in its 
effect on the rest. That mistake also explained why I thought the 
uuidwas identical between at least two of the cloned flowfiles. What I 
would wish for, and am probably not strong enough to write, would be a 
synthesis of the session methods read() and write() and how best to use 
them (one-to-one, one-to-many, etc.). Javadoc is too paratactic by 
nature, the NiFi Developer's Guide almost silent on these methods. If it 
were not for the many existing examples using these methods, it would be 
hard to learn to do even simple things. I did look for something closer 
to what I needed to do, but unsuccessfully.


Thanks again. If anything, the NiFi mailing lists are a place both for 
great information and being treated well.


Russ

On 8/25/20 12:24 PM, Mark Payne wrote:

Russ,

Several comments here. I’ve included them inline, below.

Hope it’s helpful.

Thanks
-Mark



On Aug 25, 2020, at 2:09 PM, Russell Bateman  wrote:

Thanks for your suggestions, Matt.

I decided to keep the original flowfile only upon failure. So, I have the 
embedded-document file and the serialized POJOs created from processing the non 
embedded-document part as the result if successful. (Condensed code at end...)

Now I have three questions...

1. I seem not to have placated NiFi with the assurance that I have transferred 
or disposed of all three flowfiles suitably. I get:

java.lang.AssertionError: 
org.apache.nifi.processor.exception.FlowFileHandlingException: Cannot commit 
session because the following FlowFiles have not been removed or transferred: 
[2]

This is probably because at the end of the block, you catch Exception and then 
route the original FlowFile to failure. But you’ve already cloned it and didn’t 
deal with the clone.

*Which of the three flowfiles does [2] refer to? Or does it just mean I botched 
two flowfiles. *

2. session.clone()generates a new flowfile with the identical uuid. I don't 
think I want the result to be two flowfiles with the same uuid. I am binding 
them together so I can associate them later using attribute embedded-document. 
*Should I/How do I force cloning to acquire new **uuid**s?*

This appears to actually be a bug in the mock framework. It *should* have a 
unique uuid, and would in a running NiFi instance. Feel free to file a Jira for 
that.

3. A question on theory... *Wouldn't all of this cloning be expensive* and I 
should just clone for one of the new files and then mangle the original 
flowfile to become the other?

session.clone() is not particularly expensive. It’s just creating a new 
FlowFile object. It doesn’t clone the FlowFile’s contents.

That said, it is probably more appropriate to call session.create(flowFile), 
rather than session.clone(flowFile). It makes little difference in practice but 
what you’re really doing is forking a child, and that will come across more 
cleanly in the Provenance lineage that is generated if using 
session.create(flowFile).

Additional comments in code below.



Thanks,
Russ


@Override
public void onTrigger( final ProcessContext context, final ProcessSession 
session ) throws ProcessException
{
   FlowFile flowfile = session.get();

   if( flowfile == null )
   {
 context.yield();

No need to yield here. Let the framework handle the scheduling. 
ProcessContext.yield() is meant for cases where you’re communicating with some 
external service, for instance, and you know the service is unavailable or rate 
limiting you or something like that. You can’t make any progress, so tell NiFi 
to not bother wasting CPU cycles with this Processor.

 return;
   }

   try
   {
 final String UUID = flowfile.getAttribute( NiFiUtilities.UUID );

 FlowFile document = session.clone( flowfile );

*// excerpt and write the embedded document to a new flowfile...*
 session.write( document, new OutputStreamCallback()
 {
   @Override public void process( OutputStream outputStream )
   {
 // read from the original flowfile copying to the output flowfile...
 session.read( flowfile, new InputStreamCallback()
 {
   @Override public void process( InputStream inputStream ) throws 
IOException
   {
...
   }
 } );
   }
 } );

 FlowFile concepts = session.clone( flowfile );

 AtomicReference< ConceptList > conceptListHolder = new AtomicReference<>();

*// parse the concepts into a POJO list...*
 session.read( concepts, new InputStreamCallback()
 {
   final ConceptList conceptList = conceptListHolder.get();

   @Override public void process( InputStream inputStream ) throws 
IOException
   {
 ...
   }
 } );

*// write out the concept POJOs serialized...*
 s

Re: From one flowfile to two...

[Russell Bateman]

Thanks for your suggestions, Matt.

I decided to keep the original flowfile only upon failure. So, I have 
the embedded-document file and the serialized POJOs created from 
processing the non embedded-document part as the result if successful. 
(Condensed code at end...)


Now I have three questions...

1. I seem not to have placated NiFi with the assurance that I have 
transferred or disposed of all three flowfiles suitably. I get:


java.lang.AssertionError: 
org.apache.nifi.processor.exception.FlowFileHandlingException: Cannot 
commit session because the following FlowFiles have not been removed or 
transferred: [2]


*Which of the three flowfiles does [2] refer to? Or does it just mean I 
botched two flowfiles. *


2. session.clone()generates a new flowfile with the identical uuid. I 
don't think I want the result to be two flowfiles with the same uuid. I 
am binding them together so I can associate them later using attribute 
embedded-document. *Should I/How do I force cloning to acquire new 
**uuid**s?*


3. A question on theory... *Wouldn't all of this cloning be expensive* 
and I should just clone for one of the new files and then mangle the 
original flowfile to become the other?


Thanks,
Russ


@Override
public void onTrigger( final ProcessContext context, final 
ProcessSession session ) throws ProcessException

{
  FlowFile flowfile = session.get();

  if( flowfile == null )
  {
    context.yield();
    return;
  }

  try
  {
    final String UUID = flowfile.getAttribute( NiFiUtilities.UUID );

    FlowFile document = session.clone( flowfile );

*    // excerpt and write the embedded document to a new flowfile...*
    session.write( document, new OutputStreamCallback()
    {
  @Override public void process( OutputStream outputStream )
  {
    // read from the original flowfile copying to the output 
flowfile...

    session.read( flowfile, new InputStreamCallback()
    {
  @Override public void process( InputStream inputStream ) 
throws IOException

  {
   ...
  }
    } );
  }
    } );

    FlowFile concepts = session.clone( flowfile );

    AtomicReference< ConceptList > conceptListHolder = new 
AtomicReference<>();


*    // parse the concepts into a POJO list...*
    session.read( concepts, new InputStreamCallback()
    {
  final ConceptList conceptList = conceptListHolder.get();

  @Override public void process( InputStream inputStream ) throws 
IOException

  {
    ...
  }
    } );

*    // write out the concept POJOs serialized...*
    session.write( concepts, new OutputStreamCallback()
    {
  @Override public void process( OutputStream outputStream )
  {
    ...
  }
    } );

    document = session.putAttribute( document, "embedded-document", UUID );
    concepts = session.putAttribute( document, "embedded-document", UUID );
    session.transfer( document, DOCUMENT );
    session.transfer( concepts, CONCEPTS );
    session.remove( flowfile );
  }
  catch( Exception e )
  {
    session.transfer( flowfile, FAILURE );
  }
}

On 8/24/20 4:52 PM, Matt Burgess wrote:

Russell,

session.read() won't overwrite any contents of the incoming flow file,
but write() will. For #2, are you doing any processing on the file? If
not, wouldn't that be the original flowfile anyway? Or do you want it
to be a different flowfile on purpose (so you can send the incoming
flowfile to a different relationship)? You can use session.clone() to
create a new flowfile that has the same content and attributes from
the incoming flowfile, then handle that separately from the incoming
(original) flowfile. For #1, you could clone() the original flowfile
and do the read/process/write as part of a session.write(FlowFile,
StreamCallback) call, then you're technically reading the "new" file
content (which is the same of course) and overwriting it on the way
out.

Regards,
Matt

On Mon, Aug 24, 2020 at 6:37 PM Russell Bateman  wrote:

I am writing a custom processor that, upon processing a flowfile,
results  in two new flowfiles (neither keeping the exact, original
content) out two different relationships. I might like to route the
original flowfile to a separate relationship.

FlowFile original = session.get();

Do I need to call session.create()for the two new files?

  1. session.read()of original file's contents, not all of the way
 through, but send the processed output from what I do read as
 flowfile 1.
  2. session.read()of original file's contents and send resulting output
 as flowfile 2.
  3. session.transfer()of original flowfile.

I look at all of these session.read()and session.write()calls and I'm a
bit confused as to which to use that won't lose the original flowfile's
content after #1 so I can start over again in #2.

Thanks.

From one flowfile to two...

[Russell Bateman]

I am writing a custom processor that, upon processing a flowfile, 
results  in two new flowfiles (neither keeping the exact, original 
content) out two different relationships. I might like to route the 
original flowfile to a separate relationship.


FlowFile original = session.get();

Do I need to call session.create()for the two new files?

1. session.read()of original file's contents, not all of the way
   through, but send the processed output from what I do read as
   flowfile 1.
2. session.read()of original file's contents and send resulting output
   as flowfile 2.
3. session.transfer()of original flowfile.

I look at all of these session.read()and session.write()calls and I'm a 
bit confused as to which to use that won't lose the original flowfile's 
content after #1 so I can start over again in #2.


Thanks.

Re: Suggestions for splitting, then reassembling documents

[Russell Bateman]

Hey, thanks, Jason. I will give this approach a try. I confess I had not 
even thought of /Wait/ for that. Thanks too for pointing out side effects.


Russ

On 8/21/20 5:46 AM, Sherman, Jason wrote:

This sounds like a good use case for wait/notify, which I've used
successfully multiple times.  Once the document is split, the original
document part would sit at the wait processor until a notify processor
signals the completion of the flow.  I would first try using the original
files UUID for the wait/notify signal.

Also, you can set attributes on the notify processor that can get added to
the document at the wait processor. Then, use that added information to
build whatever output you need with the original document.

However, with such large documents, this will likely slow down the
processing for the XML portion, depending on how much processing they have
to go through.

Cheers,
Jason
--
Jason C. Sherman, CSSLP, CISSP
Owner
Logical Software Solutions, LLC
Solid. Secure. Software.

http://logicalsoftware.co/
.co?  Yes, your data isn't always what you expect.  We'll make sense of it.

https://www.linkedin.com/in/lss-js/


On Tue, Aug 18, 2020 at 12:38 PM Russell Bateman 
wrote:


I am writing custom processors that juggle medical documents (in a more
or less proprietary format). The document are always XML and contain
two, major parts:

  1. an original document which may be text, HL7v2 or XML and may contain
 HTML between  ... , could be many megabytes in
size
  2. XML structure representing data extracted from (a) in myriad XML
 elements, rarely more than a few hundred kilobytes in size

I am using XStreamto serialize #2 after I've parsed it into POJOs for
later use. It's too over the top to base-64 encode #1 to survive
serialization by XStreamand it buys me nothing except the convenience of
making #1 conceptually identical to #2. Since I don't need to dig down
into #1 or analyze it, and it's so big, processing it at all is costly
and undesirable.

What I thought I'd investigate is the possibility of splitting
 ... (#1) into a separate flowfile to be
reassembled by a later processor, but with (literally) millions of these
files flowing through NiFi, I wonder about the advisability of splitting
them up then hoping I can unite the correct parts and how to accomplish
that (discrete attribute ids on constituent parts?).

The reassembly would involve deserializing #2, working with that data to
generate a new document (HL7v4/FHIR, other formats) along with
reinserting #1.

Yes, I have examined /SplitXml/ and /SplitContent/, but I need to do
much more than just split the flowfile at the time I have it in my
hands, hence, a custom processor. Similarly, /MergeContent/ will not be
helpful for reassembly.

So, specifically, I can easily generate a flowfile attribute, an id that
discretely identifies these two now separate documents as suitable to
weld back together. However, I have not yet experimented with flowfiles
randomly (?) showing up together later in the flow within easy reach of
one processor for reassembly. Obviously, /Split/- and /MergeContent/
must be in the habit of dealing with this situation, but I have no
experience with them outside my primitive imagination.

I'm asking for suggestions, best practice, gotchas, warnings or any
other thoughts.

Russ

Suggestions for splitting, then reassembling documents

[Russell Bateman]

I am writing custom processors that juggle medical documents (in a more 
or less proprietary format). The document are always XML and contain 
two, major parts:


1. an original document which may be text, HL7v2 or XML and may contain
   HTML between  ... , could be many megabytes in size
2. XML structure representing data extracted from (a) in myriad XML
   elements, rarely more than a few hundred kilobytes in size

I am using XStreamto serialize #2 after I've parsed it into POJOs for 
later use. It's too over the top to base-64 encode #1 to survive 
serialization by XStreamand it buys me nothing except the convenience of 
making #1 conceptually identical to #2. Since I don't need to dig down 
into #1 or analyze it, and it's so big, processing it at all is costly 
and undesirable.


What I thought I'd investigate is the possibility of splitting 
 ... (#1) into a separate flowfile to be 
reassembled by a later processor, but with (literally) millions of these 
files flowing through NiFi, I wonder about the advisability of splitting 
them up then hoping I can unite the correct parts and how to accomplish 
that (discrete attribute ids on constituent parts?).


The reassembly would involve deserializing #2, working with that data to 
generate a new document (HL7v4/FHIR, other formats) along with 
reinserting #1.


Yes, I have examined /SplitXml/ and /SplitContent/, but I need to do 
much more than just split the flowfile at the time I have it in my 
hands, hence, a custom processor. Similarly, /MergeContent/ will not be 
helpful for reassembly.


So, specifically, I can easily generate a flowfile attribute, an id that 
discretely identifies these two now separate documents as suitable to 
weld back together. However, I have not yet experimented with flowfiles 
randomly (?) showing up together later in the flow within easy reach of 
one processor for reassembly. Obviously, /Split/- and /MergeContent/ 
must be in the habit of dealing with this situation, but I have no 
experience with them outside my primitive imagination.


I'm asking for suggestions, best practice, gotchas, warnings or any 
other thoughts.


Russ

Re: Change Version not possible due to setting a Parameter Context for a Process Group

[Russell Bateman]

Forgive me for asking, but I'm curious. NiFi Registry 0.5.0 is fully two 
versions behind the current one. Wouldn't using the latest be a 
precursor to sorting out problems? Bugs may have been fixed. Or, is 
there something good known to be expected from the older Registry that 
the newest one doesn't offer in terms of migrating from variables to 
parameters?


On 8/17/20 6:04 AM, josef.zahn...@swisscom.com wrote:


Hi guys

We are using NiFi 1.11.4 with NiFi Registry 0.5.0. We are trying to 
migrate from variables to parameters.


As soon as we are adding a “Process Group Parameter Context” to an 
existing process group and change something else to be able to commit 
it to the NiFi Registry, it seems that the commit gets corrupt. We 
can’t pull the new version on the other NiFI cluster, error message is 
below:


/«Failed to update flow to new version due to 
org.apache.nifi.web.util.LifecycleManagementException: Failed to 
update Flow on all nodes in cluster due to An unexpected error has 
occurred. Please check the logs for additional details.//»/


A screenshot of a social media post Description automatically generated

Where can we find the log messages mentioned in the error messages? 
nifi-app.log shows not more than what the GUI shows. How can we 
troubleshoot this or is there any known bug?


We already added some Parameter Context to other Process Groups in the 
past (may be in another NiFi Version?), so in general it was working.


Thanks in advance, Josef

Live development of custom processors and JAVA_HOME--confirm best practice

[Russell Bateman]

When installing NiFi in production, Ansible can be used to set up 
JAVA_HOME. There is zero problem for users of NiFi.


However, from a development host, given the now rapid cadence of Java 
releases, we sometimes run into problems launching a private 
installation of NiFi in the course of testing or debugging our custom 
processors because tools used minute-to-minute and all day long like 
IntelliJ IDEA (and others) march on requiring later and later Java versions.


I found an old JIRA issue that suggests a solution for NiFi 0.1.0, that 
of working around the problem by setting java=  in /conf/bootstrap.conf/ 
to point to a valid Java 1.8 JRE/JDK. This sounds good to me, but the 
version is very old.


Is this still best practice?

Re: Need to know if there is multiple template support

[Russell Bateman]

Since the advent of the NiFi Registry, templates are sort of deprecated. 
The Registry is a brilliant design far better able to support what you 
seem to be asking for.


By the way, this is much more a "user" question you might have asked in 
that forum rather than this "dev" forum.


Cheers!

On 8/6/20 2:44 AM, Rahul Vasan wrote:

Im looking to create multiple templates for multiple client with different
processor flows for each of them. Is it possible?

Re: PUT SQL error attribute

[Russell Bateman]

I guess my implicit point was that, since you have the ability to reach 
into the logging, via /${NIFI_ROOT}///conf/logback.xml/, you could cause 
what's logged to be written in place of or in addition to /nifi-app.log/ 
somewhere else if you desire.



On 8/3/20 2:27 PM, KhajaAsmath Mohammed wrote:
Yes errors are being written to log but it would be useful if part of 
error was sent as attribute in failure queue. That will help us to 
decide on what processor should be called instead of checking logs.


Thanks,
Asmath

On Mon, Aug 3, 2020 at 3:24 PM Russell Bateman <mailto:r...@windofkeltia.com>> wrote:


Pardon me, I meant: /${NIFI_ROOT}///logs///nifi-app.log ./

On 8/3/20 2:23 PM, Russell Bateman wrote:

Mohammed,

I didn't write /PutSQL/ and I haven't dug around to look at its
source code, but I wouldn't be surprised to learn that those
errors were already written by it to
/${NIFI_ROOT}/nifi-app.log/ too.

Russ

On 8/3/20 8:56 AM, KhajaAsmath Mohammed wrote:

Hi,

I generally get errors on putsql on the top right for this
processor. Is there a way to get errors inside the attribute so
that I can use it and log in our audit table.

Any other alternative? Put Database gives this attribute but it
is not possible to get with PUTSQL.

Thanks,
Asmath

Re: PUT SQL error attribute

[Russell Bateman]

Pardon me, I meant: /${NIFI_ROOT}///logs///nifi-app.log ./

On 8/3/20 2:23 PM, Russell Bateman wrote:

Mohammed,

I didn't write /PutSQL/ and I haven't dug around to look at its source 
code, but I wouldn't be surprised to learn that those errors were 
already written by it to /${NIFI_ROOT}/nifi-app.log/ too.


Russ

On 8/3/20 8:56 AM, KhajaAsmath Mohammed wrote:

Hi,

I generally get errors on putsql on the top right for this processor. 
Is there a way to get errors inside the attribute so that I can use 
it and log in our audit table.


Any other alternative? Put Database gives this attribute but it is 
not possible to get with PUTSQL.


Thanks,
Asmath

Re: PUT SQL error attribute

[Russell Bateman]

Mohammed,

I didn't write /PutSQL/ and I haven't dug around to look at its source 
code, but I wouldn't be surprised to learn that those errors were 
already written by it to /${NIFI_ROOT}/nifi-app.log/ too.


Russ

On 8/3/20 8:56 AM, KhajaAsmath Mohammed wrote:

Hi,

I generally get errors on putsql on the top right for this processor. 
Is there a way to get errors inside the attribute so that I can use it 
and log in our audit table.


Any other alternative? Put Database gives this attribute but it is not 
possible to get with PUTSQL.


Thanks,
Asmath

Re: Failing to update custom processor properties names, displayNames, etc.

[Russell Bateman]

Andy,

You're right. I was a caching issue with Chrome. I don't know why I 
didn't think to try that--forest for the trees, I guess. Thank you.


Russ

On 7/17/20 5:58 PM, Andy LoPresto wrote:

Russell,

Have you verified this is not a browser caching issue? They are pernicious and 
it sounds like this could be an example. If you’re sure it’s not, verify the 
API calls using your browser’s Developer Tools to see what properties are 
actually being returned by the server when inspecting a component to see if the 
correct values are present. If so, it’s likely a UI bug, and if not, the new 
code is not being properly loaded/used by NiFi’s server application.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


On Jul 17, 2020, at 3:48 PM, Russell Bateman  wrote:

If I have changed a custom processor's PropertyDescriptor.nameand/or 
.displayName,including changes I have made to my /additionalDetails.html, /and 
I have:

- removed that processor from my test flow or removed /flow.xml.gz/ altogether
- removed my NAR from /${NIFI_ROOT}///custom-lib/ and bounced NiFi with the new 
custom NAR copied

yet nothing changes (in usage or in additional details), what am I overlooking?

Let's set aside the vaguely confusing semantic distinction between nameand 
displayName, I just want NiFi to forget my new custom processor completely and 
then accept my new version as if brand new including all the changes I have 
made.

Thanks for any suggestions.

Failing to update custom processor properties names, displayNames, etc.

[Russell Bateman]

If I have changed a custom processor's PropertyDescriptor.nameand/or 
.displayName,including changes I have made to my 
/additionalDetails.html, /and I have:


- removed that processor from my test flow or removed flow.xml.gz altogether
- removed my NAR from /${NIFI_ROOT}///custom-lib/ and bounced NiFi with 
the new custom NAR copied


yet nothing changes (in usage or in additional details), what am I 
overlooking?


Let's set aside the vaguely confusing semantic distinction between 
nameand displayName, I just want NiFi to forget my new custom processor 
completely and then accept my new version as if brand new including all 
the changes I have made.


Thanks for any suggestions.

Re: Derby as DBCP service, error from Kerberos?

[Russell Bateman]

Oopsie! Okay, too early for my eyes to pick up on that difference. 
Indeed, it works now. Thanks a million, Bryan!


On 7/14/20 8:25 AM, Bryan Bende wrote:

The one I referenced is actually "nifi-kerberos-credentials-service-api"
and you have "nifi-kerberos-credentials-service".

On Tue, Jul 14, 2020 at 10:24 AM Russell Bateman 
wrote:


Thanks for the responses. I did have this dependency already before
mailing to the forum:

  
  1.11.0
  ...
  
org.apache.nifi
nifi-kerberos-credentials-service
${nifi.version}
  

Other thoughts? I tried debugging through this as both test scope and no
specified scope. The result is the same.


On 7/14/20 8:12 AM, Matt Burgess wrote:

Don't forget to include that service with "test" scope so it doesn't
get included in the "real" bundle.

On Tue, Jul 14, 2020 at 9:49 AM Bryan Bende  wrote:

It looks like you are missing a dependency in your project...



https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/pom.xml#L50-L55

On Mon, Jul 13, 2020 at 5:24 PM Russell Bateman 
wrote:


I'm trying to use Apache Derby as the DBCP controller in JUnit tests.
For the first test, I start off vetting my ability to inject Derby as
the DBCP controller I want to use. But, right off, I get this Kerberos
error. I wasn't trying to use Kerberos, but maybe I'm missing
configuration to tell that to DBCPConnectionPool?

public void test() throws Exception
{
 final DBCPConnectionPool service = new DBCPConnectionPool();

   *java.lang.NoClassDefFoundError:
org/apache/nifi/kerberos/KerberosCredentialsService at


org.apache.nifi.dbcp.DBCPConnectionPool.(DBCPConnectionPool.java:243)

at

com.windofkeltia.processor.TestWithDerby.test(TestWithDerby.java:111)

< 26 internal calls> Caused by: java.lang.ClassNotFoundException:
org.apache.nifi.kerberos.KerberosCredentialsService < 2 internal calls
at


java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)

at


java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)

at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) ...*

 runner.addControllerService( "Derby service", service );
 runner.setProperty( service, DBCPConnectionPool.DATABASE_URL,
"jdbc:derby:memory:sampledb;create=true" );
 runner.setProperty( service, DBCPConnectionPool.DB_USER,

  "sa" );

 runner.setProperty( service, DBCPConnectionPool.DB_PASSWORD,

  "sa" );

 runner.setProperty( service, DBCPConnectionPool.DB_DRIVERNAME,
"org.apache.derby.jdbc.EmbeddedDriver" );
 runner.enableControllerService( service );
 runner.assertValid( service );

 final DBCPService derbyService = ( DBCPService )
runner.getProcessContext()

.getControllerServiceLookup()

.getControllerService( "Derby service" );

 // get and verify connections to Derby...
 for( int count = 0; count < 10; count++ )
 {
   final Connection connection = service.getConnection();
   if( VERBOSE )
 System.out.println( connection );
   assertNotNull( connection );
   assertValidConnectionDerby( connection, count );
 }

 final Map< String, String > properties = new HashMap<>();
 runner.setProperty( TestWithDerby.DBCP_SERVICE, "Derby service" );
 runner.setIncomingConnection( false );
 runner.setIncomingConnection( false );
 runner.run();
}

Re: Derby as DBCP service, error from Kerberos?

[Russell Bateman]

Thanks for the responses. I did have this dependency already before 
mailing to the forum:


    
    1.11.0
    ...
    
  org.apache.nifi
  *nifi-kerberos-credentials-service*
  ${nifi.version}
    

Other thoughts? I tried debugging through this as both test scope and no 
specified scope. The result is the same.



On 7/14/20 8:12 AM, Matt Burgess wrote:

Don't forget to include that service with "test" scope so it doesn't
get included in the "real" bundle.

On Tue, Jul 14, 2020 at 9:49 AM Bryan Bende  wrote:

It looks like you are missing a dependency in your project...

https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/pom.xml#L50-L55

On Mon, Jul 13, 2020 at 5:24 PM Russell Bateman 
wrote:


I'm trying to use Apache Derby as the DBCP controller in JUnit tests.
For the first test, I start off vetting my ability to inject Derby as
the DBCP controller I want to use. But, right off, I get this Kerberos
error. I wasn't trying to use Kerberos, but maybe I'm missing
configuration to tell that to DBCPConnectionPool?

public void test() throws Exception
{
final DBCPConnectionPool service = new DBCPConnectionPool();

  *java.lang.NoClassDefFoundError:
org/apache/nifi/kerberos/KerberosCredentialsService at
org.apache.nifi.dbcp.DBCPConnectionPool.(DBCPConnectionPool.java:243)

at com.windofkeltia.processor.TestWithDerby.test(TestWithDerby.java:111)
< 26 internal calls> Caused by: java.lang.ClassNotFoundException:
org.apache.nifi.kerberos.KerberosCredentialsService < 2 internal calls >
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)

at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)

at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) ...*

runner.addControllerService( "Derby service", service );
runner.setProperty( service, DBCPConnectionPool.DATABASE_URL,
"jdbc:derby:memory:sampledb;create=true" );
runner.setProperty( service, DBCPConnectionPool.DB_USER,   "sa" );
runner.setProperty( service, DBCPConnectionPool.DB_PASSWORD,   "sa" );
runner.setProperty( service, DBCPConnectionPool.DB_DRIVERNAME,
"org.apache.derby.jdbc.EmbeddedDriver" );
runner.enableControllerService( service );
runner.assertValid( service );

final DBCPService derbyService = ( DBCPService )
runner.getProcessContext()

.getControllerServiceLookup()

.getControllerService( "Derby service" );

// get and verify connections to Derby...
for( int count = 0; count < 10; count++ )
{
  final Connection connection = service.getConnection();
  if( VERBOSE )
System.out.println( connection );
  assertNotNull( connection );
  assertValidConnectionDerby( connection, count );
}

final Map< String, String > properties = new HashMap<>();
runner.setProperty( TestWithDerby.DBCP_SERVICE, "Derby service" );
runner.setIncomingConnection( false );
runner.setIncomingConnection( false );
runner.run();
}

Derby as DBCP service, error from Kerberos?

[Russell Bateman]

I'm trying to use Apache Derby as the DBCP controller in JUnit tests. 
For the first test, I start off vetting my ability to inject Derby as 
the DBCP controller I want to use. But, right off, I get this Kerberos 
error. I wasn't trying to use Kerberos, but maybe I'm missing 
configuration to tell that to DBCPConnectionPool?


public void test() throws Exception
{
  final DBCPConnectionPool service = new DBCPConnectionPool();

*java.lang.NoClassDefFoundError: 
org/apache/nifi/kerberos/KerberosCredentialsService at 
org.apache.nifi.dbcp.DBCPConnectionPool.(DBCPConnectionPool.java:243) 
at com.windofkeltia.processor.TestWithDerby.test(TestWithDerby.java:111) 
< 26 internal calls> Caused by: java.lang.ClassNotFoundException: 
org.apache.nifi.kerberos.KerberosCredentialsService < 2 internal calls > 
at 
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583) 
at 
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) 
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) ...*


  runner.addControllerService( "Derby service", service );
  runner.setProperty( service, DBCPConnectionPool.DATABASE_URL,  
"jdbc:derby:memory:sampledb;create=true" );
  runner.setProperty( service, DBCPConnectionPool.DB_USER,   "sa" );
  runner.setProperty( service, DBCPConnectionPool.DB_PASSWORD,   "sa" );
  runner.setProperty( service, DBCPConnectionPool.DB_DRIVERNAME, 
"org.apache.derby.jdbc.EmbeddedDriver" );
  runner.enableControllerService( service );
  runner.assertValid( service );

  final DBCPService derbyService = ( DBCPService ) runner.getProcessContext()
 
.getControllerServiceLookup()
 .getControllerService( 
"Derby service" );

  // get and verify connections to Derby...
  for( int count = 0; count < 10; count++ )
  {
final Connection connection = service.getConnection();
if( VERBOSE )
  System.out.println( connection );
assertNotNull( connection );
assertValidConnectionDerby( connection, count );
  }

  final Map< String, String > properties = new HashMap<>();
  runner.setProperty( TestWithDerby.DBCP_SERVICE, "Derby service" );
  runner.setIncomingConnection( false );
  runner.setIncomingConnection( false );
  runner.run();
}

Re: So, Derby works...

[Russell Bateman]

The mock driver supports a sort of bogus URL ("jdbc:mock") that it 
recognizes for its purpose which is only to respond to a single query 
with canned data (whereas I'll be able to populate Derby with as much 
and whatever data I ever want). Of course, its driver doesn't ever do 
anything "real," though it supports a few metadata methods (and the 
MockConnectionimplements Connectionand supports a whole pile of methods 
that stub out most actions and unimaginatively return the canned data).


Yes, your suggestion #3 works well and solves that particular problem of 
an empty driver list. (The list is empty instead of containing the mock 
driver because I separated the packages. DriverManagerdoes some 
reflection desperately looking for classes that might have drivers.) The 
list now has the Derby driver in it:


Down inside DriverManager's private getConnection(), walking the driver 
list using the IntelliJ IDEA debugger, I see:


   *aDriver* = {DriverInfo@2097}
   "driver[className=org.apache.derby.jdbc.*AutoloadedDriver*@6bb4dd34]"
   *con* = {EmbeddedConnection@2585}
   "org.apache.derby.impl.jdbc.*EmbedConnection*@1414013111 (XID =
   164), (SESSIONID = 1), (DATABASE = memory:sys), (DRDAID = null) "

I have to admit that I'm not to the point yet where I grok why I'm 
seeing AutoloadedDriverwhen I forced Java to verify EmbeddedDriver. I 
got no errors doing that; I did it just before making the call to 
getConnection():


   Class.forName( "org.apache.derby.jdbc.EmbeddedDriver" );
   DriverManager.getConnection(
   "jdbc:derby:memory:sampledb;created=true", "sa", "sa" );

I have learned a great deal more about JDBC drivers than I had planned 
thanks to deciding to use Derby, replacing this old test mock and with 
your help which I hugely appreciate!


Russ


On 7/8/20 5:15 PM, Rick Hillegas wrote:
It's hard to say what's going on, but the instability in your 
experiments is hard to reconcile against the deterministic code paths 
involved in JDBC autoloading.


I can only speculate about what is causing this instability:

1) What JDBC URLs does your MockDriver claim to support?

2) Is something tricky being done to the classpath, creating a 
situation in which the Derby jars are not visible when you invoke 
DriverManager.getConnection()?


3) What happens when you do a 
Class.forName("org.apache.derby.jdbc.EmbeddedDriver")? What are the 
chain of errors raised? For the record, this is the recommended JDBC 
way to fault in a driver which was missed at autoloading time. 
Autoloading occurs on the very first call to 
DriverManager.getConnection() during the lifetime of the JVM.


Hope this helps,
-Rick

On 7/8/20 10:37 AM, Russell Bateman wrote:
...again, don't know what I changed.* I'm only depending on 
/derby-10.15.2.0.jar/.


What I think is going on is an artifact of being side-by-side with 
another JDBC driver implementation.


I was hoping to keep the original mocked test driver working 
alongside the Derby implementation at least until I'm able to abolish 
it in every case. Just as used to be the case, it's registered thus:


    DriverManager.registerDriver( new MockDriver() );

However, this is not how it works for Derby. It just happens on the 
basis of having the JAR linked (in /pom.xml/). The problem comes 
that, because the MockDriveris explicitly registered, when we go to 
create a Derby connection:


    java.sql.Connection connection = DriverManager.getConnection( 
"jdbc:derby:memory:sampledb;create=true" );


...DriverManagertakes as license to create an instance of the old 
test mock because that is the (only) driver in its 
registeredDriverslist. *I moved the Derby-based code out of the same 
Java package to solve this?


Thanks again. I'm sorry for having made this annoying.

Russ

So, Derby works...

[Russell Bateman]

...again, don't know what I changed.* I'm only depending on 
/derby-10.15.2.0.jar/.


What I think is going on is an artifact of being side-by-side with 
another JDBC driver implementation.


I was hoping to keep the original mocked test driver working alongside 
the Derby implementation at least until I'm able to abolish it in every 
case. Just as used to be the case, it's registered thus:


    DriverManager.registerDriver( new MockDriver() );

However, this is not how it works for Derby. It just happens on the 
basis of having the JAR linked (in /pom.xml/). The problem comes that, 
because the MockDriveris explicitly registered, when we go to create a 
Derby connection:


    java.sql.Connection connection = DriverManager.getConnection( 
"jdbc:derby:memory:sampledb;create=true" );


...DriverManagertakes as license to create an instance of the old test 
mock because that is the (only) driver in its registeredDriverslist. *I 
moved the Derby-based code out of the same Java package to solve this?


Thanks again. I'm sorry for having made this annoying.

Russ