Re: [ANNOUNCE] Apache NiFi 1.27.0 Released

[Russell Bateman]

On the Downloads page , is it 
intentional to offer--under Binaries--source titles?


This is for both v2 and v1. Looking at view-source, it's because title=mistakenly (?) uses "source" in the name. When I click to 
download, I *do* get the binary and not the source, however. So that's 
working right. It's only that it's slightly disturbing to see "source" 
when my mouse hovers over the download button.


Specifically, hover your mouse over the NiFi Standard 1.27.0button and 
you'll see what I mean.


--just in case we didn't mean to do this.

Russ

On 7/8/24 17:14, David Handermann wrote:

Apache NiFi is an easy to use, powerful, and reliable system to
process and distribute data.

https://nifi.apache.org

The release artifacts can be downloaded from the project website.

https://nifi.apache.org/download/

Maven artifacts have been released and mirrored according to ASF
artifact distribution processes.

Issues resolved in Apache NiFi 1.27.0 are listed in Jira Release Notes.

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12354832

Highlights of the release are available on the project wiki page:

https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.27.0

Regards,
The Apache NiFi Team

Re: Who or what sets up NIFI_HOME as a system environment variable?

[Russell Bateman]

I have an update on this question which remained quite rightly 
unanswered since it was my own peculiar problem.


It resulted from the not-so-clever way our NiFi systemdservice was put 
together 6+ years ago which was avoided using /nifi.sh/ (and 
/nifi-env.sh/).NIFI_HOMEsimply did not exist once NiFi was launched.


In the much more recent Docker container instance as well as from the 
development host command line directly (/nifi.sh/), NIFI_HOMEis present.



On 6/7/24 15:46, Russell Bateman wrote:
It's hard for me to determine this for customers who may already have 
installed Apache NiFi and just drop my NAR in or count on my Docker 
container. I'd like a clearer idea of who or what is responsible for that.

Who or what sets up NIFI_HOME as a system environment variable?

[Russell Bateman]

It's hard for me to determine this for customers who may already have 
installed Apache NiFi and just drop my NAR in or count on my Docker 
container. I'd like a clearer idea of who or what is responsible for that.

Re: SELinux and NiFi

[Russell Bateman]

That's what lies in those "SELinux policies."

I think it's simple: Use SELinux to lock the filesystem (and other 
stuff) up so no one can get in or go around. Then create specific 
policies that allow, in this case, NiFi, access to its filesystem (like 
//opt/nifi/current-nifi//) so that it can do work. Obviously, when you 
install NiFi, things can get complicated like where do you want each 
repository to live--you'll have to provide NiFi access to each place, no 
longer a single filesystem.


This is handled by DevOps guys and not me (I just write custom 
processors), but if you get real pointed, I can ask them better 
questions they can answer.


Russ


On 3/8/24 15:04, Mike Thomsen wrote:
I think the admin told me that even a simple nifi.sh start won’t work. 
Just won’t even start the script and it is marked executable. I was 
wondering if there were any gotchas to getting a basic setup running.



Sent from my iPhone

On Mar 8, 2024, at 4:29 PM, Russell Bateman  
wrote:


 We have run on CentOS with SELinux set to enforcing and have run 
NiFi in that environment for probably 8 or 9 years now. We do install 
some SELinux policies that allow NiFi to access the filesystem 
underneath itself and not outside that filesystem.


What specifically are you asking?

On 3/8/24 14:04, Mike Thomsen wrote:
Does anyone have experience setting up NiFi w/ SELinux set to 
"enforcing?"

Re: SELinux and NiFi

[Russell Bateman]

We have run on CentOS with SELinux set to enforcing and have run NiFi in 
that environment for probably 8 or 9 years now. We do install some 
SELinux policies that allow NiFi to access the filesystem underneath 
itself and not outside that filesystem.


What specifically are you asking?

On 3/8/24 14:04, Mike Thomsen wrote:

Does anyone have experience setting up NiFi w/ SELinux set to "enforcing?"

Re: NiFi failing to start

[Russell Bateman]

In case you or someone else wishes only to run, develop, start, stop, 
start over, etc., and doesn't care to authenticate a (non-production) 
installation, I have followed this since NiFi 1.14 and last used it for 
1.19:


https://www.javahotchocolate.com/notes/nifi.html#20210716

If this doesn't work it's usually because the properties file has become 
too modified. Start over with a fresh download.


Russ


On 12/28/22 12:03, Chris Sampson wrote:
I think you will need to remove/comment out the references to 
single-user-provider in authorisers.xml and login-providers.xml as 
well as removing it from nifi.properties (see the comments in these 
files as they're provided in the nifi distributions).


If you are using 2-way TLS authentication then I don't think you need 
to configure anything else, but remember that all of your nifi 
instances in your cluster (if applicable) will need to trust one 
another's certificates along with all user certificates - the easiest 
way of doing this is typically to trust a common CA that issues all 
the nifi instance and user certs. This could be nifi-toolkit, but 
beware that the CA used by toolkit is auto-generated on startup, so 
you need to retain and configure the same CA for toolkit of you plan 
to use it to issue new certs in future.


On Wed, 28 Dec 2022, 17:32 James McMahon,  wrote:

I continue to experience errors when I try to start my nifi 1.16.3
instance. I have followed this guide in an effort to use the
toolkit to generate self-0signed certs for user admin, signed by a
nifi truststore:

Apache NiFi Walkthroughs


I seem to be having issues with this in my nifi.properties:
nifi.security.user.authorizer=single-user-authorizer

When I set it to nothing, it tells me this is required. When I set
it to single-user-authorizer, this error results in the log:
 Error creating bean with name 'authorizer': FactoryBean threw
exception on object creation; nested exception is
java.lang.Exception: The specified authorizer
'single-user-authorizer' could not be found.

I suspect my authorizers.xml and/or my
login-identity-providers.xml files are misconfigured. How should
those two config files be structured if I wish to run a secure
nifi instance where mith my self-signed certs, generated using the
nifi toolkit?

Re: On configuring SSLContextService...

[Russell Bateman]

If you'll permit, I want to cap this thread I started off a bit by a) 
thanking the many who contributed to it and b) summing up the solution I 
am using based on that help.


Here are the command lines and germane instructions. For localhost 
below, substitute the DNS name (or, at least, //etc/hosts/ name) of the 
VM/hardware running Tomcat. Substitute your own password for "changeit" 
and modify any other details according to need.


*1. Generate Tomcat a keystore with certificate and key inside plus a 
/subject alternative name/ (SAN)--crucial for the client's use.*
keytool -genkeypair -keyalg RSA -keysize 2048 -validity 365 -dname 
"CN=tomcat" -ext san=dns:localhost -alias tomcat -keystore tomcat.jks 
-storepass changeit -keypass changeit


**2. Inspect**Tomcat's new keystore. You're looking to see the SAN.*
*keytool -list -v -keystore tomcat.jks -storepass changeit
*
3. Configure this keystore in Tomcat's **/conf/server.xml/**via a 
definition.*



  
    
  


*4. Get Tomcat's certificate "live." (Tomcat must be running with the 
new certificate.) In addition to getting the certificate, this should 
preserve the crucial SAN from step #1.*

openssl s_client -connect localhost:8443 -showcerts > client.cer

*5. Import that certificate into a keystore (that will be used in the 
client's trust store).*
keytool -importcert -file client.cer -alias tomcat -keystore 
client-truststore.jks -keypass changeit -storepass changeit -noprompt


*6. Verify the client's trust store. Again, you're looking to see the SAN.*
keytool -list -v -keystore client-truststore.jks -storepass changeit


The two artifacts to take away are /tomcat.jks/, for Tomcat's use, and 
/client-truststore.jks/, for the client's use.


In the case of this thread, the "client" in question was Apache NiFi's 
/InvokeHTTP/ and the configuration was done partly in that processor and 
partly in the accompanying /SSLContextService/ (I used 
/StandardRestrictedSSLContextService/). Those configurations looked like 
this:


/InvokeHTTP/:
HTTP URL: https://localhost:8443//servicename/

/SSLContextService/:
Truststore Filename: /client-truststore.jks/  (this must be a 
full path in your filesystem)

Truststore Password: changeit
Truststore Type: JKS
TLS Protocol: TLS

Re: On configuring SSLContextService...

[Russell Bateman]

That is an important goal to me as well (giving back as I have been 
given to). Last week, I had obligations outside of work that took up 
some time.


I have made progress in that (either of the) /SSLContextService/ options 
now validates my artifacts, however, I'm not out of the woods yet and 
have slowed my attack on this problem to deepen my personal 
understanding and knowledge of TLS, key, certificate, keystore, trust 
store, etc. theory.


At present, with a working /SSLContextService/, I'm investigating the 
following (failure) error back from /InvokeHTTP/ when I expected a response:


   Request Processing failed:
   java.net.ConnectException: Failed to connect to localhost/127.0.0.1:8443
   - Caused by java.net.ConnectException: Connection refused

I believe this is because Tomcat as now configured isn't accepting HTTPS.

I'm engaged in an attempt to understand the whole--generating of the 
self-signed certificate for use by Tomcat, transformation of same into a 
trust store certificate (what you have helped with). I fear that what I 
generated for Tomcat's use, which resides in a keystore referenced from 
/server.xml/, is not correct.


Russ

On 8/9/22 07:28, Paul Grey wrote:
To aid future searches, I wanted to follow up on the mailing list and 
make sure you had worked through your problem.



On Tue, Aug 2, 2022 at 3:41 PM Paul Grey  wrote:

Just tried out these (command line) steps; hope they work for you,
too.

(1) openssl s_client -connect localhost:8443 -showcerts
This will dump a bunch of text to stdout.  You want to grab the
text between these two text lines:
-BEGIN CERTIFICATE-
...
-END CERTIFICATE-
... and save that text to a file "trust.cer".

(2) openssl x509 -in trust.cer -text
This will verify that you got the right text.

(3) keytool -importcert -keystore trust.jks -file trust.cer -alias 1
This will create a JKS keystore that contains your certificate. 
The command will ask for a password (twice to confirm); pick
something easy.

(4) Enter "Truststore Filename" (/full/path/to/trust.jks) ,
"Truststore Password", and "Truststore Type" (JKS) into your
StandardSSLContextService properties.

Cheers!


On Tue, Aug 2, 2022 at 11:33 AM Nathan Gough 
wrote:

That's right Russ, if you purchased a commercially signed
certificate, InvokeHTTP should work by default. With your self
signed certificate, you will need an SSL context service, and
I believe you will need to insert the self signed certificate
into the trust store. I suggest using KeystoreExplorer to
create a trust store, import the self signed certificate, save
that trust store file and then point to that file in the SSL
context service. Let us know if you have issues with that part.

The screenshot error message you sent is showing that
InvokeHTTP does not trust the remote server when it says
'unable to find valid certfiication path to requested target'
(a pretty confusing error in my opinion).

X509 key/certs and key/trust store stuff is pretty tricky to
understand the first time around.

Nathan


On Tue, Aug 2, 2022, 11:00 AM Russell Bateman
 wrote:

Yes, of course. And, therefore, /SSLContextService/ is
required was my conclusion. I see the point more clearly
now, but my conclusion seems inescapable. To forego
/SSLContextService/ we would have to purchase a
commercially signed certificate for use by Tomcat, right?

I will experiment with just somehow injecting the
self-signed certificate we created into the trust store
certificate? --which I thought I had done already, but
/SSLContextService/ has steadfastly refused to accept
anything I throw at it. (I reiterate that this is my first
TLS rodeo; I have never had to do this kind of work
before. I greatly appreciate your patience.)

Russ

On 8/1/22 19:03, Paul Grey wrote:

1.  You've instructed your browser to accept the
(untrusted) certificate that is being presented by Tomcat.
untrusted.cert.png

2. You've supplied the "--insecure" flag to curl.
https://curl.se/docs/manpage.html#-k

3.  The NiFi equivalent to these two instructions is to
provide a truststore, which contains a record specifying
    the certificate being served by your Tomcat server.


On Mon, Aug 1, 2022 at 6:27 PM Russell Bateman
 wrote:


Okay. I'm hoping this is true, but it's not what I'm
seeing. It's not dawning on me yet what I'm doing wrong.

Re: On configuring SSLContextService...

[Russell Bateman]

Just a note (for later readers of this thread)...

My experience now with this trick seems to say that, as long as "https" 
is in the URL, a /SSLContextService/ must be supplied. As a URL with 
"https" and port number 8443 is the only way I have to engage TLS at the 
far end, I must live with /SSLContextService/'s requirements.


On 7/26/22 19:17, Paul Grey wrote:

leave the InvokeHTTP property SSLContextService blank.

On configuring SSLContextService...

[Russell Bateman]

I have hesitated between providing some huge tl;dr exposé and something 
shorter. I'll do shorter here.


0. For now, I'm using "changeit" below as password rolling a self-signed 
certificate for key, key store and trust store.
1. I have a service running in Tomcat that I hit via HTTPs because the 
content always involves personal health information.
2. I use a key store containing my certificate. No trust store is needed 
or involved in Tomcat.

3. I need to hit my Tomcat service using /InvokeHTTP/ in my flow.
4. This means configuring an instance of 
/Standard[Restricted]SSLContextService/.
5. The SSL context service insists on a defined key store with key 
password and key store password.
6. The SSL context service insists on a defined trust store. The best I 
have been able to do is to roll the key store certificate into a trust 
store.
7. When either key- or trust store file is missing, the SSL context 
service complains that a resource is missing (for key store or trust store).

8. Once both files/resources exist, all three passwords appear crucial.
9. Despite password used to create key and certificates, it is always 
wrong according to SSL context service validator which consistently issues:


   /Keystore Properties is invalid because invalid keystore password or
   type specified for file __.//
   //Truststore Properties is invalid because invalid truststore
   password or type specified for file __./

It would be nice to see a step-by-step illustration of creating the key, 
key store and trust store artifacts required by SSL context service and 
perhaps the full configuration of the SSL context service.


Other notes:

1. I seem to get pretty far toward a solution using Java's keytool.
2. I don't get very far using openssl.
3. I get even less traction trying to use NiFi's TLS toolkit to solve this.
4. I guess I could simply write my own SSL context service that doesn't 
require a trust store?


Huge thanks for any help or comments.

Russ

P.S. I have a scratch sheet that reveals how I created artifacts and 
thought through the problem at:


https://www.javahotchocolate.com/notes/keytool-experience.html

Re: Placement and specification of certificates for StandardRestrictedSSLContextService

[Russell Bateman]

David,

Sadly, this is my experience. "changeit" works for me. And I tried 
reconfiguring the three passwords in 
/StandardRestrictedSSLContextService/ to no avail.


   ~/dev/nifi/nifi-1.15.0/conf $ *keytool -list -v -keystore
   mdmi-keystore.jks*
   Enter keystore password: *changeit*
   Keystore type: PKCS12
   Keystore provider: SUN

   Your keystore contains 1 entry

   Alias name: mdmi
   Creation date: Jul 20, 2022
   Entry type: PrivateKeyEntry
   Certificate chain length: 1
   Certificate[1]:
   Owner: CN=windofkeltia.com, OU=Unknown, O=Wind of Keltia, L=Provo,
   ST=UT, C=US
   Issuer: CN=windofkeltia.com, OU=Unknown, O=Wind of Keltia, L=Provo,
   ST=UT, C=US
   Serial number: 1e7288f7
   Valid from: Wed Jul 20 15:39:23 MDT 2022 until: Thu Jul 20 15:39:23
   MDT 2023
   Certificate fingerprints:
     SHA1: B9:58:6E:C1:0D:DA:1D:CF:7D:02:16:54:F2:FA:1F:C4:19:01:F5:1B
     SHA256:
   
FF:0B:3B:4A:59:69:9B:B8:B3:23:1F:4E:72:03:C7:24:11:A9:DF:11:C6:76:89:32:44:F7:12:A4:26:F5:9D:4B
   Signature algorithm name: SHA256withRSA
   Subject Public Key Algorithm: 2048-bit RSA key
   Version: 3

   Extensions:

   #1: ObjectId: 2.5.29.14 Criticality=false
   SubjectKeyIdentifier [
   KeyIdentifier [
   : 69 63 BD 7E 67 A1 EC 0A   54 3C 61 2F 51 D7 64 46 
   ic..g...T
Hi Russell,

Thanks for describing the steps used to generate the keystore and 
truststore files.


The validation warnings on StandardRestrictedSSLContextService appear 
to indicate that the configured password properties do not match the 
keystore and truststore passwords.


It would be helpful to enter the password properties again and confirm 
that there are no trailing spaces.


The following keytool commands can also be used to verify the passwords:

keytool -list -v -keystore mdmi-keystore.jks
keytool -list -v -keystore mdmi-truststore.jks

The configuration appears to be correct, so confirming the password on 
both files is a good next step.

Placement and specification of certificates for StandardRestrictedSSLContextService

[Russell Bateman]

I'm trying to set up TLS for a service using /InvokeHTTP/ against an 
external-to-NiFi Tomcat-based service and I have configured 
/StandardRestrictedSSLContextService/ thus:


https://www.javahotchocolate.com/notes/nifi-images/mdmi-standard-ssl-context-service.png

...which results in the errors shown here:

https://www.javahotchocolate.com/notes/nifi-images/s-sslcontextservice.png

Do the NiFi errors mean that "changeit" can't be used as a password?

At the risk of over-simplifying their placement, I dropped them into 
/${NIFI_ROOT}/conf/.


   ~/dev/nifi/nifi-1.15.0/conf $ *ll mdmi**
   -rw-rw-r-- 1 russ russ  899 Jul 20 15:40 mdmi-keystore.crt
   -rw-rw-r-- 1 russ russ 2725 Jul 20 15:39 *mdmi-keystore.jks*
   -rw-rw-r-- 1 russ russ 1255 Jul 20 15:53 *mdmi-truststore.jks*

/mdmi-keystore.crt/ is self-signed for now and (for now) I have used 
"changeit":


   ~/dev/nifi/nifi-1.15.0/conf $ *keytool -genkey -keyalg RSA -alias
   mdmi -keystore mdmi-keystore.jks -validity 365 -keysize 2048*
   Enter keystore password:  changeit
   Re-enter new password:  changeit
   What is your first and last name?
   ...

   ~/dev/nifi/nifi-1.15.0/conf $ *keytool -export -alias mdmi -file
   mdmi-**keystore.crt -keystore mdmi-keystore.jks -storepass changeit*
   Certificate stored in file 
   ~/dev/nifi/nifi-1.15.0/conf $ *keytool -import -noprompt
   -trustcacerts**-alias mdmi -file mdmi-keystore.crt -keystore
   mdmi-truststore.jks**-storepass changeit*
   Certificate was added to keystore

This all works fine via curl or Postman outside of NiFi for hitting the 
service (I put the keytool artifacts into /${CATALINA_BASE}/conf/and 
note this in /${CATALINA_BASE}/conf/server.xml/).


When it comes to TLS in NiFi, this is my first rodeo. I'm open to 
suggestions on any other this. Thanks.

Re: how to setup nifi.content.repository.archive.max.retention.period in a Nifi docker image ?

[Russell Bateman]

Thanks, Breno!

On 2/22/22 09:54, Breno Cesar wrote:

Nicolas,

Nice, thank you for sharing your final solution.

Russel,

I'm not a docker/container expert, BUT in my humble opinion, in my day 
to day i run an 'exec -it /bin/bash' and change manually if the 
container has an sh environment, and if it doesn't, I do it with 
sed/awk or another text processor that is usually inside the containers.



*Breno *



Em ter., 22 de fev. de 2022 às 11:38, Nicolas Belot 
 escreveu:


Hi Breno.

Thanks for your feedback.

I think I will override the nifi image by adding my own entrypoint
( which will call the nifi start.sh at the end)

In my entrypoint, I will call another script that will mimic the
nifi script.sh. In this new script , I will add something like :

prop_replace
‘nifi.content.repository.archive.max.retention.period'
"${NIFI_CONTENT_REPOSITORY_ARCHIVE_MAX_RETENTION_PERIOD:-’12 hours”}"

Regards

*From:* Breno Cesar 
*Sent:* Tuesday, February 22, 2022 3:07 PM
*To:* users@nifi.apache.org
*Subject:* Re: how to setup
nifi.content.repository.archive.max.retention.period in a Nifi
docker image ?

*CAUTION:*External Email : Be wary of clicking links or if this
claims to be internal.

Hi Nicolas,
As far as i know, there is no variable for this config.The
documentation has a lack about this topic and does not explain
about it.
Doing some "googling", i found that someone already maped this
variables.


https://github.com/dprophet/nifi/blob/master/nifi-docker/dockerhub/CONFIGURATION.md



Assuming this configuration you need is a tunnig, and will not be
done frequenly, as far as I know, you can change it in the default
nifi files on a clean docker image, and export it for later use.

*Breno *

Em ter., 22 de fev. de 2022 às 10:01, Nicolas Belot
 escreveu:

Hello everyone.

I use a docker image of nifi and I need to tune
nifi.content.repository.archive.max.retention.period.

Is there a way to simply  set it up through an env variable ?

I read the start.sh script but
nifi.content.repository.archive.max.retention.period  does not
appear in this script.

Regards

N.

Re: how to setup nifi.content.repository.archive.max.retention.period in a Nifi docker image ?

[Russell Bateman]

Breno,

While we're on this topic, what's best practice for changing something 
like "the default nifi files on a clean docker image, ..."? Use sed or 
awk from a RUN command? (This is really a Docker question, but you 
raised it. Anything you suggest would be helpful.)


Russ

On 2/22/22 07:07, Breno Cesar wrote:

Hi Nicolas,
As far as i know, there is no variable for this config.The 
documentation has a lack about this topic and does not explain about it.

Doing some "googling", i found that someone already maped this variables.

https://github.com/dprophet/nifi/blob/master/nifi-docker/dockerhub/CONFIGURATION.md

Assuming this configuration you need is a tunnig, and will not be done 
frequenly, as far as I know, you can change it in the default nifi 
files on a clean docker image, and export it for later use.


*Breno *

Em ter., 22 de fev. de 2022 às 10:01, Nicolas Belot 
 escreveu:


Hello everyone.

I use a docker image of nifi and I need to tune
nifi.content.repository.archive.max.retention.period.

Is there a way to simply  set it up through an env variable ?

I read the start.sh script but
nifi.content.repository.archive.max.retention.period  does not
appear in this script.

Regards

N.

Re:

[Russell Bateman]

Ismaël,

Please send an e-mail to users-unsubscr...@nifi.apache.org.

Thanks.

On 12/20/21 8:45 AM, ismaelmartin.ri...@post.ch wrote:


unsubscribe

Re: How to restrict custom processor execution time

[Russell Bateman]

Sanjeet,

It occurred to me that you may not be getting replies because you report 
"building a custom processor" and that's the subject of the Apache NiFi 
Developers' forum. Try posting there.


Best regards,

Russ

On 8/10/21 7:33 AM, sanjeet rath wrote:

Hi ,

I am building a custom processor and there is restriction i want to 
put for the  processor that it should not schedule to run 2 times in 1 
hour time period.


I can acheive this by passing "run schedule" 60 mins.

Is there any other way i can do in my custom processor code, So that 
it won't allow the user to select "run schedule " time less than 60 
mins.basically similar to we can restrict the procesor to execute on 
prinary node.


Any other thought is really helpfull.

Thanks,
Sanjeet

Re: odd performance behavior 1.14

[Russell Bateman]

(I'm sorry, I meant to say that I read that the Java Flight Recorder is 
now available for unlicensed--in the original Oracle sense--for use.)


On 8/2/21 12:44 PM, Russell Bateman wrote:

Scott,

I believe I read somewhere in the last year. I found this:

https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u

And, I used it to look into a problem I had with Apache NiFi a few 
years ago. For what it's worth, my experience is recorded here:


https://www.javahotchocolate.com/notes/jfr.html

Hope some of this helps.

Russ

On 8/2/21 12:20 PM, scott wrote:
I'm using openjdk-11.0.7.10-4 as I was on the previous version of 
NiFi. I'll look around for a free Java profiler to use to dig deeper.

Thanks,
Scott

On Sat, Jul 31, 2021 at 7:56 PM Joe Witt <mailto:joe.w...@gmail.com>> wrote:


Scott

Nope this sounds pretty dang unique

What JVM?   May need to attach a profiler.

I have seen buried exceptions happening at massive rates causing
horrid performance among a few other scenarios but nothing
specific to 1.14

thanks

On Sat, Jul 31, 2021 at 4:01 PM scott mailto:tcots8...@gmail.com>> wrote:

Hi All,
I upgraded to 1.14 last week and within a few days I started
to see some pretty odd behavior, I'm hoping someone has
either seen it before or could point me to a deeper level of
troubleshooting.

Here are the symptoms I observed.
* Performance issues:
    Processor performance very poor. Even simple processors
like router and updateattribute went from being able to
process 100,000recs/min to 100recs/min or stop all together,
but not consistently.
    Processors needing to be force killed, even simple ones
like updateattribute.

* Weirdness. One of my routers lost its mind and
didn't recognize the routes configured anymore. It changed
all the arrows to dotted lines except for the default. I
ended up copying it and replacing it with the copy, no
changes mind you, but it worked fine.

* Errors: I have not found any obvious errors in the nifi
logs that could explain this, but one error keeps repeating
in the logs: SQLServerDriver is not found . I have dozens of
processors that use SQL Server, all seem to be working fine.
This is not tied to a particular processor's configuration. I
don't think this is related.

* Server resources fine. I use htop and sar to troubleshoot
hardware issues usually, all looks normal. I added 1/3 more
memory to the JVM, now at 24GB, just for good measure, but
that had no effect.

Is it possible there are some hidden performance issues going
on within the JVM I need a special tool to see?

Any help would be greatly appreciated.

Thanks,
Scott

Re: odd performance behavior 1.14

[Russell Bateman]

Scott,

I believe I read somewhere in the last year. I found this:

https://developers.redhat.com/blog/2020/08/25/get-started-with-jdk-flight-recorder-in-openjdk-8u

And, I used it to look into a problem I had with Apache NiFi a few years 
ago. For what it's worth, my experience is recorded here:


    https://www.javahotchocolate.com/notes/jfr.html

Hope some of this helps.

Russ

On 8/2/21 12:20 PM, scott wrote:
I'm using openjdk-11.0.7.10-4 as I was on the previous version of 
NiFi. I'll look around for a free Java profiler to use to dig deeper.

Thanks,
Scott

On Sat, Jul 31, 2021 at 7:56 PM Joe Witt > wrote:


Scott

Nope this sounds pretty dang unique

What JVM?   May need to attach a profiler.

I have seen buried exceptions happening at massive rates causing
horrid performance among a few other scenarios but nothing
specific to 1.14

thanks

On Sat, Jul 31, 2021 at 4:01 PM scott mailto:tcots8...@gmail.com>> wrote:

Hi All,
I upgraded to 1.14 last week and within a few days I started
to see some pretty odd behavior, I'm hoping someone has either
seen it before or could point me to a deeper level of
troubleshooting.

Here are the symptoms I observed.
* Performance issues:
    Processor performance very poor. Even simple processors
like router and updateattribute went from being able to
process 100,000recs/min to 100recs/min or stop all together,
but not consistently.
    Processors needing to be force killed, even simple ones
like updateattribute.

* Weirdness. One of my routers lost its mind and
didn't recognize the routes configured anymore. It changed all
the arrows to dotted lines except for the default. I ended up
copying it and replacing it with the copy, no changes
mind you, but it worked fine.

* Errors: I have not found any obvious errors in the nifi logs
that could explain this, but one error keeps repeating in the
logs: SQLServerDriver is not found . I have dozens of
processors that use SQL Server, all seem to be working fine.
This is not tied to a particular processor's configuration. I
don't think this is related.

* Server resources fine. I use htop and sar to troubleshoot
hardware issues usually, all looks normal. I added 1/3 more
memory to the JVM, now at 24GB, just for good measure, but
that had no effect.

Is it possible there are some hidden performance issues going
on within the JVM I need a special tool to see?

Any help would be greatly appreciated.

Thanks,
Scott

Re: Problem with the GetFile processor deleting my entire installation

[Russell Bateman]

Oh, sorry, to finish the answer, yes, you do need to be *very* careful 
how you specify the Input Directory and File Filter properties; the last 
one is a regular expression. It's true that the documentation is less 
than flag-waving or hair-lighting-on-fire as it presents its help on 
filling those in.


Russ

On 6/4/21 11:16 AM, Russell Bateman wrote:
Sorry for this behavior of /GetFile/ which is purposeful. If you 
configure to keep the files instead of removing them, you'll keep 
getting the same files ingested over and over again as flow files. 
It's just how it is.


The secret was to read the help blurb when configuring this processor.

Hope this helps,

Russ

On 6/4/21 10:44 AM, Ruth, Thomas wrote:


Hello,

I recently built a 3-node NiFi cluster in my organization as a 
proof-of-concept for some work we are doing. I used version 1.13.2 
and installed it onto 3 CentOS 7.9 systems. In my organization, I 
don’t have root access to the system, so I used a different user 
called “nfadm” to install and run the product. I don’t remember 
seeing anything in the documentation that stated that this would be 
an issue.


I am also new to NiFi, and was relying heavily on the Admin 
documentation on the web site for instructions to set up the OS and 
NiFi installations. I configured certificate-based security and 
distributed them to my users. I also configured policies for groups 
that I thought were OK for them from a development standpoint.


I had an incident occur yesterday in which a user, who is also new to 
NiFi, ran a component called “GetFile” for the filesystem “/” with 
the default settings (Recurse=true, KeepSourceFile=false). Well, this 
essentially ran “rm -rf /” as the user that owns all the installation 
files and files in the various repositories, nfadm, the same user 
running the NiFi processes. This deleted all the installation and 
configuration files for the entire cluster, making it completely 
useless now.


I am surprised to find out that NiFi allowed a user to basically wipe 
out all the files the user running the NiFi server had access to. I 
would expect much higher security to be present in a default system. 
I have some questions that hopefully you can help me with:


Is this a known issue in NiFi?

Am I doing something wrong when configuring or installing NiFi?

Is there a section in the documentation that warns me of this type of 
scenario?


Thanks in advance for your help with this,

Tom Ruth

Sr. Data Engineer

Optum, Inc.

E: thomas.r...@optum.com


This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the 
intended

recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.

Re: Problem with the GetFile processor deleting my entire installation

[Russell Bateman]

Sorry for this behavior of /GetFile/ which is purposeful. If you 
configure to keep the files instead of removing them, you'll keep 
getting the same files ingested over and over again as flow files. It's 
just how it is.


The secret was to read the help blurb when configuring this processor.

Hope this helps,

Russ

On 6/4/21 10:44 AM, Ruth, Thomas wrote:


Hello,

I recently built a 3-node NiFi cluster in my organization as a 
proof-of-concept for some work we are doing. I used version 1.13.2 and 
installed it onto 3 CentOS 7.9 systems. In my organization, I don’t 
have root access to the system, so I used a different user called 
“nfadm” to install and run the product. I don’t remember seeing 
anything in the documentation that stated that this would be an issue.


I am also new to NiFi, and was relying heavily on the Admin 
documentation on the web site for instructions to set up the OS and 
NiFi installations. I configured certificate-based security and 
distributed them to my users. I also configured policies for groups 
that I thought were OK for them from a development standpoint.


I had an incident occur yesterday in which a user, who is also new to 
NiFi, ran a component called “GetFile” for the filesystem “/” with the 
default settings (Recurse=true, KeepSourceFile=false). Well, this 
essentially ran “rm -rf /” as the user that owns all the installation 
files and files in the various repositories, nfadm, the same user 
running the NiFi processes. This deleted all the installation and 
configuration files for the entire cluster, making it completely 
useless now.


I am surprised to find out that NiFi allowed a user to basically wipe 
out all the files the user running the NiFi server had access to. I 
would expect much higher security to be present in a default system. I 
have some questions that hopefully you can help me with:


Is this a known issue in NiFi?

Am I doing something wrong when configuring or installing NiFi?

Is there a section in the documentation that warns me of this type of 
scenario?


Thanks in advance for your help with this,

Tom Ruth

Sr. Data Engineer

Optum, Inc.

E: thomas.r...@optum.com


This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.

Re: NiFi 1.11.4 Custom Processor Development

[Russell Bateman]

I'm sorry. I mistyped (numbers soup). I meant to say NiFi 1.2.0, a 
version several years ago already.


On 4/13/21 9:04 AM, Russell Bateman wrote:
There shouldn't be any problem. I have many custom processors in a NAR 
I haven't rebuilt since 1.12 and we use them successfully. They're 
just missing versions because we had not yet upgraded the Maven we 
used to build the NAR (not exactly relevant, I know, but I thought I'd 
I would point this out).



On 4/13/21 8:55 AM, nathan.engl...@bt.com wrote:

Hi Joe,

Thanks for that information.

Will there be any issue in running a Custom NAR file built against 
1.9.2 on a different versioned cluster. We haven’t experienced 
anything yet, but would hate to spend ages in debugging an issue that 
is the result of this!


Kind Regards,

Nathan

*From:* Joe Witt 
*Sent:* Tuesday, April 13, 2021 5:48 pm
*To:* users@nifi.apache.org
*Subject:* Re: NiFi 1.11.4 Custom Processor Development
Hello

In moving to support maven 3.8 we found only https based repos are 
allowed by default.  In addressing this we updated the build to move 
away from bintray and another location as well.


This is now the case when you build master. You could try to cherry 
pick and edit the key commits or hang tight for us to kick out a 1.14 
release.  No time table for that yet though.


Thanks

On Tue, Apr 13, 2021 at 7:45 AM <mailto:nathan.engl...@bt.com>> wrote:


Hi There,

I am in the process of upgrading the version of the
nifi-nar-bundle from 1.9.2 to 1.12.1 however I’ve hit a
unexpected issue.

We develop on an offline platform, with Maven Central Mirrored
using a Nexus Repository Manager, with all dependencies delivered
through this mechanism. Today when I have changed the version
from 1.9.2 to 1.12.1 I have found that the parent (I assume for
the groovy processor) has a dependency on groovy-eclipse-batch
version 2.5.4-01 which seems to be only available from a Groovy
Repository (https://dl.bintray.com/groovy/maven/

<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdl.bintray.com%2Fgroovy%2Fmaven%2F&data=04%7C01%7Cnathan.english%40bt.com%7Cedb498f9213242aee05108d8fe8b3c57%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637539221253446565%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9T1Ii386gRcclHLZw6Dw9%2FPEal1hrqaNGDjWAgJ6Hz8%3D&reserved=0>)
and not maven central.

I also noticed that bintray will be closing shortly
 
(https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/

<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fjfrog.com%2Fblog%2Finto-the-sunset-bintray-jcenter-gocenter-and-chartcenter%2F&data=04%7C01%7Cnathan.english%40bt.com%7Cedb498f9213242aee05108d8fe8b3c57%7Ca7f356889c004d5eba4129f146377ab0%7C0%7C0%7C637539221253456560%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8nphGAZUMxc0QX19bvZMrhm6JdLBGpmdxktTSjsiccs%3D&reserved=0>),
so I guess the dependency will need correcting before then?

Kind Regards,

Nathan

Nathan English
Applications Specialist - Cyber Delivery & DevOps

Re: NiFi 1.11.4 Custom Processor Development

[Russell Bateman]

There shouldn't be any problem. I have many custom processors in a NAR I 
haven't rebuilt since 1.12 and we use them successfully. They're just 
missing versions because we had not yet upgraded the Maven we used to 
build the NAR (not exactly relevant, I know, but I thought I'd I would 
point this out).



On 4/13/21 8:55 AM, nathan.engl...@bt.com wrote:

Hi Joe,

Thanks for that information.

Will there be any issue in running a Custom NAR file built against 
1.9.2 on a different versioned cluster. We haven’t experienced 
anything yet, but would hate to spend ages in debugging an issue that 
is the result of this!


Kind Regards,

Nathan

*From:* Joe Witt 
*Sent:* Tuesday, April 13, 2021 5:48 pm
*To:* users@nifi.apache.org
*Subject:* Re: NiFi 1.11.4 Custom Processor Development
Hello

In moving to support maven 3.8 we found only https based repos are 
allowed by default.  In addressing this we updated the build to move 
away from bintray and another location as well.


This is now the case when you build master. You could try to cherry 
pick and edit the key commits or hang tight for us to kick out a 1.14 
release.  No time table for that yet though.


Thanks

On Tue, Apr 13, 2021 at 7:45 AM > wrote:


Hi There,

I am in the process of upgrading the version of the
nifi-nar-bundle from 1.9.2 to 1.12.1 however I’ve hit a unexpected
issue.

We develop on an offline platform, with Maven Central Mirrored
using a Nexus Repository Manager, with all dependencies delivered
through this mechanism. Today when I have changed the version from
1.9.2 to 1.12.1 I have found that the parent (I assume for the
groovy processor) has a dependency on groovy-eclipse-batch version
2.5.4-01 which seems to be only available from a Groovy Repository
(https://dl.bintray.com/groovy/maven/

)
and not maven central.

I also noticed that bintray will be closing shortly
 
(https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/

),
so I guess the dependency will need correcting before then?

Kind Regards,

Nathan

Nathan English
Applications Specialist - Cyber Delivery & DevOps

Re: Tuning for flow with lots of processors

[Russell Bateman]

Eric, Mark,

This is an example of an exchange in this forum that is of immeasurable 
help to to on-lookers. Thanks for discussing it so thoroughly!


Russ

On 11/23/20 5:55 PM, Eric Secules wrote:

Hello everyone,

I was wondering if there was a metric for the amount of time 
tImer-driven processors spend in a queue ready and waiting to be run. 
I use NiFi in an atypical way and my flow has over 2000 processors 
running on a single node, but there are usually less than 10 
connections that have one or more flowfiles in them at any given time.


I have a theory that the number of processors in use is slowing down 
the system overall. But I would need to see some more metrics to know 
whether that's the case and tell whether anything I am doing is 
helping. Are there some logs that I could look for or internal stats I 
could poke at with a debugger?


Should I be able to see increased throughput by increasing the number 
of timer-driven threads, or is there a different mechanism responsible 
for going through all the runnable processors to see whether they have 
input to process. I also noticed "nifi.bored.yield.duration" would it 
be good to increase the yield duration in this setting?


Thanks,
Eric

Re: Run Nifi in IntelliJ to debug?

[Russell Bateman]

Hmmm... It's rare that I debug NiFi code. And it's also rare that I 
debug my own in that context since the NiFi test runner allows me to 
fend off most surprises via my JUnit tests.


I think back in 2016, I was debugging a start-up problem involving NiFi 
start-up and incompatibility with the Java Flight Recorder. As I recall, 
I downloaded the relevant NiFi code sources matching the version of NiFi 
I was debugging remotely. I remember ultimately making a slight (and 
only temporary) change to NiFi start-up that fixed the problem. At that 
point I must have been building my own copy to have seen it fixed.. It 
had to do with the order in which NiFi was getting command-line 
arguments making it so the JFR wasn't running. I'd have to dig back to 
figure out what I was doing, but it's probably not too relevant to what 
you need to do.


What do you need to see in this?

Russ

On 10/26/20 5:38 PM, Darren Govoni wrote:
Correct. Primarily the nifi-web-api module and AccessResource class. 
For starters.


Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android <https://aka.ms/ghei36>

------------
*From:* Russell Bateman 
*Sent:* Monday, October 26, 2020 7:37:13 PM
*To:* Darren Govoni ; users@nifi.apache.org 


*Subject:* Re: Run Nifi in IntelliJ to debug?
Darren,

This is just Apache NiFi code out of NARs you want to step through or 
is it yours? You haven't stripped debug information or anything, right?


Russ

On 10/26/20 5:30 PM, Darren Govoni wrote:

Kevin/Russel

Thanks for the info. I did set things up this way.

IntelliJ does connect to the nifi jvm and nifi runs and works but 
intellij isnt breaking on code it should.


I did set the module where the code/classes are located (in the 
remote connection dialog) and i see the exception im tracking print 
on the console output but intellij never breaks.


Is there an extra step needed? Generate sources?

For future it would be nice if there was a maven goal for debug.

Much appreciated!
Darren

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android <https://aka.ms/ghei36>
------------
*From:* Russell Bateman  
<mailto:r...@windofkeltia.com>

*Sent:* Monday, October 26, 2020 4:09:50 PM
*To:* users@nifi.apache.org <mailto:users@nifi.apache.org> 
 <mailto:users@nifi.apache.org>; Darren Govoni 
 <mailto:dar...@ontrenet.com>

*Subject:* Re: Run Nifi in IntelliJ to debug?
Darren,

I was out this morning and didn't see your plea until I got in just 
now. Here's a step by step I wrote up for both IntelliJ IDEA and 
Eclipse (I'm more an IntelliJ guy). It also covers using an IP tunnel.


https://www.javahotchocolate.com/notes/nifi.html#20160323

On 10/26/20 9:52 AM, Darren Govoni wrote:

Hi
   Is it possible to run Nifi from inside IntelliJ with debugging 
such that I can hit the app from my browser and trigger breakpoints?


If anyone has done this can you please share any info?

Thanks in advance!
Darren

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android <https://aka.ms/ghei36>

Re: Run Nifi in IntelliJ to debug?

[Russell Bateman]

Darren,

This is just Apache NiFi code out of NARs you want to step through or is 
it yours? You haven't stripped debug information or anything, right?


Russ

On 10/26/20 5:30 PM, Darren Govoni wrote:

Kevin/Russel

Thanks for the info. I did set things up this way.

IntelliJ does connect to the nifi jvm and nifi runs and works but 
intellij isnt breaking on code it should.


I did set the module where the code/classes are located (in the remote 
connection dialog) and i see the exception im tracking print on the 
console output but intellij never breaks.


Is there an extra step needed? Generate sources?

For future it would be nice if there was a maven goal for debug.

Much appreciated!
Darren

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android <https://aka.ms/ghei36>
----
*From:* Russell Bateman 
*Sent:* Monday, October 26, 2020 4:09:50 PM
*To:* users@nifi.apache.org ; Darren Govoni 


*Subject:* Re: Run Nifi in IntelliJ to debug?
Darren,

I was out this morning and didn't see your plea until I got in just 
now. Here's a step by step I wrote up for both IntelliJ IDEA and 
Eclipse (I'm more an IntelliJ guy). It also covers using an IP tunnel.


https://www.javahotchocolate.com/notes/nifi.html#20160323

On 10/26/20 9:52 AM, Darren Govoni wrote:

Hi
   Is it possible to run Nifi from inside IntelliJ with debugging 
such that I can hit the app from my browser and trigger breakpoints?


If anyone has done this can you please share any info?

Thanks in advance!
Darren

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android <https://aka.ms/ghei36>

Re: Run Nifi in IntelliJ to debug?

[Russell Bateman]

Darren,

I was out this morning and didn't see your plea until I got in just now. 
Here's a step by step I wrote up for both IntelliJ IDEA and Eclipse (I'm 
more an IntelliJ guy). It also covers using an IP tunnel.


https://www.javahotchocolate.com/notes/nifi.html#20160323

On 10/26/20 9:52 AM, Darren Govoni wrote:

Hi
   Is it possible to run Nifi from inside IntelliJ with debugging such 
that I can hit the app from my browser and trigger breakpoints?


If anyone has done this can you please share any info?

Thanks in advance!
Darren

Sent from my Verizon, Samsung Galaxy smartphone
Get Outlook for Android 

Re: Hive NAR not loading because of snappy?

[Russell Bateman]

Our current installations are running on NiFi 1.1.2, I think. Recently, 
the company wanted to move up in the world. So, we don't have that 
continuous experience. We started on 0.7.1.


We're running Java 8 on CentOS 8, I think.

Personally, I run Linux Mint (Ubuntu, corrected) for development. I have 
never before had any reason to think that NiFi on CentOS was any 
different than on Fedora, Ubuntu or Mint (among the Linux boxes I have 
run on over the last 4-5 years).


On 10/13/20 6:35 PM, Matt Burgess wrote:
Understood. But outside of a permissions issue or usage of 
CompressContent or something else that would try to load the same 
version of Snappy, I’m not sure what’s going on there. AFAIK other 
folks install this on centos without issue. The related snappy issues 
were supposedly resolved well before 1.11.4. Did it used to work on 
previous versions? Have you tried the latest release to see if it 
works? Has the version of Java changed?


Sent from my iPhone

On Oct 13, 2020, at 8:13 PM, Russell Bateman  
wrote:


 We didn't want to make deletion or modification of a shipped 
component to be a required part of its installation since we don't 
produce the NiFi download. We'd rather install it as it comes.


On 10/13/20 4:45 PM, Matt Burgess wrote:
Ouch! It does happen on the loading of the NAR to ensure the native 
library gets loaded. If you are not using Hive I’d think you could 
safely delete the nifi-hive-nar and it shouldn’t happen. Hard to 
tell why the native library couldn’t be installed though.



On Oct 13, 2020, at 6:26 PM, Russell Bateman 
 wrote:


 No, we don't even use (nor have we ever used) Hive in our flows. 
It's just there and we didn't want to modify the NiFi download. 
Should this not even happen if we're not using it?


On 10/13/20 4:24 PM, Matt Burgess wrote:
Ugh now I remember, that version of Hive uses a version of Snappy 
that doesn’t create a unique path under /tmp, do you have multiple 
PutHiveStreaming processors in the flow? I don’t think that works 
because we can’t load a single native library into multiple 
classloaders.



On Oct 13, 2020, at 6:15 PM, Russell Bateman 
 wrote:


 I see

-rwxr-xr-x. 1 nifi nifi  48432 Oct 13 13:48 
snappy-1.0.5-libsnappyjava.so


in //tmp/. Therefore, not a permissions issue? Launching this way 
works:


$ ( export 
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so" 
&& /opt/nifi/bin/nifi.sh start )


but it's not something we want to do (in case that shared object 
disappears from //tmp/).



On 10/13/20 3:42 PM, Matt Burgess wrote:
IIRC this is likely a permissions issue, Xerial Snappy tries to 
unzip the native library to the location pointed to by 
“java.io.tempdir” which on *nix defaults to /tmp. Does the NiFi 
user have write access to that directory? If not you can change 
the Java temp dir or set it specifically for Snappy (I don’t 
have the property on hand but a quick Google should find it)


Regards,
Matt

Sent from my iPhone

On Oct 13, 2020, at 5:36 PM, Russell Bateman 
 wrote:


 Should I be seeing this in the log of a vanilla NiFi 
installation on CentOS?


ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to 
java.util.ServiceConfigurationError: 
org.apache.nifi.processor.Processor: Provider 
org.apache.nifi.processors.hive.PutHiveStreaming could not be 
initialized

.
.
.
Caused by: org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null


"Snappy" makes me think Ubuntu. This is CentOS. Did I pull down 
the wrong /nifi-1.11.4-bin.zip/?

Re: Hive NAR not loading because of snappy?

[Russell Bateman]

We didn't want to make deletion or modification of a shipped component 
to be a required part of its installation since we don't produce the 
NiFi download. We'd rather install it as it comes.


On 10/13/20 4:45 PM, Matt Burgess wrote:
Ouch! It does happen on the loading of the NAR to ensure the native 
library gets loaded. If you are not using Hive I’d think you could 
safely delete the nifi-hive-nar and it shouldn’t happen. Hard to tell 
why the native library couldn’t be installed though.



On Oct 13, 2020, at 6:26 PM, Russell Bateman  
wrote:


 No, we don't even use (nor have we ever used) Hive in our flows. 
It's just there and we didn't want to modify the NiFi download. 
Should this not even happen if we're not using it?


On 10/13/20 4:24 PM, Matt Burgess wrote:
Ugh now I remember, that version of Hive uses a version of Snappy 
that doesn’t create a unique path under /tmp, do you have multiple 
PutHiveStreaming processors in the flow? I don’t think that works 
because we can’t load a single native library into multiple 
classloaders.



On Oct 13, 2020, at 6:15 PM, Russell Bateman 
 wrote:


 I see

-rwxr-xr-x. 1 nifi    nifi 48432 Oct 13 13:48 
snappy-1.0.5-libsnappyjava.so


in //tmp/. Therefore, not a permissions issue? Launching this way 
works:


$ ( export 
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so" 
&& /opt/nifi/bin/nifi.sh start )


but it's not something we want to do (in case that shared object 
disappears from //tmp/).



On 10/13/20 3:42 PM, Matt Burgess wrote:
IIRC this is likely a permissions issue, Xerial Snappy tries to 
unzip the native library to the location pointed to by 
“java.io.tempdir” which on *nix defaults to /tmp. Does the NiFi 
user have write access to that directory? If not you can change 
the Java temp dir or set it specifically for Snappy (I don’t have 
the property on hand but a quick Google should find it)


Regards,
Matt

Sent from my iPhone

On Oct 13, 2020, at 5:36 PM, Russell Bateman 
 wrote:


 Should I be seeing this in the log of a vanilla NiFi 
installation on CentOS?


ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to 
java.util.ServiceConfigurationError: 
org.apache.nifi.processor.Processor: Provider 
org.apache.nifi.processors.hive.PutHiveStreaming could not be 
initialized

.
.
.
Caused by: org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null


"Snappy" makes me think Ubuntu. This is CentOS. Did I pull down 
the wrong /nifi-1.11.4-bin.zip/?

Re: Hive NAR not loading because of snappy?

[Russell Bateman]

No, we don't even use (nor have we ever used) Hive in our flows. It's 
just there and we didn't want to modify the NiFi download. Should this 
not even happen if we're not using it?


On 10/13/20 4:24 PM, Matt Burgess wrote:
Ugh now I remember, that version of Hive uses a version of Snappy that 
doesn’t create a unique path under /tmp, do you have multiple 
PutHiveStreaming processors in the flow? I don’t think that works 
because we can’t load a single native library into multiple classloaders.



On Oct 13, 2020, at 6:15 PM, Russell Bateman  
wrote:


 I see

-rwxr-xr-x. 1 nifi    nifi  48432 Oct 13 13:48 
snappy-1.0.5-libsnappyjava.so


in //tmp/. Therefore, not a permissions issue? Launching this way works:

$ ( export 
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so" 
&& /opt/nifi/bin/nifi.sh start )


but it's not something we want to do (in case that shared object 
disappears from //tmp/).



On 10/13/20 3:42 PM, Matt Burgess wrote:
IIRC this is likely a permissions issue, Xerial Snappy tries to 
unzip the native library to the location pointed to by 
“java.io.tempdir” which on *nix defaults to /tmp. Does the NiFi user 
have write access to that directory? If not you can change the Java 
temp dir or set it specifically for Snappy (I don’t have the 
property on hand but a quick Google should find it)


Regards,
Matt

Sent from my iPhone

On Oct 13, 2020, at 5:36 PM, Russell Bateman 
 wrote:


 Should I be seeing this in the log of a vanilla NiFi installation 
on CentOS?


ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to 
java.util.ServiceConfigurationError: 
org.apache.nifi.processor.Processor: Provider 
org.apache.nifi.processors.hive.PutHiveStreaming could not be 
initialized

.
.
.
Caused by: org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null


"Snappy" makes me think Ubuntu. This is CentOS. Did I pull down the 
wrong /nifi-1.11.4-bin.zip/?

Re: Hive NAR not loading because of snappy?

[Russell Bateman]

I see

-rwxr-xr-x. 1 nifi    nifi  48432 Oct 13 13:48 
snappy-1.0.5-libsnappyjava.so


in //tmp/. Therefore, not a permissions issue? Launching this way works:

$ ( export 
LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/snappy-1.0.5-libsnappyjava.so" && 
/opt/nifi/bin/nifi.sh start )


but it's not something we want to do (in case that shared object 
disappears from //tmp/).



On 10/13/20 3:42 PM, Matt Burgess wrote:
IIRC this is likely a permissions issue, Xerial Snappy tries to unzip 
the native library to the location pointed to by “java.io.tempdir” 
which on *nix defaults to /tmp. Does the NiFi user have write access 
to that directory? If not you can change the Java temp dir or set it 
specifically for Snappy (I don’t have the property on hand but a quick 
Google should find it)


Regards,
Matt

Sent from my iPhone

On Oct 13, 2020, at 5:36 PM, Russell Bateman  
wrote:


 Should I be seeing this in the log of a vanilla NiFi installation 
on CentOS?


ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to 
java.util.ServiceConfigurationError: 
org.apache.nifi.processor.Processor: Provider 
org.apache.nifi.processors.hive.PutHiveStreaming could not be initialized

.
.
.
Caused by: org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null


"Snappy" makes me think Ubuntu. This is CentOS. Did I pull down the 
wrong /nifi-1.11.4-bin.zip/?

Hive NAR not loading because of snappy?

[Russell Bateman]

Should I be seeing this in the log of a vanilla NiFi installation on CentOS?

ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to 
java.util.ServiceConfigurationError: 
org.apache.nifi.processor.Processor: Provider 
org.apache.nifi.processors.hive.PutHiveStreaming could not be initialized

.
.
.
Caused by: org.xerial.snappy.SnappyError: 
[FAILED_TO_LOAD_NATIVE_LIBRARY] null


"Snappy" makes me think Ubuntu. This is CentOS. Did I pull down the 
wrong /nifi-1.11.4-bin.zip/?

Re: Change Version not possible due to setting a Parameter Context for a Process Group

[Russell Bateman]

Forgive me for asking, but I'm curious. NiFi Registry 0.5.0 is fully two 
versions behind the current one. Wouldn't using the latest be a 
precursor to sorting out problems? Bugs may have been fixed. Or, is 
there something good known to be expected from the older Registry that 
the newest one doesn't offer in terms of migrating from variables to 
parameters?


On 8/17/20 6:04 AM, josef.zahn...@swisscom.com wrote:


Hi guys

We are using NiFi 1.11.4 with NiFi Registry 0.5.0. We are trying to 
migrate from variables to parameters.


As soon as we are adding a “Process Group Parameter Context” to an 
existing process group and change something else to be able to commit 
it to the NiFi Registry, it seems that the commit gets corrupt. We 
can’t pull the new version on the other NiFI cluster, error message is 
below:


/«Failed to update flow to new version due to 
org.apache.nifi.web.util.LifecycleManagementException: Failed to 
update Flow on all nodes in cluster due to An unexpected error has 
occurred. Please check the logs for additional details.//»/


A screenshot of a social media post Description automatically generated

Where can we find the log messages mentioned in the error messages? 
nifi-app.log shows not more than what the GUI shows. How can we 
troubleshoot this or is there any known bug?


We already added some Parameter Context to other Process Groups in the 
past (may be in another NiFi Version?), so in general it was working.


Thanks in advance, Josef

Re: PUT SQL error attribute

[Russell Bateman]

I guess my implicit point was that, since you have the ability to reach 
into the logging, via /${NIFI_ROOT}///conf/logback.xml/, you could cause 
what's logged to be written in place of or in addition to /nifi-app.log/ 
somewhere else if you desire.



On 8/3/20 2:27 PM, KhajaAsmath Mohammed wrote:
Yes errors are being written to log but it would be useful if part of 
error was sent as attribute in failure queue. That will help us to 
decide on what processor should be called instead of checking logs.


Thanks,
Asmath

On Mon, Aug 3, 2020 at 3:24 PM Russell Bateman <mailto:r...@windofkeltia.com>> wrote:


Pardon me, I meant: /${NIFI_ROOT}///logs///nifi-app.log ./

On 8/3/20 2:23 PM, Russell Bateman wrote:

Mohammed,

I didn't write /PutSQL/ and I haven't dug around to look at its
source code, but I wouldn't be surprised to learn that those
errors were already written by it to
/${NIFI_ROOT}/nifi-app.log/ too.

Russ

On 8/3/20 8:56 AM, KhajaAsmath Mohammed wrote:

Hi,

I generally get errors on putsql on the top right for this
processor. Is there a way to get errors inside the attribute so
that I can use it and log in our audit table.

Any other alternative? Put Database gives this attribute but it
is not possible to get with PUTSQL.

Thanks,
Asmath

Re: PUT SQL error attribute

[Russell Bateman]

Pardon me, I meant: /${NIFI_ROOT}///logs///nifi-app.log ./

On 8/3/20 2:23 PM, Russell Bateman wrote:

Mohammed,

I didn't write /PutSQL/ and I haven't dug around to look at its source 
code, but I wouldn't be surprised to learn that those errors were 
already written by it to /${NIFI_ROOT}/nifi-app.log/ too.


Russ

On 8/3/20 8:56 AM, KhajaAsmath Mohammed wrote:

Hi,

I generally get errors on putsql on the top right for this processor. 
Is there a way to get errors inside the attribute so that I can use 
it and log in our audit table.


Any other alternative? Put Database gives this attribute but it is 
not possible to get with PUTSQL.


Thanks,
Asmath

Re: PUT SQL error attribute

[Russell Bateman]

Mohammed,

I didn't write /PutSQL/ and I haven't dug around to look at its source 
code, but I wouldn't be surprised to learn that those errors were 
already written by it to /${NIFI_ROOT}/nifi-app.log/ too.


Russ

On 8/3/20 8:56 AM, KhajaAsmath Mohammed wrote:

Hi,

I generally get errors on putsql on the top right for this processor. 
Is there a way to get errors inside the attribute so that I can use it 
and log in our audit table.


Any other alternative? Put Database gives this attribute but it is not 
possible to get with PUTSQL.


Thanks,
Asmath

Re: NiFi Expression Language in UpdateAttribute

[Russell Bateman]

That solves confusion I had after and despite reading the NiFi 
Expression Language documentation. Having written documentation before, 
I get why it's never ending and danger-fraught to write "bigger" 
examples in such documentation, and we resist doing it, but that's what 
I needed to clear up in my head the combination of how the two 
constructs, endsWithand ifElse, work together.


Many thanks!

On 6/11/20 8:24 AM, Shawn Weeks wrote:


Your syntax is incorrect. Try this instead.

${filename:endsWith('xml'):ifElse('XML','JSON')}

*From: *Russell Bateman 
*Reply-To: *"users@nifi.apache.org" 
*Date: *Thursday, June 11, 2020 at 9:15 AM
*To: *NiFi Users 
*Subject: *NiFi Expression Language in UpdateAttribute

I wnat to create an attribute, format-type, which I hoped would be set 
based on the in-coming flowfile name:


    ${ ${filename:endsWith( 'xml' )} : ifElse( 'XML', 'JSON' ) }

If the in-coming flowfile name is /sample.xml/, the result is 
nevertheless and always


format-type = JSON

I want it to be XML, obviously. What have I screwed up in my NEL syntax?

(Yes, I have tried taking out all the white space.)

Thanks,
Russ

NiFi Expression Language in UpdateAttribute

[Russell Bateman]

I wnat to create an attribute, format-type, which I hoped would be set 
based on the in-coming flowfile name:


    ${ ${filename:endsWith( 'xml' )} : ifElse( 'XML', 'JSON' ) }

If the in-coming flowfile name is /sample.xml/, the result is 
nevertheless and always


format-type = JSON

I want it to be XML, obviously. What have I screwed up in my NEL syntax?

(Yes, I have tried taking out all the white space.)

Thanks,
Russ

Re: NiFi Registry web documentation broken on Linux for Chrome and Firefox...

[Russell Bateman]

Yes.

It turns out to be unique to the NiFi Registry documentation.

For example, if I follow that link in either Chrome or Firefox, I can 
scroll to the bottom if I do it right away. A few seconds later, 
however, whether I rescroll to the top and back down to the bottom or 
just wait for it, suddenly, there is no way to see anything except 
what's in the current browser window's displayed content, no way to 
scroll, the elevator is long, solid grey, the arrows at top and bottom 
of the scroll bar do nothing, etc.


Here I am posting this observation, but I have just discovered that the 
scroll bar (on the right-hand side) belongs not to the content, but to 
the wide navigational thumb at the left. I can scroll after all as long 
as the cursor is in the content region, but if it drifts to the 
navigational thumb or I try to use the scrollbar, I cannot scroll.


So, I suppose this behavior is peculiar to whatever framework (I'm a 
back-end Java guy) is in use for this set of browser pages. I looked for 
other NiFi documents exhibiting this peculiarity, but none exhibit this 
problem and none make use of the navigational thumb at the left that the 
registry documentation presents (and whose contents are:


- NiFi Registry Documentation
-
General
  Getting Started
  User Guide
  Admin Guide

Developer
  REST API)

I thought I'd report this; it's not that important.

Russ


On 12/11/19 3:28 PM, Ken Swanson wrote:

Russell,

are you talking about 
https://nifi.apache.org/docs/nifi-registry-docs/index.html ?


(FWIW, no problem for that link on MacOS Firefox)

-Ken

On Wed, Dec 11, 2019 at 12:26 PM Russell Bateman 
mailto:r...@windofkeltia.com>> wrote:


Not long upon beginning to read this document, it becomes
impossible to scroll it. Sometimes, backing up to the first page
and clicking on a hyperlink (Starting NiFi Registry, I Started
NiFi Registry. Now What?, etc.) lands you where you want to read
and you can scroll, but this is usually short-lived. The scroll
bar elevator is always the same height as the scroll bar, as if
your window height is also the total content available. Even when
you land somewhere and are able to scroll, this ability will
disappear after a little while.

Or, I'm a monkey's uncle. It could be. Maybe my environment?

$ uname -a
Linux nargothrond 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/os-release
NAME="Linux Mint"
VERSION="19 (Tara)"
...
VERSION_CODENAME=tara
UBUNTU_CODENAME=bionic

Google Chrome: 73.0.3683.103 64-bit
Firefox:   70.0.1    64-bit

I don't have a perfect handle on the exact behavior, but it is
very hard to read straight through the document. I thought I'd ask
if anyone else has trouble.

Russ

NiFi Registry web documentation broken on Linux for Chrome and Firefox...

[Russell Bateman]

Not long upon beginning to read this document, it becomes impossible to 
scroll it. Sometimes, backing up to the first page and clicking on a 
hyperlink (Starting NiFi Registry, I Started NiFi Registry. Now What?, 
etc.) lands you where you want to read and you can scroll, but this is 
usually short-lived. The scroll bar elevator is always the same height 
as the scroll bar, as if your window height is also the total content 
available. Even when you land somewhere and are able to scroll, this 
ability will disappear after a little while.


Or, I'm a monkey's uncle. It could be. Maybe my environment?

   $ uname -a
   Linux nargothrond 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1
   05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
   $ cat /etc/os-release
   NAME="Linux Mint"
   VERSION="19 (Tara)"
   ...
   VERSION_CODENAME=tara
   UBUNTU_CODENAME=bionic

   Google Chrome: 73.0.3683.103 64-bit
   Firefox:   70.0.1    64-bit

I don't have a perfect handle on the exact behavior, but it is very hard 
to read straight through the document. I thought I'd ask if anyone else 
has trouble.


Russ

Re: Content repository data filling up disk...

[Russell Bateman]

0.7.4. I just got here a few weeks ago and have plans to move everything 
up to 1.9.2, which I'm running myself. I'm probably a little premature 
asking about what to look for here. I'll try to ask more direct 
questions next. Thanks.


On 6/19/19 9:17 AM, Mark Payne wrote:

Russell,

I would also be curious what version of NiFi you are running. Version 
1.9.1 introduced a bug [1]  that resulted
in the content repository not being properly cleaned up, which could 
cause you to run out of disk space.


Thanks
-Mark

[1] https://issues.apache.org/jira/browse/NIFI-6150

On Jun 19, 2019, at 11:12 AM, Russell Bateman <mailto:r...@windofkeltia.com>> wrote:


Joe,

I am at the beginning of this, I admit. It's useful to understand the 
bit about combining content. Our application is for ETL of medical 
documents, contains myriad and complex interactions between points, 
the documents leave NiFi at points, then return at others before 
finally being entrusted to a database/search engine. I'm trying to 
corral information on the flow, but it's confusing and hard to know 
just where to start my observations, what to look for, etc. This is 
why I ask. Thanks.


On 6/19/19 8:58 AM, Joe Witt wrote:

Russell

If data remains in the content repository beyond the specified 
archive values then it suggests there is content remaining in the 
flow that is not yet eligible to be removed/deleted.  This is not 
always a direct "500 MB of content waiting for delivery results in 
500 MB of content in the content repos" though.  That is due to how 
the content archiving works and that we tend to combine the content 
of many flowfiles into a single physical file on the file system.


It would be necessary to understand a great deal more detail about 
your case, flow, config to help more specifically.


Thanks

On Wed, Jun 19, 2019 at 10:54 AM Russell Bateman 
mailto:r...@windofkeltia.com>> wrote:


Just in general, when this data begins to collect without
clearing itself out, what direction might I be looking for the
cause? Ordinarily, in our application, content doesn't collect
flooding the disk and threaten to bring the server down.

Thanks for any comments.

Re: Content repository data filling up disk...

[Russell Bateman]

Joe,

I am at the beginning of this, I admit. It's useful to understand the 
bit about combining content. Our application is for ETL of medical 
documents, contains myriad and complex interactions between points, the 
documents leave NiFi at points, then return at others before finally 
being entrusted to a database/search engine. I'm trying to corral 
information on the flow, but it's confusing and hard to know just where 
to start my observations, what to look for, etc. This is why I ask. Thanks.


On 6/19/19 8:58 AM, Joe Witt wrote:

Russell

If data remains in the content repository beyond the specified archive 
values then it suggests there is content remaining in the flow that is 
not yet eligible to be removed/deleted.  This is not always a direct 
"500 MB of content waiting for delivery results in 500 MB of content 
in the content repos" though.  That is due to how the content 
archiving works and that we tend to combine the content of many 
flowfiles into a single physical file on the file system.


It would be necessary to understand a great deal more detail about 
your case, flow, config to help more specifically.


Thanks

On Wed, Jun 19, 2019 at 10:54 AM Russell Bateman 
mailto:r...@windofkeltia.com>> wrote:


Just in general, when this data begins to collect without clearing
itself out, what direction might I be looking for the cause?
Ordinarily, in our application, content doesn't collect flooding
the disk and threaten to bring the server down.

Thanks for any comments.

Content repository data filling up disk...

[Russell Bateman]

Just in general, when this data begins to collect without clearing 
itself out, what direction might I be looking for the cause? Ordinarily, 
in our application, content doesn't collect flooding the disk and 
threaten to bring the server down.


Thanks for any comments.

Re: How can I change the content-type of a flow file in a script processor?

[Russell Bateman]

Eric,

Just a reminder not to forget that this modifies the flowfile and you 
must gather/carry that modification in a practical sense. Therefore, 
your code will be


    flowfile = session.putAttribute( flowfile, "content-type", 
"application/json" );


and thence you'll use the updated flowfile the next time you need it 
(such as when you call session.transfer( flowfile, SUCCESS )).


Hope this reminder is useful,

Russ

On 12/07/2017 07:12 PM, Chris Herrera wrote:

Hi Eric,

If its for the content viewer you might want to use mime.type. 
additionally you can use session.putAttribute(flowFile, 
“content-type”, "application/json”);


Regards,
Chris


On Dec 7, 2017, 8:10 PM -0600, Eric Chaves , wrote:

Hi guys,

I'm trying to programatically set the content-type of a flow-file by 
calling flowFile.putAttribute('content-type', 'application/json') but 
it's not working.


How can I change my flow-file content-type?

Regards,

Eric

Re: Transformations using Nifi

[Russell Bateman]

Aruna,

I don't think there is any generalized NiFi training course yet. I 
started writing a book a year ago, but it's a pretty thankless task and 
takes a lot of time. I mostly write custom processors so I tend to see 
the world differently from most "user" users of NiFi. When I answer 
questions, which I don't do too much, I often tell an impractical story 
that doesn't answer the question asked.


Now, I haven't done much database work at all, so I'm not going to be 
too much help to you. However, looking at your questions, particularly 
the one about obtaining the current date when the flow is run (#2) and 
also #3, I would suggest that you study the NiFi Expression Language. 
This would allow you to insert the "now" date into your flow at some point.


The other suggestions I have are to experiment, which I'm sure you're 
doing, and Google hard for help. For this forum, which is filled with 
very nice and helpful people, you'll tend to get a lot more and better 
help if you come in with a very specific question rather than a list of 
things or a general, "help me" sort of plea.


Cheers,

Russ

P.S. NiFi absolutely rocks, which you'll see as soon as you get over 
your initial hump here. But, you're on the right track.



On 10/12/2017 08:16 AM, Aruna Sankaralingam wrote:


Hi,

Could you someone please help me with these requirements in the email 
below?


Thanks

Aruna

*From:*Aruna Sankaralingam [mailto:aruna.sankaralin...@cormac-corp.com]
*Sent:* Wednesday, October 11, 2017 11:26 AM
*To:* users@nifi.apache.org
*Subject:* Transformations using Nifi

I am trying to see what kind of transformations that can be done in 
nifi and how.


Now I have a basic flow that takes CSV from the local dir and puts 
into s3 and loads into postgres database.


There are 4 columns in my test file 3 of which are string and one is 
an integer field. I would like to do the following before I load the 
data into postgres. If someone can help me on how to go about these, 
it will be great.


1.Convert one of the string columns to upper case

/For converting to upper case, I was told to use the Update Record 
processor but  my version is 1.2.0 and the update record processor is 
not available. /


2.Postgres has an extra column called “Load_Date” in which I would 
like to load the current date with timestamp when the flow is run


3.If the integer column has more than 5 digits, I would like to take 
only the first 5 digits and load to the table


4.There is a look up table in postgres. I would like to check if the 
first column value is present in the look up table and if yes, proceed 
ahead and if not ignore the record


I am trying to learn nifi so I would really appreciate any kind of 
help here. Is there any training available online that I can take in 
order to understand and do all these?

Re: Nifi Flows

[Russell Bateman]

Aruna,

Welcome to NiFi!

There are multiple options. Two of my favorite are:

1) If you want to "archive" the first flow and no longer continue to use 
it, save it as a template.


2) If you want both flows running alongside each other, but don't want 
the noise on your UI canvas, push the first flow down under a process 
group, then create your new flow under its own process group.


Hope this helps.

Russ


On 10/11/2017 08:32 AM, Aruna Sankaralingam wrote:


Hi

I have created one Nifi flow that takes a file and loads a database.

Now if I want to create another different flow not related to what I 
already did, is it possible to save the 1^st flow and create a new 
flow in a new window or something?


Thanks

Aruna

Re: nifi 1.4 - variables

[Russell Bateman]

Can't say which is more useful--registry or variables, but I'm delighted 
to have both now. The registry was already a huge step toward solving 
problems of this sort. These variables are also very, very welcome! I 
love the scope semantic that comes along with them.


--what Uwe said about NiFi. NiFi just rocks!


On 10/04/2017 09:45 AM, Uwe Geercken wrote:

cool. I will check.
I really like this - to get rid of the hardcoding of paths e.g.
Nifi gets better and better.
Rgds,
Uwe
*Gesendet:* Mittwoch, 04. Oktober 2017 um 17:39 Uhr
*Von:* "Bryan Bende" 
*An:* users@nifi.apache.org
*Betreff:* Re: Re: nifi 1.4 - variables
Thanks!

There is a JIRA to add additional documentation to the user guide [1].
For the previously existing capabilities with the properties file,
there is some mention in the admin guide [2].

The variables work in a hierarchical manner, meaning if a variable
"foo" is declared at the root group and also declared inside a process
group, components inside the process group will use the value of "foo"
from within the process group.

If you test this out, the UI should indicate which one is being
used... In the above example, if you looked at the variables from
within the process group, you should see "foo" listed twice, but the
one from the root group would have a line through it indicating that
it is being overridden.

-Bryan

[1] https://issues.apache.org/jira/browse/NIFI-4462
[2] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#custom_properties



On Wed, Oct 4, 2017 at 11:26 AM, Uwe Geercken  wrote:
> thank you Bryan.
>
> is there documentation around for the variables?
>
> If I add variables to the processor group and I want to reference a 
variable

> of the parent flow, will I have to add the scope somehow like
> ${scope.variable}?
>
> If I have variables in different hierarchies of the flow with the 
same name,

> which one will be used?
>
>
> P.S. I really like your blog :-)
>
> Rgds,
>
> Uwe
>
>
> Gesendet: Mittwoch, 04. Oktober 2017 um 16:40 Uhr
> Von: "Bryan Bende" 
> An: users@nifi.apache.org
> Betreff: Re: nifi 1.4 - variables
> Hello,
>
> You can use them in any fields that support expression language, just
> like variables in a properties file
> (nifi.variable.registry.properties= in nifi.properties) or system
> properties in bootstrap.conf.
>
> In the properties file approach you had to restart NiFi for any
> changes in the properties to be picked up.
>
> In the new approach you no longer need to restart anything yourself,
> it will be handled for you.
>
> -Bryan
>
> On Wed, Oct 4, 2017 at 8:20 AM, Uwe Geercken  
wrote:

>> Hello,
>>
>> in 1.4 when you right-click the processor group then there is an entry
>> "variables". I have defined some, but I wonder how I can use them.
>>
>> I have not found documentation on this. Does somebody have details or a
>> link to documentation or an example?
>>
>> Rgds,
>>
>> Uwe

Re: Missing nifi-app.log files

[Russell Bateman]

James,

It's the case that, NiFi running, deleting a log file will result in 
that file no longer existing and no longer written to again until NiFi 
is restarted. This is my observation anyway.


Hope this observation is useful.

Russ


On 08/17/2017 02:11 PM, James McMahon wrote:
Thank you Joe. I agree and will monitor it closely going forward. I 
suspect there were some external factors at play here.


On Thu, Aug 17, 2017 at 4:05 PM, Joe Witt > wrote:


Ok if 50,000 is the max then i'm doubtful that it ran out.

In the event of exhaustion of allowed open file handle count NiFi will
run but its behavior will be hard to reason over.  That means it
cannot create any new files or open existing files but can merely
operate using the handles it already has. It is a situation to avoid.

As far as what actually happened resulting in logfile issues it is not
easy to tell at this stage but should be monitored for system state
when it happens again.

Thanks


On Thu, Aug 17, 2017 at 1:02 PM, James McMahon
mailto:jsmcmah...@gmail.com>> wrote:
> 50,000.
> Is NiFi robust enough that it can continue to run without the
log file for
> write attempts?
> It is back up and running like a champ now, so I will keep an
eye on it.
>
> On Thu, Aug 17, 2017 at 3:40 PM, Joe Witt mailto:joe.w...@gmail.com>> wrote:
>>
>> It sounds like a case of exhausted file handles.
>>
>> Ulimit -a
>>
>> How many open files are allowed for the user nifi runs as?
>>
>> On Aug 17, 2017 12:26 PM, "James McMahon" mailto:jsmcmah...@gmail.com>> wrote:
>>>
>>> Our nifi instance appeared to be running fine but we noticed
that there
>>> were no log files for today in the logs subdirectory. We could
not find any
>>> nifi logs for today anywhere on our system.
>>>
>>> I was surprised that NiFi continued to run. Has anyone
experienced such
>>> behavior?
>>>
>>> How is NiFi able to continue to run without a nifi-app.log -
do all its
>>> log messages effectively go to bit bucket heaven?
>>>
>>> I ultimately did an orderly shutdown via
>>> service nifi stop
>>> and an orderly start via
>>> service nifi start
>>> after which the log files were there as expected.
>>>
>>> Thanks in advance for any insights. -Jim
>>>
>>>
>>>
>
>

Re: New "hierarchical" controller service system confusing

[Russell Bateman]

Thanks for the suggestion, Matt. I created NIFI-4251.

On 08/01/2017 09:39 AM, Matt Gilman wrote:

Russell,

Thanks for the suggestion on the improved Controller Service UX. Would 
you mind filing a JIRA for this improvement?


In 1.x, the user can see the components referencing a Controller 
Service in 1 of 3 places. The references are shown in the read-only 
details dialog, the service configuration dialog, and the 
enable/disable service dialog. The references will include Processors, 
Reporting Tasks, and Controller Services (and components that 
reference those services and so on).


Thanks

Matt


On Mon, Jul 31, 2017 at 12:45 PM, Russell Bateman 
mailto:r...@windofkeltia.com>> wrote:


Friends,

I find the new (well, since 0.7.x -> 1.x) way of associating
controller services based on the process group a bit disorienting.
When one follows the right arrow from a consuming processor and
lands on the Process Group Configurationpage where the services
are configured, one is obliged to click on the Generaltab to
figure out whether the controller service is at the root (the case
for services whose flows were upgraded from the 0.7.x world) or
for a specific process group (the case for configuration
accomplished post-upgrade/native 1.x).

I realize that this will be a dwindling problem, but there's
enough white space in this dialog to justify putting the Process
Group Name to the right of the dialog title (Process Group
Configuration) while on the Controller Servicestab, even without
the momentary confusion of origin (that is, a flow from 0.7.x vs.
a "native" 1.x flow). So, I consider that this enhancement holds
regardless of whether upgraded flows are present or not.

At the same time, I'm tempted to ask how, when looking on the
Controller Services tab/page, one figures out which processor
instance(s) is (are) consuming a particular controller service I'm
looking at?

Russ

New "hierarchical" controller service system confusing

[Russell Bateman]

Friends,

I find the new (well, since 0.7.x -> 1.x) way of associating controller 
services based on the process group a bit disorienting. When one follows 
the right arrow from a consuming processor and lands on the Process 
Group Configurationpage where the services are configured, one is 
obliged to click on the Generaltab to figure out whether the controller 
service is at the root (the case for services whose flows were upgraded 
from the 0.7.x world) or for a specific process group (the case for 
configuration accomplished post-upgrade/native 1.x).


I realize that this will be a dwindling problem, but there's enough 
white space in this dialog to justify putting the Process Group Name to 
the right of the dialog title (Process Group Configuration) while on the 
Controller Servicestab, even without the momentary confusion of origin 
(that is, a flow from 0.7.x vs. a "native" 1.x flow). So, I consider 
that this enhancement holds regardless of whether upgraded flows are 
present or not.


At the same time, I'm tempted to ask how, when looking on the Controller 
Services tab/page, one figures out which processor instance(s) is (are) 
consuming a particular controller service I'm looking at?


Russ

Re: Jetty failure to start: org.jasypt.exceptions.EncryptionOperationNotPossibleException

[Russell Bateman]

Sorry I took so long...

I created NIFI-4237. A community person might want to polish my JIRA entry.

Thanks,
Russ


On 07/26/2017 11:47 AM, Andy LoPresto wrote:

Russell,

Thanks for following up and documenting this. If you are willing to 
file a Jira, we can hopefully improve the error messaging to make this 
easier for users to diagnose, and as there is already a ticket 
(NIFI-3116 [1]) to remove Jasypt (the underlying library which is 
generating the stacktrace), they may be done in conjunction. Thanks.


[1] https://issues.apache.org/jira/browse/NIFI-3116

Andy LoPresto
alopre...@apache.org <mailto:alopre...@apache.org>
/alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com>/
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jul 26, 2017, at 8:39 AM, Russell Bateman <mailto:r...@windofkeltia.com>> wrote:


Follow-up...

We use openJRE, so the JCE problem doesn't affect us.

The problem was as Mark suggested: Our Ansible instructions upgraded 
NiFi and created a newnifi.sensitive.props.key. In /nifi.properties/ 
this property, if extant, is used to encrypt sensitive properties in 
/flow.xml.gz/. Thus, upon relaunching NiFi, the wrong key was used to 
decrypt resulting in the reported failure to start, /flow.xml.gz/ is 
no longer useful.


How did we solve it?

We looked in the /nifi.properties.rpmsave/ file, what RPM does with a 
file it's changed, and copied the old key from this property to paste 
in over the newly generated key in /nifi.properties/. Relaunched, 
NiFi worked with no problem. The full solution, in our case, is to 
insist in Ansible that it not generate for and replace 
nifi.sensitive.props.key with a new key.


Many thanks to Mark and Joe for their very immediate and useful help 
saving us much time down!


Russ


On 07/26/2017 07:53 AM, Russell Bateman wrote:
Thanks for these suggestions, guys. I've only come in this morning 
to this complaint on a customer's production server to which I don't 
have access. So, I'm at the beginning of it, but I've never seen 
this before and thought I'd ask in the meantime. Your suggestions 
are invaluable; I'm sure that something like what you say must be 
going on. I'll confer with the DevOps guys when they get in for the day.


Many thanks,

Russ

On 07/26/2017 07:46 AM, Joe Witt wrote:

Has the version of java being used changed by chance on the system?
And if so, or perhaps even if not, were the JCE extensions
installed/configured previously and now it is not?  Other than that
the only other thing that comes to mind is if the sensitive properties
key was changed

On Wed, Jul 26, 2017 at 9:40 AM, Russell Bateman  wrote:

I'm getting this stack trace reported. I'm completely unfamiliar with this
problem or what could cause it--never having seen it before. I could use
some help here.

Thanks.

2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
Failed to start web server... shutting down.
org.apache.nifi.encrypt.EncryptionException:
org.jasypt.exceptions.EncryptionOperationNotPossibleException
 at
org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.serialization.FlowFromDOMFactory.getProperties(FlowFromDOMFactory.java:411)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.serialization.FlowFromDOMFactory.getControllerService(FlowFromDOMFactory.java:96)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.fingerprint.FingerprintFactory.addFlowControllerFingerprint(FingerprintFactory.java:211)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.fingerprint.FingerprintFactory.createFingerprint(FingerprintFactory.java:176)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.fingerprint.FingerprintFactory.createFingerprint(FingerprintFactory.java:146)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.StandardFlowSynchronizer.checkFlowInheritability(StandardFlowSynchronizer.java:1335)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.StandardFlowSynchronizer.checkFlowInheritability(StandardFlowSynchronizer.java:1325)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:240)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1461)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.StandardFlowService.

Re: Jetty failure to start: org.jasypt.exceptions.EncryptionOperationNotPossibleException

[Russell Bateman]

Follow-up...

We use openJRE, so the JCE problem doesn't affect us.

The problem was as Mark suggested: Our Ansible instructions upgraded 
NiFi and created a newnifi.sensitive.props.key. In /nifi.properties/ 
this property, if extant, is used to encrypt sensitive properties in 
/flow.xml.gz/. Thus, upon relaunching NiFi, the wrong key was used to 
decrypt resulting in the reported failure to start, /flow.xml.gz/ is no 
longer useful.


How did we solve it?

We looked in the /nifi.properties.rpmsave/ file, what RPM does with a 
file it's changed, and copied the old key from this property to paste in 
over the newly generated key in /nifi.properties/. Relaunched, NiFi 
worked with no problem. The full solution, in our case, is to insist in 
Ansible that it not generate for and replace nifi.sensitive.props.key 
with a new key.


Many thanks to Mark and Joe for their very immediate and useful help 
saving us much time down!


Russ


On 07/26/2017 07:53 AM, Russell Bateman wrote:
Thanks for these suggestions, guys. I've only come in this morning to 
this complaint on a customer's production server to which I don't have 
access. So, I'm at the beginning of it, but I've never seen this 
before and thought I'd ask in the meantime. Your suggestions are 
invaluable; I'm sure that something like what you say must be going 
on. I'll confer with the DevOps guys when they get in for the day.


Many thanks,

Russ

On 07/26/2017 07:46 AM, Joe Witt wrote:

Has the version of java being used changed by chance on the system?
And if so, or perhaps even if not, were the JCE extensions
installed/configured previously and now it is not?  Other than that
the only other thing that comes to mind is if the sensitive properties
key was changed

On Wed, Jul 26, 2017 at 9:40 AM, Russell Bateman  wrote:

I'm getting this stack trace reported. I'm completely unfamiliar with this
problem or what could cause it--never having seen it before. I could use
some help here.

Thanks.

2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
Failed to start web server... shutting down.
org.apache.nifi.encrypt.EncryptionException:
org.jasypt.exceptions.EncryptionOperationNotPossibleException
 at
org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.serialization.FlowFromDOMFactory.getProperties(FlowFromDOMFactory.java:411)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.serialization.FlowFromDOMFactory.getControllerService(FlowFromDOMFactory.java:96)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.fingerprint.FingerprintFactory.addFlowControllerFingerprint(FingerprintFactory.java:211)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.fingerprint.FingerprintFactory.createFingerprint(FingerprintFactory.java:176)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.fingerprint.FingerprintFactory.createFingerprint(FingerprintFactory.java:146)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.StandardFlowSynchronizer.checkFlowInheritability(StandardFlowSynchronizer.java:1335)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.StandardFlowSynchronizer.checkFlowInheritability(StandardFlowSynchronizer.java:1325)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:240)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1461)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:678)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:508)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:69)
~[na:na]
 at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
 at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533)
~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
 at
org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
 at

Re: Jetty failure to start: org.jasypt.exceptions.EncryptionOperationNotPossibleException

[Russell Bateman]

Thanks for these suggestions, guys. I've only come in this morning to 
this complaint on a customer's production server to which I don't have 
access. So, I'm at the beginning of it, but I've never seen this before 
and thought I'd ask in the meantime. Your suggestions are invaluable; 
I'm sure that something like what you say must be going on. I'll confer 
with the DevOps guys when they get in for the day.


Many thanks,

Russ

On 07/26/2017 07:46 AM, Joe Witt wrote:

Has the version of java being used changed by chance on the system?
And if so, or perhaps even if not, were the JCE extensions
installed/configured previously and now it is not?  Other than that
the only other thing that comes to mind is if the sensitive properties
key was changed

On Wed, Jul 26, 2017 at 9:40 AM, Russell Bateman  wrote:

I'm getting this stack trace reported. I'm completely unfamiliar with this
problem or what could cause it--never having seen it before. I could use
some help here.

Thanks.

2017-07-25 23:23:31,148 WARN [main] org.apache.nifi.web.server.JettyServer
Failed to start web server... shutting down.
org.apache.nifi.encrypt.EncryptionException:
org.jasypt.exceptions.EncryptionOperationNotPossibleException
 at
org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.serialization.FlowFromDOMFactory.getProperties(FlowFromDOMFactory.java:411)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.serialization.FlowFromDOMFactory.getControllerService(FlowFromDOMFactory.java:96)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.fingerprint.FingerprintFactory.addFlowControllerFingerprint(FingerprintFactory.java:211)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.fingerprint.FingerprintFactory.createFingerprint(FingerprintFactory.java:176)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.fingerprint.FingerprintFactory.createFingerprint(FingerprintFactory.java:146)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.StandardFlowSynchronizer.checkFlowInheritability(StandardFlowSynchronizer.java:1335)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.StandardFlowSynchronizer.checkFlowInheritability(StandardFlowSynchronizer.java:1325)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:240)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1461)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:678)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:508)
~[nifi-framework-core-1.1.2.jar:1.1.2]
 at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:69)
~[na:na]
 at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
 at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533)
~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
 at
org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
 at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345)
~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
 at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404)
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
 at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366)
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
 at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
 at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
 at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520)
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
 at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
~[jetty-uti

Jetty failure to start: org.jasypt.exceptions.EncryptionOperationNotPossibleException

[Russell Bateman]

I'm getting this stack trace reported. I'm completely unfamiliar with 
this problem or what could cause it--never having seen it before. I 
could use some help here.


Thanks.

2017-07-25 23:23:31,148 WARN [main] 
org.apache.nifi.web.server.JettyServer Failed to start web server... 
shutting down.
org.apache.nifi.encrypt.EncryptionException: 
org.jasypt.exceptions.EncryptionOperationNotPossibleException
at 
org.apache.nifi.encrypt.StringEncryptor.decrypt(StringEncryptor.java:149) ~[nifi-framework-core-1.1.2.jar:1.1.2]
at 
org.apache.nifi.controller.serialization.FlowFromDOMFactory.decrypt(FlowFromDOMFactory.java:474) 
~[nifi-framework-core-1.1.2.jar:1.1.2]
at 
org.apache.nifi.controller.serialization.FlowFromDOMFactory.getProperties(FlowFromDOMFactory.java:411) 
~[nifi-framework-core-1.1.2.jar:1.1.2]
at 
org.apache.nifi.controller.serialization.FlowFromDOMFactory.getControllerService(FlowFromDOMFactory.java:96) 
~[nifi-framework-core-1.1.2.jar:1.1.2]
at 
org.apache.nifi.fingerprint.FingerprintFactory.addFlowControllerFingerprint(FingerprintFactory.java:211) 
~[nifi-framework-core-1.1.2.jar:1.1.2]
at 
org.apache.nifi.fingerprint.FingerprintFactory.createFingerprint(FingerprintFactory.java:176) 
~[nifi-framework-core-1.1.2.jar:1.1.2]
at 
org.apache.nifi.fingerprint.FingerprintFactory.createFingerprint(FingerprintFactory.java:146) 
~[nifi-framework-core-1.1.2.jar:1.1.2]
at 
org.apache.nifi.controller.StandardFlowSynchronizer.checkFlowInheritability(StandardFlowSynchronizer.java:1335) 
~[nifi-framework-core-1.1.2.jar:1.1.2]
at 
org.apache.nifi.controller.StandardFlowSynchronizer.checkFlowInheritability(StandardFlowSynchronizer.java:1325) 
~[nifi-framework-core-1.1.2.jar:1.1.2]
at 
org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:240) 
~[nifi-framework-core-1.1.2.jar:1.1.2]
at 
org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1461) 
~[nifi-framework-core-1.1.2.jar:1.1.2]
at 
org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:83) 
~[nifi-framework-core-1.1.2.jar:1.1.2]
at 
org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:678) 
~[nifi-framework-core-1.1.2.jar:1.1.2]
at 
org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:508) 
~[nifi-framework-core-1.1.2.jar:1.1.2]
at 
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:69) 
~[na:na]
at 
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837) 
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533) 
~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810) 
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345) 
~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404) 
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366) 
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772) 
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262) 
~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520) 
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) 
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) 
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) 
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) 
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) 
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) 
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
at 
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) 
~[jett

Re: disk space cleanup for running NIFI

[Russell Bateman]

Ben,

I took these notes last spring that may be useful to you.

http://www.javahotchocolate.com/notes/nifi.html#20170428

Hope this helps,

Russ


On 07/12/2017 08:38 PM, 尹文才 wrote:
Hi guys, I have a question about cleaning up the disk space used by 
NIFI from time to time.
As you know NIFI saves a lot to disks, like the repository folders. I 
checked the NIFI official admin guide and I know the content 
repository supports
toggling content archiving. So in order to save disk space, I could 
adjust the content archiving parameters or simply turn it off. My 
question is I didn't
any similar options for the other repository folders, do they support 
archiving as well? What are the best practices to keep the disk space 
used by NIFI
as low as possible? (I'm running NIFI in a single machine and 
sometimes I could see disk out of space error in NIFI bulletin board) 
Thanks.


Regards,
Ben

Re: Service controller list NiFi 0.x to 1.x

[Russell Bateman]

As promised I'm posting back that all of our controller services are 
migrated from any flow.xml.gx created by 0.7.2 and put into the 
top-level process group so that we have no problem accessing them. Our 
biggest problem is then education on how best to develop flows with 
controller services beginning in NiFi 1.x. We're producing documentation 
internally for that.


Thanks.

On 05/16/2017 03:30 PM, Russell Bateman wrote:
We're still testing our migration from 0.7.2 to 1.1.2 which is 
imminent. We still don't know how much of this is going to have to be 
reconfigured. Our down-streamers tend to be on the grouchy side, so 
we're testing to see what's going to irritate them. Were it not for 
the magic ids, I'd feel safe writing a Python script to run through 
their flow tarballs fixing the controller-setting impedance 
mismatches. Who knows; maybe there will be nothing to change although 
we know for certain of a /DBConnectionPool/ property change.


I will come back here in a few hours or tomorrow with an update on how 
it's going. I'll try to keep the noise down because what may alarm us 
shouldn't be a cause for alarm for everyone.


Thanks for the replies,

Russ

On 05/16/2017 02:47 PM, Michael Moser wrote:

Hi Russ,

In NiFi 1.2.0, there are 3 Reporting Tasks
(SiteToSiteBulletionReportingTask, SiteToSiteProvenanceReportingTask,
and SiteToSiteStatusReportingTask) that have the capability of using a
Controller Service (StandardSSLContextService).  So yes, the Global ->
Controller Settings has a limited set of use cases, but they are very
important.  This becomes even more true considering the extensibility
of NiFi components.  Who can predict what new reporting tasks might be
created to meet end user's needs, and what controller services those
tasks might need?

This is definitely high on the list of things that catch NiFi users
off guard, as they transition from the 0.x versions into the 1.x
versions.  I can say that once I got used to using the Operate Palette
(especially for defining controller services!) then it didn't seem as
bad.

I'll see what I can add to the 0.x to 1.0.0 Migration Guide [1] to
call this out specifically.

-- Mike

[1] -https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance



On Tue, May 16, 2017 at 3:53 PM, Russell Bateman  wrote:

Thanks, Andrew!

(From [1]:) "This means that the service will be available to all Processors
and Controller Services defined in that Process Group and below."

In my experience, this isn't true. If I create a controller via the General
menu in the very root of my NiFi canvas, configure its name in Settings,
calling it Jack, then I create a new process group, then configure a new
processor in that group, when I try to configure to use the controller, Jack
is not among the options.

In order for the statement above to be true, I have to create it via the
gear icon in the Operate menu/palette.

Is this not what you see?

So, beginning sometime in 1.x, the Controller Settings option in the General
menu became useless, even at the top level except for "all ReportingTasks
and services defined in the Controller Settings." But, when would any
reporting task or other service defined be able to benefit? Never if I'm any
judge.

I think this is much more than a mere documentation issue. I wonder if
removing the Controller Services... option from the General menu would not
be the most important thing to do (even before documenting the gear icon in
the Operate menu).

Russ


On 05/16/2017 10:05 AM, Andrew Lim wrote:

Hi Russell,

Thanks for your question.

Yes, working with Controller Services has definitely changed in 1.x compared
to 0.x NiFi.  Matt Gilman wrote a nice article about how Controller Service
scoping was updated in 1.x with the introduction of Multi-Tenant
Authorization and also discusses the recent improvements made in NiFi 1.2 to
alleviate some of the user confusion around scoping [1].   If you would like
to see further details, the parent Jira for the improvements can be found
here [2].

I think there is opportunity to improve the Apache documentation we have
around this functionality, so I just filed a new Jira [3].

Let us know if you have any more questions.

Thanks,

Drew

[1]
https://community.hortonworks.com/articles/90259/understanding-controller-service-availability-in-a.html
[2]https://issues.apache.org/jira/browse/NIFI-3128
[3]https://issues.apache.org/jira/browse/NIFI-3911



On May 16, 2017, at 11:28 AM, Russell Bateman  wrote:

It appears to me that that, unlike what happened in NiFi 0.x, in 1.x when I
look at controller services via the General menu -> Controller Services,
what I see is totally different from what I see when I configure controller
services for a processor.

If I use the General menu to set up my controller services, I do not see nor
am I given the option of using them in particular for process

Re: Service controller list NiFi 0.x to 1.x

[Russell Bateman]

It's because product planning can't be done overnight. It was a fight 
(months ago) to get higher approval to move to1.x. 1.1.2was the current 
release when approval was given. We'll plan to move ahead to 1.2.xin a 
few months motivated by the next release. In development, I tend to keep 
up, but product moves behind. As soon as my code for this release is 
locked, I'll start a new project and move up--months ahead of everyone else.


We're not clustering yet. I think the make-over of the authentication 
files and this issue were really the only ones that gave us pause in 
making the move. I pushed hard because I'm tired of going back and forth 
between the UI changes. However, as principally a write of custom 
processors and a controller service or two, not much has really changed 
for me.


NiFi's a great framework!

Russ

On 05/16/2017 03:37 PM, Andy LoPresto wrote:

Russell,

Is there a reason you are planning your migration to 1.1.2 and not 
1.2.0? There are a number of significant feature and bug fixes that 
went into 1.2.0 that will likely make your experience much better.



Andy LoPresto
alopre...@apache.org <mailto:alopre...@apache.org>
/alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com>/
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On May 16, 2017, at 2:30 PM, Russell Bateman <mailto:r...@windofkeltia.com>> wrote:


We're still testing our migration from 0.7.2 to 1.1.2 which is 
imminent. We still don't know how much of this is going to have to be 
reconfigured. Our down-streamers tend to be on the grouchy side, so 
we're testing to see what's going to irritate them. Were it not for 
the magic ids, I'd feel safe writing a Python script to run through 
their flow tarballs fixing the controller-setting impedance 
mismatches. Who knows; maybe there will be nothing to change although 
we know for certain of a /DBConnectionPool/ property change.


I will come back here in a few hours or tomorrow with an update on 
how it's going. I'll try to keep the noise down because what may 
alarm us shouldn't be a cause for alarm for everyone.


Thanks for the replies,

Russ

On 05/16/2017 02:47 PM, Michael Moser wrote:

Hi Russ,

In NiFi 1.2.0, there are 3 Reporting Tasks
(SiteToSiteBulletionReportingTask, SiteToSiteProvenanceReportingTask,
and SiteToSiteStatusReportingTask) that have the capability of using a
Controller Service (StandardSSLContextService).  So yes, the Global ->
Controller Settings has a limited set of use cases, but they are very
important.  This becomes even more true considering the extensibility
of NiFi components.  Who can predict what new reporting tasks might be
created to meet end user's needs, and what controller services those
tasks might need?

This is definitely high on the list of things that catch NiFi users
off guard, as they transition from the 0.x versions into the 1.x
versions.  I can say that once I got used to using the Operate Palette
(especially for defining controller services!) then it didn't seem as
bad.

I'll see what I can add to the 0.x to 1.0.0 Migration Guide [1] to
call this out specifically.

-- Mike

[1] -https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance



On Tue, May 16, 2017 at 3:53 PM, Russell Bateman  wrote:

Thanks, Andrew!

(From [1]:) "This means that the service will be available to all Processors
and Controller Services defined in that Process Group and below."

In my experience, this isn't true. If I create a controller via the General
menu in the very root of my NiFi canvas, configure its name in Settings,
calling it Jack, then I create a new process group, then configure a new
processor in that group, when I try to configure to use the controller, Jack
is not among the options.

In order for the statement above to be true, I have to create it via the
gear icon in the Operate menu/palette.

Is this not what you see?

So, beginning sometime in 1.x, the Controller Settings option in the General
menu became useless, even at the top level except for "all ReportingTasks
and services defined in the Controller Settings." But, when would any
reporting task or other service defined be able to benefit? Never if I'm any
judge.

I think this is much more than a mere documentation issue. I wonder if
removing the Controller Services... option from the General menu would not
be the most important thing to do (even before documenting the gear icon in
the Operate menu).

Russ


On 05/16/2017 10:05 AM, Andrew Lim wrote:

Hi Russell,

Thanks for your question.

Yes, working with Controller Services has definitely changed in 1.x compared
to 0.x NiFi.  Matt Gilman wrote a nice article about how Controller Service
scoping was updated in 1.x with the introduction of Multi-Tenant
Authorization and also discusses the recent improvements made in NiFi 1.2 to
alleviate some of t

Re: Service controller list NiFi 0.x to 1.x

[Russell Bateman]

We're still testing our migration from 0.7.2 to 1.1.2 which is imminent. 
We still don't know how much of this is going to have to be 
reconfigured. Our down-streamers tend to be on the grouchy side, so 
we're testing to see what's going to irritate them. Were it not for the 
magic ids, I'd feel safe writing a Python script to run through their 
flow tarballs fixing the controller-setting impedance mismatches. Who 
knows; maybe there will be nothing to change although we know for 
certain of a /DBConnectionPool/ property change.


I will come back here in a few hours or tomorrow with an update on how 
it's going. I'll try to keep the noise down because what may alarm us 
shouldn't be a cause for alarm for everyone.


Thanks for the replies,

Russ

On 05/16/2017 02:47 PM, Michael Moser wrote:

Hi Russ,

In NiFi 1.2.0, there are 3 Reporting Tasks
(SiteToSiteBulletionReportingTask, SiteToSiteProvenanceReportingTask,
and SiteToSiteStatusReportingTask) that have the capability of using a
Controller Service (StandardSSLContextService).  So yes, the Global ->
Controller Settings has a limited set of use cases, but they are very
important.  This becomes even more true considering the extensibility
of NiFi components.  Who can predict what new reporting tasks might be
created to meet end user's needs, and what controller services those
tasks might need?

This is definitely high on the list of things that catch NiFi users
off guard, as they transition from the 0.x versions into the 1.x
versions.  I can say that once I got used to using the Operate Palette
(especially for defining controller services!) then it didn't seem as
bad.

I'll see what I can add to the 0.x to 1.0.0 Migration Guide [1] to
call this out specifically.

-- Mike

[1] - https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance



On Tue, May 16, 2017 at 3:53 PM, Russell Bateman  wrote:

Thanks, Andrew!

(From [1]:) "This means that the service will be available to all Processors
and Controller Services defined in that Process Group and below."

In my experience, this isn't true. If I create a controller via the General
menu in the very root of my NiFi canvas, configure its name in Settings,
calling it Jack, then I create a new process group, then configure a new
processor in that group, when I try to configure to use the controller, Jack
is not among the options.

In order for the statement above to be true, I have to create it via the
gear icon in the Operate menu/palette.

Is this not what you see?

So, beginning sometime in 1.x, the Controller Settings option in the General
menu became useless, even at the top level except for "all ReportingTasks
and services defined in the Controller Settings." But, when would any
reporting task or other service defined be able to benefit? Never if I'm any
judge.

I think this is much more than a mere documentation issue. I wonder if
removing the Controller Services... option from the General menu would not
be the most important thing to do (even before documenting the gear icon in
the Operate menu).

Russ


On 05/16/2017 10:05 AM, Andrew Lim wrote:

Hi Russell,

Thanks for your question.

Yes, working with Controller Services has definitely changed in 1.x compared
to 0.x NiFi.  Matt Gilman wrote a nice article about how Controller Service
scoping was updated in 1.x with the introduction of Multi-Tenant
Authorization and also discusses the recent improvements made in NiFi 1.2 to
alleviate some of the user confusion around scoping [1].   If you would like
to see further details, the parent Jira for the improvements can be found
here [2].

I think there is opportunity to improve the Apache documentation we have
around this functionality, so I just filed a new Jira [3].

Let us know if you have any more questions.

Thanks,

Drew

[1]
https://community.hortonworks.com/articles/90259/understanding-controller-service-availability-in-a.html
[2] https://issues.apache.org/jira/browse/NIFI-3128
[3] https://issues.apache.org/jira/browse/NIFI-3911



On May 16, 2017, at 11:28 AM, Russell Bateman  wrote:

It appears to me that that, unlike what happened in NiFi 0.x, in 1.x when I
look at controller services via the General menu -> Controller Services,
what I see is totally different from what I see when I configure controller
services for a processor.

If I use the General menu to set up my controller services, I do not see nor
am I given the option of using them in particular for processors I'm
configuring. Instead, I appear to get a "Process Group Configuration and a
list of controller services which are not the ones I'm looking for (because
when I set them up, I gave them "special" names or renamed names I could
recognize apart from any other use).

Note: I'm more of a processor and controller service author than an
experienced user of NiFi, so I may just be hopelessly confused.

My ques

Re: Service controller list NiFi 0.x to 1.x

[Russell Bateman]

Thanks, Andrew!

(From [1]:) "/This means that the service will be available to all 
Processors and Controller Services defined in that Process Group _and 
below_./"


In my experience, this isn't true. If I create a controller _via the 
General menu_ in the very root of my NiFi canvas, configure its name in 
Settings, calling it Jack, then I create a new process group, then 
configure a new processor in that group, when I try to configure to use 
the controller, Jack is not among the options.


In order for the statement above to be true, I have to create it _via 
the gear icon_ in the Operate menu/palette.


Is this not what you see?

So, beginning sometime in 1.x, the Controller Settings option in the 
General menu became useless, even at the top level except for "all 
ReportingTasks and services defined in the Controller Settings." But, 
when would any reporting task or other service defined be able to 
benefit? Never if I'm any judge.


I think this is much more than a mere documentation issue. I wonder if 
removing the Controller Services... option from the General menu would 
not be the most important thing to do (even before documenting the gear 
icon in the Operate menu).


Russ


On 05/16/2017 10:05 AM, Andrew Lim wrote:

Hi Russell,

Thanks for your question.

Yes, working with Controller Services has definitely changed in 1.x compared to 
0.x NiFi.  Matt Gilman wrote a nice article about how Controller Service 
scoping was updated in 1.x with the introduction of Multi-Tenant Authorization 
and also discusses the recent improvements made in NiFi 1.2 to alleviate some 
of the user confusion around scoping [1].   If you would like to see further 
details, the parent Jira for the improvements can be found here [2].

I think there is opportunity to improve the Apache documentation we have around 
this functionality, so I just filed a new Jira [3].

Let us know if you have any more questions.

Thanks,

Drew

[1] 
https://community.hortonworks.com/articles/90259/understanding-controller-service-availability-in-a.html
[2] https://issues.apache.org/jira/browse/NIFI-3128
[3] https://issues.apache.org/jira/browse/NIFI-3911




On May 16, 2017, at 11:28 AM, Russell Bateman  wrote:

It appears to me that that, unlike what happened in NiFi 0.x, in 1.x when I look 
at controller services via the General menu -> Controller Services, what I see 
is totally different from what I see when I configure controller services for a 
processor.

If I use the General menu to set up my controller services, I do not see nor am I given the 
option of using them in particular for processors I'm configuring. Instead, I appear to get a 
"Process Group Configuration and a list of controller services which are not the ones I'm 
looking for (because when I set them up, I gave them "special" names or renamed 
names I could recognize apart from any other use).

Note: I'm more of a processor and controller service author than an experienced 
user of NiFi, so I may just be hopelessly confused.

My question is what's the point of being able to configure controller services "globally" 
or "generally" if you can't reach them when you need them?

Please confirm that I'm not just smoking funny weed and that this is different, 
in fact, from how it worked in 0.7.1.

Thanks.

Service controller list NiFi 0.x to 1.x

[Russell Bateman]

It appears to me that that, unlike what happened in NiFi 0.x, in 1.x 
when I look at controller services via the General menu -> Controller 
Services, what I see is totally different from what I see when I 
configure controller services for a processor.


If I use the General menu to set up my controller services, I do not see 
nor am I given the option of using them in particular for processors I'm 
configuring. Instead, I appear to get a "Process Group Configuration and 
a list of controller services which are not the ones I'm looking for 
(because when I set them up, I gave them "special" names or renamed 
names I could recognize apart from any other use).


Note: I'm more of a processor and controller service author than an 
experienced user of NiFi, so I may just be hopelessly confused.


My question is what's the point of being able to configure controller 
services "globally" or "generally" if you can't reach them when you need 
them?


Please confirm that I'm not just smoking funny weed and that this is 
different, in fact, from how it worked in 0.7.1.


Thanks.

Re: Is it possible to reference python requests module in ExecuteScript?

[Russell Bateman]

The very first custom processor I wrote was called ExecutePythonScript, 
right after working through the NiFi Rocks! JSON example. Then,  I began 
to grok the scope of free, existing, standard processors, among them 
/ExecuteProcess/ and /ExecuteStreamCommand/. Those did pretty much 
exactly what I was trying to do and I threw my code away when I found 
that this base was already covered. Months later, this 
Jython/ExecuteScript investigation exercise was my revisiting the issue 
for my users.


What my users employ out of, then back into NiFi is a proprietary 
filesystem in-box. It's a sophisticated mechanism that amounts to a very 
tight and sharing-safe version of GetFile and PutFile with decent 
metadata support (hey, wow, sort of like the NiFi flowfile). We've used 
it for many years. When we adopted NiFi just over a year ago, we 
interfaced it with custom processors /PutInbox/ and /GetInb//ox/. Once 
the flowfile goes to the filesystem, our folk are able to run their 
Python scripts on them. Thence, they return the result to a different 
in-box from which our NiFi ETL brings it back in as a flowfile. These 
guys have a hammer and everything else looks like a nail to them 
(they're no different than most of the rest of us in that respect).


I don't personally like this because it's not a clean flow. It suffers 
from not being very turn-key, requires a high degree of understanding on 
the part of those that use this hybrid approach and makes it impossible 
to think of, measure, and plan for our NiFi ETL process on many levels 
(like provenance, performance, SLA, etc.). We constantly have to 
translate attributes into in-box metadata, then translate back.


I have no opinion about whether a better Python (or other language) 
script is a good thing for NiFi generally, I just know that if 
/ExecuteScript/ were a no-brainer to use, my guys would use it and it 
would keep them away from these little black-hole, /Interstellar/ games 
in NiFi where the flowfiles leave, then reemerge.


I only say this in response to Matt's response to us. My original 
contribution to this thread was only to provide our experience in case 
it was somehow relevant. As I've said, I look to providing everything we 
need here to stay in NiFi from beginning to end of our ETL process to 
gain its _visibility_ and _unity_. Originally, I was promoting using 
some kind of AMQP to solve our ETL needs. One day, we stumbled upon 
NiFi. Since then, I've fallen in love with everything it does better 
than a home-brewed queue-messaging approach tying together inevitably 
disparate ETL applications. I especially love the UI, 
AbstractProcessorand the unit-test framework. It's pretty much a dream 
come true for me.


Russ


On 05/05/2017 05:24 PM, Matt Burgess wrote:

Russell et al,

I'd like to mention a couple of things here re: the context of the
scripting processors:

1) The scripting processors were designed to use the JSR-223 spec for
interfacing with a scripting language, in order to make adding new
languages easier (ideally you just add the dependency to the scripting
NAR). However the drawback is that the desired language must implement
this spec, and Python does not but Jython does.

2) NiFi is a Java application, Python is a native one. Either you'd
have to bring your own Python (and configure the processor with the
location of it, and then you're pretty close to just using
ExecuteStreamCommand), or we'd have to package a version for each OS
(if we are allowed to), and still have to shell out to it from Java to
the OS (again, that's what ExecuteStreamCommand does).  There is some
experimental work with JyNI to bridge the gap, but it's not ready for
primetime.

Having said that, there's a PR out there for a Groovy-specific
scripting processor (ExecuteGroovyScript), which isn't limited by the
JSR-223 spec and can make full use of all the Groovy goodness. That's
made easier because Groovy is a JVM scripting language.  However for
Python we could make the user experience better by having an
ExecutePythonScript processor; perhaps under the hood it is just a
glorified ExecuteStreamCommand that lets you put the script into a
processor property like ExecuteScript, but still shells out to a
python on the OS.  What do you think?

Regards,
Matt


On Fri, May 5, 2017 at 7:12 PM, Joe Witt  wrote:

It is worth discussing whether there is sufficient value to warrant
keeping jython/python support in the processors or whether we should
pull it.  It is certainly something we can document as being highly
limited but we don't really know how limited.  Frankly given the
performance I've seen with it I'd be ok removing it entirely.  One is
better off calling the script via a system call.  Groovy is one that
I've seen perform very well and be fully featured.

On Fri, May 5, 2017 at 6:38 PM, Russell Bateman  wrote:

We really want to use ExecuteS

Re: Is it possible to reference python requests module in ExecuteScript?

[Russell Bateman]

We really want to use /ExecuteScript/ because our end users are 
Pythonistas. They tend to punctuate their flows with the equivalent of 
/PutFile/ and /GetFile/ with Python scripts doing stuff on flowfiles 
that pass out of NiFi before returning into NiFi.


However, we find it nearly impossible to replace even the tamest of 
undertakings. If there were a good set of NiFi/Python shims that, from 
PyCharm, etc., gave us the ability to prototype, test and debug before 
copying and pasting into /ExecuteScript/, that would be wonderful. It 
hasn't worked out that way. Most of our experience is copying, pasting 
into the processor property, only to find something wrong, sometimes 
syntax, sometimes something runtime.


On their behalf, I played with this processor a few hours a while back. 
Another colleague too. Googling this underused tool hasn't been helpful, 
so the overall experience is negative so far. I can get most of the 
examples out there to work, but as soon as I try to do "real" work from 
my point of view, my plans sort of cave in.


Likely the Groovy and/or Ruby options are better? But, we're not Groovy 
or Ruby guys here. I understand the problems with this tool and so I 
understand what the obstacles are to it growing stronger. The problems 
won't yield to a few hours one Saturday afternoon. Better 
problem-logging underneath and better- and more lenient Python support 
on top. The second one is tough, though.


My approach is to minimize those black holes these guys put into their 
flows by creating custom processors for what I can't solve using 
standard processors.


Trying not to be too negative here...


On 05/05/2017 04:09 PM, Andre wrote:

Mike,

I believe it is possible to use requests under jython, however the 
process isn't very intuitive.


I know one folk that if I recall correctly has used it. Happy to try 
to find out how it is done.


Cheers

On Sat, May 6, 2017 at 4:57 AM, Mike Harding > wrote:


Hi All, I'm now looking at using ExecuteScript and python engine
to execute HTTP requests using the requests module. I've tried
referencing requests the module but when I try to import requests
I get a module reference error.
I downloaded the module from here >
https://pypi.python.org/pypi/requests

Not sure why it isnt picking it up. Ive tried referencing the
directory and the .py directly with no success.
Any ideas where im going wrong?
Cheers,
Mike

Re: Quickly find queues with data...

[Russell Bateman]

Ah, I see. Thanks very much. (I spend too much time writing custom 
processors and not enough time dog-fooding so when I do, I tend to choke 
like a beginning NiFi user.)


On 05/05/2017 03:37 PM, Joe Witt wrote:

Go here: 
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Summary_Page
  It shows you how to get the summary view.  Click on connections.
Sort on the column of interest.

Thanks

On Fri, May 5, 2017 at 5:35 PM, Russell Bateman  wrote:

Is there some NiFi power-user trick to quick-finding queues that have
flowfiles sitting in them?

I have a pretty big flow with lots of processor groups, processors, input-
and output-ports, etc. and I ingest some files. I realize, however, that
they don't all show up at the end, so I must go looking.

I need to find out which queues contain the ones that never made it to the
end. Have I no other choice than to inspect every processor group and every
queue visually in search of my lost files?

I know that processors that failed will have a red blob in their upper-right
corner and that I'll find that the queue in front of them may contain files.
I also know that a stopped processor's (i.e.: a processor I never started)
preceding queue may contain files. But, is there a faster way than to
inspect each processor, queue, etc?

Thanks for any comments, experience, tricks, etc.

Quickly find queues with data...

[Russell Bateman]

Is there some NiFi power-user trick to quick-finding queues that have 
flowfiles sitting in them?


I have a pretty big flow with lots of processor groups, processors, 
input- and output-ports, etc. and I ingest some files. I realize, 
however, that they don't all show up at the end, so I must go looking.


I need to find out which queues contain the ones that never made it to 
the end. Have I no other choice than to inspect every processor group 
and every queue visually in search of my lost files?


I know that processors that failed will have a red blob in their 
upper-right corner and that I'll find that the queue in front of them 
may contain files. I also know that a stopped processor's (i.e.: a 
processor I never started) preceding queue may contain files. But, is 
there a faster way than to inspect each processor, queue, etc?


Thanks for any comments, experience, tricks, etc.

Re: Best practices for backing up an instance of NiFi?

[Russell Bateman]

Mark,

This echoes if not answers a concern I've had with the whole concept of 
backing up the repositories.


Second, our users have just about completely shut provenance down as not 
useful enough to justify the disk space and crunch (in a desperate, 
vague expression of "throw everything non-essential overboard") in order 
to wring every last cycle out of the installation. We run NiFi for our 
ETL on an almost dedicated host (but no clustering yet).


Thank you very much,

Russ


On 04/28/2017 02:51 PM, Mark Payne wrote:


Given that the Content & FlowFile Repositories are intended to be 
rather short-lived,


I'm not sure that you'd really have need to backup the repository (if 
you were to pause the flow


in order to back it up, for instance, you could likely process the 
data just as fast as you could


back it up - and at that point you'd be finished with it). That being 
said, for a production use case,


I would certainly recommend running on a RAID configuration that 
provides redundancy so that if


a disk were to go bad you'd still be able to access the data.


For the Provenance Repository, there actually exists a Reporting Task 
that can send the data via


Site-to-Site so that it can be exfilled however you see fit in your flow.

Best practices for backing up an instance of NiFi?

[Russell Bateman]

Been Googling hard on this (NiFi docs, Horton docs, these forums, 
stackoverflow) and I'm not seeing anything about specific wisdom 
surrounding backing up /content_repository/ and /flowfile_repository/, 
why I would, why I wouldn't, turning NiFi off when I do, etc.


I completely get the rest of the picture and have experience reproducing 
a NiFi instance, including flows, making sure that flows match data 
already in repositories, configuration, logs, etc. But, I was hoping to 
find wisdom on the rather huge undertaking of backing up the 4 
repositories, must we pause NiFi when we do it, etc.


I'm being asked for guidance by my IT staff who are ready to hop on this 
question now. I've been able to give them very pointed details on 
everything except the repositories.


Any sharing is welcome.

Thanks,

Russ

Re: How to configure logging via SLF4J ?

[Russell Bateman]

Take a look at ${NIFI_ROOT}//conf/logback.xml/. If you've developed a 
processor and are asking about it, likely this question would fit better 
in d...@nifi.apache.org, no?


Hope this helps.


On 04/11/2017 08:24 AM, Vince Cole wrote:

Hi

I have developed a processor which uses qpid-jms-client to send
messages to ActiveMQ. The jms client uses SLF4J and I need to
configure that via NIFI... how?


Thanks

Re: RouteOnContent--how to specify the promised dynamic relationship?

[Russell Bateman]

Aldrin,

Wow, I was all set to push back on this, but I re-read what I'd once 
found on NiFi Rocks (especially the highlighted sentence):


   "This processor applies user-added regular expressions to the
   content of a FlowFile and routes a copy of the FlowFile to each
   destination whose regular expression matches. *The user adds
   properties where the name is the relationship that the FlowFile
   should follow if it matches the regular expression, which is defined
   as the property’s value.* User-defined properties do support the
   NiFi Expression Language, but in such cases, the results are
   interpreted as literal values, not regular expressions."

which was suddenly hugely clearer when I read it after reading what you 
wrote me here. The link you provide, for its sister processor (and with 
which I have the experience of hundreds of times using it), was 
misleading to the point that I thought for a moment to push back.


Once I'd digested the NiFi Rocks blurb, it fell into place. I don't know 
if this constitutes a suggestion for /RouteOnContent/'s usage or not. 
Now that I grok it, I don't need it, but the next poor schmuck? (Well, 
I've already hit an all-time high for stupid questions this week --  
;-)  .)


Thanks!

Russ

P.S. Yeah, as the domain says, NiFi totally rocks!

On 03/30/2017 11:42 AM, Aldrin Piri wrote:

Russell,

Your expectations are inline with the functionality of the processor.  
You can add dynamic relationship via a dynamic property in the 
processor properties tab.  The plus (+) located at the top right will 
let you first specify a property name (Treated as the relationship) 
and its value would be the regex of choice.  After hitting apply, the 
relationship will be available to connect to other components.


This is covered for RouteOnAttribute in the User Guide [1].

[1] 
http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#properties-tab


On Thu, Mar 30, 2017 at 1:32 PM, Russell Bateman 
mailto:r...@windofkeltia.com>> wrote:


I'm working in NiFi 1.1.1. In this processor's usage, I see:


  Dynamic Relationships:

A Dynamic Relationship may be created based on how the user
configures the Processor.

NameDescription
Name from Dynamic Property  FlowFiles that match the Dynamic
Property's Regular Expression


However, I cannot see in configuration any option to create a
relationship other than the "unmatched" one that already exists.
I've connect up the "unmatched" route, but find no opportunity
before or during my attempt to wire the "matches" route to
anything (there isn't a "matches" route).

Have I misunderstood this processor's purpose?

Thanks.

RouteOnContent--how to specify the promised dynamic relationship?

[Russell Bateman]

I'm working in NiFi 1.1.1. In this processor's usage, I see:


 Dynamic Relationships:

A Dynamic Relationship may be created based on how the user configures 
the Processor.


NameDescription
Name from Dynamic Property 	FlowFiles that match the Dynamic Property's 
Regular Expression



However, I cannot see in configuration any option to create a 
relationship other than the "unmatched" one that already exists. I've 
connect up the "unmatched" route, but find no opportunity before or 
during my attempt to wire the "matches" route to anything (there isn't a 
"matches" route).


Have I misunderstood this processor's purpose?

Thanks.

Re: Providing Config and Attributes to NiFi

[Russell Bateman]

James,

I'm not certain what Joe intends here, as it works for me.

I've used it in NiFi 1.1.1 and have notes on it (just today, actually 
) for my guys 
to use, however, my example uses two processors you don't have, a way to 
create an arbitrary flowfile and a no-op processor (could have been any 
processor only stopped).


Better than my notes is this nice how-to by Yolanda from last September:

https://community.hortonworks.com/articles/57304/supporting-custom-properties-for-expression-langua.html

Hope this helps,

Russ

On 03/24/2017 11:07 AM, Joe Witt wrote:

We made our initial capability available but we still have more work
to go to reach the full vision of that feature proposal/vision as well
as related visions for extensions and flow versions.

On Fri, Mar 24, 2017 at 1:02 PM, Matt Foley  wrote:

Hi Joe,
The specification document in the wiki is written in future tense, but is it in 
fact what got implemented in the two jiras (all linked below)?
Thanks,
--Matt

On 3/24/17, 5:50 AM, "Joe Witt"  wrote:

 James,

 In NiFi 1.x line we introduced the variable registry concept for
 exactly this purpose. Right now it reads from a properties file but we
 intend to also grow this to support other more dynamic/central
 options.

 https://issues.apache.org/jira/browse/NIFI-2208
 https://issues.apache.org/jira/browse/NIFI-2449
 https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry

 Thanks
 JOe

 On Fri, Mar 24, 2017 at 8:47 AM, James McMahon  
wrote:
 > I would like to allow my workflow administrators to control NiFi 
attribute
 > and configuration parameters using an editable configuration text file 
read
 > in at start up - similar to nifi.properties. How can I do this?
 >
 > An example would be the hostname and port for our AMQP broker, or for
 > HandleHttpRequest.   -Jim

Re: Aw: Re: new Nifi Processors

[Russell Bateman]

I too find GPL to be confusing enough that I (and I am not alone) 
consider it to be poisonous fruit that I simply do not touch.


I earn my living working as an employee developing software for 
companies that sell it, or rights to use it, for money and do not 
publish their product source code any more than Coca Cola gives away the 
recipe for its flagship beverage.


As I understand it, if I consume a JAR that falls under GPL in my work, 
even if I only consume the JAR's functionality and do not modify it, 
however small and "insignificant" that JAR's contribution may be to the 
whole, that use opens my employer to a lawsuit because my source code is 
in essence and in fact built atop that GPL'd component.


Maybe my interpretation is born of irrational paranoia, but it's how it 
looks to me. To me, GPL means software to be used by academics and 
people who develop software for their health only.


Thoughts? (Yeah, this isn't really the forum for it.)


On 03/01/2017 04:40 AM, Uwe Geercken wrote:

Hello,
what would be the appropriate way to license the processors? Is it an 
annotation, a seperate license file or something else?

To the GPL3: This is what wikipedia says:

The *GNU General Public License* (*GNU GPL* or *GPL*) is a widely used 
free software license 
, which 
guarantees end users  the 
freedom to run, study, share and modify the software.^[6] 
 
The license was originally written by Richard Stallman 
 of the Free Software 
Foundation  
(FSF) for the GNU Project , 
and grants the recipients of a computer program 
 the rights of the 
Free Software Definition 
.^[7] 
 
The GPL is a copyleft  
license, which means that derivative work 
 can only be 
distributed under the same license terms. This is in distinction to 
permissive free software licenses 
, of 
which the BSD licenses  
and the MIT License  are 
widely used examples. GPL was the first copyleft license for general use.


Historically, the GPL license family has been one of the most popular 
software licenses in the free and open-source software 
 
domain.^[6] 
 
^[8] 
 
^[9] 
 
^[10] 
 
^[11] 
 
^[12] 
 
^[13] 
 
Prominent free software programs licensed under the GPL include the 
Linux kernel  and the GNU 
Compiler Collection 
 (GCC). David 
A. Wheeler  argues 
that the copyleft provided by the GPL was crucial to the success of 
Linux -based systems, 
giving the programmers who contributed to the kernel the assurance 
that their work would benefit the whole world and remain free, rather 
than being exploited by software companies that would not have to give 
anything back to the community.^[14] 



The ruleengine is under GPL3. So users acan freely use, embed or share 
it. It is only derivative work, that needs to be distributed under the 
same lisence terms. So what would be the problem with the Nifi 
processor? Can somebody explain that to me.
Also, I would be glad to have a quick explanation of what the core 
differences or advantages are of Apache 2.0 versus GPL3. That would 
help me understand the issue better.

Greetings and thanks for feedback.
Uwe
*Gesendet:* Mittwoch, 01. März 2017 um 03:33 Uhr
*Von:* "Angry Duck Studio" 
*An:* users@nifi.apache.org
*Betreff:* Re: new Nifi Processors
Hi, Uwe,
These look useful. However, typically custom processors are either 
Apache 2.0 or MIT licensed. These don't se

Re: Deadletter

[Russell Bateman]

No confusion, Nick. I hear you. In our case...

We attempt to reroute flowfile back through processors that, for 
whatever reason, might bring them to a state in which they can be 
successful (much to explain there, but...) and others we route to a 
similar, "deadletter" store where their contents are examined by hand 
and changes made (to the flow and processors used, to original document 
contents, etc.), then readmitted to the flow later.


(Note: We have many custom processors doing very special things.)

I'm personally all ears on this thread--eager to hear what others will 
say. Thanks for hosting it!


Russ

On 03/01/2017 01:38 PM, Nick Carenza wrote:
Sorry for the confusion, I meant to put emphasis on the _you_, as in 
'you all' or other users of nifi. I am looking to get insight into 
solutions other have implemented to deal with failures.


- Nick

On Wed, Mar 1, 2017 at 12:29 PM, Oleg Zhurakousky 
mailto:ozhurakou...@hortonworks.com>> 
wrote:


Nick

Since you’ve already designed Process Group (PG) that is specific
to failed flow files, I am not sure I understand your last
question “. . .How do you manage failure relationships?. . .”.
I am assuming that within your global flow all failure
relationships are sent to this PG, which essentially is a Dead
Letter Storage.

Are you asking about how do you get more information from the
failed Flow Files  (i.e., failure location, reason etc)?

Cheers
Oleg


On Mar 1, 2017, at 3:21 PM, Nick Carenza
mailto:nick.care...@thecontrolgroup.com>> wrote:

I have a lot of processors in my flow, all of which can, and do,
route flowfiles to their failure relationships at some point.

In the first iteration of my flow, I routed every failure
relationship to an inactive DebugFlow but monitoring these was
difficult, I wouldn't get notifications when something started to
fail and if the queue got filled up it would apply backpressure
and prevent new, good flowfiles from being processed.

Not only was that just not a good way to handle failures, but my
flow was littered with all of these do-nothing processors and was
an eye sore. So then I tried routing processor failure
relationships into themselves which tidied up my flow but caused
nifi to go berserk when a failure occurred because the failure
relationship is not penalized (nor should it be) and most
processors don't provide a 'Retry' relationship (InvokeHttp being
a notable exception). But really, most processors wouldn't
conceivable succeed if they were tried again. I mostly just
wanted the flowfiles to sit there until I had a chance to check
out why they failed and fix them manually.

This leads me to https://issues.apache.org/jira/browse/NIFI-3351
. I think I need
a way to store failed flowfiles, fix them and reprocess them. The
process group I am currently considering implementing everywhere is:

Input Port [Failed Flowfile] --> PutS3 deadletter///${uuid} --> PutSlack
ListS3 deadletter/// -->
FetchS3 -> Output Port [Fixed]

This gives me storage of failed messages logically grouped and in
a place that wont block up my flow since s3 never goes down,
err... wait. Configurable process groups or template like
https://issues.apache.org/jira/browse/NIFI-1096
 would make this
easier to reuse.

How do you manage failure relationships?

- Nick

Re: Deadlocks after upgrade from 0.6.1 to 1.1.1

[Russell Bateman]

Mikhail,

I have a short article with step-by-step information and comments on how 
I profile NiFi. You'll want the latest NiFi release, however, because 
the Java Flight Recorder JVM arguments are very order-dependent. (I'm 
assuming that NiFi 1.1.2 and 0.7.2 have the fix for 
/conf/bootstrap.conf/ numeric-argument order.) I've been using this for 
a couple of months and finally got around to writing it up from my 
personal notes in a more usable form:


http://www.javahotchocolate.com/notes/jfr.html

I hope this is helpful.

Russ

On 02/16/2017 10:18 PM, Mikhail Sosonkin wrote:
Been a while since I've used a profiler, but I'll give it a shot when 
I get to a place with faster internet link :)


On Fri, Feb 17, 2017 at 12:08 AM, Tony Kurc > wrote:


Mike, also if what Joe asked with the backpressure is "not being
applied", if you're good with a profiler, I think joe and I both
gravitated to 0x0006c533b770 being locked in at

org.apache.nifi.provenance.PersistentProvenanceRepository.persistRecord(PersistentProvenanceRepository.java:757).
It would be interesting to see if that section is taking longer
over time.

On Thu, Feb 16, 2017 at 11:56 PM, Joe Witt mailto:joe.w...@gmail.com>> wrote:

Mike

One more thing...can you please grab a couple more thread
dumps for us
with 5 to 10 mins between?

I don't see a deadlock but do suspect either just crazy slow
IO going
on or a possible livelock.  The thread dump will help narrow
that down
a bit.

Can you run 'iostat -xmh 20' for a bit (or its equivalent) on the
system too please.

Thanks
Joe

On Thu, Feb 16, 2017 at 11:52 PM, Joe Witt mailto:joe.w...@gmail.com>> wrote:
> Mike,
>
> No need for more info.  Heap/GC looks beautiful.
>
> The thread dump however, shows some problems.  The provenance
> repository is locked up.  Numerous threads are sitting here
>
> at

java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
> at

org.apache.nifi.provenance.PersistentProvenanceRepository.persistRecord(PersistentProvenanceRepository.java:757)
>
> This means these are processors committing their sessions
and updating
> provenance but they're waiting on a readlock to provenance. 
This lock

> cannot be obtained because a provenance maintenance thread is
> attempting to purge old events and cannot.
>
> I recall us having addressed this so am looking to see when
that was
> addressed.  If provenance is not critical for you right now
you can
> swap out the persistent implementation with the volatile
provenance
> repository.  In nifi.properties change this line
>
>

nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
>
> to
>
>

nifi.provenance.repository.implementation=org.apache.nifi.provenance.VolatileProvenanceRepository
>
> The behavior reminds me of this issue which was fixed in 1.x
> https://issues.apache.org/jira/browse/NIFI-2395

>
> Need to dig into this more...
>
> Thanks
> Joe
>
> On Thu, Feb 16, 2017 at 11:36 PM, Mikhail Sosonkin
mailto:mikh...@synack.com>> wrote:
>> Hi Joe,
>>
>> Thank you for your quick response. The system is currently
in the deadlock
>> state with 10 worker threads spinning. So, I'll gather the
info you
>> requested.
>>
>> - The available space on the partition is 223G free of 500G
(same as was
>> available for 0.6.1)
>> - java.arg.3=-Xmx4096m in bootstrap.conf
>> - thread dump and jstats are here
>>
https://gist.github.com/nologic/1ac064cb42cc16ca45d6ccd1239ce085

>>
>> Unfortunately, it's hard to predict when the decay starts
and it takes too
>> long to have to monitor the system manually. However, if
you still need,
>> after seeing the attached dumps, the thread dumps while it
decays I can set
>> up a timer script.
>>
>> Let me know if you need any more info.
>>
>> Thanks,
>> Mike.
>>
>>
>> On Thu, Feb 16, 2017 at 9:54 PM, Joe Witt
mailto:joe.w...@gmail.com>> wrote:
>>>
>>> Mike,
>>>
>>> Can you capture a series of thread dumps as the gradual
decay occurs
>>> and signal at what point they were generated specif

Re: Outputting flowfiles to disk

[Russell Bateman]

Or, 3?

   -> MergeContent
  +-> PutFile
   AttributesToJson > MergeContent


Or, 4?

   Join the ranks of custom processor writers and write one to do
   exactly what you want--good idea if this a pretty permanent part of
   your roadmap.


Hope this helps.

Russ

On 02/15/2017 03:18 PM, Kiran wrote:

Hello,
Within my NiFi flows for the error scenarios I would really like the 
option of outputting the flow file to an error directory (the 
outputted file contains the flow file contents and well as the 
attributes).
This way once the error has been resolved I can replay the FlowFile by 
reading it back in which would read the contents as well as the flow 
file attributes.
From looking through the processor list the only way I can see to do 
this is by:
1. Using the AttributesToJson to output the attributes and separately 
output the flowfile contents. Then read both the contents and JSON 
file back in and parse the JSON back into attributes.
2. Use a groovy script within the ExecuteScript processor to combine 
the attributes and contents together. Then output the results to disk. 
When reading it back in use another groovy script to parse the file 
and populate the attributes and contents.

My preferred option is number 2.
Can I confirm that I haven't missed anything obvious.
Thanks,
Brian

 
	Virus-free. www.avast.com 
 

Re: Writing back through a python stream callback when the flowfile content is a mix of character and binary

[Russell Bateman]

There is also a /SplitContent/ processor. Assuming you can recognize the 
boundaries of the different data types, you can split them up into 
separate flowfiles. Then you /MergeContent/ them back together later.



On 02/02/2017 04:19 PM, James McMahon wrote:
This is very helpful Russell, but in my case each file is a mix of 
data types. So even if i determine that the flowfile is a mix, I'd 
still have to be poised to tackle it it my ExecuteScript script. Good 
suggestion, though, and one I can use in other ways in my workflows.


I do hope someone can tell me what I can do in my callback write back 
to handle all. I'd like to better understand this error I'm getting, 
too.  -Jim


On Thu, Feb 2, 2017 at 6:02 PM, Russell Bateman <mailto:r...@windofkeltia.com>> wrote:


Could you use /RouteOnContent/ to determine what sort of content
you're dealing with, then branch to different /ExecuteScript/
processors rigged to different Python scripts?

Hope this comment is helpful.


On 02/02/2017 03:38 PM, James McMahon wrote:


I have a flowfile that has tagged character information I need to
get at throughout the first few sections of the file. I need to
use regex in python to select some of those values and to
transform others. I am using an ExecuteScript processor to
execute my python code. Here is my approach:

= = = = =

class PyStreamCallback(StreamCallback) :

   def __init__ (self) :

   def process(self, inputSteam, outputStream) :

  stuff = IOUtils.toString(inputStream,
StandardCharsets.UTF_8)  # what happens to my binary and extreme
chars when they get passed through this step?

 .

 . (transform and pick out select content)

 .

outputStream.write(bytearray(stuff.encode(‘utf-8’ # am I
using the wrong functions to put my text chars and my binary and
my extreme chars back on the stream as a byte stream? What should
I be doing to handle the variety of data?

flowFile = session.get()

if (flowFile!= None)

   incoming = flowFile.getAttribute(‘filename’)

logging.info <http://logging.info>(‘about to process file: %s’,
incoming)

   flowFile = session.write(flowFile, PyStreamCallback())   #
line 155 in my code

session.transfer(flowFile, REL_SUCCESS)

   session.commit()

= = = = =

When my incoming flowfile is all character content - such as
tagged xml - my code works fine. All the flowfiles that also
contain some binary data and/or characters at the extremes such
as foreign language characters don’t work. They error out. I
suspect it has to do with the way I am writing back to the
flowfile stream.

Here is the error I am getting:

Org.apache.nifi.processor.exception.ProcessException:
javax.script.ScriptException: TypeError: write(): 1^st arg can’t
be coerced to int, byte[] in 

Re: Writing back through a python stream callback when the flowfile content is a mix of character and binary

[Russell Bateman]

Could you use /RouteOnContent/ to determine what sort of content you're 
dealing with, then branch to different /ExecuteScript/ processors rigged 
to different Python scripts?


Hope this comment is helpful.


On 02/02/2017 03:38 PM, James McMahon wrote:


I have a flowfile that has tagged character information I need to get 
at throughout the first few sections of the file. I need to use regex 
in python to select some of those values and to transform others. I am 
using an ExecuteScript processor to execute my python code. Here is my 
approach:


= = = = =

class PyStreamCallback(StreamCallback) :

def __init__ (self) :

def process(self, inputSteam, outputStream) :

stuff = IOUtils.toString(inputStream, StandardCharsets.UTF_8) # what 
happens to my binary and extreme chars when they get passed through 
this step?


.

. (transform and pick out select content)

.

outputStream.write(bytearray(stuff.encode(‘utf-8’ # am I using 
the wrong functions to put my text chars and my binary and my extreme 
chars back on the stream as a byte stream? What should I be doing to 
handle the variety of data?


flowFile = session.get()

if (flowFile!= None)

incoming = flowFile.getAttribute(‘filename’)

logging.info (‘about to process file: %s’, incoming)

flowFile = session.write(flowFile, PyStreamCallback())   # line 155 in 
my code


session.transfer(flowFile, REL_SUCCESS)

session.commit()

= = = = =

When my incoming flowfile is all character content - such as tagged 
xml - my code works fine. All the flowfiles that also contain some 
binary data and/or characters at the extremes such as foreign language 
characters don’t work. They error out. I suspect it has to do with the 
way I am writing back to the flowfile stream.


Here is the error I am getting:

Org.apache.nifi.processor.exception.ProcessException: 
javax.script.ScriptException: TypeError: write(): 1^st arg can’t be 
coerced to int, byte[] in 

Re: Deleting millions of files from a queue...

[Russell Bateman]

Sure. I dropped a sample one into my notes for today here 
<http://www.javahotchocolate.com/notes/nifi.html#20170111>. However, 
it's not in a NAR. I assume you can package it up and do that? If you 
need help doing that, see some notes on this here 
<http://www.javahotchocolate.com/notes/nifi-project.html>. Please change 
the package name and make any other changes as you see fit.



On 01/11/2017 03:53 PM, Kevin Verhoeven wrote:


Russell,

Would you be able to make the counting processor you wrote (that 
maintains named NiFi counters) available to the public? This feature 
would be very useful to me. I need a counter that could survive a 
cluster restart for a few of my workflows. Currently I write out to an 
SQL server, but this is expensive to maintain.


Regards,

Kevin

*From:*Russell Bateman [mailto:r...@windofkeltia.com]
*Sent:* Wednesday, January 11, 2017 1:18 PM
*To:* users@nifi.apache.org
*Subject:* Re: Deleting millions of files from a queue...

Yes, I had thought of that, but I needed to know how many reached the 
end and I lamely thought I could look at the queue size not thinking 
about how emptying the queue wouldn't be instantaneous.


I finally just put a counting processor we wrote (that maintains named 
NiFi counters) in each of the flows I'm testing, then let the 
flowfiles drain out into the bit bucket.


This question really only arose because I was concentrating on my 
testing and not the bigger picture of what hundreds of millions of 
flowfiles might do to me in the end. So, I guess this is sort of a 
wasted thread except that (I'm hoping) it will be Googlable by someone 
else down the road who wonders as I did.


Thanks

On 01/11/2017 12:08 AM, Lee Laim wrote:

Russ,

This sort of deviates from your original question, but Would
applying a flowfile expiration time on the connection (during
experimentation)  work with your flow?  This would keep the queue
more manageable.

    On Jan 10, 2017, at 4:35 PM, Russell Bateman
mailto:r...@windofkeltia.com>> wrote:

To update this thread, ...

1. Setting up a no-op processor to "drain" the queue doesn't
seem to present any speed advantage over right-clicking the
queue and choosing Empty queue.
2. Removing the flowfile and provenance repositories (cd
flowfile_repository ; rm -rf *) is instantaneous.
3. However, removing the content repository from the
filesystem via console isn't immediate. It does take time. It
appears that it may not be taking as long as either method in
#1 above, but it does take a very long time. I had over a
hundred million files being emptied when I started this thread
and I'm still only down just under 40% left as I write this
final volley 2 hours after trying to delete them using the
filesystem (CentOS 7, CPU inactive, 128Gb memory, 56 cores,
hard drive--not SSD).
4. I don't dare delete the /database_repository/.
5. I'm assuming that once all three repositories are gone,
I'll be able to restart NiFi without any damage to what I
expect /flowfile.xml.gz/ and the rest of the /conf//
subdirectory are safe-guarding for me.

There may not be any instantaneous solution anyone can offer
short of renaming the content repository subdirectory, setting
up a background task to smoke it, and creating a new content
repository to start afresh with. I haven't tried that yet.

It's likely a better idea to think ahead about this and
provide for draining the queues as the flowfiles reach them.
If all you want is a count of successful outcomes, you could
do that with a no-op processor that counts as it goes and puts
the number somewhere for safe-keeping. I wouldn't be doing
this if I weren't trying to make some observations on
performance, processing loads, etc., in short, testing.

If I experience anything nasty or noteworthy from this point
    on [4,5], I'll come back and update this thread again.

On 01/10/2017 02:50 PM, Russell Bateman wrote:

In my case, I'm experimenting with huge flows and huge
numbers of files. I wasn't thinking about how much work
I'd create for myself by storing up files in a queue at
the end (or, in some cases, at intermediate points) when I
might want to clean house and start over.

So, I can just bring NiFi down, smoke the repos, then
restart safely?

On 01/10/2017 02:39 PM, Joe Witt wrote:

Millions or gajillions will indeed take a while as
they have to swap in as presently implemented.  We
could certainly optimize that if is a common need.

B

Re: Deleting millions of files from a queue...

[Russell Bateman]

Yes, I had thought of that, but I needed to know how many reached the 
end and I lamely thought I could look at the queue size not thinking 
about how emptying the queue wouldn't be instantaneous.


I finally just put a counting processor we wrote (that maintains named 
NiFi counters) in each of the flows I'm testing, then let the flowfiles 
drain out into the bit bucket.


This question really only arose because I was concentrating on my 
testing and not the bigger picture of what hundreds of millions of 
flowfiles might do to me in the end. So, I guess this is sort of a 
wasted thread except that (I'm hoping) it will be Googlable by someone 
else down the road who wonders as I did.


Thanks

On 01/11/2017 12:08 AM, Lee Laim wrote:

Russ,
This sort of deviates from your original question, but Would applying 
a flowfile expiration time on the connection (during experimentation) 
 work with your flow?  This would keep the queue more manageable.


On Jan 10, 2017, at 4:35 PM, Russell Bateman <mailto:r...@windofkeltia.com>> wrote:



To update this thread, ...

1. Setting up a no-op processor to "drain" the queue doesn't seem to 
present any speed advantage over right-clicking the queue and 
choosing Empty queue.
2. Removing the flowfile and provenance repositories (cd 
flowfile_repository ; rm -rf *) is instantaneous.
3. However, removing the content repository from the filesystem via 
console isn't immediate. It does take time. It appears that it may 
not be taking as long as either method in #1 above, but it does take 
a very long time. I had over a hundred million files being emptied 
when I started this thread and I'm still only down just under 40% 
left as I write this final volley 2 hours after trying to delete them 
using the filesystem (CentOS 7, CPU inactive, 128Gb memory, 56 cores, 
hard drive--not SSD).

4. I don't dare delete the /database_repository/.
5. I'm assuming that once all three repositories are gone, I'll be 
able to restart NiFi without any damage to what I expect 
/flowfile.xml.gz/ and the rest of the /conf// subdirectory are 
safe-guarding for me.


There may not be any instantaneous solution anyone can offer short of 
renaming the content repository subdirectory, setting up a background 
task to smoke it, and creating a new content repository to start 
afresh with. I haven't tried that yet.


It's likely a better idea to think ahead about this and provide for 
draining the queues as the flowfiles reach them. If all you want is a 
count of successful outcomes, you could do that with a no-op 
processor that counts as it goes and puts the number somewhere for 
safe-keeping. I wouldn't be doing this if I weren't trying to make 
some observations on performance, processing loads, etc., in short, 
testing.


If I experience anything nasty or noteworthy from this point on 
[4,5], I'll come back and update this thread again.



On 01/10/2017 02:50 PM, Russell Bateman wrote:
In my case, I'm experimenting with huge flows and huge numbers of 
files. I wasn't thinking about how much work I'd create for myself 
by storing up files in a queue at the end (or, in some cases, at 
intermediate points) when I might want to clean house and start over.


So, I can just bring NiFi down, smoke the repos, then restart safely?


On 01/10/2017 02:39 PM, Joe Witt wrote:
Millions or gajillions will indeed take a while as they have to 
swap in as presently implemented. We could certainly optimize that 
if is a common need.


Blowing away the repos will certainly do the trick and be faster.  
Though is clearly a blunt instrument.


Do you think we need an express queue killer option?

On Jan 10, 2017 1:32 PM, "Russell Bateman" <mailto:r...@windofkeltia.com>> wrote:


If I'm experimenting and have gajillions of flowfiles in a
queue that takes a very long time to empty from the UI, is
there a quicker way? I can certainly bounce NiFi, delete files,
both, etc.

Re: Deleting millions of files from a queue...

[Russell Bateman]

To update this thread, ...

1. Setting up a no-op processor to "drain" the queue doesn't seem to 
present any speed advantage over right-clicking the queue and choosing 
Empty queue.
2. Removing the flowfile and provenance repositories (cd 
flowfile_repository ; rm -rf *) is instantaneous.
3. However, removing the content repository from the filesystem via 
console isn't immediate. It does take time. It appears that it may not 
be taking as long as either method in #1 above, but it does take a very 
long time. I had over a hundred million files being emptied when I 
started this thread and I'm still only down just under 40% left as I 
write this final volley 2 hours after trying to delete them using the 
filesystem (CentOS 7, CPU inactive, 128Gb memory, 56 cores, hard 
drive--not SSD).

4. I don't dare delete the /database_repository/.
5. I'm assuming that once all three repositories are gone, I'll be able 
to restart NiFi without any damage to what I expect /flowfile.xml.gz/ 
and the rest of the /conf// subdirectory are safe-guarding for me.


There may not be any instantaneous solution anyone can offer short of 
renaming the content repository subdirectory, setting up a background 
task to smoke it, and creating a new content repository to start afresh 
with. I haven't tried that yet.


It's likely a better idea to think ahead about this and provide for 
draining the queues as the flowfiles reach them. If all you want is a 
count of successful outcomes, you could do that with a no-op processor 
that counts as it goes and puts the number somewhere for safe-keeping. I 
wouldn't be doing this if I weren't trying to make some observations on 
performance, processing loads, etc., in short, testing.


If I experience anything nasty or noteworthy from this point on [4,5], 
I'll come back and update this thread again.



On 01/10/2017 02:50 PM, Russell Bateman wrote:
In my case, I'm experimenting with huge flows and huge numbers of 
files. I wasn't thinking about how much work I'd create for myself by 
storing up files in a queue at the end (or, in some cases, at 
intermediate points) when I might want to clean house and start over.


So, I can just bring NiFi down, smoke the repos, then restart safely?


On 01/10/2017 02:39 PM, Joe Witt wrote:
Millions or gajillions will indeed take a while as they have to swap 
in as presently implemented.  We could certainly optimize that if is 
a common need.


Blowing away the repos will certainly do the trick and be faster.  
Though is clearly a blunt instrument.


Do you think we need an express queue killer option?

On Jan 10, 2017 1:32 PM, "Russell Bateman" <mailto:r...@windofkeltia.com>> wrote:


If I'm experimenting and have gajillions of flowfiles in a queue
that takes a very long time to empty from the UI, is there a
quicker way? I can certainly bounce NiFi, delete files, both, etc.

Re: Deleting millions of files from a queue...

[Russell Bateman]

I'm trying your suggestion right now. It would certainly be an easy way 
to avoid accumulation, but, in terms of voiding a queue with millions of 
files saved up until I'm ready to lose them (at the end of a test run), 
it doesn't seem any improvement in speed over just right-clicking and 
emptying a queue.



On 01/10/2017 02:40 PM, Jonathan Telfer wrote:
If I want a sink node to get rid of flow files while I’m messing 
around I add a ‘dev/null' update attribute processor that does 
nothing. Set the output to automatically terminate and just connect 
the queue to it and start it up. If you want to retain some output to 
look at just stop it processing.




On 10 Jan 2017, at 21:32, Russell Bateman <mailto:r...@windofkeltia.com>> wrote:


If I'm experimenting and have gajillions of flowfiles in a queue that 
takes a very long time to empty from the UI, is there a quicker way? 
I can certainly bounce NiFi, delete files, both, etc.

Re: Deleting millions of files from a queue...

[Russell Bateman]

In my case, I'm experimenting with huge flows and huge numbers of files. 
I wasn't thinking about how much work I'd create for myself by storing 
up files in a queue at the end (or, in some cases, at intermediate 
points) when I might want to clean house and start over.


So, I can just bring NiFi down, smoke the repos, then restart safely?


On 01/10/2017 02:39 PM, Joe Witt wrote:
Millions or gajillions will indeed take a while as they have to swap 
in as presently implemented.  We could certainly optimize that if is a 
common need.


Blowing away the repos will certainly do the trick and be faster.  
Though is clearly a blunt instrument.


Do you think we need an express queue killer option?

On Jan 10, 2017 1:32 PM, "Russell Bateman" <mailto:r...@windofkeltia.com>> wrote:


If I'm experimenting and have gajillions of flowfiles in a queue
that takes a very long time to empty from the UI, is there a
quicker way? I can certainly bounce NiFi, delete files, both, etc.

Deleting millions of files from a queue...

[Russell Bateman]

If I'm experimenting and have gajillions of flowfiles in a queue that 
takes a very long time to empty from the UI, is there a quicker way? I 
can certainly bounce NiFi, delete files, both, etc.

Re: GetFile: why must source directory be writeable...

[Russell Bateman]

I see that it's okay for the files themselves to be read-only, but it 
seems heavy-handed to enforce writeability on the parent subdirectory, no?


On 01/06/2017 02:00 PM, Russell Bateman wrote:
...when Keep Source Fileis set to trueand the sources are being 
protected anyway against deletion? Is this an oversight that should be 
JIRA'd?

GetFile: why must source directory be writeable...

[Russell Bateman]

...when Keep Source Fileis set to trueand the sources are being 
protected anyway against deletion? Is this an oversight that should be 
JIRA'd?

Re: Making FlowFiles environment independent

[Russell Bateman]

Without completely understanding your scenario, may I suggest you insert 
UpdateAttributesto do this before PutTCP? It supports Expression Language.


Hope this helps.

Russ

On 12/12/2016 01:04 AM, ddewaele wrote:

We have a flowfile that contains a number of environment specific values
(ports / hostnames / .).

Am I correct in saying that there is no immediate variable registry
somewhere in nifi, and that all of these environment specific items need to
be passed as environment variables or java system properties ?

I understand that the nifi expression language allows us to retrieve
environment variables / system properties, but a number of processor don't
support the expression language for fields that do contain environment
specific values (like the PutTCP processor hostname property).

How should we go about updating those ?



--
View this message in context: 
http://apache-nifi-users-list.2361937.n4.nabble.com/Making-FlowFiles-environment-independent-tp409.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.

NiFi versions and remote process groups

[Russell Bateman]

Should I be able to wire a remote process group on a NiFi 1.0.0 canvas 
to an input port on a 0.7.1 canvas? (Neither is a cluster.)


I have an input port in 0.7.1, but when I click the "circle-arrow" icon 
on my NiFi 1.0.0 remote process group and drag it, I get


'NiFi Flow' does not have any output ports (true, it doesn't)
'NiFi Flow' does not have any input ports (false, there is one).

When I try to connect the output of a processor in my NiFi 1.0.0 
instance, I also get

'NiFi Flow' does not have any input ports.

I presume 'NiFi Flow' refers to my running NiFi 0.7.1 instance (since 
that's its name in the title bar of the browser).


I've examined:

   
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Remote_Group_Transmission
   https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#site-to-site
   
https://community.hortonworks.com/articles/16461/nifi-understanding-how-to-use-process-groups-and-r.html

Thanks.

NiFi 1.0.0 canvas background

[Russell Bateman]

Now that I'm working in the 1.x world, there's no way I can restore the 
nice, light grid from 0.x to 1.x in place of the heavy, dull "fabric" 
background of the canvas in the new version, is there?


I've Googled and prowled around the interface with no success.

Also, wasn't there a way to align rectangles? I thought I remembered 
seeing that in 0.x, but couldn't find how to do it there or in 1.x. 
Maybe I was having a LibreOffice Draw-inspired dream or something.


Thanks,

Russ

Re: What else besides port conflict will keep NiFi 1.0.0 from running?

[Russell Bateman]

Sure. NIFI-2993 Issue clearer message when debug port conflicts

Thanks!


On 11/04/2016 06:25 PM, Andy LoPresto wrote:
That’s a tricky one. Great catch Bryan — good news that it had gotten 
you before so you knew what it was, bad news that it happens to 
anyone. I think this something we can catch during bootstrap and 
provide a better error for. Russ, feel free to file a Jira for that.


Andy LoPresto
alopre...@apache.org <mailto:alopre...@apache.org>
/alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com>/
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 4, 2016, at 12:31 PM, Bryan Rosander <mailto:brosan...@apache.org>> wrote:


No worries.  That's bitten me more than once :-).  It would be nice 
of the JVM to print at least the port it's trying to bind to to make 
it a little more obvious.


On Fri, Nov 4, 2016 at 3:29 PM, Russell Bateman 
<mailto:russell.bate...@perfectsearchcorp.com>> wrote:


Bryan,

Yes, thank you very much, I did and I wasn't even thinking about
that (port 8000 no less).

Embarrassed,

Russ

On 11/04/2016 11:47 AM, Bryan Rosander wrote:

Hey Russ,

It looks like you may have configured NiFi for remote
debugging.  If you have port conflicts on the port Java tries to
listen on for remote debugging, you'll see an error just like that.

Thanks,
Bryan

    On Fri, Nov 4, 2016 at 1:46 PM, Russell Bateman
mailto:russell.bate...@perfectsearchcorp.com>> wrote:

I feel sheepish asking this question, but I don't know where
to turn. Now ready to move up to NiFi 1.0.0 from 0.7.1,
where I've been running successfully since 0.4.0 nearly a
year ago, and with no changes ever (in my test environment)
to /nifi.properties/ except:

nifi.web.http.port=9091

I can't get NiFi to launch. The error from
/logs/nifi-bootstrap.log/ is:

*~/dev/nifi/nifi-1.0.0/logs $ cat nifi-bootstrap.log*
2016-11-04 11:32:32,217 INFO [main]
o.a.n.b.NotificationServiceManager Successfully loaded the
following 0 services: []
2016-11-04 11:32:32,221 INFO [main]
org.apache.nifi.bootstrap.RunNiFi Registered no Notification
Services for Notification Type NIFI_STARTED
2016-11-04 11:32:32,221 INFO [main]
org.apache.nifi.bootstrap.RunNiFi Registered no Notification
Services for Notification Type NIFI_STOPPED
2016-11-04 11:32:32,221 INFO [main]
org.apache.nifi.bootstrap.RunNiFi Registered no Notification
Services for Notification Type NIFI_DIED
2016-11-04 11:32:32,222 INFO [main]
org.apache.nifi.bootstrap.Command Apache NiFi is not
currently running
2016-11-04 11:32:35,684 INFO [main]
o.a.n.b.NotificationServiceManager Successfully loaded the
following 0 services: []
2016-11-04 11:32:35,687 INFO [main]
org.apache.nifi.bootstrap.RunNiFi Registered no Notification
Services for Notification Type NIFI_STARTED
2016-11-04 11:32:35,688 INFO [main]
org.apache.nifi.bootstrap.RunNiFi Registered no Notification
Services for Notification Type NIFI_STOPPED
2016-11-04 11:32:35,688 INFO [main]
org.apache.nifi.bootstrap.RunNiFi Registered no Notification
Services for Notification Type NIFI_DIED
2016-11-04 11:32:35,695 INFO [main]
org.apache.nifi.bootstrap.Command Starting Apache NiFi...
2016-11-04 11:32:35,695 INFO [main]
org.apache.nifi.bootstrap.Command Working Directory:
/home/russ/dev/nifi/nifi-1.0.0
2016-11-04 11:32:35,696 INFO [main]
org.apache.nifi.bootstrap.Command Command:
/home/russ/dev/jdk1.8.0_25/bin/java -classpath

/home/russ/dev/nifi/nifi-1.0.0/./conf:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-properties-loader-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/jcl-over-slf4j-1.7.12.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-api-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/logback-core-1.1.3.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/logback-classic-1.1.3.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-framework-api-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/jul-to-slf4j-1.7.12.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-documentation-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-runtime-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/slf4j-api-1.7.12.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-properties-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/bcprov-jdk15on-1.54.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/log4j-over-sl
f

4j-1.7.12.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/commons-lang3-3.4.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-nar-utils-1.0.0.jar
-Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m
-Xms512m
-agentlib:jdwp=transport=dt_socket,server=y,su

Re: What else besides port conflict will keep NiFi 1.0.0 from running?

[Russell Bateman]

Bryan,

Yes, thank you very much, I did and I wasn't even thinking about that 
(port 8000 no less).


Embarrassed,

Russ

On 11/04/2016 11:47 AM, Bryan Rosander wrote:

Hey Russ,

It looks like you may have configured NiFi for remote debugging.  If 
you have port conflicts on the port Java tries to listen on for remote 
debugging, you'll see an error just like that.


Thanks,
Bryan

On Fri, Nov 4, 2016 at 1:46 PM, Russell Bateman 
<mailto:russell.bate...@perfectsearchcorp.com>> wrote:


I feel sheepish asking this question, but I don't know where to
turn. Now ready to move up to NiFi 1.0.0 from 0.7.1, where I've
been running successfully since 0.4.0 nearly a year ago, and with
no changes ever (in my test environment) to /nifi.properties/ except:

nifi.web.http.port=9091

I can't get NiFi to launch. The error from
/logs/nifi-bootstrap.log/ is:

*~/dev/nifi/nifi-1.0.0/logs $ cat nifi-bootstrap.log*
2016-11-04 11:32:32,217 INFO [main]
o.a.n.b.NotificationServiceManager Successfully loaded the
following 0 services: []
2016-11-04 11:32:32,221 INFO [main]
org.apache.nifi.bootstrap.RunNiFi Registered no Notification
Services for Notification Type NIFI_STARTED
2016-11-04 11:32:32,221 INFO [main]
org.apache.nifi.bootstrap.RunNiFi Registered no Notification
Services for Notification Type NIFI_STOPPED
2016-11-04 11:32:32,221 INFO [main]
org.apache.nifi.bootstrap.RunNiFi Registered no Notification
Services for Notification Type NIFI_DIED
2016-11-04 11:32:32,222 INFO [main]
org.apache.nifi.bootstrap.Command Apache NiFi is not currently running
2016-11-04 11:32:35,684 INFO [main]
o.a.n.b.NotificationServiceManager Successfully loaded the
following 0 services: []
2016-11-04 11:32:35,687 INFO [main]
org.apache.nifi.bootstrap.RunNiFi Registered no Notification
Services for Notification Type NIFI_STARTED
2016-11-04 11:32:35,688 INFO [main]
org.apache.nifi.bootstrap.RunNiFi Registered no Notification
Services for Notification Type NIFI_STOPPED
2016-11-04 11:32:35,688 INFO [main]
org.apache.nifi.bootstrap.RunNiFi Registered no Notification
Services for Notification Type NIFI_DIED
2016-11-04 11:32:35,695 INFO [main]
org.apache.nifi.bootstrap.Command Starting Apache NiFi...
2016-11-04 11:32:35,695 INFO [main]
org.apache.nifi.bootstrap.Command Working Directory:
/home/russ/dev/nifi/nifi-1.0.0
2016-11-04 11:32:35,696 INFO [main]
org.apache.nifi.bootstrap.Command Command:
/home/russ/dev/jdk1.8.0_25/bin/java -classpath

/home/russ/dev/nifi/nifi-1.0.0/./conf:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-properties-loader-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/jcl-over-slf4j-1.7.12.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-api-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/logback-core-1.1.3.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/logback-classic-1.1.3.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-framework-api-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/jul-to-slf4j-1.7.12.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-documentation-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-runtime-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/slf4j-api-1.7.12.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-properties-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/bcprov-jdk15on-1.54.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/log4j-over-slf4j-1.7.12.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/commons-lang3-3.4.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-nar-utils-1.0.0.jar
-Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000
-Dsun.net.http.allowRestrictedHeaders=true
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
-XX:+UseG1GC -Djava.protocol.handler.pkgs=sun.net.www.protocol

-Dnifi.properties.file.path=/home/russ/dev/nifi/nifi-1.0.0/./conf/nifi.properties
-Dnifi.bootstrap.listen.port=33110 -Dapp=NiFi

-Dorg.apache.nifi.bootstrap.config.log.dir=/home/russ/dev/nifi/nifi-1.0.0/logs
org.apache.nifi.NiFi
*2016-11-04 11:32:35,747 ERROR [NiFi logging handler]
org.apache.nifi.StdErr ERROR: transport error 202: bind failed:
Address already in use**
**2016-11-04 11:32:35,747 ERROR [NiFi logging handler]
org.apache.nifi.StdErr ERROR: JDWP Transport dt_socket failed to
initialize, TRANSPORT_INIT(510)**
**2016-11-04 11:32:35,747 ERROR [NiFi logging handler]
org.apache.nifi.StdErr JDWP exit error
AGENT_ERROR_TRANSPORT_INIT(197): No transports initialized
[debugInit.c:750]**
**2016-11-04 11:32:35,747 INFO [NiFi logging handler]
org.apache.nifi.StdOut FATAL ERROR in native method: JDWP No
transports initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)*
2016-11-04 11:32:36,739 INFO [main]
org.apache.nifi.bootstrap.RunNiFi NiFi never started. Will

What else besides port conflict will keep NiFi 1.0.0 from running?

[Russell Bateman]

I feel sheepish asking this question, but I don't know where to turn. 
Now ready to move up to NiFi 1.0.0 from 0.7.1, where I've been running 
successfully since 0.4.0 nearly a year ago, and with no changes ever (in 
my test environment) to /nifi.properties/ except:


nifi.web.http.port=9091

I can't get NiFi to launch. The error from /logs/nifi-bootstrap.log/ is:

*~/dev/nifi/nifi-1.0.0/logs $ cat nifi-bootstrap.log*
2016-11-04 11:32:32,217 INFO [main] o.a.n.b.NotificationServiceManager 
Successfully loaded the following 0 services: []
2016-11-04 11:32:32,221 INFO [main] org.apache.nifi.bootstrap.RunNiFi 
Registered no Notification Services for Notification Type NIFI_STARTED
2016-11-04 11:32:32,221 INFO [main] org.apache.nifi.bootstrap.RunNiFi 
Registered no Notification Services for Notification Type NIFI_STOPPED
2016-11-04 11:32:32,221 INFO [main] org.apache.nifi.bootstrap.RunNiFi 
Registered no Notification Services for Notification Type NIFI_DIED
2016-11-04 11:32:32,222 INFO [main] org.apache.nifi.bootstrap.Command 
Apache NiFi is not currently running
2016-11-04 11:32:35,684 INFO [main] o.a.n.b.NotificationServiceManager 
Successfully loaded the following 0 services: []
2016-11-04 11:32:35,687 INFO [main] org.apache.nifi.bootstrap.RunNiFi 
Registered no Notification Services for Notification Type NIFI_STARTED
2016-11-04 11:32:35,688 INFO [main] org.apache.nifi.bootstrap.RunNiFi 
Registered no Notification Services for Notification Type NIFI_STOPPED
2016-11-04 11:32:35,688 INFO [main] org.apache.nifi.bootstrap.RunNiFi 
Registered no Notification Services for Notification Type NIFI_DIED
2016-11-04 11:32:35,695 INFO [main] org.apache.nifi.bootstrap.Command 
Starting Apache NiFi...
2016-11-04 11:32:35,695 INFO [main] org.apache.nifi.bootstrap.Command 
Working Directory: /home/russ/dev/nifi/nifi-1.0.0
2016-11-04 11:32:35,696 INFO [main] org.apache.nifi.bootstrap.Command 
Command: /home/russ/dev/jdk1.8.0_25/bin/java -classpath 
/home/russ/dev/nifi/nifi-1.0.0/./conf:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-properties-loader-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/jcl-over-slf4j-1.7.12.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-api-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/logback-core-1.1.3.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/logback-classic-1.1.3.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-framework-api-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/jul-to-slf4j-1.7.12.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-documentation-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-runtime-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/slf4j-api-1.7.12.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-properties-1.0.0.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/bcprov-jdk15on-1.54.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/log4j-over-slf4j-1.7.12.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/commons-lang3-3.4.jar:/home/russ/dev/nifi/nifi-1.0.0/./lib/nifi-nar-utils-1.0.0.jar 
-Dorg.apache.jasper.compiler.disablejsr199=true -Xmx512m -Xms512m 
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 
-Dsun.net.http.allowRestrictedHeaders=true 
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -XX:+UseG1GC 
-Djava.protocol.handler.pkgs=sun.net.www.protocol 
-Dnifi.properties.file.path=/home/russ/dev/nifi/nifi-1.0.0/./conf/nifi.properties 
-Dnifi.bootstrap.listen.port=33110 -Dapp=NiFi 
-Dorg.apache.nifi.bootstrap.config.log.dir=/home/russ/dev/nifi/nifi-1.0.0/logs 
org.apache.nifi.NiFi
*2016-11-04 11:32:35,747 ERROR [NiFi logging handler] 
org.apache.nifi.StdErr ERROR: transport error 202: bind failed: Address 
already in use**
**2016-11-04 11:32:35,747 ERROR [NiFi logging handler] 
org.apache.nifi.StdErr ERROR: JDWP Transport dt_socket failed to 
initialize, TRANSPORT_INIT(510)**
**2016-11-04 11:32:35,747 ERROR [NiFi logging handler] 
org.apache.nifi.StdErr JDWP exit error AGENT_ERROR_TRANSPORT_INIT(197): 
No transports initialized [debugInit.c:750]**
**2016-11-04 11:32:35,747 INFO [NiFi logging handler] 
org.apache.nifi.StdOut FATAL ERROR in native method: JDWP No transports 
initialized, jvmtiError=AGENT_ERROR_TRANSPORT_INIT(197)*
2016-11-04 11:32:36,739 INFO [main] org.apache.nifi.bootstrap.RunNiFi 
NiFi never started. Will not restart NiFi


And yet,

# lsof -i | grep 9091

produces nothing for port 9091 (nor does netstat -lptu). So, port 9091 
is not in use on my host, yet the error leads me to believe that this is 
the problem. (Indeed, I've seen this error before, or something like it, 
when running pre-1.0 NiFi and recognizing that I had a port conflict 
which, corrected, always led to happiness.)


Put-downs, wolf whistles and cat calls accepted (and welcome!), but what 
am I doing wrong or where should I look?


Thanks,
Russ

Re: Build failure under CentOS 6.7

[Russell Bateman]

Michael,

You don't have to build NiFi yourself to get going, you can just 
download pre-built "binaries". It's written in Java, so there are no 
platform gotchas at all:


https://nifi.apache.org/download.html


On 10/18/2016 07:36 AM, Giordano, Michael wrote:


I am currently following the NiFi QuickStart to get my first NiFi 
server up and running under CentOS 6.7. I am getting compile failures 
and I’m not sure why. I started from an empty directory as root (with 
full permissions) on a test machine with SELinux disabled.


# git clone https://github.com/apache/nifi.git

Initialized empty Git repository in /tmp/nifi/.git/

remote: Counting objects: 102987, done.

remote: Compressing objects: 100% (496/496), done.

remote: Total 102987 (delta 230), reused 0 (delta 0), pack-reused 102467

Receiving objects: 100% (102987/102987), 64.93 MiB | 10.24 MiB/s, done.

Resolving deltas: 100% (45888/45888), done.

# cd nifi

# git checkout master

# export MAVEN_OPTS="-Xms1024m -Xmx3076m -XX:MaxPermSize=256m"

# mvn -T C2.0 clean install



[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.2:compile 
(default-compile) on project nifi-api: Compilation failure -> [Help 1]


[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.2:compile 
(default-compile) on project nifi-logging-utils: Compilation failure 
-> [Help 1]


[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.2:compile 
(default-compile) on project nifi-properties: Compilation failure -> 
[Help 1]


cat /etc/redhat-release

CentOS release 6.7 (Final)

# mvn -version

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 
2015-11-10T11:41:47-05:00)


Maven home: /usr/local/apache-maven-3.3.9

Java version: 1.8.0_101, vendor: Oracle Corporation

Java home: 
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.101-3.b13.el6_8.x86_64/jre


Default locale: en_US, platform encoding: UTF-8

OS name: "linux", version: "2.6.32-573.el6.x86_64", arch: "amd64", 
family: "unix"


Any help is deeply appreciated.

Thanks,

Mike G.

This communication, along with its attachments, is considered 
confidential and proprietary to Vistronix.  It is intended only for 
the use of the person(s) named above.  Note that unauthorized 
disclosure or distribution of information not generally known to the 
public is strictly prohibited.  If you are not the intended recipient, 
please notify the sender immediately.

Re: Nifi hardware recommendation

[Russell Bateman]

Yeah, I spent a bit of time this morning before posting looking for a 
magic 8-10Gb advisory and generally for GC gotchas related to larger 
heap sizes in the 64-bit world, but couldn't find any. We're using 12Gb 
right now for NiFi and haven't noticed any trouble. We vaguely conceive 
of increasing this amount in the future as needed as our servers tend to 
run large amounts of memory.


The statement yesterday on this thread warning against using that much 
is what sent me into Google-it mode. I think this advice is a red herring.


Russ

On 10/14/2016 03:03 PM, Corey Flowers wrote:
We actually use heap sizes from 32 to 64Gb for ours but our volumes 
and graphs are both extremely large. Although I believe the smaller 
heap sizes were a limitation of the garbage collection in Java 7. We 
also moved to ssd drives, which did help through put quite a bit. Our 
systems were actually requesting the creation and removal of file 
handles faster than traditional disks could keep up with (we believe). 
In addition, unlike with traditional drives where we tired to minimize 
caching, we actually forced more disk caching when we moved to ssds. 
Still waiting to see the results of that on our volumes, although it 
does seemed to have help. Also remember, depending on how you code 
them, individual processors can use system memory outside of the heap. 
So you need to take that into consideration when designing the servers.


Sent from my iPhone

On Oct 14, 2016, at 1:36 PM, Joe Witt <mailto:joe.w...@gmail.com>> wrote:



Russ,

You can definitely find a lot of material on the Internet about Java 
heap sizes, types of garbage collectors, application usage patterns.  
By all means please do experiment with different sizes appropriate 
for your case.  We're not saying NiFi itself has any problem with 
large heaps.


Thanks
Joe

On Fri, Oct 14, 2016 at 12:44 PM, Russell Bateman 
<mailto:russell.bate...@perfectsearchcorp.com>> wrote:


Ali,

"not recommended to dedicate more than 8-10 GM to JVM heap space"
by whom? Do you have links/references establishing this? I
couldn't find anyone saying this or why.

Russ

On 10/13/2016 05:47 PM, Ali Nazemian wrote:

Hi,

I have another question regarding the hardware recommendation.
As far as I found out, Nifi uses on-heap memory currently, and
it will not try to load the whole object in memory. From the
garbage collection perspective, it is not recommended to
dedicate more than 8-10 GB to JVM heap space. In this case, may
I say spending money on system memory is useless? Probably 16 GB
per each system is enough according to this architecture. Unless
some architecture changes appear in the future to use off-heap
memory as well. However, I found some articles about best
practices, and in terms of memory recommendation it does not
make sense. Would you please clarify this part for me?
Thank you very much.

Best regards,
Ali


On Thu, Oct 13, 2016 at 11:38 PM, Ali Nazemian
mailto:alinazem...@gmail.com>> wrote:

Thank you very much.
I would be more than happy to provide some benchmark results
after the implementation.
Sincerely yours,
Ali

On Thu, Oct 13, 2016 at 11:32 PM, Joe Witt
mailto:joe.w...@gmail.com>> wrote:

Ali,

I agree with your assumption.  It would be great to test
that out and provide some numbers but intuitively I agree.

I could envision certain scatter/gather data flows that
could challenge that sequential access assumption but
honestly with how awesome disk caching is in Linux these
days in think practically speaking this is the right way
to think about it.

Thanks
Joe

On Thu, Oct 13, 2016 at 8:29 AM, Ali Nazemian
mailto:alinazem...@gmail.com>>
wrote:

Dear Joe,

Thank you very much. That was a really great
explanation.
I investigated the Nifi architecture, and it seems
that most of the read/write operations for flow file
repo and provenance repo are random. However, for
content repo most of the read/write operations are
sequential. Let's say cost does not matter. In this
case, even choosing SSD for content repo can not
provide huge performance gain instead of HDD. Am I
right? Hence, it would be better to spend content
repo SSD money on network infrastructure.

Best regards,
Ali

On Thu, Oct 13, 2016 at 10:22 PM, Joe Witt
mailto:joe.w...@gmail.com>> wrote:

Ali,

You have a lot of nice res

Re: Nifi hardware recommendation

[Russell Bateman]

Ali,

"not recommended to dedicate more than 8-10 GM to JVM heap space" by 
whom? Do you have links/references establishing this? I couldn't find 
anyone saying this or why.


Russ

On 10/13/2016 05:47 PM, Ali Nazemian wrote:

Hi,

I have another question regarding the hardware recommendation. As far 
as I found out, Nifi uses on-heap memory currently, and it will not 
try to load the whole object in memory. From the garbage collection 
perspective, it is not recommended to dedicate more than 8-10 GB to 
JVM heap space. In this case, may I say spending money on system 
memory is useless? Probably 16 GB per each system is enough according 
to this architecture. Unless some architecture changes appear in the 
future to use off-heap memory as well. However, I found some articles 
about best practices, and in terms of memory recommendation it does 
not make sense. Would you please clarify this part for me?

Thank you very much.

Best regards,
Ali


On Thu, Oct 13, 2016 at 11:38 PM, Ali Nazemian > wrote:


Thank you very much.
I would be more than happy to provide some benchmark results after
the implementation.
Sincerely yours,
Ali

On Thu, Oct 13, 2016 at 11:32 PM, Joe Witt mailto:joe.w...@gmail.com>> wrote:

Ali,

I agree with your assumption.  It would be great to test that
out and provide some numbers but intuitively I agree.

I could envision certain scatter/gather data flows that could
challenge that sequential access assumption but honestly with
how awesome disk caching is in Linux these days in think
practically speaking this is the right way to think about it.

Thanks
Joe

On Thu, Oct 13, 2016 at 8:29 AM, Ali Nazemian
mailto:alinazem...@gmail.com>> wrote:

Dear Joe,

Thank you very much. That was a really great explanation.
I investigated the Nifi architecture, and it seems that
most of the read/write operations for flow file repo and
provenance repo are random. However, for content repo most
of the read/write operations are sequential. Let's say
cost does not matter. In this case, even choosing SSD for
content repo can not provide huge performance gain instead
of HDD. Am I right? Hence, it would be better to spend
content repo SSD money on network infrastructure.

Best regards,
Ali

On Thu, Oct 13, 2016 at 10:22 PM, Joe Witt
mailto:joe.w...@gmail.com>> wrote:

Ali,

You have a lot of nice resources to work with there. 
I'd recommend the series of RAID-1 configuration

personally provided you keep in mind this means you
can only lose a single disk for any one partition.  As
long as they're being monitored and would be quickly
replaced this in practice works well.  If there could
be lapses in monitoring or time to replace then it is
perhaps safer to go with more redundancy or an
alternative RAID type.

I'd say do the OS, app installs w/user and audit db
stuff, application logs on one physical RAID volume. 
Have a dedicated physical volume for the flow file

repository.  It will not be able to use all the space
but it certainly could benefit from having no other
contention.  This could be a great thing to have SSDs
for actually.  And for the remaining volumes split
them up for content and provenance as you have. You
get to make the overall performance versus retention
decision. Frankly, you have a great system to work
with and I suspect you're going to see excellent
results anyway.

Conservatively speaking expect say 50MB/s of
throughput per volume in the content repository so if
you end up with 8 of them could achieve upwards of
400MB/s sustained. You'll also then want to make sure
you have a good 10G based network setup as well.  Or,
you could dial back on the speed tradeoff and simply
increase retention or disk loss tolerance.  Lots of
ways to play the game.

There are no published SSD vs HDD performance
benchmarks that I am aware of though this is a good
idea.  Having a hybrid of SSDs and HDDs could offer a
really solid performance/retention/cost tradeoff.  For
example having SSDs for the
OS/logs/provenance/flowfile with HDDs for the content
- that would be quite nice.  At that rate to take full
ad

Re: [DISCUSS] Slack team for Apache NiFi

[Russell Bateman]

In my opinion, Bryan brings up early a huge point given the paucity of 
NiFi documentation, discussion and samples out there in Googleland. It 
would be a big loss not to have questions and answers show up in searches.


On 10/10/2016 12:46 PM, Bryan Bende wrote:

My only concern with any kind of chat/channel is the loss of searchable
public content.
Right now the mailing list posts show up in Google searches and can be
found through other various tools, which is valuable for people searching
for information related to an issue they are seeing.
If people start transitioning to chat-based communication, all the
conversations there will likely only be found through that tool, and will
probably not be organized into individual threads of questions.

I have not used Slack or Gitter much so I can't really say which is better,
but if I remember correctly Gitter only required you to sign-in with your
GitHub account and after that you could go into the community without
approval, which seems more open to me, but I really don't know.

On Mon, Oct 10, 2016 at 2:28 PM, John Wiesel  wrote:


Good evening,

I am a new member to the community so I do not know much about the current
needs.
Still I'd like to point at gitter.im which has an open-sourced IRC bridge
(https://irc.gitter.im/) for the heavy IRC users.
Overall imho a good Slack alternative, its aim is to provide chatrooms for
developers.

Best
John


On Mon, Oct 10, 2016 at 4:53 PM, Matt Burgess 
wrote:


All,

I'd like to revisit the idea of having (and promoting) a Slack team
for the Apache NiFi community. We broached the subject in an email
thread a while back [1]. The email lists are great and IRC per se is
still popular in the open-source community, but I think for folks that
are more comfortable in the Slack environment, this would be another
good avenue for the community to get together.

We could perhaps add integrations with the email list(s), IRC
channel(s), GitHub, etc. as sort of a one-stop shop for NiFi
communication, or just use it for online collaboration.

I'm thinking it would be invite-only (to avoid bots, spam, etc.) but
that it would be fully open to the community, just an email to an
admin (such as myself) to request an invitation (kind of like a manual
version of the Apache Mailing List subscribe bot). The code of conduct
would be the Apache Software Foundation's CoC [2], as it is a
"communication channel used by our communities". I have seen this same
invite-only approach work well for other ASF projects with Slack
teams.

I grabbed a URL [3] just in case the community would like to proceed
(nifi.slack.com is taken, but if someone in the community owns it and
wants to use that instead, it's all good). The PMC would be
owners/admins, my email address (mattyb...@apache.org) could be the
destination for invite requests, and we could discuss what
integrations, access rights (creating new channels, e.g.), etc. are
appropriate.


Thanks,
Matt

[1] https://mail-archives.apache.org/mod_mbox/nifi-users/201511.
mbox/%3CCA+w4d5RJvxvFEnGVsexCSmEvzL_TxG_Ne2KEiFsrGEapCwoOAQ@
mail.gmail.com%3E

[2] https://www.apache.org/foundation/policies/conduct

[3] https://apachenifi.slack.com

Re: UI: feedback on the processor 'color' in NiFi 1.0

[Russell Bateman]

After thinking on it a bit, I agree that Manish' suggestion could be a 
good idea as an option (the way /additionalDetails.html/ is an option). 
It would be easier if they were /.png/ files rather than formal icon 
files only with a "width x length" limit.


My two cents,

Russ

On 09/28/2016 12:57 AM, Manish Gupta 8 wrote:


I think one of the things that will really help in complex data flow 
from UI perspective is “colored icons” on each processor. Not sure if 
this already part of 1.0, but from my experience, icons definitely 
help a lot in quickly understanding complex flows. Those icons can be 
fixed (embedded within the nar) or may be dynamic (user defined icon 
file for different processors) – just a suggestion.


Regards,

Manish

*From:*Andrew Grande [mailto:apere...@gmail.com]
*Sent:* Tuesday, September 20, 2016 10:40 PM
*To:* users@nifi.apache.org
*Subject:* Re: UI: feedback on the processor 'color' in NiFi 1.0

No need to go wild, changing processor colors should be enough, IMO. 
PG and RPG are possible candidates, but they are different enough 
already, I guess.


What I heard quite often was to differentiate between regular 
processors, incoming sources of data and out only (data producers?). 
Maybe even with a shape?


Andrew

On Tue, Sep 20, 2016, 12:35 PM Rob Moran > wrote:


Good points. I was thinking a label would be tied to the group of
components to which it was applied, but that could also introduce
problems as things move and are added to a flow.

So would you all expect to be able to change the color of every
component type, or just processors?

Andrew - your comment about coloring terminators red is
interesting as well. What are some other parts of a flow you might
use color to identify? Along with backpressure, we could explore
other ways to call these things out so users do not come up with
their own methods. Perhaps there are layer options, like on a map
(e.g., "show terrain" or "show traffic").


Rob

On Tue, Sep 20, 2016 at 11:23 AM, Andrew Grande
mailto:apere...@gmail.com>> wrote:

I agree. Labels are great for grouping, beyond PGs. Processor
colors individually add value. E.g. flow terminator colored in
red was a very common pattern I used. Besides, labels are not
grouped with components, so moving things and re-arranging is
a pain.

Andrew

On Tue, Sep 20, 2016, 11:21 AM Joe Skora mailto:jsk...@gmail.com>> wrote:

Rob,

The labelling functionality you described sounds very
useful in general.  But, I miss the processor color too.

I think labels are really useful for identifying groups of
components and areas in the flow, but I worry that needing
to use them in volume for processor coloring will increase
the API and browser canvas load for elements that don't
actually affect the flow.

On Tue, Sep 20, 2016 at 10:40 AM, Rob Moran
mailto:rmo...@gmail.com>> wrote:

What if we promote the use of Labels as a way to
highlight things. We could add functionality to expand
their usefulness as a way to highlight things on the
canvas. I believe that is their intended use.

Today you can create a label and change its color to
highlight single or multiple components. Even better
you can do it for any component (not just processors).

To expand on functionality, I'm imagining a context
menu and palette action to "Label" a selected
component or components. This would prompt a user to
pick a background and add text which would place a
label around everything once it's applied.


Rob

On Mon, Sep 19, 2016 at 6:42 PM, Jeff
mailto:jtsw...@gmail.com>> wrote:

I was thinking, in addition to changing the color
of the icon on the processor, that the color of
the drop shadow could be changed as well.  That
would provide more contrast, but preserve
readability, in my opinion.

On Mon, Sep 19, 2016 at 6:39 PM Andrew Grande
mailto:apere...@gmail.com>>
wrote:

Hi All,

Rolling with UI feedback threads. This time
I'd like to discuss how NiFi 'lost' its
ability to change processor boxes color. I.e.
as you can see from a screenshot attached, it
does change color for the processor in the
flow overview panel, but the processor itself
only changes the icon in the top-left o

Re: Processor to send flowfile to two different destinations?

[Russell Bateman]

Thanks, Andrew. I should have tried that first.

On 09/13/2016 04:19 PM, Andrew Grande wrote:


It's actually very simple - connect a processor output to 2 or more 
other processors or ports, use the 'success' relationship if prompted 
to choose from multiple.


Andrew


On Tue, Sep 13, 2016, 6:14 PM Russell Bateman 
<mailto:russell.bate...@perfectsearchcorp.com>> wrote:


/DuplicateFlowFile/ sends multiple copies all to the
Successrelationship, mostly for testing load.

How does one legitimately send two copies of the same flowfile to
different relationships to result essentially in two parallel
workflows? (I'm probably missing some simple understanding here...)

Russ

Processor to send flowfile to two different destinations?

[Russell Bateman]

/DuplicateFlowFile/ sends multiple copies all to the 
Successrelationship, mostly for testing load.


How does one legitimately send two copies of the same flowfile to 
different relationships to result essentially in two parallel workflows? 
(I'm probably missing some simple understanding here...)


Russ

Re: Failed to amend logback.xml to force reporting task log entries to different file...

[Russell Bateman]

Incidentally, I took the information on how to do this from:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.controller.ControllerStatusReportingTask/additionalDetails.html

On Thu, Jul 14, 2016 at 11:02 AM, Russell Bateman <
russell.bate...@perfectsearchcorp.com> wrote:

> First, I'm using ControllerStatusReportingTask whose output I see in
> *nifi-app.log.* This seems to do report what I want to see. However, I'd
> like the status reports to go out to a different log file. I bounced NiFi
> after enhancing the log-back file:
>
> I inserted this near the top of *conf/logback.xml*, just after the
> definition of the BOOTSTRAP_FILE rolling appender:
>
> 
>  class="ch.qos.logback.core.rolling.RollingFileAppender">
>   logs/nifi-controller-status.log
>class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
> *
> ./logs/nifi-controller-status_%d.log*
> 30
>   
>   
> %date %level [%thread] %logger{40} %msg%n
>   
> 
>
> ...and I inserted this nearer the bottom of *conf/logback.xml:*
>
> 
>  level="INFO" additivity="false">
>   
> 
>
> The status still comes out in *nifi-app.log*; *nifi-controller-status.log,
> *which is created in the *logs* subdirectory, remains 0-length. What am I
> missing here?
>
> Thanks,
> Russ
>

Failed to amend logback.xml to force reporting task log entries to different file...

[Russell Bateman]

First, I'm using ControllerStatusReportingTask whose output I see in
*nifi-app.log.* This seems to do report what I want to see. However, I'd
like the status reports to go out to a different log file. I bounced NiFi
after enhancing the log-back file:

I inserted this near the top of *conf/logback.xml*, just after the
definition of the BOOTSTRAP_FILE rolling appender:



  logs/nifi-controller-status.log
  
*
./logs/nifi-controller-status_%d.log*
30
  
  
%date %level [%thread] %logger{40} %msg%n
  


...and I inserted this nearer the bottom of *conf/logback.xml:*



  


The status still comes out in *nifi-app.log*;
*nifi-controller-status.log, *which
is created in the *logs* subdirectory, remains 0-length. What am I missing
here?

Thanks,
Russ

Best practice: template subdirectory?

[Russell Bateman]

What's everybody using as a subdirectory for templates copied over from 
one's own and other folks' work? Googling, I haven't found any opinions 
on this. I know that to import a template I can just browse for it, but 
where are some of you keeping them? Surely there's a smarter place than 
Downloads or Desktop?

Re: How to effectively log the data flow in NiFi?

[Russell Bateman]

Mark,

When did this happen? I'm looking at (an as yet unchanged from original 
download) /co//nf///logback.xml/ in 0.6.1 and I'm seeing:


  - no string of characters "org.apache.nifi.processors" and
  - INFO all over in places like  level="INFO"/>


What am I missing? Was your comment valid from 0.7.0-onward?

Russ

On 06/10/2016 10:53 AM, Mark Payne wrote:

Hi Huagen,

This is typically the type of logging you will see in NiFi. Each 
processor will generally log at an INFO
level what it is doing for each FlowFile. Unfortunately, though, this 
can become extremely verbose,
and many people want that logging toned down, so in the master branch 
of NiFi, the minimum log
level for processors is set to WARN instead of INFO. You can change 
this by updating the conf/logback.xml
and setting the log level of "org.apache.nifi.processors" to INFO 
instead of WARN.


The main reason that we have changed the default log level though is 
that in NiFi, it is very rare to need
to go through all of the tedious labor of grepping through logs. 
Instead, the recommended approach is to
use the Data Provenance features [1]. This will allow you to search 
for data of interest to you and see exactly
how it was processed throughout the flow. Additionally, this provides 
you access to the FlowFile attributes as
they were each step along the way, and the ability to click-to-content 
to see how the data looked at that point
in the flow as well. It also allows you to visualize what happened to 
the data, even if it is split into many smaller
pieces of data (potentially with different filenames) or merged 
together with other data, so that you don't have

to worry about the filename.

I hope this helps!

-Mark

[1] 
http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#data-provenance



On Jun 10, 2016, at 11:58 AM, Huagen peng > wrote:


Hi,

I would like to learn about some better practices on logging.  Here 
is what I would imagine in an ideal log for a flow like fetching 
files from SFTP, processing the files in certain way, and then saving 
the file to the disk.  In the log, I would see that the SFTP step is 
triggered, with the filename in clear text.  I would then see that 
the file processing is started, and that the file is saved.  If there 
are errors, I would also see the errors in the log as well.


How would I achieve that or something close to that in NiFi?

Thanks,

Huagen