Re: Using NiFi Expression Language outside of NiFi

[Tim Zimmerman]

Sorry for the delay in responding.
After looking at a few different options it looks like it will be pretty
easy to accomplish what we need using  Java Unified Expression Language
  .





--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: Using NiFi Expression Language outside of NiFi

[Tim Zimmerman]

Thanks to all for the suggestions. I am always impressed at how quickly and
thoughtfully the responses I have gotten here have been.

I will do some more research and post back here on what we end up doing.



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: Using NiFi Expression Language outside of NiFi

[Tim Zimmerman]

Joe,
  Thank you for your thoughts. I have gotten a few suggestions. So, I will
do some research to see what makes the most sense for my current project. If
making a standalone NiFi expression library were to be a thing I would be
happy to help out and give something back.



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: Using NiFi Expression Language outside of NiFi

[Tim Zimmerman]

Thank you. This is where I started last night. 

As far as other nifi-* dependencies,  the nifi-expression-language
dependency has brought in the nifi-core and nifi-api modules, and then some
other transitive dependencies. So, that is not bad.

I have a few other options to look at and I will pursue this more over the
weekend and see how it goes.



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: Using NiFi Expression Language outside of NiFi

[Tim Zimmerman]

Spring Expression Language is on my list to be evaluated. I am just putting
together the thoughts for this new functionality in an existing process.

I looked to NiFi EL first just because we are already using NiFi and some
users/developers are already familiar with the syntax.



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Using NiFi Expression Language outside of NiFi

[Tim Zimmerman]

The NiFi Expression Language is quite powerful. I am working on another
project where I would love to have this sort of functionality and am
wondering how hard it would be to use nifi-expression-language outside of
NiFi, in some other Java app.

I have looked at some other alternatives (e.g. JEXL, JUEL) but we are
already using NiFi and so are familiar with the syntax. And it seems more
powerful than some of the alternatives.

How feasible is this idea?



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: Additional configuration properties for DBCPConnectionPool

[Tim Zimmerman]

Matt,
  Thank you, and I apologize , I forgot to provide the version number. I am
using 1.5.0. I have modified the DBCPConnectionPool to expose a few
properties that I think will help.

  I will create an Improvement Jira after I confirm the changes that I made
truly help me resolve the issue.



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: Additional configuration properties for DBCPConnectionPool

[Tim Zimmerman]

Matt,
  Thank you for the quick response. I do not think these are connection
properties but properties of the connection pool. The properties I am
looking to set are the BasicDataSource Configuration parameters such as
minIdle and minEvictableIdleTimeMillis.

  I do use the validation query but the problem is the connection to the
database is still valid while, I may get this wrong but in layman's terms,
the kerberos authentication for the user who has connected is "stale". The
result is, NiFi is able to query the database, create temp tables etc but
when it attempts to perform the bulk insert of a file on a third (NAS)
server it fails. I suppose my validation query could be an attempt at some
bulk insert but that seems a little inefficient.

  Right now I have a test set up with four connection pools and four flows.
Two ran for some time, maybe a week but are now continually failing. The
other two continue to work. The difference is that the two that continue to
work have either had the pool reduced to zero at some point due to idle
connections being closed or the connections were never allowed to idle i.e.
executing the bulk insert every 10 seconds.

  The two that are failing were allowed to maintain a minimum of one
connection and ran less frequently. One runs every three hours the other
once a week. While the bulk insert for these fails, querying the database
and creating/dropping tables continues to succeed.



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Additional configuration properties for DBCPConnectionPool

[Tim Zimmerman]

Without going into too much detail (unless it would be useful). I am facing
an issue where long standing connections are eventually failing when
SQLServer attempts to bulk insert a file from NAS (we are using
integratedSecurity and the SQLServer is configured with kerberos
authentication and delegation). The process does work but at some point the
bulk insert is denied access to the file and it appears to be an issue with
the kerberos ticket becoming stale. 

The solution seems to be to time out connections that have been idle for
some time and allow the pool to reduce to zero connections. This forces a
new connection and authorization when needed. The connection seems to work
for as long as needed and after it is idle for some time it is removed from
the pool.

These additional configurations are available in the underlying
DBCPConnectionPool but are not exposed through the service configuration. Is
there a specific reason they have not been exposed? I have downloaded the 
source and made the changes myself but before I go any further I would like
to make sure I am not being too clever for my own good.

Also, if there is not a specific reason they have not been exposed, would
this be something that would be considered for a future release?



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/

Re: Importing template into 1.2.0

[Tim Zimmerman]

Andy,

  The processor is EvaluateJsonPath. The error message is :

Multiple versions of org.apache.nifi.processors.standard.EvaluateJsonPath exist

When I try to drop a new processor on the canvase I can see there are in fact 
two version of EvaluateJsonPath 0.0.1 and 1.2.0. I did not look at all the 
processors but I see 0.0.1 and 1.2.0 versions for quite a few of the processors.


  I thought I could add the bundle information to the template. But thanks for 
the suggestion of moving the flow.xml.gz. I will do that. Probably a lot easier 
too!


Thanks,

  tim


From: Andy LoPresto 
Sent: Thursday, May 11, 2017 11:52:05 AM
To: users@nifi.apache.org
Subject: Re: Importing template into 1.2.0

Hi Tim.

What processor has multiple versions and which is the desired target? As your 
1.2.0 installation is a clean install, I would actually recommend just 
exporting the complete flow.xml.gz file from your existing 1.1.2 instance and 
placing it in the $NIFI_HOME/conf directory for the 1.2.0 instance.

Bryan Bende also wrote a good article describing the feature and some of the 
background if you haven’t seen it yet [1].

[1] 
http://bryanbende.com/development/2017/05/10/apache-nifi-1-2-0-component-versioning


Andy LoPresto
alopre...@apache.org<mailto:alopre...@apache.org>
alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On May 11, 2017, at 4:47 PM, Tim Zimmerman 
mailto:iceman...@hotmail.com>> wrote:

First thanks for the 1.2.0 release. There are a few things I have been looking 
forward to (wait/notify).

Now, I have a question. I was running 1.1.2 locally and experimenting with a 
few ideas. After creating a template and exporting one of those ideas, I 
unpacked 1.2.0 and started it up.

I am able to import the template fine but when I try to place it on the canvas 
I get dialog that there is more than one version of the processor on the path 
and the template does not appear.

I understand this is due to the new versioning and I thought maybe if I just 
hacked in the  element I could get this template working but I did that 
and tried again but it still does not seem to be working.

How should I be migrating 1.1.2 work into 1.2.0?

Re: Importing template into 1.2.0

[Tim Zimmerman]

Matt,

  Thanks for the details. I was aware of the new support for multiple versions 
(and looking forward to the Registry). I just thought I could manually update 
the exported xml.


  Copying over the flow.xml.gz worked and was much easier.


Thanks again,

  tim


From: Matt Gilman 
Sent: Thursday, May 11, 2017 12:00:32 PM
To: users@nifi.apache.org
Subject: Re: Importing template into 1.2.0

Tim,

Just wanted to provide a few more details. Prior to 1.2.0 the flow and 
templates did not contain any details about the NAR (bundle) it was packaged 
in. In 1.2.0, since your template did not contain any bundle details NiFi 
attempted to find a compatible Processor (matched by class name only). If the 
instance detects that more than one Processor exists with that class name you 
will receive the error you described.

Re-exporting the template in 1.2.0 will include the relevant bundle details for 
future use.

Hope this helps

Matt

On Thu, May 11, 2017 at 7:52 PM, Andy LoPresto 
mailto:alopre...@apache.org>> wrote:
Hi Tim.

What processor has multiple versions and which is the desired target? As your 
1.2.0 installation is a clean install, I would actually recommend just 
exporting the complete flow.xml.gz file from your existing 1.1.2 instance and 
placing it in the $NIFI_HOME/conf directory for the 1.2.0 instance.

Bryan Bende also wrote a good article describing the feature and some of the 
background if you haven’t seen it yet [1].

[1] 
http://bryanbende.com/development/2017/05/10/apache-nifi-1-2-0-component-versioning


Andy LoPresto
alopre...@apache.org<mailto:alopre...@apache.org>
alopresto.apa...@gmail.com<mailto:alopresto.apa...@gmail.com>
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On May 11, 2017, at 4:47 PM, Tim Zimmerman 
mailto:iceman...@hotmail.com>> wrote:

First thanks for the 1.2.0 release. There are a few things I have been looking 
forward to (wait/notify).

Now, I have a question. I was running 1.1.2 locally and experimenting with a 
few ideas. After creating a template and exporting one of those ideas, I 
unpacked 1.2.0 and started it up.

I am able to import the template fine but when I try to place it on the canvas 
I get dialog that there is more than one version of the processor on the path 
and the template does not appear.

I understand this is due to the new versioning and I thought maybe if I just 
hacked in the  element I could get this template working but I did that 
and tried again but it still does not seem to be working.

How should I be migrating 1.1.2 work into 1.2.0?

Importing template into 1.2.0

[Tim Zimmerman]

First thanks for the 1.2.0 release. There are a few things I have been looking 
forward to (wait/notify).


Now, I have a question. I was running 1.1.2 locally and experimenting with a 
few ideas. After creating a template and exporting one of those ideas, I 
unpacked 1.2.0 and started it up.


I am able to import the template fine but when I try to place it on the canvas 
I get dialog that there is more than one version of the processor on the path 
and the template does not appear.


I understand this is due to the new versioning and I thought maybe if I just 
hacked in the  element I could get this template working but I did that 
and tried again but it still does not seem to be working.


How should I be migrating 1.1.2 work into 1.2.0?

Re: Issues with debugging

[Tim Zimmerman]

Pierre,

  Thanks, yes I did make sure I download the corresponding source.  And even 
tried to match up the stacktrace with the source and it all seemed to tie 
together.


  I just responded to Bryan that this morning all is working as expected. The 
issue may have been with NetBeans. I waited quite some time after connecting 
but I am thinking now that it never actually submitted the breakpoints.


  Regardless, I do appreciate everyone's input and I am happily back to working 
on my flows.


Thanks again,

  tim


From: Pierre Villard 
Sent: Thursday, April 27, 2017 1:33:32 AM
To: users@nifi.apache.org
Subject: Re: Issues with debugging

Hi,

Dummy question but... are you sure the code of the running NiFi is exactly 
matching the code where you specified breakpoints? If this is not matching, 
that could be the reason.

Re: Issues with debugging

[Tim Zimmerman]

Bryan,

  Thanks for the quick response. Yes I thought it quite strange myself. I was 
trying to debug a processor rather than any bootstrap process. I am pretty sure 
I was connected to the correct process. I did not see any additional java 
processes (other than my IDE) and when I disconnected I saw output in NiFi 
indicating that it was listening to for connections on 8187.


  This morning I tried to reproduce everything I had done yesterday and it 
works using either my port or the original port regardless of whether I suspend 
the process on start up or not. Basically, it all works as expected this 
morning.


  I don't have an explanation for the change in behavior. Yesterday I did 
restart my IDE (NetBeans) but that did not seem to make any difference. This 
morning I did not restart anything (just put my laptop to sleep last night) and 
it all seems to work.


  I apologize for submitting before I tested it again this morning. And I 
sincerely appreciate the help.


tim






From: Bryan Rosander 
Sent: Thursday, April 27, 2017 1:22:24 AM
To: users@nifi.apache.org
Subject: Re: Issues with debugging

Hey Tim,

That's pretty strange that your breakpoints aren't being hit.  In the past when 
things like that have happened to me, it's usually due to me successfully 
connecting but to the wrong JVM.

If you're trying to debug code related to the bootstrap process itself (RunNiFi 
or anything it calls), you'd need to put the debug arg in nifi.sh or nifi.bat 
depending on OS, not in bootstrap.conf.  Our bootstrap process runs in a 
separate JVM from NiFi.

If that isn't the issue, you could run jps  to list all Java processes and be 
sure that only the NiFi instance(s) you expect are running are listed.

If that doesn't show anything unexpected, would you mind setting suspend=y in 
the above property and restarting NiFi?  Having suspend=y will make the NiFi 
JVM wait on a debugger connection before proceeding with startup which would 
rule out things like accidentally connecting to another local NiFi or Java 
process, etc.

Thanks,
Bryan

On Thu, Apr 27, 2017 at 9:11 AM, Tim Zimmerman 
mailto:iceman...@hotmail.com>> wrote:

Not sure if this is a bug or misunderstanding on my part.


I was trying to enable debugging so that I could troubleshoot a problem. I 
modified bootstrap.conf to enable debugging. I simply uncommented the 
java.arg.debug line and changed the  port to 8187. I was able to attach to the 
process at 8187 but my breakpoints were never hit. If Ieave the address alone, 
set to 8000 it worked as expected.


The following worked as expected: 
java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000

With address changed to 8187, debugger attached but never stopped on 
breakpoints : 
java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8187

Input is appreciated.

Thanks,
  tim

Issues with debugging

[Tim Zimmerman]

Not sure if this is a bug or misunderstanding on my part.


I was trying to enable debugging so that I could troubleshoot a problem. I 
modified bootstrap.conf to enable debugging. I simply uncommented the 
java.arg.debug line and changed the  port to 8187. I was able to attach to the 
process at 8187 but my breakpoints were never hit. If Ieave the address alone, 
set to 8000 it worked as expected.


The following worked as expected: 
java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000

With address changed to 8187, debugger attached but never stopped on 
breakpoints : 
java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8187

Input is appreciated.

Thanks,
  tim