Re: ActiveMQ trust store issues

[Joe Gresock]

Thanks Michael!  I'll try it out.

On Wed, Jan 11, 2017 at 5:19 PM, Michael Moser  wrote:

> Joe,
>
> Have you seen https://issues.apache.org/jira/browse/NIFI-3230?  You might
> try the latest code on the master branch to see if it fixes your problem.
> The JIRA ticket also offers a workaround by using a failover URI.
>
> -- Mike
>
>
> On Wed, Jan 11, 2017 at 8:20 PM, Joe Gresock  wrote:
>
> > Hi folks,
> >
> > I'm using PutJMS to try to send messages to an ActiveMQ broker over
> SSL.  I
> > verified that the trust store referenced in my ssl-context controller
> > service does indeed contain the issuer DN of the broker's certificate,
> but
> > I get the error "PKIX path building failed:
> > sun.security.provider.certpath.SunCertPathBuilderException: unable to
> find
> > valid certification path to requested target".
> >
> > On a whim, I tried adding the truststore location and password to
> > bootstrap.conf:
> > java.arg.17=-Djavax.net.ssl.trustStore=...
> > java.arg.18=-Djavax.net.ssl.trustStorePassword=...
> >
> > And this time the SSL connection actually worked.  Therefore, it looks
> like
> > somehow the ActiveMQ connection factory is not accepting my trust store
> > information from my controller service.  Has anyone else observed this
> > behavior?
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty.  I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want.  I can
> do
> > all this through him who gives me strength.*-Philippians 4:12-13*
> >
>



-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.*-Philippians 4:12-13*

Re: ActiveMQ trust store issues

[Michael Moser]

Joe,

Have you seen https://issues.apache.org/jira/browse/NIFI-3230?  You might
try the latest code on the master branch to see if it fixes your problem.
The JIRA ticket also offers a workaround by using a failover URI.

-- Mike


On Wed, Jan 11, 2017 at 8:20 PM, Joe Gresock  wrote:

> Hi folks,
>
> I'm using PutJMS to try to send messages to an ActiveMQ broker over SSL.  I
> verified that the trust store referenced in my ssl-context controller
> service does indeed contain the issuer DN of the broker's certificate, but
> I get the error "PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to requested target".
>
> On a whim, I tried adding the truststore location and password to
> bootstrap.conf:
> java.arg.17=-Djavax.net.ssl.trustStore=...
> java.arg.18=-Djavax.net.ssl.trustStorePassword=...
>
> And this time the SSL connection actually worked.  Therefore, it looks like
> somehow the ActiveMQ connection factory is not accepting my trust store
> information from my controller service.  Has anyone else observed this
> behavior?
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.*-Philippians 4:12-13*
>

Re: MergeContent error on restart - not the most recent flow file in this session

[bmichaud]

I just realized that I had to recopy the lib folder from the new version
(1.1.1), as all the stack traces were referencing 1.1.0 code. 

This change where I cannot link symbolically to the latest version of the
software is already causing problems. I have to change my deployment and
upgrade process because of it.





--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/MergeContent-error-on-restart-not-the-most-recent-flow-file-in-this-session-tp14436p14440.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: MergeContent error on restart - not the most recent flow file in this session

[bmichaud]

Mark Payne wrote
> I've not seen this issue, but I created a JIRA [1] to address it. Will try
> to replicate it locally and see
> if I can understand what is happening. When this happens, do you see the
> processor continually
> spewing this Exception, rolling back, and trying again?

Thank you. I do see the same exception many times, repeatedly. And it
finally failed like this:

2017-01-11 14:06:47,157 INFO [StandardProcessScheduler Thread-8]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully swapped out 1
FlowFiles from
FlowFileQueue[id=536e3794-0157-1000-8f4b-303d154b73eb] to Swap File
/app_2/runtime/nifi/./flowfile_repository/swap/1484165206865-536e3794-0157-1000-8f
4b-303d154b73eb-3b22b90e-a691-4e8b-9474-64159092d8a7.swap
2017-01-11 14:06:47,291 INFO [StandardProcessScheduler Thread-6]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully swapped out 1
FlowFiles from
FlowFileQueue[id=e6af072c-fda8-4596-3baa-67a4f0dd0691] to Swap File
/app_2/runtime/nifi/./flowfile_repository/swap/1484165207150-e6af072c-fda8-4596-3b
aa-67a4f0dd0691-e7e015d9-b13a-4542-acb9-c143dfafc1eb.swap
2017-01-11 14:06:47,334 INFO [StandardProcessScheduler Thread-8]
o.a.n.c.r.WriteAheadFlowFileRepository Successfully swapped out 1
FlowFiles from
FlowFileQueue[id=536e3794-0157-1000-8f4b-303d154b73eb] to Swap File
/app_2/runtime/nifi/./flowfile_repository/swap/1484165207232-536e3794-0157-1000-8f4b-303d154b73eb-6fb35722-ef7f-44b6-b930-b875831b6407.swap
2017-01-11 14:06:47,449 INFO [pool-1-thread-1]
o.apache.nifi.controller.FlowController Controller has been terminated
successfully.
2017-01-11 14:06:47,456 INFO [pool-1-thread-1]
o.a.n.w.c.ApplicationStartupContextListener Flow service termination
completed.
2017-01-11 14:06:47,457 INFO [pool-1-thread-1] /nifi-api Closing Spring root
WebApplicationContext
2017-01-11 14:06:47,465 ERROR [StandardProcessScheduler Thread-6]
org.apache.nifi.util.ReflectionUtils Failed while invoking annotated method
'public final void org.apache.nifi.processor.util.bin.BinFiles.resetState()'
with arguments '[]'.
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.8.0_65]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[na:1.8.0_65]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_65]
at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_65]
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotations(ReflectionUtils.java:233)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotation(ReflectionUtils.java:85)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.StandardProcessorNode$2.run(StandardProcessorNode.java:1373)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_65]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[na:1.8.0_65]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_65]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[na:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
Caused by: java.lang.IllegalStateException: Partition is closed
at
org.wali.MinimalLockingWriteAheadLog$Partition.update(MinimalLockingWriteAheadLog.java:945)
~[nifi-write-ahead-log-1.1.0.jar:1.1.0]
at
org.wali.MinimalLockingWriteAheadLog.update(MinimalLockingWriteAheadLog.java:238)
~[nifi-write-ahead-log-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.swapFlowFilesOut(WriteAheadFlowFileRepository.java:318)
~[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.FileSystemSwapManager.swapOut(FileSystemSwapManager.java:135)
~[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.StandardFlowFileQueue.writeSwapFilesIfNecessary(StandardFlowFileQueue.java:582)
~[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.StandardFlowFileQueue.put(StandardFlowFileQueue.java:282)
~[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.repository.StandardProcessSession.rollback(StandardProcessSession.java:959)

Re: MergeContent error on restart - not the most recent flow file in this session

[Mark Payne]

Hi Ben,

I've not seen this issue, but I created a JIRA [1] to address it. Will try to 
replicate it locally and see
if I can understand what is happening. When this happens, do you see the 
processor continually
spewing this Exception, rolling back, and trying again?

This error itself would not result in data loss. However, clearing the queue 
certainly would.

Thanks
-Mark


[1] https://issues.apache.org/jira/browse/NIFI-3329




On Jan 11, 2017, at 1:21 PM, bmichaud 
> wrote:

This error occurred in NiFi 1.1.1 (single node) right after an upgrade from
1.1.0, so I thought it was due to the upgrade, but now I see that it happens
if I restart as well and there is data in the flow.

This MergeContent processor is sorting incoming FlowFiles into bins based on
a correlation attribute and dumping out the accumulated uber-FlowFile after
one hour or 500K messages are accumulated.

Questions:
(1) Should I clear out all the flow files from the flow (terminate intake
and let everything drain out)?
(2) Will data be lost when this error occurs?

Error:
---
2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
o.a.n.processors.standard.MergeContent
MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] Processor
Administratively Yielded for 1 sec due to processing failure
2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] due to uncaught
Exception: org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=07f3d75a-3347-45da-8dd4-fecf775d1dcf,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1484151577298-1971,
container=default, section=947], offset=566526,
length=64],offset=0,name=8551433629757608.tsv,size=64] is not the most
recent version of this FlowFile within this session
(StandardProcessSession[id=133767])
2017-01-11 10:53:39,563 WARN [Timer-Driven Process Thread-9]
o.a.n.c.t.ContinuallyRunProcessorTask
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=07f3d75a-3347-45da-8dd4-fecf775d1dcf,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1484151577298-1971,
container=default, section=947], offset=566526,
length=64],offset=0,name=8551433629757608.tsv,size=64] is not the most
recent version of this FlowFile within this session
(StandardProcessSession[id=133767])
   at
org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1121)
~[nifi-framework-core-1.1.0.jar:1.1.0]
   at
org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1102)
~[nifi-framework-core-1.1.0.jar:1.1.0]
   at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142)
~[na:na]
   at
org.apache.nifi.processor.util.bin.BinManager.offer(BinManager.java:194)
~[na:na]
   at
org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:279)
~[na:na]
   at
org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178)
~[na:na]
   at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
~[nifi-framework-core-1.1.0.jar:1.1.0]
   at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.1.0.jar:1.1.0]
   at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.1.0.jar:1.1.0]
   at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.1.0.jar:1.1.0]
   at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_65]
   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_65]
   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_65]
   at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_65]
   at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_65]
   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_65]
   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]




--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/MergeContent-error-on-restart-not-the-most-recent-flow-file-in-this-session-tp14436.html
Sent from the Apache NiFi Developer List mailing list archive at 
Nabble.com.

ActiveMQ trust store issues

[Joe Gresock]

Hi folks,

I'm using PutJMS to try to send messages to an ActiveMQ broker over SSL.  I
verified that the trust store referenced in my ssl-context controller
service does indeed contain the issuer DN of the broker's certificate, but
I get the error "PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find
valid certification path to requested target".

On a whim, I tried adding the truststore location and password to
bootstrap.conf:
java.arg.17=-Djavax.net.ssl.trustStore=...
java.arg.18=-Djavax.net.ssl.trustStorePassword=...

And this time the SSL connection actually worked.  Therefore, it looks like
somehow the ActiveMQ connection factory is not accepting my trust store
information from my controller service.  Has anyone else observed this
behavior?

-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.*-Philippians 4:12-13*

MergeContent error on restart - not the most recent flow file in this session

[bmichaud]

This error occurred in NiFi 1.1.1 (single node) right after an upgrade from
1.1.0, so I thought it was due to the upgrade, but now I see that it happens
if I restart as well and there is data in the flow. 

This MergeContent processor is sorting incoming FlowFiles into bins based on
a correlation attribute and dumping out the accumulated uber-FlowFile after
one hour or 500K messages are accumulated.

Questions:
(1) Should I clear out all the flow files from the flow (terminate intake
and let everything drain out)?
(2) Will data be lost when this error occurs?

Error:
---
2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
o.a.n.processors.standard.MergeContent
MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] Processor
Administratively Yielded for 1 sec due to processing failure
2017-01-11 10:53:39,561 WARN [Timer-Driven Process Thread-9]
o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
MergeContent[id=b9898788-1c11-45d9-8646-44b1dceb92d5] due to uncaught
Exception: org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=07f3d75a-3347-45da-8dd4-fecf775d1dcf,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1484151577298-1971,
container=default, section=947], offset=566526,
length=64],offset=0,name=8551433629757608.tsv,size=64] is not the most
recent version of this FlowFile within this session
(StandardProcessSession[id=133767])
2017-01-11 10:53:39,563 WARN [Timer-Driven Process Thread-9]
o.a.n.c.t.ContinuallyRunProcessorTask
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=07f3d75a-3347-45da-8dd4-fecf775d1dcf,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1484151577298-1971,
container=default, section=947], offset=566526,
length=64],offset=0,name=8551433629757608.tsv,size=64] is not the most
recent version of this FlowFile within this session
(StandardProcessSession[id=133767])
at
org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1121)
~[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1102)
~[nifi-framework-core-1.1.0.jar:1.1.0]
at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142)
~[na:na]
at
org.apache.nifi.processor.util.bin.BinManager.offer(BinManager.java:194)
~[na:na]
at
org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:279)
~[na:na]
at
org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178)
~[na:na]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099)
~[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.1.0.jar:1.1.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_65]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_65]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_65]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_65]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]




--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/MergeContent-error-on-restart-not-the-most-recent-flow-file-in-this-session-tp14436.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Where does javax.net.debug output go?

[Joe Gresock]

No, we have not called setOut anywhere before now, and no customizations in
logging.properties.  I'll see if I can reproduce this with a vanilla
installation of NiFi and I'll get back to you.

On Wed, Jan 11, 2017 at 7:51 PM, Andy LoPresto  wrote:

> Have you had to call System.setOut() anywhere else in the code base?
> Anything in your custom processors that redirects output streams? Any
> custom configurations in $JAVA_HOME/jre/lib/logging.properties?
>
> This isn’t something I’ve encountered before — we get questions about log
> threshold, but I’ve never heard of the standard out/error streams not
> writing correctly, or the entire nifi-bootstrap.log being empty.
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jan 11, 2017, at 11:38 AM, Joe Gresock  wrote:
>
> CentOS 6.8 with openjdk version "1.8.0_111".  I did also try running nifi
> directly using nifi.sh start, to the same effect.
>
> I was able to work around the issue by calling System.setOut(...) inside
> one of my custom processors, so I at least have the debugging info I need.
> However, it would still be nice to get the standard out working again.
>
> On Wed, Jan 11, 2017 at 7:19 PM, Andy LoPresto 
> wrote:
>
> Joe,
>
> Sorry you are experiencing difficulty with this. SSL debugging is a
> feature I use often. You should see this output in logs/nifi-bootstrap.log.
> Ensure you do not have another Java arg with a conflicting argument number
> in bootstrap.conf (although as you have put it as 16, I think you should be
> safe against a default install). If it is not getting *any* output, I’d
> toggle that Java arg and restart. If it’s still nothing, I’d try running
> NiFi directly, rather than as a service.
>
> Can you please let us know what OS, hardware, and Java version you are
> using? Thanks.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jan 11, 2017, at 9:48 AM, Joe Gresock  wrote:
>
> Hi folks,
>
> I'm trying to turn on SSL verbose logging using
> java.arg.16=-Djavax.net.debug=ssl,handshake in my bootstrap.conf (nifi
> version 1.1.0).  As far as I can remember, this output goes to either
> stdout or stderr (I forget which).
>
> As a result, I'm trying to find the output in my nifi-bootstrap.log file,
> but this log file doesn't appear to be getting any output whatsoever (not
> even at startup time).  I am using the original logback.xml, with respect
> to the BOOTSTRAP_FILE appender and related loggers.
>
> Does anyone know where the javax.net.debug logging should go?  Also, should
> I see anything in my nifi-bootstrap.log file?
>
> I am starting nifi using "sudo service nifi start".
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.*-Philippians 4:12-13*
>
>
>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.*-Philippians 4:12-13*
>
>
>


-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.*-Philippians 4:12-13*

Re: Where does javax.net.debug output go?

[Andy LoPresto]

Have you had to call System.setOut() anywhere else in the code base? Anything 
in your custom processors that redirects output streams? Any custom 
configurations in $JAVA_HOME/jre/lib/logging.properties?

This isn’t something I’ve encountered before — we get questions about log 
threshold, but I’ve never heard of the standard out/error streams not writing 
correctly, or the entire nifi-bootstrap.log being empty.

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jan 11, 2017, at 11:38 AM, Joe Gresock  wrote:
> 
> CentOS 6.8 with openjdk version "1.8.0_111".  I did also try running nifi
> directly using nifi.sh start, to the same effect.
> 
> I was able to work around the issue by calling System.setOut(...) inside
> one of my custom processors, so I at least have the debugging info I need.
> However, it would still be nice to get the standard out working again.
> 
> On Wed, Jan 11, 2017 at 7:19 PM, Andy LoPresto  > wrote:
> 
>> Joe,
>> 
>> Sorry you are experiencing difficulty with this. SSL debugging is a
>> feature I use often. You should see this output in logs/nifi-bootstrap.log.
>> Ensure you do not have another Java arg with a conflicting argument number
>> in bootstrap.conf (although as you have put it as 16, I think you should be
>> safe against a default install). If it is not getting *any* output, I’d
>> toggle that Java arg and restart. If it’s still nothing, I’d try running
>> NiFi directly, rather than as a service.
>> 
>> Can you please let us know what OS, hardware, and Java version you are
>> using? Thanks.
>> 
>> 
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com  
>> >*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
>> On Jan 11, 2017, at 9:48 AM, Joe Gresock  wrote:
>> 
>> Hi folks,
>> 
>> I'm trying to turn on SSL verbose logging using
>> java.arg.16=-Djavax.net.debug=ssl,handshake in my bootstrap.conf (nifi
>> version 1.1.0).  As far as I can remember, this output goes to either
>> stdout or stderr (I forget which).
>> 
>> As a result, I'm trying to find the output in my nifi-bootstrap.log file,
>> but this log file doesn't appear to be getting any output whatsoever (not
>> even at startup time).  I am using the original logback.xml, with respect
>> to the BOOTSTRAP_FILE appender and related loggers.
>> 
>> Does anyone know where the javax.net.debug logging should go?  Also, should
>> I see anything in my nifi-bootstrap.log file?
>> 
>> I am starting nifi using "sudo service nifi start".
>> 
>> --
>> I know what it is to be in need, and I know what it is to have plenty.  I
>> have learned the secret of being content in any and every situation,
>> whether well fed or hungry, whether living in plenty or in want.  I can do
>> all this through him who gives me strength.*-Philippians 4:12-13*
>> 
>> 
>> 
> 
> 
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.*-Philippians 4:12-13*



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Where does javax.net.debug output go?

[Joe Gresock]

CentOS 6.8 with openjdk version "1.8.0_111".  I did also try running nifi
directly using nifi.sh start, to the same effect.

I was able to work around the issue by calling System.setOut(...) inside
one of my custom processors, so I at least have the debugging info I need.
However, it would still be nice to get the standard out working again.

On Wed, Jan 11, 2017 at 7:19 PM, Andy LoPresto  wrote:

> Joe,
>
> Sorry you are experiencing difficulty with this. SSL debugging is a
> feature I use often. You should see this output in logs/nifi-bootstrap.log.
> Ensure you do not have another Java arg with a conflicting argument number
> in bootstrap.conf (although as you have put it as 16, I think you should be
> safe against a default install). If it is not getting *any* output, I’d
> toggle that Java arg and restart. If it’s still nothing, I’d try running
> NiFi directly, rather than as a service.
>
> Can you please let us know what OS, hardware, and Java version you are
> using? Thanks.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jan 11, 2017, at 9:48 AM, Joe Gresock  wrote:
>
> Hi folks,
>
> I'm trying to turn on SSL verbose logging using
> java.arg.16=-Djavax.net.debug=ssl,handshake in my bootstrap.conf (nifi
> version 1.1.0).  As far as I can remember, this output goes to either
> stdout or stderr (I forget which).
>
> As a result, I'm trying to find the output in my nifi-bootstrap.log file,
> but this log file doesn't appear to be getting any output whatsoever (not
> even at startup time).  I am using the original logback.xml, with respect
> to the BOOTSTRAP_FILE appender and related loggers.
>
> Does anyone know where the javax.net.debug logging should go?  Also, should
> I see anything in my nifi-bootstrap.log file?
>
> I am starting nifi using "sudo service nifi start".
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.*-Philippians 4:12-13*
>
>
>


-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.*-Philippians 4:12-13*

Re: Where does javax.net.debug output go?

[Andy LoPresto]

Joe,

Sorry you are experiencing difficulty with this. SSL debugging is a feature I 
use often. You should see this output in logs/nifi-bootstrap.log. Ensure you do 
not have another Java arg with a conflicting argument number in bootstrap.conf 
(although as you have put it as 16, I think you should be safe against a 
default install). If it is not getting *any* output, I’d toggle that Java arg 
and restart. If it’s still nothing, I’d try running NiFi directly, rather than 
as a service.

Can you please let us know what OS, hardware, and Java version you are using? 
Thanks.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jan 11, 2017, at 9:48 AM, Joe Gresock  wrote:
> 
> Hi folks,
> 
> I'm trying to turn on SSL verbose logging using
> java.arg.16=-Djavax.net.debug=ssl,handshake in my bootstrap.conf (nifi
> version 1.1.0).  As far as I can remember, this output goes to either
> stdout or stderr (I forget which).
> 
> As a result, I'm trying to find the output in my nifi-bootstrap.log file,
> but this log file doesn't appear to be getting any output whatsoever (not
> even at startup time).  I am using the original logback.xml, with respect
> to the BOOTSTRAP_FILE appender and related loggers.
> 
> Does anyone know where the javax.net.debug logging should go?  Also, should
> I see anything in my nifi-bootstrap.log file?
> 
> I am starting nifi using "sudo service nifi start".
> 
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.*-Philippians 4:12-13*



signature.asc
Description: Message signed with OpenPGP using GPGMail

Where does javax.net.debug output go?

[Joe Gresock]

Hi folks,

I'm trying to turn on SSL verbose logging using
java.arg.16=-Djavax.net.debug=ssl,handshake in my bootstrap.conf (nifi
version 1.1.0).  As far as I can remember, this output goes to either
stdout or stderr (I forget which).

As a result, I'm trying to find the output in my nifi-bootstrap.log file,
but this log file doesn't appear to be getting any output whatsoever (not
even at startup time).  I am using the original logback.xml, with respect
to the BOOTSTRAP_FILE appender and related loggers.

Does anyone know where the javax.net.debug logging should go?  Also, should
I see anything in my nifi-bootstrap.log file?

I am starting nifi using "sudo service nifi start".

-- 
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.*-Philippians 4:12-13*

Re: NiFi cannot start due to log permissions error

[bmichaud]

Andy LoPresto-2 wrote
> The illegal key size error is almost certainly due to the length of the
> keystore/truststore password, but the ideal solution is not to decrease
> the password length, but rather to either install the Unlimited Strength
> Jurisdiction Policy files if possible, and/or switch to using a JKS
> keystore rather than PKCS12. PKCS12 without the USJ policies limit the
> keystore password length to 7 characters, which is *not* sufficiently
> strong against modern computing capability.

Thank you very much for that. I was able to resolve the SSL issues. 



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-cannot-start-due-to-log-permissions-error-tp14413p14430.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: local NiFi and MiNiFi - yml generation errors

[Bryan Rosander]

Hi Pushkar,

In the tutorial you link to, it looks like before generating the template,
you need to select just subsection of the flow shown.

Currently MiNiFi does not support Input or Output ports in the root process
group so if you were to create a template from the whole flow, it would be
invalid.

Thanks,
Bryan

On Wed, Jan 11, 2017 at 6:07 AM, Pushkara R  wrote:

> Hi,
>
> I was using this tutorial
>  minifi.html>
> to get started with NiFi and MiNiFi. At step 11 in the linked tutorial, I
> tried creating the yml for the given template. I couldn't find the
> config.sh file in the latest source of minifi-0.2.0
>  which is why I downloaded
> minifi-toolkit-0.1.0
>  0.1.0/minifi-toolkit-0.1.0-bin.tar.gz>
> which has the config.sh.
>
> Executing 'bin/config.sh transform  ' gave the
> following error
> -
> config.sh: JAVA_HOME not set; results may vary
>
> Java home:
> MiNiFi Toolkit home: /home/pushkar/workplace/minifi-toolkit-0.1.0
>
>
> There are validation errors with the template, still outputting YAML but it
> will need to be edited.
> 'Input Ports' in section 'top level' because must be empty in root group as
> external input/output ports are currently unsupported
> -
> Could someone please help me debug this error? I'd also like to ask if the
> linked tutorial is still valid or is it dated?
>
> Thanks
> Pushkar
>

Re: MiNiFi event-based scheduling?

[Andrew Christianson]

All,

Please review/merge pull request 
https://github.com/apache/nifi-minifi-cpp/pull/40 which adds event-based 
scheduling.

Regards,

Andy

P.S. Thanks for the info Mark (missed your email earlier). I did notice the 
bored duration seemed to be implemented in minifi-cpp. Thankfully, we now have 
both. This "true" event scheduler based on native C++ threading constructs 
should get us significantly lower latency than even 10ms.


From: Mark Payne 
Sent: Friday, January 6, 2017 10:59:01 AM
To: dev@nifi.apache.org
Subject: Re: MiNiFi event-based scheduling?

Andy,

Of note - with NiFi, the way that the Timer-Based Scheduling agent works, if 
you set a schedule of "0 secs"
NiFi will determine whether or not there is any work that can be done by the 
processor. If the processor has
no work to do, it is considered "bored" and isn't scheduled again for 10 
milliseconds (this is configurable in
nifi.properties, using the "nifi.bored.yield.duration" property, of which the 
default is 10 milliseconds). This is
designed to avoid hammering the CPU.

Off the top of my head, I am not sure if MiNiFi supports this feature or not, 
though. Any idea, Aldrin?

Thanks
-Mark


> On Jan 6, 2017, at 10:51 AM, Aldrin Piri  wrote:
>
> Great!  Thanks for taking the time to do so, much appreciated.
>
> On Fri, Jan 6, 2017 at 10:48 AM, Andrew Christianson <
> andrew.christian...@nextcentury.com> wrote:
>
>> Got it. Added this as MINIFI-182 with a link to the wiki for design ideas.
>>
>> 
>> From: Aldrin Piri 
>> Sent: Friday, January 6, 2017 10:03:06 AM
>> To: dev
>> Subject: Re: MiNiFi event-based scheduling?
>>
>> Hey Andy,
>>
>> Your inspection would prove correct and it does not appear we have a JIRA
>> as of yet either.  If you would be interested in creating one, please do,
>> otherwise I shall see to it that it gets added.
>>
>> We are still working to get some of the more foundational framework
>> components established but certainly can see how this would be helpful.
>> Definitely should track this so that it isn't lost in the shuffle and the
>> framework can be refactored to support it.
>>
>> Thanks!
>> --aldrin
>>
>> On Fri, Jan 6, 2017 at 9:56 AM, Andrew Christianson <
>> andrew.christian...@nextcentury.com> wrote:
>>
>>> We noticed that MiNiFi currently only supports timer-based scheduling.
>>> Looking at the code, one could set the timer interval to 0, but this
>> would
>>> result in a hot loop burning 100% of a core. As such, there seems to be
>> no
>>> way presently to have flow file records processed as soon as incoming
>>> records populate the queue. Further, it looks like the FlowController
>> code
>>> is hard-coded against the timer-based scheduling agent, and unfortunately
>>> there is no obvious way to plug-in an alternative scheduler.
>>>
>>> Are there any near/mid-term plans to implement event-based scheduling?
>>>
>>> -Andy
>>>
>>

local NiFi and MiNiFi - yml generation errors

[Pushkara R]

Hi,

I was using this tutorial

to get started with NiFi and MiNiFi. At step 11 in the linked tutorial, I
tried creating the yml for the given template. I couldn't find the
config.sh file in the latest source of minifi-0.2.0
 which is why I downloaded
minifi-toolkit-0.1.0

which has the config.sh.

Executing 'bin/config.sh transform  ' gave the
following error
-
config.sh: JAVA_HOME not set; results may vary

Java home:
MiNiFi Toolkit home: /home/pushkar/workplace/minifi-toolkit-0.1.0


There are validation errors with the template, still outputting YAML but it
will need to be edited.
'Input Ports' in section 'top level' because must be empty in root group as
external input/output ports are currently unsupported
-
Could someone please help me debug this error? I'd also like to ask if the
linked tutorial is still valid or is it dated?

Thanks
Pushkar