[jira] [Created] (NIFI-2040) JMSConnectionFactoryProvider - "MQ Client Libraries path" property documentation is incorrect

[Christopher McDermott (JIRA)]

Christopher McDermott created NIFI-2040:
---

 Summary: JMSConnectionFactoryProvider - "MQ Client Libraries path" 
property documentation is incorrect
 Key: NIFI-2040
 URL: https://issues.apache.org/jira/browse/NIFI-2040
 Project: Apache NiFi
  Issue Type: Bug
  Components: Documentation & Website
Affects Versions: 0.6.0
Reporter: Christopher McDermott


The documentation on JMSConnectionFactoryProvider says that the "MQ Client 
Libraries path" path property is optional with 
org.apache.activemq.ActiveMQConnectionFactory provider.  That is incorrect. I 
couple good examples would be helpful.  Showing how to configure an AMQ 
provider would be most helpful since that is probably the most widely used.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (NIFI-2015) Corrupted flow file leads to a wedged flow

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330586#comment-15330586
 ] 

Christopher McDermott edited comment on NIFI-2015 at 6/15/16 5:03 PM:
--

Oleg,

I think the problem could be reproduced by 

# Run a flow with a stopped processor to queue up some flow files 
# Stop NiFi
# Overwrite with 0's all of the allocated blocks of a queued flow file
# Restart NiFi
# Start the stopped processor 

If I can find time I will try this.

In the meantime I am satisfied with your suggestion.

Thanks,
Chris


was (Author: ch...@mcdermott.net):
Oleg,

I think the problem could be reproduced by running a flow with a stopped 
processor to queue up some flow files.  Then truncate, to 0 length, every file 
in the content repository. Then restart the flow.  If I can find time I will 
try this.

In the meantime I am satisfied with your suggestion.

Thanks,
Chris

> Corrupted flow file leads to a wedged flow
> --
>
> Key: NIFI-2015
> URL: https://issues.apache.org/jira/browse/NIFI-2015
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.1
>Reporter: Christopher McDermott
>Assignee: Oleg Zhurakousky
>Priority: Critical
>
> Encountered the following error.  Once encountered the processor in question 
> cannot make any progress.  The incoming connection cannot be emptied or even 
> listed.  Note this issue is not to address the corrupted flow file but rather 
> the handling of the corrupted file.
> 2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] 
> o.a.n.p.attributes.UpdateAttribute
> java.lang.IllegalStateException: Cannot create Provenance Event Record 
> because FlowFile UUID is not set
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
>  ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (NIFI-2015) Corrupted flow file leads to a wedged flow

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330505#comment-15330505
 ] 

Christopher McDermott edited comment on NIFI-2015 at 6/15/16 4:59 PM:
--

Does NiFi rely on the rename idiom to achieve atomic writes?

{code}
fd=open(“file.new”); write(fd, data); close(fd); rename(“file.new”, “file”);
{code}

If so I think this in concert with using ext4 with default options, may be the 
ultimate cause of my problem.  I basically crashed the OS with NiFi running.  
This led to a bunch zero'ed out files which is not unexpected with ext4, at 
least without some specific tuning.

[http://www.pointsoftware.ch/en/4-ext4-vs-ext3-filesystem-and-why-delayed-allocation-is-bad/]


was (Author: ch...@mcdermott.net):
Does NiFi rely on the rename idiom to achieve atomic writes?

{code}
fd=open(“file.new”); write(fd, data); close(fd); rename(“file.new”, “file”);
{code}

If so I think this in concert with using ext4 with default options, may be the 
ultimate cause of my problem.  I basically crashed the OS with NiFi running.  
This led to a bunch of zero length files, which is not unexpected with ext4, at 
least without some specific tuning.

[http://www.pointsoftware.ch/en/4-ext4-vs-ext3-filesystem-and-why-delayed-allocation-is-bad/]

> Corrupted flow file leads to a wedged flow
> --
>
> Key: NIFI-2015
> URL: https://issues.apache.org/jira/browse/NIFI-2015
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.1
>Reporter: Christopher McDermott
>Assignee: Oleg Zhurakousky
>Priority: Critical
> Fix For: 1.0.0, 0.7.0
>
>
> Encountered the following error.  Once encountered the processor in question 
> cannot make any progress.  The incoming connection cannot be emptied or even 
> listed.  Note this issue is not to address the corrupted flow file but rather 
> the handling of the corrupted file.
> 2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] 
> o.a.n.p.attributes.UpdateAttribute
> java.lang.IllegalStateException: Cannot create Provenance Event Record 
> because FlowFile UUID is not set
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
>  ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_45]
> at 
> java.util.concurre

[jira] [Comment Edited] (NIFI-2015) Corrupted flow file leads to a wedged flow

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330586#comment-15330586
 ] 

Christopher McDermott edited comment on NIFI-2015 at 6/14/16 9:00 PM:
--

Oleg,

I think the problem could be reproduced by running a flow with a stopped 
processor to queue up some flow files.  Then truncate, to 0 length, every file 
in the content repository. Then restart the flow.  If I can find time I will 
try this.

In the meantime I am satisfied with your suggestion.

Thanks,
Chris


was (Author: ch...@mcdermott.net):
I think the problem could be reproduced by running a flow with a stopped 
processor to queue up some flow files.  Then truncate, to 0 length, every file 
in the content repository. Then restart the flow.  If I can find time I will 
try this.

> Corrupted flow file leads to a wedged flow
> --
>
> Key: NIFI-2015
> URL: https://issues.apache.org/jira/browse/NIFI-2015
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.1
>Reporter: Christopher McDermott
>Assignee: Oleg Zhurakousky
>Priority: Critical
> Fix For: 1.0.0, 0.7.0
>
>
> Encountered the following error.  Once encountered the processor in question 
> cannot make any progress.  The incoming connection cannot be emptied or even 
> listed.  Note this issue is not to address the corrupted flow file but rather 
> the handling of the corrupted file.
> 2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] 
> o.a.n.p.attributes.UpdateAttribute
> java.lang.IllegalStateException: Cannot create Provenance Event Record 
> because FlowFile UUID is not set
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
>  ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2015) Corrupted flow file leads to a wedged flow

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330586#comment-15330586
 ] 

Christopher McDermott commented on NIFI-2015:
-

I think the problem could be reproduced by running a flow with a stopped 
processor to queue up some flow files.  Then truncate, to 0 length, every file 
in the content repository. Then restart the flow.  If I can find time I will 
try this.

> Corrupted flow file leads to a wedged flow
> --
>
> Key: NIFI-2015
> URL: https://issues.apache.org/jira/browse/NIFI-2015
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.1
>Reporter: Christopher McDermott
>Assignee: Oleg Zhurakousky
>Priority: Critical
> Fix For: 1.0.0, 0.7.0
>
>
> Encountered the following error.  Once encountered the processor in question 
> cannot make any progress.  The incoming connection cannot be emptied or even 
> listed.  Note this issue is not to address the corrupted flow file but rather 
> the handling of the corrupted file.
> 2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] 
> o.a.n.p.attributes.UpdateAttribute
> java.lang.IllegalStateException: Cannot create Provenance Event Record 
> because FlowFile UUID is not set
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
>  ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (NIFI-2015) Corrupted flow file leads to a wedged flow

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330505#comment-15330505
 ] 

Christopher McDermott edited comment on NIFI-2015 at 6/14/16 8:56 PM:
--

Does NiFi rely on the rename idiom to achieve atomic writes?

{code}
fd=open(“file.new”); write(fd, data); close(fd); rename(“file.new”, “file”);
{code}

If so I think this in concert with using ext4 with default options, may be the 
ultimate cause of my problem.  I basically crashed the OS with NiFi running.  
This led to a bunch of zero length files, which is not unexpected with ext4, at 
least without some specific tuning.

[http://www.pointsoftware.ch/en/4-ext4-vs-ext3-filesystem-and-why-delayed-allocation-is-bad/]


was (Author: ch...@mcdermott.net):
Does NiFi rely on the rename idiom (fd=open(“file.new”); write(fd, data); 
close(fd); rename(“file.new”, “file”);) to achieve atomic writes?

If so I think this in concert with using ext4 with default options, may be the 
ultimate cause of my problem.  I basically crashed the OS with NiFi running.  
This led to a bunch of zero length files, which is not unexpected with ext4, at 
least without some specific tuning.

[http://www.pointsoftware.ch/en/4-ext4-vs-ext3-filesystem-and-why-delayed-allocation-is-bad/]

> Corrupted flow file leads to a wedged flow
> --
>
> Key: NIFI-2015
> URL: https://issues.apache.org/jira/browse/NIFI-2015
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.1
>Reporter: Christopher McDermott
>Assignee: Oleg Zhurakousky
>Priority: Critical
> Fix For: 1.0.0, 0.7.0
>
>
> Encountered the following error.  Once encountered the processor in question 
> cannot make any progress.  The incoming connection cannot be emptied or even 
> listed.  Note this issue is not to address the corrupted flow file but rather 
> the handling of the corrupted file.
> 2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] 
> o.a.n.p.attributes.UpdateAttribute
> java.lang.IllegalStateException: Cannot create Provenance Event Record 
> because FlowFile UUID is not set
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
>  ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.Threa

[jira] [Resolved] (NIFI-2023) CompressContent Processor should not always log decompression failures as an ERROR

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott resolved NIFI-2023.
-
Resolution: Workaround

> CompressContent Processor should not always log decompression failures as an 
> ERROR
> --
>
> Key: NIFI-2023
> URL: https://issues.apache.org/jira/browse/NIFI-2023
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Core UI
>Affects Versions: 0.6.1
>Reporter: Christopher McDermott
>Priority: Minor
>
> Sometimes you don't you don't know if file compression is used or what kind 
> of compression is used.  In these cases it is normal I think to chain 
> together a bunch CompressContent processors via the the failed output to 
> attempt decompression. If a stage fails in the chain it is not desirable to 
> emit and ERROR bulletin.  
> Perhaps a configuration parameter could be added to control the level at 
> which the failure is logged.  See NIFI-1724 for a similar enhancement request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2023) CompressContent Processor should not always log decompression failures as an ERROR

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330575#comment-15330575
 ] 

Christopher McDermott commented on NIFI-2023:
-

Ahh.  That's brilliant, Joe.  I didn't even know about IdentifyMimeType.  
Thanks.  I will withdraw this JIRA.

> CompressContent Processor should not always log decompression failures as an 
> ERROR
> --
>
> Key: NIFI-2023
> URL: https://issues.apache.org/jira/browse/NIFI-2023
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Core UI
>Affects Versions: 0.6.1
>Reporter: Christopher McDermott
>Priority: Minor
>
> Sometimes you don't you don't know if file compression is used or what kind 
> of compression is used.  In these cases it is normal I think to chain 
> together a bunch CompressContent processors via the the failed output to 
> attempt decompression. If a stage fails in the chain it is not desirable to 
> emit and ERROR bulletin.  
> Perhaps a configuration parameter could be added to control the level at 
> which the failure is logged.  See NIFI-1724 for a similar enhancement request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2023) CompressContent Processor should not always log decompression failures as an ERROR

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-2023:

Description: 
Sometimes you don't you don't know if file compression is used or what kind of 
compression is used.  In these cases it is normal I think to chain together a 
bunch CompressContent processors via the the failed output to attempt 
decompression. If a stage fails in the chain it is not desirable to emit and 
ERROR bulletin.  

Perhaps a configuration parameter could be added to control the level at which 
the failure is logged.  See NIFI-1724 for a similar enhancement request.

  was:
Sometimes you don't you don't know if file compression is used or what kind of 
compression is used.  In these cases is normal I think to chain together a 
bunch of failed outputs to attempt decompression. If a stage fails in the chain 
it is not desirable to emit and ERROR bulletin.  

Perhaps a configuration parameter could be added to control the level at which 
the failure is logged.  See NIFI-1724 for a similar enhancement request.


> CompressContent Processor should not always log decompression failures as an 
> ERROR
> --
>
> Key: NIFI-2023
> URL: https://issues.apache.org/jira/browse/NIFI-2023
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Core UI
>Affects Versions: 0.6.1
>Reporter: Christopher McDermott
>Priority: Minor
>
> Sometimes you don't you don't know if file compression is used or what kind 
> of compression is used.  In these cases it is normal I think to chain 
> together a bunch CompressContent processors via the the failed output to 
> attempt decompression. If a stage fails in the chain it is not desirable to 
> emit and ERROR bulletin.  
> Perhaps a configuration parameter could be added to control the level at 
> which the failure is logged.  See NIFI-1724 for a similar enhancement request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2023) CompressContent Processor should not always log decompression failures as an ERROR

[Christopher McDermott (JIRA)]

Christopher McDermott created NIFI-2023:
---

 Summary: CompressContent Processor should not always log 
decompression failures as an ERROR
 Key: NIFI-2023
 URL: https://issues.apache.org/jira/browse/NIFI-2023
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework, Core UI
Affects Versions: 0.6.1
Reporter: Christopher McDermott
Priority: Minor


Sometimes you don't you don't know if file compression is used or what kind of 
compression is used.  In these cases is normal I think to chain together a 
bunch of failed outputs to attempt decompression. If a stage fails in the chain 
it is not desirable to emit and ERROR bulletin.  

Perhaps a configuration parameter could be added to control the level at which 
the failure is logged.  See NIFI-1724 for a similar enhancement request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2015) Corrupted flow file leads to a wedged flow

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330505#comment-15330505
 ] 

Christopher McDermott commented on NIFI-2015:
-

Does NiFi rely on the rename idiom (fd=open(“file.new”); write(fd, data); 
close(fd); rename(“file.new”, “file”);) to achieve atomic writes?

If so I think this in concert with using ext4 with default options, may be the 
ultimate cause of my problem.  I basically crashed the OS with NiFi running.  
This led to a bunch of zero length files, which is not unexpected with ext4, at 
least without some specific tuning.

[http://www.pointsoftware.ch/en/4-ext4-vs-ext3-filesystem-and-why-delayed-allocation-is-bad/]

> Corrupted flow file leads to a wedged flow
> --
>
> Key: NIFI-2015
> URL: https://issues.apache.org/jira/browse/NIFI-2015
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.1
>Reporter: Christopher McDermott
>Assignee: Oleg Zhurakousky
>Priority: Critical
> Fix For: 1.0.0, 0.7.0
>
>
> Encountered the following error.  Once encountered the processor in question 
> cannot make any progress.  The incoming connection cannot be emptied or even 
> listed.  Note this issue is not to address the corrupted flow file but rather 
> the handling of the corrupted file.
> 2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] 
> o.a.n.p.attributes.UpdateAttribute
> java.lang.IllegalStateException: Cannot create Provenance Event Record 
> because FlowFile UUID is not set
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
>  ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2015) Corrupted flow file leads to a wedged flow

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330188#comment-15330188
 ] 

Christopher McDermott commented on NIFI-2015:
-

Some additional notes.  When this happened the connection had 20K or so flow 
files that were not going anywhere.  I changed changed the 'prioritizer' 
(newest first, I think) on the connection which allowed all but 20 files to 
drain.  These were once again stuck.  I stopped the connection and the 
processor which it was feeding. I connected the connection to a new 
LogAttributes processor which auto-terminated.  I started that processor and 
the connection and the queue drained completely.  Note, that I could not list 
or empty the connection.  I presume this was because the bad file was at the 
head of the queue.  When I attempted to empty queue I got a message which I 
don't fully recall but I think it said something about a "missing segment".  
When I tried to empty the queue I got the message about the missing FlowFile 
UUID.  I hope this additional info will help.

> Corrupted flow file leads to a wedged flow
> --
>
> Key: NIFI-2015
> URL: https://issues.apache.org/jira/browse/NIFI-2015
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.1
>Reporter: Christopher McDermott
>Assignee: Oleg Zhurakousky
>Priority: Critical
> Fix For: 1.0.0, 0.7.0
>
>
> Encountered the following error.  Once encountered the processor in question 
> cannot make any progress.  The incoming connection cannot be emptied or even 
> listed.  Note this issue is not to address the corrupted flow file but rather 
> the handling of the corrupted file.
> 2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] 
> o.a.n.p.attributes.UpdateAttribute
> java.lang.IllegalStateException: Cannot create Provenance Event Record 
> because FlowFile UUID is not set
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
>  ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2015) Corrupted flow file leads to a wedged flow

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15330039#comment-15330039
 ] 

Christopher McDermott commented on NIFI-2015:
-

[~markap14] It was an RouteOnAttribute processors.  I believe the FlowFile was 
corrupted due to some nasty fault testing and I am not sure our file system 
journaling was turned on.


> Corrupted flow file leads to a wedged flow
> --
>
> Key: NIFI-2015
> URL: https://issues.apache.org/jira/browse/NIFI-2015
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.1
>Reporter: Christopher McDermott
>Assignee: Oleg Zhurakousky
>Priority: Critical
> Fix For: 1.0.0, 0.7.0
>
>
> Encountered the following error.  Once encountered the processor in question 
> cannot make any progress.  The incoming connection cannot be emptied or even 
> listed.  Note this issue is not to address the corrupted flow file but rather 
> the handling of the corrupted file.
> 2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] 
> o.a.n.p.attributes.UpdateAttribute
> java.lang.IllegalStateException: Cannot create Provenance Event Record 
> because FlowFile UUID is not set
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412)
>  ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
>  ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2006) Null attribute values causing NPEs in provenance repo

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329633#comment-15329633
 ] 

Christopher McDermott commented on NIFI-2006:
-

[~ozhurakousky], I don't believe that this is the same issue as NIFI-2015.  
There is no NPE in that case rather it raises an ISE when checking for the 
FlowFile UUID.  Maybe the fix is the same but on the surface it seems quite 
different.

> Null attribute values causing NPEs in provenance repo
> -
>
> Key: NIFI-2006
> URL: https://issues.apache.org/jira/browse/NIFI-2006
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Brandon DeVries
>Assignee: Oleg Zhurakousky
>Priority: Critical
> Fix For: 1.0.0, 0.7.0
>
>
> A null attribute value (as you would see for a removed attribute) causes the 
> provenance repo to thrown an NPE here:
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java#L1789
> (as "entry.getValue()" is null...)
> suggested fix is to change:
> {code}
> if (entry.getValue().length() > maxAttributeChars) {
> {code}
> to:
> {code}
> if (entry.getValue() != null && entry.getValue().length() > 
> maxAttributeChars) {
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2015) Corrupted flow file leads to a wedged flow

[Christopher McDermott (JIRA)]

Christopher McDermott created NIFI-2015:
---

 Summary: Corrupted flow file leads to a wedged flow
 Key: NIFI-2015
 URL: https://issues.apache.org/jira/browse/NIFI-2015
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 0.6.1
Reporter: Christopher McDermott
Priority: Critical


Encountered the following error.  Once encountered the processor in question 
cannot make any progress.  The incoming connection cannot be emptied or even 
listed.  Note this issue is not to address the corrupted flow file but rather 
the handling of the corrupted file.

2016-06-13 17:33:30,275 ERROR [Timer-Driven Process Thread-4] 
o.a.n.p.attributes.UpdateAttribute
java.lang.IllegalStateException: Cannot create Provenance Event Record because 
FlowFile UUID is not set
at 
org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.assertSet(StandardProvenanceEventRecord.java:700)
 ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:721)
 ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.provenance.StandardProvenanceEventRecord$Builder.build(StandardProvenanceEventRecord.java:412)
 ~[nifi-data-provenance-utils-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.repository.StandardProcessSession.updateProvenanceRepo(StandardProcessSession.java:634)
 ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:295)
 ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:283)
 ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28)
 ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
 ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
[na:1.8.0_45]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [na:1.8.0_45]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [na:1.8.0_45]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_45]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1660) Enhance the expression language with jsonPath function

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319078#comment-15319078
 ] 

Christopher McDermott commented on NIFI-1660:
-

[~joewitt] This should be good to go now.  

> Enhance the expression language with jsonPath function
> --
>
> Key: NIFI-1660
> URL: https://issues.apache.org/jira/browse/NIFI-1660
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Christopher McDermott
>Assignee: Joseph Witt
>Priority: Minor
> Fix For: 1.0.0, 0.7.0
>
>
> jsonPath would evaluate a JSON path provided, as an argument, against the 
> subject.
> Example
> {quote}
> $\{kafka.key:jsonPath('$.foo.bar')\}
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1660) Enhance the expression language with jsonPath function

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15319069#comment-15319069
 ] 

Christopher McDermott commented on NIFI-1660:
-

I've created a second pull request for 0.x

> Enhance the expression language with jsonPath function
> --
>
> Key: NIFI-1660
> URL: https://issues.apache.org/jira/browse/NIFI-1660
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Christopher McDermott
>Assignee: Joseph Witt
>Priority: Minor
> Fix For: 1.0.0, 0.7.0
>
>
> jsonPath would evaluate a JSON path provided, as an argument, against the 
> subject.
> Example
> {quote}
> $\{kafka.key:jsonPath('$.foo.bar')\}
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1827) PutKafka attempts to write to non-existent partition.

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264524#comment-15264524
 ] 

Christopher McDermott commented on NIFI-1827:
-

I believe the way we hit this situation is the the topic has some number of 
partitions, say, 5.  The topic is deleted by an outside party. When this 
happened say, index was 3.   PutKafka runs again and because the topic is not 
there it is auto-created with 1 partition.  Now index will never equal the 
number of partitions and will and will never be reset.  

Clearly this is not a normal situation.  It happened to me because I  hacking 
around in a development environment and easiest way to for me to do something 
was to delete the topic.  

> PutKafka attempts to write to non-existent partition. 
> --
>
> Key: NIFI-1827
> URL: https://issues.apache.org/jira/browse/NIFI-1827
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Christopher McDermott
>
> PutKafka attempts to write to non-existent partition.  I have not verified 
> yet but I think the problem can be triggered by deleting a topic while the 
> processors is running, and then recreating the topic with the same name.  
> Since the problem has occurred I have not been able to make it go away.  I've 
> recreated the processor in the flow but the new processor exhibits the same 
> behavior.  
> 2016-04-29 12:00:53,550 ERROR [Timer-Driven Process Thread-1] 
> o.apache.nifi.processors.kafka.PutKafka
> java.lang.IllegalArgumentException: Invalid partition given with record: 4 is 
> not in the range [0...1].
> at 
> org.apache.kafka.clients.producer.internals.Partitioner.partition(Partitioner.java:52)
>  ~[na:na]
> at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:333) 
> ~[na:na]
> at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:248) 
> ~[na:na]
> at 
> org.apache.nifi.processors.kafka.KafkaPublisher.toKafka(KafkaPublisher.java:203)
>  ~[na:na]
> at 
> org.apache.nifi.processors.kafka.KafkaPublisher.publish(KafkaPublisher.java:137)
>  ~[na:na]
> at 
> org.apache.nifi.processors.kafka.PutKafka$1.process(PutKafka.java:300) 
> ~[na:na]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1807)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1778)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.processors.kafka.PutKafka.onTrigger(PutKafka.java:296) 
> ~[na:na]
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>  ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1827) PutKafka attempts to write to non-existent partition.

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264427#comment-15264427
 ] 

Christopher McDermott commented on NIFI-1827:
-

I'm not exactly sure how things got in this situation but stepping through the 
following code in the debugger ({{RoundRobinPartitioner}}) I see that  
{{index}} has become greater than {{numberOfPartitons}} and 
{{numberOfPartitions}} has a correct value.

{code}
private int next(int numberOfPartitions) {
if (index == numberOfPartitions) {
index = 0;
}
int indexToReturn = index++;
return indexToReturn;
}
{code}

This causes {{index}} to never reset to {{0}}.

> PutKafka attempts to write to non-existent partition. 
> --
>
> Key: NIFI-1827
> URL: https://issues.apache.org/jira/browse/NIFI-1827
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Christopher McDermott
>
> PutKafka attempts to write to non-existent partition.  I have not verified 
> yet but I think the problem can be triggered by deleting a topic while the 
> processors is running, and then recreating the topic with the same name.  
> Since the problem has occurred I have not been able to make it go away.  I've 
> recreated the processor in the flow but the new processor exhibits the same 
> behavior.  
> 2016-04-29 12:00:53,550 ERROR [Timer-Driven Process Thread-1] 
> o.apache.nifi.processors.kafka.PutKafka
> java.lang.IllegalArgumentException: Invalid partition given with record: 4 is 
> not in the range [0...1].
> at 
> org.apache.kafka.clients.producer.internals.Partitioner.partition(Partitioner.java:52)
>  ~[na:na]
> at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:333) 
> ~[na:na]
> at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:248) 
> ~[na:na]
> at 
> org.apache.nifi.processors.kafka.KafkaPublisher.toKafka(KafkaPublisher.java:203)
>  ~[na:na]
> at 
> org.apache.nifi.processors.kafka.KafkaPublisher.publish(KafkaPublisher.java:137)
>  ~[na:na]
> at 
> org.apache.nifi.processors.kafka.PutKafka$1.process(PutKafka.java:300) 
> ~[na:na]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1807)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1778)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.processors.kafka.PutKafka.onTrigger(PutKafka.java:296) 
> ~[na:na]
> at 
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>  ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
>  ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
>  [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_45]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_45]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-1827) PutKafka attempts to write to non-existent partition.

[Christopher McDermott (JIRA)]

Christopher McDermott created NIFI-1827:
---

 Summary: PutKafka attempts to write to non-existent partition. 
 Key: NIFI-1827
 URL: https://issues.apache.org/jira/browse/NIFI-1827
 Project: Apache NiFi
  Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Christopher McDermott


PutKafka attempts to write to non-existent partition.  I have not verified yet 
but I think the problem can be triggered by deleting a topic while the 
processors is running, and then recreating the topic with the same name.  Since 
the problem has occurred I have not been able to make it go away.  I've 
recreated the processor in the flow but the new processor exhibits the same 
behavior.  

2016-04-29 12:00:53,550 ERROR [Timer-Driven Process Thread-1] 
o.apache.nifi.processors.kafka.PutKafka
java.lang.IllegalArgumentException: Invalid partition given with record: 4 is 
not in the range [0...1].
at 
org.apache.kafka.clients.producer.internals.Partitioner.partition(Partitioner.java:52)
 ~[na:na]
at 
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:333) 
~[na:na]
at 
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:248) 
~[na:na]
at 
org.apache.nifi.processors.kafka.KafkaPublisher.toKafka(KafkaPublisher.java:203)
 ~[na:na]
at 
org.apache.nifi.processors.kafka.KafkaPublisher.publish(KafkaPublisher.java:137)
 ~[na:na]
at 
org.apache.nifi.processors.kafka.PutKafka$1.process(PutKafka.java:300) ~[na:na]
at 
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1807)
 ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1778)
 ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.processors.kafka.PutKafka.onTrigger(PutKafka.java:296) ~[na:na]
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
 ~[nifi-api-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1059)
 ~[nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:123)
 [nifi-framework-core-0.7.0-SNAPSHOT.jar:0.7.0-SNAPSHOT]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_45]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
[na:1.8.0_45]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [na:1.8.0_45]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [na:1.8.0_45]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_45]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1661) Expression language should support a random function

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245687#comment-15245687
 ] 

Christopher McDermott commented on NIFI-1661:
-

I'll update the PR.  I would like to avoid negative numbers.  I think it just 
makes it more complicated than it needs to be.  I'll use Math.abs() to address 
it.

> Expression language should support a random function
> 
>
> Key: NIFI-1661
> URL: https://issues.apache.org/jira/browse/NIFI-1661
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Christopher McDermott
>Assignee: Joe Skora
>Priority: Minor
> Fix For: 0.7.0
>
>
> Random would be a subjectless function that generates a 32? bit random 
> number.  The random number generator need not be cryptographically secure. 
> A use case would be for random be to generate a random pattern across some 
> number of file paths to be used with putFile.
> Example - puts files randomly to /my-dirs/dir-1, /my-dirs/dir-2, 
> /my-dirs/dir-3
> /my-dirs/dir-${random():mod(3):plus(1)}/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (NIFI-1764) NullPointerException in PutKafka for failed segments with no delimiter

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240015#comment-15240015
 ] 

Christopher McDermott edited comment on NIFI-1764 at 4/13/16 9:10 PM:
--

Here is a likely patch.  There is other similar code in 
SplittableMessageContext.java  that guards against NPE when converting byte[] 
to String.

{code}
diff --git 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
index 3b5eb4f..19e06ea 100644
--- 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
+++ 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
@@ -1,4 +1,5 @@
 /*
+
  * Licensed to the Apache Software Foundation (ASF) under one or more
  * contributor license agreements.  See the NOTICE file distributed with
  * this work for additional information regarding copyright ownership.
@@ -393,7 +394,7 @@ public class PutKafka extends AbstractProcessor {
 attributes.put(ATTR_FAILED_SEGMENTS, new 
String(failedSegments.toByteArray(), StandardCharsets.UTF_8));
 attributes.put(ATTR_TOPIC, messageContext.getTopicName());
 attributes.put(ATTR_KEY, messageContext.getKeyBytesAsString());
-attributes.put(ATTR_DELIMITER, new 
String(messageContext.getDelimiterBytes(), StandardCharsets.UTF_8));
+attributes.put(ATTR_DELIMITER, 
messageContext.getDelimeterBytesAsString());
 return attributes;
 }

diff --git 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
index d5f1c0b..4d63b7b 100644
--- 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
+++ 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
@@ -16,6 +16,7 @@
  */
 package org.apache.nifi.processors.kafka;

+import java.nio.charset.Charset;
 import java.nio.charset.StandardCharsets;
 import java.util.BitSet;

@@ -108,6 +109,13 @@ final class SplittableMessageContext {
 }

 /**
+ * Returns the delimiter bytes as String
+ */
+String getDelimeterBytesAsString() {
+return this.delimiterBytes != null ? new String(this.delimiterBytes, 
StandardCharsets.UTF_8) : null;
+}
+
+/**
  * Returns the key bytes as String
  */
 String getKeyBytesAsString() {
{code}


was (Author: ch...@mcdermott.net):
Here is a likely patch.  There is other similar code in 
SplittableMessageContext.java  that guards against NPE when converting byte[] 
to String.


diff --git 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
index 3b5eb4f..19e06ea 100644
--- 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
+++ 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
@@ -1,4 +1,5 @@
 /*
+
  * Licensed to the Apache Software Foundation (ASF) under one or more
  * contributor license agreements.  See the NOTICE file distributed with
  * this work for additional information regarding copyright ownership.
@@ -393,7 +394,7 @@ public class PutKafka extends AbstractProcessor {
 attributes.put(ATTR_FAILED_SEGMENTS, new 
String(failedSegments.toByteArray(), StandardCharsets.UTF_8));
 attributes.put(ATTR_TOPIC, messageContext.getTopicName());
 attributes.put(ATTR_KEY, messageContext.getKeyBytesAsString());
-attributes.put(ATTR_DELIMITER, new 
String(messageContext.getDelimiterBytes(), StandardCharsets.UTF_8));
+attributes.put(ATTR_DELIMITER, 
messageContext.getDelimeterBytesAsString());
 return attributes;
 }

diff --git 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
index d5f1c0b..4d63b7b 100644
--- 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
+++ 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-

[jira] [Comment Edited] (NIFI-1764) NullPointerException in PutKafka for failed segments with no delimiter

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240015#comment-15240015
 ] 

Christopher McDermott edited comment on NIFI-1764 at 4/13/16 9:09 PM:
--

Here is a likely patch.  There is other similar code in 
SplittableMessageContext.java  that guards against NPE when converting byte[] 
to String.


diff --git 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
index 3b5eb4f..19e06ea 100644
--- 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
+++ 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
@@ -1,4 +1,5 @@
 /*
+
  * Licensed to the Apache Software Foundation (ASF) under one or more
  * contributor license agreements.  See the NOTICE file distributed with
  * this work for additional information regarding copyright ownership.
@@ -393,7 +394,7 @@ public class PutKafka extends AbstractProcessor {
 attributes.put(ATTR_FAILED_SEGMENTS, new 
String(failedSegments.toByteArray(), StandardCharsets.UTF_8));
 attributes.put(ATTR_TOPIC, messageContext.getTopicName());
 attributes.put(ATTR_KEY, messageContext.getKeyBytesAsString());
-attributes.put(ATTR_DELIMITER, new 
String(messageContext.getDelimiterBytes(), StandardCharsets.UTF_8));
+attributes.put(ATTR_DELIMITER, 
messageContext.getDelimeterBytesAsString());
 return attributes;
 }

diff --git 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
index d5f1c0b..4d63b7b 100644
--- 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
+++ 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
@@ -16,6 +16,7 @@
  */
 package org.apache.nifi.processors.kafka;

+import java.nio.charset.Charset;
 import java.nio.charset.StandardCharsets;
 import java.util.BitSet;

@@ -108,6 +109,13 @@ final class SplittableMessageContext {
 }

 /**
+ * Returns the delimiter bytes as String
+ */
+String getDelimeterBytesAsString() {
+return this.delimiterBytes != null ? new String(this.delimiterBytes, 
StandardCharsets.UTF_8) : null;
+}
+
+/**
  * Returns the key bytes as String
  */
 String getKeyBytesAsString() {



was (Author: ch...@mcdermott.net):
diff --git 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
index 3b5eb4f..19e06ea 100644
--- 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
+++ 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/PutKafka.java
@@ -1,4 +1,5 @@
 /*
+
  * Licensed to the Apache Software Foundation (ASF) under one or more
  * contributor license agreements.  See the NOTICE file distributed with
  * this work for additional information regarding copyright ownership.
@@ -393,7 +394,7 @@ public class PutKafka extends AbstractProcessor {
 attributes.put(ATTR_FAILED_SEGMENTS, new 
String(failedSegments.toByteArray(), StandardCharsets.UTF_8));
 attributes.put(ATTR_TOPIC, messageContext.getTopicName());
 attributes.put(ATTR_KEY, messageContext.getKeyBytesAsString());
-attributes.put(ATTR_DELIMITER, new 
String(messageContext.getDelimiterBytes(), StandardCharsets.UTF_8));
+attributes.put(ATTR_DELIMITER, 
messageContext.getDelimeterBytesAsString());
 return attributes;
 }

diff --git 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
index d5f1c0b..4d63b7b 100644
--- 
a/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
+++ 
b/nifi-nar-bundles/nifi-kafka-bundle/nifi-kafka-processors/src/main/java/org/apache/nifi/processors/kafka/SplittableMessageContext.java
@@ -16,6 +16,7 @@
  */
 package org.apache.nifi.processors.kafka;

+impor

[jira] [Updated] (NIFI-1660) Enhance the expression language with jsonPath function

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1660:

Description: 
jsonPath would evaluate a JSON path provided, as an argument, against the 
subject.

Example
{quote}
$\{kafka.key:jsonPath('$.foo.bar')\}
{quote}

  was:
jsonPath would evaluate a JSON path provided as an argument against the subject.

Example
{quote}
$\{kafka.key:jsonPath('$.foo.bar')\}
{quote}


> Enhance the expression language with jsonPath function
> --
>
> Key: NIFI-1660
> URL: https://issues.apache.org/jira/browse/NIFI-1660
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Christopher McDermott
>Priority: Minor
>
> jsonPath would evaluate a JSON path provided, as an argument, against the 
> subject.
> Example
> {quote}
> $\{kafka.key:jsonPath('$.foo.bar')\}
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1661) Expression language should support a random function

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15228326#comment-15228326
 ] 

Christopher McDermott commented on NIFI-1661:
-

I've updated the pull request with 

{code}
new Random();
{code}

Cheers

> Expression language should support a random function
> 
>
> Key: NIFI-1661
> URL: https://issues.apache.org/jira/browse/NIFI-1661
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Christopher McDermott
>Priority: Minor
> Fix For: 0.7.0
>
>
> Random would be a subjectless function that generates a 32? bit random 
> number.  The random number generator need not be cryptographically secure. 
> A use case would be for random be to generate a random pattern across some 
> number of file paths to be used with putFile.
> Example - puts files randomly to /my-dirs/dir-1, /my-dirs/dir-2, 
> /my-dirs/dir-3
> /my-dirs/dir-${random():mod(3):plus(1)}/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15226235#comment-15226235
 ] 

Christopher McDermott edited comment on NIFI-1726 at 4/5/16 1:22 PM:
-

GC Stats are attache in gcstats.out


was (Author: ch...@mcdermott.net):
GC Stats

> Archiver is not respecting 
> nifi.content.repository.archive.max.usage.percentage
> ---
>
> Key: NIFI-1726
> URL: https://issues.apache.org/jira/browse/NIFI-1726
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Christopher McDermott
> Attachments: gcutil.out, nifi-td.txt, nifi-thread-dumps.tar.gz
>
>
> Content files are taking up more space than allowed to the point where the 
> disk becomes 100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1726:

Comment: was deleted

(was: thread dumps taken every 30 seconds)

> Archiver is not respecting 
> nifi.content.repository.archive.max.usage.percentage
> ---
>
> Key: NIFI-1726
> URL: https://issues.apache.org/jira/browse/NIFI-1726
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Christopher McDermott
> Attachments: gcutil.out, nifi-td.txt, nifi-thread-dumps.tar.gz
>
>
> Content files are taking up more space than allowed to the point where the 
> disk becomes 100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1726:

Comment: was deleted

(was: Here is the garbage collection stats you asked for as well.


$ /opt/mount1/jdk1.8.0_45/bin/jstat -gcutil -h5 13745
  S0 S1 E  O  M CCSYGC YGCTFGCFGCT GCT
  4.69   0.00   1.99  38.82  96.16  87.46  80.082 00.000
0.082
)

> Archiver is not respecting 
> nifi.content.repository.archive.max.usage.percentage
> ---
>
> Key: NIFI-1726
> URL: https://issues.apache.org/jira/browse/NIFI-1726
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Christopher McDermott
> Attachments: gcutil.out, nifi-td.txt, nifi-thread-dumps.tar.gz
>
>
> Content files are taking up more space than allowed to the point where the 
> disk becomes 100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1726:

Attachment: gcutil.out

GC Stats

> Archiver is not respecting 
> nifi.content.repository.archive.max.usage.percentage
> ---
>
> Key: NIFI-1726
> URL: https://issues.apache.org/jira/browse/NIFI-1726
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Christopher McDermott
> Attachments: gcutil.out, nifi-td.txt, nifi-thread-dumps.tar.gz
>
>
> Content files are taking up more space than allowed to the point where the 
> disk becomes 100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15226221#comment-15226221
 ] 

Christopher McDermott commented on NIFI-1726:
-

Here is the garbage collection stats you asked for as well.


$ /opt/mount1/jdk1.8.0_45/bin/jstat -gcutil -h5 13745
  S0 S1 E  O  M CCSYGC YGCTFGCFGCT GCT
  4.69   0.00   1.99  38.82  96.16  87.46  80.082 00.000
0.082


> Archiver is not respecting 
> nifi.content.repository.archive.max.usage.percentage
> ---
>
> Key: NIFI-1726
> URL: https://issues.apache.org/jira/browse/NIFI-1726
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Christopher McDermott
> Attachments: nifi-td.txt, nifi-thread-dumps.tar.gz
>
>
> Content files are taking up more space than allowed to the point where the 
> disk becomes 100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15226200#comment-15226200
 ] 

Christopher McDermott commented on NIFI-1726:
-

[~joewitt] I've attached I series of thread dumps from one of my nodes where 
the disk is filled.  The file names indicate the timestamp when the dump was 
taken.  nifi-thread-dumps.tar.gz

> Archiver is not respecting 
> nifi.content.repository.archive.max.usage.percentage
> ---
>
> Key: NIFI-1726
> URL: https://issues.apache.org/jira/browse/NIFI-1726
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Christopher McDermott
> Attachments: nifi-td.txt, nifi-thread-dumps.tar.gz
>
>
> Content files are taking up more space than allowed to the point where the 
> disk becomes 100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1726:

Attachment: nifi-thread-dumps.tar.gz

thread dumps taken every 30 seconds

> Archiver is not respecting 
> nifi.content.repository.archive.max.usage.percentage
> ---
>
> Key: NIFI-1726
> URL: https://issues.apache.org/jira/browse/NIFI-1726
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Christopher McDermott
> Attachments: nifi-td.txt, nifi-thread-dumps.tar.gz
>
>
> Content files are taking up more space than allowed to the point where the 
> disk becomes 100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15225046#comment-15225046
 ] 

Christopher McDermott commented on NIFI-1726:
-

Hi Joe.  I think I had stopped all ingress into the flow before I took a thread 
dump.  I grepped all logs for "Failed to cleanup archive for container" on all  
of my 5 processing nodes and I got no hits. I have logs going back a couple of 
weeks.

grep "Failed to cleanup archive for container" /opt/mount1/nifi/logs/*

I'm still running with the archiving on.  Let me know what steps you would like 
me to take when/if the problem reoccurs.



> Archiver is not respecting 
> nifi.content.repository.archive.max.usage.percentage
> ---
>
> Key: NIFI-1726
> URL: https://issues.apache.org/jira/browse/NIFI-1726
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Christopher McDermott
> Attachments: nifi-td.txt
>
>
> Content files are taking up more space than allowed to the point where the 
> disk becomes 100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224805#comment-15224805
 ] 

Christopher McDermott edited comment on NIFI-1726 at 4/4/16 7:01 PM:
-

On 4/4/16, 11:40 AM, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)" 
 wrote:

Hi Joe,

I am running fro 16108467c19f59 plus the pull request for NIFI-1660

Will do on the dump.

Thanks,
Chris

From: Joe Witt mailto:joe.w...@gmail.com>>
Reply-To: "us...@nifi.apache.org" 
mailto:us...@nifi.apache.org>>
Date: Sunday, April 3, 2016 at 8:34 PM
To: "us...@nifi.apache.org" 
mailto:us...@nifi.apache.org>>
Subject: Re: Question about content repository archiving


Chris

What version of nifi are you on?

If you happen to see such as case again please do take a thread dump of nifi as 
well.bin/nifi.sh dump

Thanks
Joe

Thanks
Joe

On Apr 3, 2016 8:20 PM, "Andre" 
mailto:andre-li...@fucs.org>> wrote:
Chris,

Can you navigate under /opt/mount2/nifi_data/content_repository and run

du -hsc */archive

keen to see if your archives aren't being deleted after expiry...

Seems pretty much like what I experienced as well ( 
http://markmail.org/message/37a2dxtkfvhwyty7 )


On Mon, Apr 4, 2016 at 6:09 AM, McDermott, Chris Kevin (MSDU - 
STaTS/StorefrontRemote) 
mailto:chris.mcderm...@hpe.com>> wrote:
I’m having a problem where the disk on which I am storing my NiFi data keeps 
filling up.  The content repository is taking up the overwhelming majority of 
that space.  It seems that the archiver is not working the way I would expect.

$ du -sh /opt/mount2/nifi_data/*
712K /opt/mount2/nifi_data/conf
147G /opt/mount2/nifi_data/content_repository
656K /opt/mount2/nifi_data/database_repository
318M /opt/mount2/nifi_data/flowfile_repository
937M /opt/mount2/nifi_data/provenance_repository

$ df -lh | grep mount2
/dev/mapper/disk2-mount2
  148G  148G 0 100% /opt/mount2

# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=10 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=/opt/mount2/nifi_data/content_repository
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=/nifi-content-viewer/

Can anyone point out what I might be missing?

Thanks,
Chris



was (Author: ch...@mcdermott.net):
So the problem cropped up again.  On one of the nodes that is out of space I 
ran a grep to find the first errors which show up I also have a thread dump.  
Is there some place I can post these too? 



On 4/4/16, 11:40 AM, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)" 
 wrote:

Hi Joe,

I am running fro 16108467c19f59 plus the pull request for NIFI-1660

Will do on the dump.

Thanks,
Chris

From: Joe Witt mailto:joe.w...@gmail.com>>
Reply-To: "us...@nifi.apache.org" 
mailto:us...@nifi.apache.org>>
Date: Sunday, April 3, 2016 at 8:34 PM
To: "us...@nifi.apache.org" 
mailto:us...@nifi.apache.org>>
Subject: Re: Question about content repository archiving


Chris

What version of nifi are you on?

If you happen to see such as case again please do take a thread dump of nifi as 
well.bin/nifi.sh dump

Thanks
Joe

Thanks
Joe

On Apr 3, 2016 8:20 PM, "Andre" 
mailto:andre-li...@fucs.org>> wrote:
Chris,

Can you navigate under /opt/mount2/nifi_data/content_repository and run

du -hsc */archive

keen to see if your archives aren't being deleted after expiry...

Seems pretty much like what I experienced as well ( 
http://markmail.org/message/37a2dxtkfvhwyty7 )


On Mon, Apr 4, 2016 at 6:09 AM, McDermott, Chris Kevin (MSDU - 
STaTS/StorefrontRemote) 
mailto:chris.mcderm...@hpe.com>> wrote:
I’m having a problem where the disk on which I am storing my NiFi data keeps 
filling up.  The content repository is taking up the overwhelming majority of 
that space.  It seems that the archiver is not working the way I would expect.

$ du -sh /opt/mount2/nifi_data/*
712K /opt/mount2/nifi_data/conf
147G /opt/mount2/nifi_data/content_repository
656K /opt/mount2/nifi_data/database_repository
318M /opt/mount2/nifi_data/flowfile_repository
937M /opt/mount2/nifi_data/provenance_repository

$ df -lh | grep mount2
/dev/mapper/disk2-mount2
  148G  148G 0 100% /opt/mount2

# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=10 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=/opt/mount2/nifi_data/content_repository
nifi.content.repository.

[jira] [Updated] (NIFI-1724) Enhance FetchFile to allow the log level for file not founds to be configurable

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1724:

Issue Type: Improvement  (was: Bug)

> Enhance FetchFile to allow the log level for file not founds to be 
> configurable
> ---
>
> Key: NIFI-1724
> URL: https://issues.apache.org/jira/browse/NIFI-1724
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Christopher McDermott
>Priority: Minor
>
> Sometimes a file may not exist but it is not an error.  This change would 
> allow the use to configure a logging level for the message so that a file not 
> found would not alway produce a bulletin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224815#comment-15224815
 ] 

Christopher McDermott edited comment on NIFI-1726 at 4/4/16 6:58 PM:
-

Thread dump - see attachment nifi-td.txt


was (Author: ch...@mcdermott.net):
see attachment nifi-td.txt

> Archiver is not respecting 
> nifi.content.repository.archive.max.usage.percentage
> ---
>
> Key: NIFI-1726
> URL: https://issues.apache.org/jira/browse/NIFI-1726
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Christopher McDermott
> Attachments: nifi-td.txt
>
>
> Content files are taking up more space than allowed to the point where the 
> disk becomes 100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224808#comment-15224808
 ] 

Christopher McDermott edited comment on NIFI-1726 at 4/4/16 6:57 PM:
-

Here is the disk space in the archive direstories

du -s /opt/mount2/nifi_data/content_repository/*/archive | cut -f1 | paste -sd+ 
- | bc
153401368



was (Author: ch...@mcdermott.net):
Here is the disk space in the archive direstoies

du -s /opt/mount2/nifi_data/content_repository/*/archive | cut -f1 | paste -sd+ 
- | bc
153401368


> Archiver is not respecting 
> nifi.content.repository.archive.max.usage.percentage
> ---
>
> Key: NIFI-1726
> URL: https://issues.apache.org/jira/browse/NIFI-1726
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Christopher McDermott
> Attachments: nifi-td.txt
>
>
> Content files are taking up more space than allowed to the point where the 
> disk becomes 100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224815#comment-15224815
 ] 

Christopher McDermott edited comment on NIFI-1726 at 4/4/16 6:56 PM:
-

see attachment nifi-td.txt


was (Author: ch...@mcdermott.net):
Thread dump

> Archiver is not respecting 
> nifi.content.repository.archive.max.usage.percentage
> ---
>
> Key: NIFI-1726
> URL: https://issues.apache.org/jira/browse/NIFI-1726
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Christopher McDermott
> Attachments: nifi-td.txt
>
>
> Content files are taking up more space than allowed to the point where the 
> disk becomes 100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1726:

Comment: was deleted

(was: Thread dump


"ActiveMQ InactivityMonitor ReadCheckTimer" Id=97 WAITING  on 
java.util.TaskQueue@7395849f
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
at java.util.TimerThread.run(Timer.java:505)

"Cleanup Archive for default" Id=44 RUNNABLE 
at 
org.apache.nifi.controller.repository.FileSystemRepository$ArchiveInfo.toPath(FileSystemRepository.java:1486)
at 
org.apache.nifi.controller.repository.FileSystemRepository.destroyExpiredArchives(FileSystemRepository.java:1237)
at 
org.apache.nifi.controller.repository.FileSystemRepository.access$1800(FileSystemRepository.java:84)
at 
org.apache.nifi.controller.repository.FileSystemRepository$DestroyExpiredArchiveClaims.run(FileSystemRepository.java:1523)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@63437af4

"Cluster Socket Listener" Id=60 RUNNABLE  (in native code)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at 
org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:114)
at java.lang.Thread.run(Thread.java:745)

"Clustering Tasks Thread-1" Id=88 WAITING  on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@490d549
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"Clustering Tasks Thread-2" Id=89 TIMED_WAITING  on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@490d549
at sun.misc.Unsafe.park(Native Method)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"Clustering Tasks Thread-3" Id=90 WAITING  on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@490d549
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(T

[jira] [Updated] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1726:

Attachment: nifi-td.txt

Thread dump

> Archiver is not respecting 
> nifi.content.repository.archive.max.usage.percentage
> ---
>
> Key: NIFI-1726
> URL: https://issues.apache.org/jira/browse/NIFI-1726
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Christopher McDermott
> Attachments: nifi-td.txt
>
>
> Content files are taking up more space than allowed to the point where the 
> disk becomes 100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224810#comment-15224810
 ] 

Christopher McDermott commented on NIFI-1726:
-

Thread dump


"ActiveMQ InactivityMonitor ReadCheckTimer" Id=97 WAITING  on 
java.util.TaskQueue@7395849f
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.util.TimerThread.mainLoop(Timer.java:526)
at java.util.TimerThread.run(Timer.java:505)

"Cleanup Archive for default" Id=44 RUNNABLE 
at 
org.apache.nifi.controller.repository.FileSystemRepository$ArchiveInfo.toPath(FileSystemRepository.java:1486)
at 
org.apache.nifi.controller.repository.FileSystemRepository.destroyExpiredArchives(FileSystemRepository.java:1237)
at 
org.apache.nifi.controller.repository.FileSystemRepository.access$1800(FileSystemRepository.java:84)
at 
org.apache.nifi.controller.repository.FileSystemRepository$DestroyExpiredArchiveClaims.run(FileSystemRepository.java:1523)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Number of Locked Synchronizers: 1
- java.util.concurrent.ThreadPoolExecutor$Worker@63437af4

"Cluster Socket Listener" Id=60 RUNNABLE  (in native code)
at java.net.PlainSocketImpl.socketAccept(Native Method)
at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at 
org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:114)
at java.lang.Thread.run(Thread.java:745)

"Clustering Tasks Thread-1" Id=88 WAITING  on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@490d549
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"Clustering Tasks Thread-2" Id=89 TIMED_WAITING  on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@490d549
at sun.misc.Unsafe.park(Native Method)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"Clustering Tasks Thread-3" Id=90 WAITING  on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@490d549
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at 
java.util.concurrent.Thre

[jira] [Commented] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224808#comment-15224808
 ] 

Christopher McDermott commented on NIFI-1726:
-

Here is the disk space in the archive direstoies

du -s /opt/mount2/nifi_data/content_repository/*/archive | cut -f1 | paste -sd+ 
- | bc
153401368


> Archiver is not respecting 
> nifi.content.repository.archive.max.usage.percentage
> ---
>
> Key: NIFI-1726
> URL: https://issues.apache.org/jira/browse/NIFI-1726
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Christopher McDermott
>
> Content files are taking up more space than allowed to the point where the 
> disk becomes 100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

Christopher McDermott created NIFI-1726:
---

 Summary: Archiver is not respecting 
nifi.content.repository.archive.max.usage.percentage
 Key: NIFI-1726
 URL: https://issues.apache.org/jira/browse/NIFI-1726
 Project: Apache NiFi
  Issue Type: Bug
Affects Versions: 0.6.0
Reporter: Christopher McDermott


Content files are taking up more space than allowed to the point where the disk 
becomes 100% full.

I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1726) Archiver is not respecting nifi.content.repository.archive.max.usage.percentage

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15224805#comment-15224805
 ] 

Christopher McDermott commented on NIFI-1726:
-

So the problem cropped up again.  On one of the nodes that is out of space I 
ran a grep to find the first errors which show up I also have a thread dump.  
Is there some place I can post these too? 



On 4/4/16, 11:40 AM, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)" 
 wrote:

Hi Joe,

I am running fro 16108467c19f59 plus the pull request for NIFI-1660

Will do on the dump.

Thanks,
Chris

From: Joe Witt mailto:joe.w...@gmail.com>>
Reply-To: "us...@nifi.apache.org" 
mailto:us...@nifi.apache.org>>
Date: Sunday, April 3, 2016 at 8:34 PM
To: "us...@nifi.apache.org" 
mailto:us...@nifi.apache.org>>
Subject: Re: Question about content repository archiving


Chris

What version of nifi are you on?

If you happen to see such as case again please do take a thread dump of nifi as 
well.bin/nifi.sh dump

Thanks
Joe

Thanks
Joe

On Apr 3, 2016 8:20 PM, "Andre" 
mailto:andre-li...@fucs.org>> wrote:
Chris,

Can you navigate under /opt/mount2/nifi_data/content_repository and run

du -hsc */archive

keen to see if your archives aren't being deleted after expiry...

Seems pretty much like what I experienced as well ( 
http://markmail.org/message/37a2dxtkfvhwyty7 )


On Mon, Apr 4, 2016 at 6:09 AM, McDermott, Chris Kevin (MSDU - 
STaTS/StorefrontRemote) 
mailto:chris.mcderm...@hpe.com>> wrote:
I’m having a problem where the disk on which I am storing my NiFi data keeps 
filling up.  The content repository is taking up the overwhelming majority of 
that space.  It seems that the archiver is not working the way I would expect.

$ du -sh /opt/mount2/nifi_data/*
712K /opt/mount2/nifi_data/conf
147G /opt/mount2/nifi_data/content_repository
656K /opt/mount2/nifi_data/database_repository
318M /opt/mount2/nifi_data/flowfile_repository
937M /opt/mount2/nifi_data/provenance_repository

$ df -lh | grep mount2
/dev/mapper/disk2-mount2
  148G  148G 0 100% /opt/mount2

# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=10 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=/opt/mount2/nifi_data/content_repository
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=/nifi-content-viewer/

Can anyone point out what I might be missing?

Thanks,
Chris


> Archiver is not respecting 
> nifi.content.repository.archive.max.usage.percentage
> ---
>
> Key: NIFI-1726
> URL: https://issues.apache.org/jira/browse/NIFI-1726
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.0
>Reporter: Christopher McDermott
>
> Content files are taking up more space than allowed to the point where the 
> disk becomes 100% full.
> I'll add the develop list email thread in the comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-1724) Enhance FetchFile to allow the log level for file not founds to be configurable

[Christopher McDermott (JIRA)]

Christopher McDermott created NIFI-1724:
---

 Summary: Enhance FetchFile to allow the log level for file not 
founds to be configurable
 Key: NIFI-1724
 URL: https://issues.apache.org/jira/browse/NIFI-1724
 Project: Apache NiFi
  Issue Type: Bug
Reporter: Christopher McDermott
Priority: Minor


Sometimes a file may not exist but it is not an error.  This change would allow 
the use to configure a logging level for the message so that a file not found 
would not alway produce a bulletin.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (NIFI-1660) Enhance the expression language with jsonPath function

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216209#comment-15216209
 ] 

Christopher McDermott edited comment on NIFI-1660 at 3/29/16 3:52 PM:
--

[~markap14] Have you had a chance to look at this yet?  


was (Author: ch...@mcdermott.net):
[~markap14]Have you had a chance to look at this yet?  

> Enhance the expression language with jsonPath function
> --
>
> Key: NIFI-1660
> URL: https://issues.apache.org/jira/browse/NIFI-1660
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Christopher McDermott
>Priority: Minor
>
> jsonPath would evaluate a JSON path provided as an argument against the 
> subject.
> Example
> {quote}
> $\{kafka.key:jsonPath('$.foo.bar')\}
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1660) Enhance the expression language with jsonPath function

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216209#comment-15216209
 ] 

Christopher McDermott commented on NIFI-1660:
-

[~markap14]Have you had a chance to look at this yet?  

> Enhance the expression language with jsonPath function
> --
>
> Key: NIFI-1660
> URL: https://issues.apache.org/jira/browse/NIFI-1660
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Christopher McDermott
>Priority: Minor
>
> jsonPath would evaluate a JSON path provided as an argument against the 
> subject.
> Example
> {quote}
> $\{kafka.key:jsonPath('$.foo.bar')\}
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1660) Enhance the expression language with jsonPath function

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15210806#comment-15210806
 ] 

Christopher McDermott commented on NIFI-1660:
-

Joe, as a newbie I'm not 100% sure what you are asking, but the dependency is 
already in the framework nar...

$ ls 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework-nar/target/classes/META-INF/bundled-dependencies/json-path-2.0.0.jar
nifi-nar-bundles/nifi-framework-bundle/nifi-framework-nar/target/classes/META-INF/bundled-dependencies/json-path-2.0.0.jar



> Enhance the expression language with jsonPath function
> --
>
> Key: NIFI-1660
> URL: https://issues.apache.org/jira/browse/NIFI-1660
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Christopher McDermott
>Priority: Minor
>
> jsonPath would evaluate a JSON path provided as an argument against the 
> subject.
> Example
> {quote}
> $\{kafka.key:jsonPath('$.foo.bar')\}
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1660) Enhance the expression language with jsonPath function

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1660:

Summary: Enhance the expression language with jsonPath function  (was: 
Enhance the expression language support with jsonPath function)

> Enhance the expression language with jsonPath function
> --
>
> Key: NIFI-1660
> URL: https://issues.apache.org/jira/browse/NIFI-1660
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Christopher McDermott
>Priority: Minor
>
> jsonPath would evaluate a JSON path provided as an argument against the 
> subject.
> Example
> {quote}
> $\{kafka.key:jsonPath('$.foo.bar')\}
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1660) Enhance the expression language support with jsonPath function

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1660:

Description: 
jsonPath would evaluate a JSON path provided as an argument against the subject.

Example
{quote}
${kafka.key:jsonPath('$.foo.bar')}
{quote}

  was:
jsonPath would evaluate a JSON path provided as an argument against the subject.

Example

${kafka.key:jsonPath('$.foo.bar')}


> Enhance the expression language support with jsonPath function
> --
>
> Key: NIFI-1660
> URL: https://issues.apache.org/jira/browse/NIFI-1660
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Christopher McDermott
>Priority: Minor
>
> jsonPath would evaluate a JSON path provided as an argument against the 
> subject.
> Example
> {quote}
> ${kafka.key:jsonPath('$.foo.bar')}
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1660) Enhance the expression language support with jsonPath function

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1660:

Description: 
jsonPath would evaluate a JSON path provided as an argument against the subject.

Example
{quote}
$\{kafka.key:jsonPath('$.foo.bar')\}
{quote}

  was:
jsonPath would evaluate a JSON path provided as an argument against the subject.

Example
{quote}
${kafka.key:jsonPath('$.foo.bar')}
{quote}


> Enhance the expression language support with jsonPath function
> --
>
> Key: NIFI-1660
> URL: https://issues.apache.org/jira/browse/NIFI-1660
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Christopher McDermott
>Priority: Minor
>
> jsonPath would evaluate a JSON path provided as an argument against the 
> subject.
> Example
> {quote}
> $\{kafka.key:jsonPath('$.foo.bar')\}
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1660) Enhance the expression language support with jsonPath function

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1660:

Description: 
jsonPath would evaluate a JSON path provided as an argument against the subject.

Example

${kafka.key:jsonPath('$.foo.bar')}

  was:
jsonPath would evaluate a JSON path provided as an argument against the subject.

Example
${kafka.key:jsonPath('$.foo.bar')}


> Enhance the expression language support with jsonPath function
> --
>
> Key: NIFI-1660
> URL: https://issues.apache.org/jira/browse/NIFI-1660
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Christopher McDermott
>Priority: Minor
>
> jsonPath would evaluate a JSON path provided as an argument against the 
> subject.
> Example
> ${kafka.key:jsonPath('$.foo.bar')}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1660) Enhance the expression language support with jsonPath function

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1660:

Description: 
jsonPath would evaluate a JSON path provided as an argument against the subject.

Example
${kafka.key:jsonPath('$.foo.bar')}

  was:
evaluateJsonPath would evaluate a JSON path provided as an argument against the 
subject.

Example
${kafka.key:evaluateJsonPath('$.foo.bar')}


> Enhance the expression language support with jsonPath function
> --
>
> Key: NIFI-1660
> URL: https://issues.apache.org/jira/browse/NIFI-1660
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Christopher McDermott
>Priority: Minor
>
> jsonPath would evaluate a JSON path provided as an argument against the 
> subject.
> Example
> ${kafka.key:jsonPath('$.foo.bar')}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1660) Enhance the expression language support with jsonPath function

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1660:

Summary: Enhance the expression language support with jsonPath function  
(was: Enhance the expression language support evaluateJsonPath)

> Enhance the expression language support with jsonPath function
> --
>
> Key: NIFI-1660
> URL: https://issues.apache.org/jira/browse/NIFI-1660
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Christopher McDermott
>Priority: Minor
>
> evaluateJsonPath would evaluate a JSON path provided as an argument against 
> the subject.
> Example
> ${kafka.key:evaluateJsonPath('$.foo.bar')}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-1661) Expression language should support a random function

[Christopher McDermott (JIRA)]

Christopher McDermott created NIFI-1661:
---

 Summary: Expression language should support a random function
 Key: NIFI-1661
 URL: https://issues.apache.org/jira/browse/NIFI-1661
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Christopher McDermott
Priority: Minor


Random would be a subjectless function that generates a 32? bit random number.  
The random number generator need not be cryptographically secure. 

A use case would be for random be to generate a random pattern across some 
number of file paths to be used with putFile.

Example - puts files randomly to /my-dirs/dir-1, /my-dirs/dir-2, /my-dirs/dir-3

/my-dirs/dir-${random():mod(3):plus(1)}/






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1659) Enhance EvaluateJsonPath to be able to evaluate an attribute value

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1659:

Issue Type: Improvement  (was: Bug)

> Enhance EvaluateJsonPath to be able to evaluate an attribute value
> --
>
> Key: NIFI-1659
> URL: https://issues.apache.org/jira/browse/NIFI-1659
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 0.5.1
>Reporter: Christopher McDermott
>Priority: Minor
>
> Today EvaluateJsonPath evaluates the content of a flow file.  This request is 
> to enhance EvaluateJsonPath to evaluate either the flow file content, or an 
> attribute value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1642) PutKafka should validate topic expression and calculated value

[Christopher McDermott (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher McDermott updated NIFI-1642:

Issue Type: Improvement  (was: Bug)

> PutKafka should validate topic expression and calculated value
> --
>
> Key: NIFI-1642
> URL: https://issues.apache.org/jira/browse/NIFI-1642
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Christopher McDermott
>Priority: Minor
>
> PutKafka does not validate the expression supplied for the topic property, 
> like most other processors.  It should also try to validate the evaluated 
> value of the topic to see if it is a legal topic name.  Note I'm not 
> suggesting that the topic need to exist, just that the name is compliant to 
> what Kafka will accept.   This would be most helpful because if certain 
> (probably not all) illegal names are used, the Kafka client throws bizarre 
> and most unhelpful exceptions.
> -
> Chris,
> Assuming the client can validate #2 i am with you.  Please do feel
> free to fire up a JIRA for this.
> Thanks
> Joe
> On Wed, Mar 16, 2016 at 1:24 PM, McDermott, Chris Kevin (MSDU -
> STaTS/StorefrontRemote)  wrote:
> It turns out the root cause of the problem was an invalid topic name.  
> Strange error for that!
> I think there are a couple of improvements could be made to PutKafka.
> 1. Check the validity of the the expression in the topic property.
> 2. Check the validity of the topic name before attempting to write to the 
> topic.
> Chris
> On 3/16/16, 11:41 AM, "McDermott, Chris Kevin (MSDU - 
> STaTS/StorefrontRemote)"  wrote:
> Joe,
> I’ll checkout the disk-space.  We are running 0.9. If disk space is not the 
> issue we’ll give 0.8 a try.
> Thanks very much for your quick reply.
> Cheers,
> Chris
> On 3/16/16, 11:04 AM, "Joe Witt"  wrote:
> Chris,
> I have seen that when the diskspace kafka relies on is full.  We've
> seen a number of interesting exceptions recently in testing various
> configurations. But recommend checking that.
> Also, what version of Kafka broker are you using?  With Apache NiFi
> 0.5.x we moved to the kafka client 0.9.  In doing that we messed up
> support for 0.8.  So...with the upcoming release we will move back to
> the 0.8 client and thus it works great with Kafka 0.8 and 0.9 brokers
> albeit without the new SSL and Kerberos support they added in their
> 0.9 work.  We have a JIRA item to go after that for our next feature
> bearing release.
> Thanks
> Joe
> On Wed, Mar 16, 2016 at 11:01 AM, McDermott, Chris Kevin (MSDU -
> STaTS/StorefrontRemote)  wrote:
> I say strange because the timeout (63ms) is so very short.  The communication 
> timeout I’ve set is 30 sec.  Has anyone overseen this?
> 2016-03-16 14:41:38,227 ERROR [Timer-Driven Process Thread-8] 
> o.apache.nifi.processors.kafka.PutKafka 
> PutKafka[id=852c8d42-a2fa-3478-b06b-84ceb6\
> 6f8b0b] Failed to send 
> StandardFlowFileRecord[uuid=a0074162-0066-49e7-918b-cea1cfc5a955,claim=StandardContentClaim
>  [resourceClaim=StandardResour\
> ceClaim[id=1458079089737-67, container=default, section=67], offset=377796, 
> length=743],offset=0,name=2349680613178720,size=743] to Kafka; routi\
> ng to 'failure'; last failure reason reported was 
> org.apache.kafka.common.errors.TimeoutException: Failed to update metadata 
> after 63 ms.;: org.\
> apache.kafka.common.errors.TimeoutException: Failed to update metadata after 
> 63 ms.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-1660) Enhance the expression language support evaluateJsonPath

[Christopher McDermott (JIRA)]

Christopher McDermott created NIFI-1660:
---

 Summary: Enhance the expression language support evaluateJsonPath
 Key: NIFI-1660
 URL: https://issues.apache.org/jira/browse/NIFI-1660
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework
Affects Versions: 0.5.1
Reporter: Christopher McDermott
Priority: Minor


evaluateJsonPath would evaluate a JSON path provided as an argument against the 
subject.

Example
${kafka.key:evaluateJsonPath('$.foo.bar')}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-1659) Enhance EvaluateJsonPath to be able to evaluate an attribute value

[Christopher McDermott (JIRA)]

Christopher McDermott created NIFI-1659:
---

 Summary: Enhance EvaluateJsonPath to be able to evaluate an 
attribute value
 Key: NIFI-1659
 URL: https://issues.apache.org/jira/browse/NIFI-1659
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 0.5.1
Reporter: Christopher McDermott
Priority: Minor


Today EvaluateJsonPath evaluates the content of a flow file.  This request is 
to enhance EvaluateJsonPath to evaluate either the flow file content, or an 
attribute value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-1642) PutKafka should validate topic expression and calculated value

[Christopher McDermott (JIRA)]

Christopher McDermott created NIFI-1642:
---

 Summary: PutKafka should validate topic expression and calculated 
value
 Key: NIFI-1642
 URL: https://issues.apache.org/jira/browse/NIFI-1642
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Reporter: Christopher McDermott
Priority: Minor


PutKafka does not validate the expression supplied for the topic property, like 
most other processors.  It should also try to validate the evaluated value of 
the topic to see if it is a legal topic name.  Note I'm not suggesting that the 
topic need to exist, just that the name is compliant to what Kafka will accept. 
  This would be most helpful because if certain (probably not all) illegal 
names are used, the Kafka client throws bizarre and most unhelpful exceptions.

-
Chris,

Assuming the client can validate #2 i am with you.  Please do feel
free to fire up a JIRA for this.

Thanks
Joe

On Wed, Mar 16, 2016 at 1:24 PM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote)  wrote:
It turns out the root cause of the problem was an invalid topic name.  Strange 
error for that!

I think there are a couple of improvements could be made to PutKafka.

1. Check the validity of the the expression in the topic property.
2. Check the validity of the topic name before attempting to write to the topic.

Chris



On 3/16/16, 11:41 AM, "McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)" 
 wrote:

Joe,

I’ll checkout the disk-space.  We are running 0.9. If disk space is not the 
issue we’ll give 0.8 a try.

Thanks very much for your quick reply.

Cheers,
Chris



On 3/16/16, 11:04 AM, "Joe Witt"  wrote:

Chris,
I have seen that when the diskspace kafka relies on is full.  We've
seen a number of interesting exceptions recently in testing various
configurations. But recommend checking that.

Also, what version of Kafka broker are you using?  With Apache NiFi
0.5.x we moved to the kafka client 0.9.  In doing that we messed up
support for 0.8.  So...with the upcoming release we will move back to
the 0.8 client and thus it works great with Kafka 0.8 and 0.9 brokers
albeit without the new SSL and Kerberos support they added in their
0.9 work.  We have a JIRA item to go after that for our next feature
bearing release.

Thanks
Joe

On Wed, Mar 16, 2016 at 11:01 AM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote)  wrote:
I say strange because the timeout (63ms) is so very short.  The communication 
timeout I’ve set is 30 sec.  Has anyone overseen this?

2016-03-16 14:41:38,227 ERROR [Timer-Driven Process Thread-8] 
o.apache.nifi.processors.kafka.PutKafka 
PutKafka[id=852c8d42-a2fa-3478-b06b-84ceb6\
6f8b0b] Failed to send 
StandardFlowFileRecord[uuid=a0074162-0066-49e7-918b-cea1cfc5a955,claim=StandardContentClaim
 [resourceClaim=StandardResour\
ceClaim[id=1458079089737-67, container=default, section=67], offset=377796, 
length=743],offset=0,name=2349680613178720,size=743] to Kafka; routi\
ng to 'failure'; last failure reason reported was 
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata 
after 63 ms.;: org.\
apache.kafka.common.errors.TimeoutException: Failed to update metadata after 63 
ms.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1593) New Processor to make use of a template engine

[Christopher McDermott (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15201504#comment-15201504
 ] 

Christopher McDermott commented on NIFI-1593:
-

Brilliant.  I've been looking for something like this.  I'd like to see it also 
support flow file attributes as a data source, as well as JSON flow file 
content.

> New Processor to make use of a template engine
> --
>
> Key: NIFI-1593
> URL: https://issues.apache.org/jira/browse/NIFI-1593
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core UI
>Affects Versions: 0.4.1
> Environment: all
>Reporter: Uwe Geercken
>Priority: Minor
>
> Add a new processor that allows to use the functionality of a template engine 
> such as Apache Velocity.
> The user would provide either the template code or reference a template file.
> In the process the flowfile content/attributes would be merged with the 
> template and thus transforming the content.
> A use case would be:
> Get a Json file >> EvaluateJsonPath >> transform the attributes of the Json 
> file to a different format using the given template code or template file 
> (using the template engine) >> conversion into any other text based format 
> such as xml, html, text.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-1549) Bug in JMSConsumer makes it hard to debug failure in processing JMS messages.

[Christopher McDermott (JIRA)]

Christopher McDermott created NIFI-1549:
---

 Summary: Bug in JMSConsumer makes it hard to debug failure in 
processing JMS messages.
 Key: NIFI-1549
 URL: https://issues.apache.org/jira/browse/NIFI-1549
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 0.5.0
Reporter: Christopher McDermott
Priority: Trivial


Minor bug in JMSConsumer makes it hard to debug failure in processing JMS 
messages.  Fix is trivial.

{code:java}
diff --git 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JmsConsumer.java
 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JmsConsumer.java
index ea70d52..c820174 100644
--- 
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JmsConsumer.java
+++ 
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/JmsConsumer.java
@@ -197,7 +197,7 @@ public abstract class JmsConsumer extends AbstractProcessor 
{
 final byte[] messageBody = 
JmsFactory.createByteArray(message);
 out.write(messageBody);
 } catch (final JMSException e) {
-throw new ProcessException("Failed to receive JMS 
Message due to {}", e);
+throw new ProcessException("Failed to receive JMS 
Message due to " + e.getMessage(), e);
 }
 }
 });
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)