[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

2017-08-11 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124463#comment-16124463
 ] 

ASF subversion and git services commented on NIFI-4023:
---

Commit 84935d4f7840fe9b22aa929afb403718c073e627 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=84935d4 ]

NIFI-4023, NIFI-4077 This closes #2075. Addressed issue where repository was 
aging off the wrong index. When it should age off Index 1, it was removing 
Index 2. As a result, the earliest index is never aged off, and the newest 
index could potentially be aged off before it is ready to be. Also addressed 
issue where a query that attempts to read an event that has aged off throws 
FileNotFoundException (NIFI-4077) instead of skipping over the event. The 
JavaDocs indicate that the EventIterator should skip records that it cannot 
find, but SelectiveRecordReaderEventIterator throw FileNotFoundException instead

Signed-off-by: joewitt 


> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
>

[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

2017-08-11 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124468#comment-16124468
 ] 

ASF GitHub Bot commented on NIFI-4023:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2075


> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at

[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

2017-08-11 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124459#comment-16124459
 ] 

ASF GitHub Bot commented on NIFI-4023:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2075
  
+1 will merge to master


> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at

[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

2017-08-11 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124137#comment-16124137
 ] 

ASF GitHub Bot commented on NIFI-4023:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2075
  
seeing far better behavior on the flow i could cause these issues on.  Such 
as the one i posted in NIFI-3329.  Will monitor a bit more and if still good 
then merge


> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
>

[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

2017-08-11 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123926#comment-16123926
 ] 

ASF GitHub Bot commented on NIFI-4023:
--

Github user joewitt commented on the issue:

https://github.com/apache/nifi/pull/2075
  
on it.  Will let it run for a while.


> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>Assignee: Mark Payne
> Fix For: 1.4.0
>
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>

[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

2017-08-11 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123848#comment-16123848
 ] 

ASF GitHub Bot commented on NIFI-4023:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/2075

NIFI-4023, NIFI-4077: Addressed issue where repository was aging off …

…the wrong index. When it should age off Index 1, it was removing Index 2. 
As a result, the earliest index is never aged off, and the newest index could 
potentially be aged off before it is ready to be. Also addressed issue where a 
query that attempts to read an event that has aged off throws 
FileNotFoundException (NIFI-4077) instead of skipping over the event. The 
JavaDocs indicate that the EventIterator should skip records that it cannot 
find, but SelectiveRecordReaderEventIterator throw FileNotFoundException instead

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-4023

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2075.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2075


commit 7e60e9e26481237a0f1d7c9d9b056a4604177b72
Author: Mark Payne 
Date:   2017-08-11T17:08:29Z

NIFI-4023, NIFI-4077: Addressed issue where repository was aging off the 
wrong index. When it should age off Index 1, it was removing Index 2. As a 
result, the earliest index is never aged off, and the newest index could 
potentially be aged off before it is ready to be. Also addressed issue where a 
query that attempts to read an event that has aged off throws 
FileNotFoundException (NIFI-4077) instead of skipping over the event. The 
JavaDocs indicate that the EventIterator should skip records that it cannot 
find, but SelectiveRecordReaderEventIterator throw FileNotFoundException instead




> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
>

[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

2017-08-10 Thread Randy Bovay (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122855#comment-16122855
 ] 

Randy Bovay commented on NIFI-4023:
---

We are experiencing this as well.  Good to see there is already an issue 
tracking this.  
Happy to test with you more to help find a solution to this.

> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at

[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

2017-06-10 Thread Andre F de Miranda (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045815#comment-16045815
 ] 

Andre F de Miranda commented on NIFI-4023:
--

[~markap14]

I tried changing my settings from 1 GB to 10 GB

{code}
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=10 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=500 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
#WAL Provenance only settings
nifi.provenance.repository.concurrent.merge.threads=2
nifi.provenance.repository.warm.cache.frequency=1 hour
{code}

however I still observe the number of deleted file handles growing with time.

I agree with your assertion that the situation becomes worse (i.e. higher 
number of deleted handles) under load.

> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:37,425 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
>

[jira] [Commented] (NIFI-4023) WriteAheadProvenanceRepository indexing and query failure under high rate stress testing

2017-06-06 Thread Mark Payne (JIRA) 

[ 
https://issues.apache.org/jira/browse/NIFI-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16039018#comment-16039018
 ] 

Mark Payne commented on NIFI-4023:
--

[~joewitt] - i've been looking into this. I haven't been able to recreate the 
exact issue but have seen something similar, when I use the default values for 
nifi.properties, other than changing to the WriteAheadProvenanceRepository. I 
suspect both problems have the same root cause. I believe that it's because the 
default prov repo size is 1 GB and we can fill that up very quickly. We run 
cleanup every 1 minute. So if we fill up faster than we cleanup, we can end up 
in a condition where we're deleting the provenance file that we're writing to. 
This then causes some funkiness with indexes not getting cleaned up 
appropriately.

Specifically, the lucene indices are cleaned up every 1 minute. So are the 
event files. But they are on separate timers so happen at different times. 
Also, we ensure that we don't delete the index that we are writing to. If the 
repo fills up, and then the cleanup for the event files triggers, this can 
result in deleting all event files. When we then go to cleanup the indices, we 
ask the Event Store for the first event so that we can determine its ID. The 
Store has no events because all event files were deleted. As a result, the 
Lucene Index cleanup task just returns without deleting any indices. This 
further compounds the issue because now we don't cleanup old indices. So every 
time that the event file cleanup triggers, it deletes all event files.  And the 
indices are never properly cleaned up.

When I changed my prov repo size to 10 GB, I am not able to run into any issues 
anymore.

> WriteAheadProvenanceRepository indexing and query failure under high rate 
> stress testing
> 
>
> Key: NIFI-4023
> URL: https://issues.apache.org/jira/browse/NIFI-4023
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.3.0
>Reporter: Joseph Witt
>
> 2017-06-06 00:32:35,995 INFO [pool-10-thread-1] 
> org.wali.MinimalLockingWriteAheadLog 
> org.wali.MinimalLockingWriteAheadLog@5ce7ab6f checkpointed with 5737 Records 
> and 0 Swap Files in 467 milliseconds (Stop-the-world time = 172 milliseconds, 
> Clear Edit Logs time = 137 millis), max Transaction ID 5739
> 2017-06-06 00:32:35,996 INFO [pool-10-thread-1] 
> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile 
> Repository with 5737 records in 467 milliseconds
> 2017-06-06 00:33:35,418 ERROR [Index Provenance Events-2] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:748)
> 2017-06-06 00:33:36,420 ERROR [Index Provenance Events-1] 
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: 
> NativeFSLock@/Users/jwitt/build-verify/nifi-1.3.0/nifi-assembly/target/nifi-1.3.0-bin/nifi-1.3.0/provenance_repository/index-1496723454612/write.lock
>   at org.apache.lucene.store.Lock.obtain(Lock.java:89)
>   at org.apache.lucene.index.IndexWriter.(IndexWriter.java:755)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.createWriter(SimpleIndexManager.java:198)
>   at 
> org.apache.nifi.provenance.lucene.SimpleIndexManager.borrowIndexWriter(SimpleIndexManager.java:227)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:184)
>   at 
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:104)
>   at 
>