Aw: Re: Nifi - Data provenance not reporting anymore

[Uwe Geercken]

James,

 

where should I put the screenshot? This mail group does not allow to send graphics.

 

Rgds,

 

Uwe

 

Gesendet: Freitag, 24. März 2017 um 18:27 Uhr
Von: "James Wing" <jvw...@gmail.com>
An: users@nifi.apache.org
Betreff: Re: Nifi - Data provenance not reporting anymore

Uwe,
 

Can you share a screenshot of the provenance search criteria?  If you try the search the other way around, starting with all provenance events and then filtering down to the time and processor, does that change anything?
 

Thanks,
 

James

 

On Fri, Mar 24, 2017 at 8:39 AM, Uwe Geercken <uwe.geerc...@web.de> wrote:

Hello,

 

I have a production flow in place (nifi-1.1.1-RC1). It runs on a virtual machine running redhat. The reads multiple Json files (small) from a windows share/mountpoint. Various shell scripts write single lines (json) into files. The files are read, split and put into MongoDb.

 

The flow goes like this:

 

TailFile >> Split Text >> PutMongo

 

Today is 2017-03-24.

 

I have a webpage running which displays the data from the MongoDb. So I know when data arrives. And I can see that data arrived throughout the last days and also today

 

I have detected something I don't understand: TailFile shows me in the flow that it has processed flow file in the last 5 minutes. But when I show the provenance events then Nifi shows: "Oldest event available: 03/16/2017 18:02:34 CET". The list of events below it shows the most recent event as "03/24/2017 11:00:35.290 CET". But where are the events since then? I see that I have data in MongoDb after that.

 

My current nifi-app.log looks like this:

 

2017-03-24 16:28:38,472 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 27 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 12 millis), max Transaction ID 12245824
2017-03-24 16:30:38,506 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 34 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 15 millis), max Transaction ID 12247564
2017-03-24 16:32:38,540 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 11 millis), max Transaction ID 12248871

 

There are no errors in the log and when I look back around "03/24/2017 11:00:35.290 CET" there is nothing special either:

 

2017-03-24 10:56:32,547 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 34 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 17 millis), max Transaction ID 12002746
2017-03-24 10:58:32,580 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 16 millis), max Transaction ID 12004378

2017-03-24 11:00:32,624 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 43 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 20 millis), max Transaction ID 12005794
2017-03-24 11:00:35,339 INFO [Timer-Driven Process Thread-7] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 2237386
2017-03-24 11:00:50,975 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (111869 records) into single Provenance Log File ./provenance_repository/2125517.prov in 15677 milliseconds
2017-03-24 11:00:50,976 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 185608 records. In the past 5 minutes, 50 events have been written to the Provenance Repository, totaling 37.05 KB
2017-03-24 11:02:32,658 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 16 millis), max Transaction ID 12007317
2017-03-24 11:04:32,687 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 28 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 14 millis), max Transaction ID 12008733

 

 

So the flow is working but I don't have provenance events. Can somebody help?

 

Rgds,

 

Uwe