Uwe, When you issue a Provenance Query, if it finds more results than the max (the UI requests 1000 results max) then it will simply return up to the max results. It does not necessarily return the newest results. If you click on the Search button in the Provenance UI and restrict the time window to say the past 10 minutes you'll see just those results for the past 10 mins. Right now, the screenshot shows that you've got more than 1,000 hits so you'd need to narrow your search in order to get the most recent events.
Thanks -Mark On Mar 30, 2017, at 3:06 AM, Uwe Geercken <[email protected]<mailto:[email protected]>> wrote: James, here is the screenshot of the data provenance: http://imgur.com/a/29M8I and here my flow: http://imgur.com/a/nd6v7 If I clear the search in provenance it is still not showing events from the TailFile processor although it is running. Rgds, Uwe Gesendet: Freitag, 24. März 2017 um 19:27 Uhr Von: "James Wing" <[email protected]<mailto:[email protected]>> An: [email protected]<mailto:[email protected]> Betreff: Re: Nifi - Data provenance not reporting anymore Uwe, Can you share a screenshot of the provenance search criteria? If you try the search the other way around, starting with all provenance events and then filtering down to the time and processor, does that change anything? Thanks, James On Fri, Mar 24, 2017 at 8:39 AM, Uwe Geercken <[email protected]<mailto:[email protected]>> wrote: Hello, I have a production flow in place (nifi-1.1.1-RC1). It runs on a virtual machine running redhat. The reads multiple Json files (small) from a windows share/mountpoint. Various shell scripts write single lines (json) into files. The files are read, split and put into MongoDb. The flow goes like this: TailFile >> Split Text >> PutMongo Today is 2017-03-24. I have a webpage running which displays the data from the MongoDb. So I know when data arrives. And I can see that data arrived throughout the last days and also today I have detected something I don't understand: TailFile shows me in the flow that it has processed flow file in the last 5 minutes. But when I show the provenance events then Nifi shows: "Oldest event available: 03/16/2017 18:02:34 CET". The list of events below it shows the most recent event as "03/24/2017 11:00:35.290 CET". But where are the events since then? I see that I have data in MongoDb after that. My current nifi-app.log looks like this: 2017-03-24 16:28:38,472 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 27 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 12 millis), max Transaction ID 12245824 2017-03-24 16:30:38,506 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 34 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 15 millis), max Transaction ID 12247564 2017-03-24 16:32:38,540 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 11 millis), max Transaction ID 12248871 There are no errors in the log and when I look back around "03/24/2017 11:00:35.290 CET" there is nothing special either: 2017-03-24 10:56:32,547 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 34 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 17 millis), max Transaction ID 12002746 2017-03-24 10:58:32,580 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 16 millis), max Transaction ID 12004378 2017-03-24 11:00:32,624 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 43 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 20 millis), max Transaction ID 12005794 2017-03-24 11:00:35,339 INFO [Timer-Driven Process Thread-7] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 2237386 2017-03-24 11:00:50,975 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (111869 records) into single Provenance Log File ./provenance_repository/2125517.prov in 15677 milliseconds 2017-03-24 11:00:50,976 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 185608 records. In the past 5 minutes, 50 events have been written to the Provenance Repository, totaling 37.05 KB 2017-03-24 11:02:32,658 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 16 millis), max Transaction ID 12007317 2017-03-24 11:04:32,687 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 28 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 14 millis), max Transaction ID 12008733 So the flow is working but I don't have provenance events. Can somebody help? Rgds, Uwe
