Aw: Re: Nifi - Data provenance not reporting anymore

Sat, 01 Apr 2017 04:13:08 -0700

Mark,
 
I tried that. The problem seems to be more difficult than that.
 
When I query for todays date: 07:00 until midnight for the TailFile processor, then I get results, but only until 07:58. Nifi shows 143 out of 143 events on that search. If I do another search starting at 08:00 the result is empty. But I have verified that after that there were other events. The monitoring app that is fed from it shows other activities. Also the Nifi flow itself shows activities in the last 5 minutes on the graph.
 
So next week I will stop nifi, clear provenance logs and other things and restart and watch what happens. The monitoring process is not that crucial that I can't deleted the old events in nifi and start over again.
 
I will let you know what happens.
 
Rgds,
 
Uwe
 
Gesendet: Donnerstag, 30. März 2017 um 17:54 Uhr
Von: "Mark Payne" <[email protected]>
An: "[email protected]" <[email protected]>
Betreff: Re: Nifi - Data provenance not reporting anymore
Uwe,
 
When you issue a Provenance Query, if it finds more results than the max (the UI requests 1000 results max)
then it will simply return up to the max results. It does not necessarily return the newest results. If you click on the
Search button in the Provenance UI and restrict the time window to say the past 10 minutes you'll see just those
results for the past 10 mins. Right now, the screenshot shows that you've got more than 1,000 hits so you'd need
to narrow your search in order to get the most recent events.
 
Thanks
-Mark
 
 
On Mar 30, 2017, at 3:06 AM, Uwe Geercken <[email protected]> wrote:
 
James,
 
here is the screenshot of the data provenance:
 
 
and here my flow:
 
 
 
If I clear the search in provenance it is still not showing events from the TailFile processor although it is running.
 
Rgds,
 
Uwe
 
Gesendet: Freitag, 24. März 2017 um 19:27 Uhr
Von: "James Wing" <[email protected]>
An: [email protected]
Betreff: Re: Nifi - Data provenance not reporting anymore
Uwe,
 
Can you share a screenshot of the provenance search criteria?  If you try the search the other way around, starting with all provenance events and then filtering down to the time and processor, does that change anything?
 
Thanks,
 
James
 
On Fri, Mar 24, 2017 at 8:39 AM, Uwe Geercken <[email protected]> wrote:
Hello,
 
I have a production flow in place (nifi-1.1.1-RC1). It runs on a virtual machine running redhat. The reads multiple Json files (small) from a windows share/mountpoint. Various shell scripts write single lines (json) into files. The files are read, split and put into MongoDb.
 
The flow goes like this:
 
TailFile >> Split Text >> PutMongo
 
Today is 2017-03-24.
 
I have a webpage running which displays the data from the MongoDb. So I know when data arrives. And I can see that data arrived throughout the last days and also today
 
I have detected something I don't understand: TailFile shows me in the flow that it has processed flow file in the last 5 minutes. But when I show the provenance events then Nifi shows: "Oldest event available: 03/16/2017 18:02:34 CET". The list of events below it shows the most recent event as "03/24/2017 11:00:35.290 CET". But where are the events since then? I see that I have data in MongoDb after that.
 
My current nifi-app.log looks like this:
 
2017-03-24 16:28:38,472 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 27 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 12 millis), max Transaction ID 12245824
2017-03-24 16:30:38,506 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 34 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 15 millis), max Transaction ID 12247564
2017-03-24 16:32:38,540 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 11 millis), max Transaction ID 12248871
 
There are no errors in the log and when I look back around "03/24/2017 11:00:35.290 CET" there is nothing special either:
 
2017-03-24 10:56:32,547 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 34 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 17 millis), max Transaction ID 12002746
2017-03-24 10:58:32,580 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 16 millis), max Transaction ID 12004378
2017-03-24 11:00:32,624 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 43 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 20 millis), max Transaction ID 12005794
2017-03-24 11:00:35,339 INFO [Timer-Driven Process Thread-7] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 2237386
2017-03-24 11:00:50,975 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (111869 records) into single Provenance Log File ./provenance_repository/2125517.prov in 15677 milliseconds
2017-03-24 11:00:50,976 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 185608 records. In the past 5 minutes, 50 events have been written to the Provenance Repository, totaling 37.05 KB
2017-03-24 11:02:32,658 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 16 millis), max Transaction ID 12007317
2017-03-24 11:04:32,687 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 28 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 14 millis), max Transaction ID 12008733
 
 
So the flow is working but I don't have provenance events. Can somebody help?
 
Rgds,
 
Uwe
 

Reply via email to