Hello,
I have a production flow in place (nifi-1.1.1-RC1). It runs on a virtual machine running redhat. The reads multiple Json files (small) from a windows share/mountpoint. Various shell scripts write single lines (json) into files. The files are read, split and put into MongoDb.
The flow goes like this:
TailFile >> Split Text >> PutMongo
Today is 2017-03-24.
I have a webpage running which displays the data from the MongoDb. So I know when data arrives. And I can see that data arrived throughout the last days and also today
I have detected something I don't understand: TailFile shows me in the flow that it has processed flow file in the last 5 minutes. But when I show the provenance events then Nifi shows: "Oldest event available: 03/16/2017 18:02:34 CET". The list of events below it shows the most recent event as "03/24/2017 11:00:35.290 CET". But where are the events since then? I see that I have data in MongoDb after that.
My current nifi-app.log looks like this:
2017-03-24 16:28:38,472 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 27 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 12 millis), max Transaction ID 12245824
2017-03-24 16:30:38,506 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 34 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 15 millis), max Transaction ID 12247564
2017-03-24 16:32:38,540 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 11 millis), max Transaction ID 12248871
2017-03-24 16:30:38,506 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 34 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 15 millis), max Transaction ID 12247564
2017-03-24 16:32:38,540 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 11 millis), max Transaction ID 12248871
There are no errors in the log and when I look back around "03/24/2017 11:00:35.290 CET" there is nothing special either:
2017-03-24 10:56:32,547 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 34 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 17 millis), max Transaction ID 12002746
2017-03-24 10:58:32,580 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 16 millis), max Transaction ID 12004378
2017-03-24 10:58:32,580 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 16 millis), max Transaction ID 12004378
2017-03-24 11:00:32,624 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 43 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 20 millis), max Transaction ID 12005794
2017-03-24 11:00:35,339 INFO [Timer-Driven Process Thread-7] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 2237386
2017-03-24 11:00:50,975 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (111869 records) into single Provenance Log File ./provenance_repository/2125517.prov in 15677 milliseconds
2017-03-24 11:00:50,976 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 185608 records. In the past 5 minutes, 50 events have been written to the Provenance Repository, totaling 37.05 KB
2017-03-24 11:02:32,658 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 16 millis), max Transaction ID 12007317
2017-03-24 11:04:32,687 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 28 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 14 millis), max Transaction ID 12008733
2017-03-24 11:00:35,339 INFO [Timer-Driven Process Thread-7] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 2237386
2017-03-24 11:00:50,975 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (111869 records) into single Provenance Log File ./provenance_repository/2125517.prov in 15677 milliseconds
2017-03-24 11:00:50,976 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 185608 records. In the past 5 minutes, 50 events have been written to the Provenance Repository, totaling 37.05 KB
2017-03-24 11:02:32,658 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 33 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 16 millis), max Transaction ID 12007317
2017-03-24 11:04:32,687 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@2212e291 checkpointed with 91 Records and 0 Swap Files in 28 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 14 millis), max Transaction ID 12008733
So the flow is working but I don't have provenance events. Can somebody help?
Rgds,
Uwe
