Hi Noel, Based on the configuration of the processor, NiFi is supposed to delete the original file once retrieved by the processor (keep original file = false). Is the user running NiFi allowed to delete the file? If not, as per the documentation, the files will be ignored.
Thanks Pierre 2017-07-26 15:05 GMT+02:00 Juan Pablo Gardella <gardellajuanpa...@gmail.com> : > I had a similar issue but with ConsumerJMS. > > On Wed, 26 Jul 2017 at 10:04 Noel Alex Makumuli <alexnoe...@gmail.com> > wrote: > >> Hello Mark, >> Thanks for the prompt response. >> Yes the folder has the data. >> >> /data/data_warehouse/data_sources/PCT/PCT_Backfile_non_Asian/nifi/sources$ >> find -type f -iname wo\*.xml | wc -l >> 157905 >> >> The GetFile is not showing anything wrong run or attention that something >> is missing. >> >> I duplicated the whole flow and changed the sources a but. It work only >> worked for few thousands files and then it just no longer GetFile for >> extraction. >> >> I am sending couple of images of the processor GetFile configuration and >> you can advice further. >> >> Regards, >> Noel >> >> >> >> On 26 July 2017 at 15:44, Mark Payne <marka...@hotmail.com> wrote: >> >>> Noel, >>> >>> From the logs, it appears that you only have a handful of Processors. >>> The only one that is a "source" processor >>> is the GetFile. Are you sure that it is pointing to a directory that has >>> data in it? Is it possible that you've got >>> a typo there so that it isn't finding any data? Can you share the config >>> for that processor? >>> >>> Thanks >>> -Mark >>> >>> >>> On Jul 26, 2017, at 7:37 AM, Noel Alex Makumuli <alexnoe...@gmail.com> >>> wrote: >>> >>> Hello, >>> I am running Apache Nifi 1.3.0 on Ubuntu Server. >>> >>> The application is running fine and prints >>> <code> >>> 2017-07-26 14:01:54,131 INFO [main] org.apache.nifi.BootstrapListener >>> Successfully initiated communication with Bootstrap >>> 2017-07-26 14:01:54,131 INFO [main] org.apache.nifi.NiFi Controller >>> initialization took 15369655939 nanoseconds. >>> 2017-07-26 14:02:04,934 INFO [NiFi Web Server-50] >>> c.s.j.s.i.application.WebApplicationImpl >>> Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM' >>> 2017-07-26 14:02:30,370 INFO [StandardProcessScheduler Thread-5] >>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>> PutFile[id=dee71437-10c9-115d-38ff-22e157972467] to run with 1 threads >>> 2017-07-26 14:02:30,369 INFO [StandardProcessScheduler Thread-4] >>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>> PutFile[id=dee7141f-10c9-115d-dcbf-430c652f8b48] to run with 1 threads >>> 2017-07-26 14:02:30,370 INFO [StandardProcessScheduler Thread-3] >>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>> PutFile[id=dee71430-10c9-115d-8a1a-d2e0355524d6] to run with 1 threads >>> 2017-07-26 14:02:30,372 INFO [StandardProcessScheduler Thread-8] >>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>> PutFile[id=dee71426-10c9-115d-701b-e65fcd56d485] to run with 1 threads >>> 2017-07-26 14:02:30,372 INFO [StandardProcessScheduler Thread-5] >>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>> PutFile[id=dee7141c-10c9-115d-ecc1-4fb0c3d2a5a3] to run with 1 threads >>> 2017-07-26 14:02:30,374 INFO [StandardProcessScheduler Thread-4] >>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>> PutFile[id=dee71433-10c9-115d-ff6c-a0160d72c5d4] to run with 1 threads >>> 2017-07-26 14:02:30,373 INFO [StandardProcessScheduler Thread-6] >>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>> EvaluateXPath[id=dee71422-10c9-115d-bb94-79ed7852b19b] to run with 1 >>> threads >>> 2017-07-26 14:02:30,375 INFO [StandardProcessScheduler Thread-8] >>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>> PutFile[id=dee71428-10c9-115d-190e-c24016ed8015] to run with 1 threads >>> 2017-07-26 14:02:30,376 INFO [StandardProcessScheduler Thread-3] >>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>> ReplaceText[id=015d100b-704a-1013-58f4-955b7fe48f9c] to run with 1 >>> threads >>> 2017-07-26 14:02:30,381 INFO [StandardProcessScheduler Thread-7] >>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>> GetFile[id=015d10c8-dee7-1411-5904-52201a329835] to run with 1 threads >>> 2017-07-26 14:02:30,407 INFO [StandardProcessScheduler Thread-1] >>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>> PutSQL[id=74971126-015d-1000-0838-8db7f5065da7] to run with 1 threads >>> 2017-07-26 14:02:30,434 INFO [StandardProcessScheduler Thread-2] >>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>> TransformXml[id=791eee75-015d-1000-5c5d-df2bd5a4192b] to run with 1 >>> threads >>> 2017-07-26 14:02:30,585 INFO [Flow Service Tasks Thread-1] >>> o.a.nifi.controller.StandardFlowService Saved flow controller >>> org.apache.nifi.controller.FlowController@10c992ce // Another save >>> pending = false >>> 2017-07-26 14:03:50,962 INFO [Write-Ahead Local State Provider >>> Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali. >>> MinimalLockingWriteAheadLog@296a71df checkpointed with 114 Records and >>> 0 Swap Files in 27 milliseconds (Stop-the-world time = 2 milliseconds, >>> Clear Edit Logs time = 1 millis), max Transaction ID >>> 351 >>> 2017-07-26 14:03:52,616 INFO [pool-10-thread-1] >>> o.a.n.c.r.WriteAheadFlowFileRepository >>> Initiating checkpoint of FlowFile Repository >>> 2017-07-26 14:03:53,064 INFO [pool-10-thread-1] >>> org.wali.MinimalLockingWriteAheadLog >>> org.wali.MinimalLockingWriteAheadLog@76ee7301 checkpointed with 6705 >>> Records and 0 Swap Files in 448 milliseconds (Stop-the-world time = 132 >>> milliseconds, Clear Edit Logs time = 53 millis), max Transaction ID >>> 1597176945 >>> 2017-07-26 14:03:53,065 INFO [pool-10-thread-1] >>> o.a.n.c.r.WriteAheadFlowFileRepository >>> Successfully checkpointed FlowFile Repository with 6705 records in 448 >>> milliseconds >>> 2017-07-26 14:05:50,980 INFO [Write-Ahead Local State Provider >>> Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali. >>> MinimalLockingWriteAheadLog@296a71df checkpointed with 114 Records and >>> 0 Swap Files in 17 milliseconds (Stop-the-world time = 2 milliseconds, >>> Clear Edit Logs time = 2 millis), max Transaction ID >>> 351 >>> 2017-07-26 14:05:53,065 INFO [pool-10-thread-1] >>> o.a.n.c.r.WriteAheadFlowFileRepository >>> Initiating checkpoint of FlowFile Repository >>> 2017-07-26 14:05:53,477 INFO [pool-10-thread-1] >>> org.wali.MinimalLockingWriteAheadLog >>> org.wali.MinimalLockingWriteAheadLog@76ee7301 checkpointed with 6705 >>> Records and 0 Swap Files in 411 milliseconds (Stop-the-world time = 91 >>> milliseconds, Clear Edit Logs time = 63 millis), max Transaction ID >>> 1597176945 >>> 2017-07-26 14:05:53,477 INFO [pool-10-thread-1] >>> o.a.n.c.r.WriteAheadFlowFileRepository >>> Successfully checkpointed FlowFile Repository with 6705 records in 411 >>> milliseconds >>> 2017-07-26 14:07:50,998 INFO [Write-Ahead Local State Provider >>> Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali. >>> MinimalLockingWriteAheadLog@296a71df checkpointed with 114 Records and >>> 0 Swap Files in 16 milliseconds (Stop-the-world time = 2 milliseconds, >>> Clear Edit Logs time = 1 millis), max Transaction ID >>> 351 >>> 2017-07-26 14:07:53,478 INFO [pool-10-thread-1] >>> o.a.n.c.r.WriteAheadFlowFileRepository >>> Initiating checkpoint of FlowFile Repository >>> 2017-07-26 14:07:53,885 INFO [pool-10-thread-1] >>> org.wali.MinimalLockingWriteAheadLog >>> org.wali.MinimalLockingWriteAheadLog@76ee7301 checkpointed with 6705 >>> Records and 0 Swap Files in 406 milliseconds (Stop-the-world time = 86 >>> milliseconds, Clear Edit Logs time = 55 millis), max Transaction ID >>> 1597176945 >>> 2017-07-26 14:07:53,885 INFO [pool-10-thread-1] >>> o.a.n.c.r.WriteAheadFlowFileRepository >>> Successfully checkpointed FlowFile Repository with 6705 records in 406 >>> milliseconds >>> </code> >>> >>> The bootstrap is showing everything is normal only nifi-app.log is >>> printing the above logs >>> >>> While the bootstrap.log is printing like below and nothing out of the >>> ordinary or any exceptions. >>> <code> >>> ce/LOGS org.apache.nifi.NiFi >>> 2017-07-26 14:01:34,005 INFO [main] org.apache.nifi.bootstrap.Command >>> Launched Apache NiFi with Proc >>> ess ID 12143 >>> 2017-07-26 14:01:34,705 INFO [NiFi Bootstrap Command Listener] >>> org.apache.nifi.bootstrap.RunNiFi Apa >>> che NiFi now running and listening for Bootstrap requests on port 32977 >>> </code> >>> >>> I changed couple of few things on the configuration mainly the memory to >>> gain performance but nothing further.. I am no longer sure what is >>> happening. >>> onifi >>> I have attached my log file , bootstrap & nifi configuration. >>> >>> on my configuration files.. >>> i disable the # The G1GC is still considered experimental but has proven >>> to be very advantageous in providing great >>> # performance without significant "stop-the-world" delays. >>> #java.arg.13=-XX:+UseG1GC >>> >>> I saw in one thread people complaining of this. >>> >>> I have lots of files which need to get processed but nifi is just >>> standing there while it should be ingesting the file. >>> My last change on the nifi.properties file was:: >>> # nifi.flow.configuration.archive.max.storage=500 MB >>> nifi.flow.configuration.archive.max.storage=50 GB >>> Could this be the culprit..?? Because reverting back does seem to help >>> neither..!! >>> >>> I apologies if I have failed to frame my question correctly. >>> >>> Please advice. >>> -- >>> NOEL ALEX MAKUMULI >>> >>> <bootstrap.txt><nifi.txt><nifi-app.log> >>> >>> >>> >> >> >> -- >> NOEL ALEX MAKUMULI >> TANZANIA >> >> Mobile: +255 755 100 700 <+255%20755%20100%20700> ( Active) >> Mobile: +358 44 3077 817 ( Active) >> >
