Good point Pierre. Could also be the api to list directories that have so many files in them. I recommend trying on a directory with less things to list and see if that helps too as that means we need to evaluate other things.
Also, we clearly could make it easier for users to see what is going on. That it shows there is a thread on GetFile is cool but surfacing more details about what is going on would really help. We need to have a reporting API for processors to build against so they can show more complex status data beyond what bulletins and such offer. On Wed, Jul 26, 2017 at 9:12 AM, Pierre Villard <pierre.villard...@gmail.com> wrote: > Hi Noel, > > Based on the configuration of the processor, NiFi is supposed to delete the > original file once retrieved by the processor (keep original file = false). > Is the user running NiFi allowed to delete the file? If not, as per the > documentation, the files will be ignored. > > Thanks > Pierre > > > > 2017-07-26 15:05 GMT+02:00 Juan Pablo Gardella > <gardellajuanpa...@gmail.com>: >> >> I had a similar issue but with ConsumerJMS. >> >> On Wed, 26 Jul 2017 at 10:04 Noel Alex Makumuli <alexnoe...@gmail.com> >> wrote: >>> >>> Hello Mark, >>> Thanks for the prompt response. >>> Yes the folder has the data. >>> >>> >>> /data/data_warehouse/data_sources/PCT/PCT_Backfile_non_Asian/nifi/sources$ >>> find -type f -iname wo\*.xml | wc -l >>> 157905 >>> >>> The GetFile is not showing anything wrong run or attention that something >>> is missing. >>> >>> I duplicated the whole flow and changed the sources a but. It work only >>> worked for few thousands files and then it just no longer GetFile for >>> extraction. >>> >>> I am sending couple of images of the processor GetFile configuration and >>> you can advice further. >>> >>> Regards, >>> Noel >>> >>> >>> >>> On 26 July 2017 at 15:44, Mark Payne <marka...@hotmail.com> wrote: >>>> >>>> Noel, >>>> >>>> From the logs, it appears that you only have a handful of Processors. >>>> The only one that is a "source" processor >>>> is the GetFile. Are you sure that it is pointing to a directory that has >>>> data in it? Is it possible that you've got >>>> a typo there so that it isn't finding any data? Can you share the config >>>> for that processor? >>>> >>>> Thanks >>>> -Mark >>>> >>>> >>>> On Jul 26, 2017, at 7:37 AM, Noel Alex Makumuli <alexnoe...@gmail.com> >>>> wrote: >>>> >>>> Hello, >>>> I am running Apache Nifi 1.3.0 on Ubuntu Server. >>>> >>>> The application is running fine and prints >>>> <code> >>>> 2017-07-26 14:01:54,131 INFO [main] org.apache.nifi.BootstrapListener >>>> Successfully initiated communication with Bootstrap >>>> 2017-07-26 14:01:54,131 INFO [main] org.apache.nifi.NiFi Controller >>>> initialization took 15369655939 nanoseconds. >>>> 2017-07-26 14:02:04,934 INFO [NiFi Web Server-50] >>>> c.s.j.s.i.application.WebApplicationImpl Initiating Jersey application, >>>> version 'Jersey: 1.19 02/11/2015 03:25 AM' >>>> 2017-07-26 14:02:30,370 INFO [StandardProcessScheduler Thread-5] >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>>> PutFile[id=dee71437-10c9-115d-38ff-22e157972467] to run with 1 threads >>>> 2017-07-26 14:02:30,369 INFO [StandardProcessScheduler Thread-4] >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>>> PutFile[id=dee7141f-10c9-115d-dcbf-430c652f8b48] to run with 1 threads >>>> 2017-07-26 14:02:30,370 INFO [StandardProcessScheduler Thread-3] >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>>> PutFile[id=dee71430-10c9-115d-8a1a-d2e0355524d6] to run with 1 threads >>>> 2017-07-26 14:02:30,372 INFO [StandardProcessScheduler Thread-8] >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>>> PutFile[id=dee71426-10c9-115d-701b-e65fcd56d485] to run with 1 threads >>>> 2017-07-26 14:02:30,372 INFO [StandardProcessScheduler Thread-5] >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>>> PutFile[id=dee7141c-10c9-115d-ecc1-4fb0c3d2a5a3] to run with 1 threads >>>> 2017-07-26 14:02:30,374 INFO [StandardProcessScheduler Thread-4] >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>>> PutFile[id=dee71433-10c9-115d-ff6c-a0160d72c5d4] to run with 1 threads >>>> 2017-07-26 14:02:30,373 INFO [StandardProcessScheduler Thread-6] >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>>> EvaluateXPath[id=dee71422-10c9-115d-bb94-79ed7852b19b] to run with 1 >>>> threads >>>> 2017-07-26 14:02:30,375 INFO [StandardProcessScheduler Thread-8] >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>>> PutFile[id=dee71428-10c9-115d-190e-c24016ed8015] to run with 1 threads >>>> 2017-07-26 14:02:30,376 INFO [StandardProcessScheduler Thread-3] >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>>> ReplaceText[id=015d100b-704a-1013-58f4-955b7fe48f9c] to run with 1 threads >>>> 2017-07-26 14:02:30,381 INFO [StandardProcessScheduler Thread-7] >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>>> GetFile[id=015d10c8-dee7-1411-5904-52201a329835] to run with 1 threads >>>> 2017-07-26 14:02:30,407 INFO [StandardProcessScheduler Thread-1] >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>>> PutSQL[id=74971126-015d-1000-0838-8db7f5065da7] to run with 1 threads >>>> 2017-07-26 14:02:30,434 INFO [StandardProcessScheduler Thread-2] >>>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled >>>> TransformXml[id=791eee75-015d-1000-5c5d-df2bd5a4192b] to run with 1 threads >>>> 2017-07-26 14:02:30,585 INFO [Flow Service Tasks Thread-1] >>>> o.a.nifi.controller.StandardFlowService Saved flow controller >>>> org.apache.nifi.controller.FlowController@10c992ce // Another save pending >>>> = >>>> false >>>> 2017-07-26 14:03:50,962 INFO [Write-Ahead Local State Provider >>>> Maintenance] org.wali.MinimalLockingWriteAheadLog >>>> org.wali.MinimalLockingWriteAheadLog@296a71df checkpointed with 114 Records >>>> and 0 Swap Files in 27 milliseconds (Stop-the-world time = 2 milliseconds, >>>> Clear Edit Logs time = 1 millis), max Transaction ID >>>> 351 >>>> 2017-07-26 14:03:52,616 INFO [pool-10-thread-1] >>>> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >>>> Repository >>>> 2017-07-26 14:03:53,064 INFO [pool-10-thread-1] >>>> org.wali.MinimalLockingWriteAheadLog >>>> org.wali.MinimalLockingWriteAheadLog@76ee7301 checkpointed with 6705 >>>> Records >>>> and 0 Swap Files in 448 milliseconds (Stop-the-world time = 132 >>>> milliseconds, Clear Edit Logs time = 53 millis), max Transaction ID >>>> 1597176945 >>>> 2017-07-26 14:03:53,065 INFO [pool-10-thread-1] >>>> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile >>>> Repository with 6705 records in 448 milliseconds >>>> 2017-07-26 14:05:50,980 INFO [Write-Ahead Local State Provider >>>> Maintenance] org.wali.MinimalLockingWriteAheadLog >>>> org.wali.MinimalLockingWriteAheadLog@296a71df checkpointed with 114 Records >>>> and 0 Swap Files in 17 milliseconds (Stop-the-world time = 2 milliseconds, >>>> Clear Edit Logs time = 2 millis), max Transaction ID >>>> 351 >>>> 2017-07-26 14:05:53,065 INFO [pool-10-thread-1] >>>> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >>>> Repository >>>> 2017-07-26 14:05:53,477 INFO [pool-10-thread-1] >>>> org.wali.MinimalLockingWriteAheadLog >>>> org.wali.MinimalLockingWriteAheadLog@76ee7301 checkpointed with 6705 >>>> Records >>>> and 0 Swap Files in 411 milliseconds (Stop-the-world time = 91 >>>> milliseconds, >>>> Clear Edit Logs time = 63 millis), max Transaction ID 1597176945 >>>> 2017-07-26 14:05:53,477 INFO [pool-10-thread-1] >>>> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile >>>> Repository with 6705 records in 411 milliseconds >>>> 2017-07-26 14:07:50,998 INFO [Write-Ahead Local State Provider >>>> Maintenance] org.wali.MinimalLockingWriteAheadLog >>>> org.wali.MinimalLockingWriteAheadLog@296a71df checkpointed with 114 Records >>>> and 0 Swap Files in 16 milliseconds (Stop-the-world time = 2 milliseconds, >>>> Clear Edit Logs time = 1 millis), max Transaction ID >>>> 351 >>>> 2017-07-26 14:07:53,478 INFO [pool-10-thread-1] >>>> o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile >>>> Repository >>>> 2017-07-26 14:07:53,885 INFO [pool-10-thread-1] >>>> org.wali.MinimalLockingWriteAheadLog >>>> org.wali.MinimalLockingWriteAheadLog@76ee7301 checkpointed with 6705 >>>> Records >>>> and 0 Swap Files in 406 milliseconds (Stop-the-world time = 86 >>>> milliseconds, >>>> Clear Edit Logs time = 55 millis), max Transaction ID 1597176945 >>>> 2017-07-26 14:07:53,885 INFO [pool-10-thread-1] >>>> o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile >>>> Repository with 6705 records in 406 milliseconds >>>> </code> >>>> >>>> The bootstrap is showing everything is normal only nifi-app.log is >>>> printing the above logs >>>> >>>> While the bootstrap.log is printing like below and nothing out of the >>>> ordinary or any exceptions. >>>> <code> >>>> ce/LOGS org.apache.nifi.NiFi >>>> 2017-07-26 14:01:34,005 INFO [main] org.apache.nifi.bootstrap.Command >>>> Launched Apache NiFi with Proc >>>> ess ID 12143 >>>> 2017-07-26 14:01:34,705 INFO [NiFi Bootstrap Command Listener] >>>> org.apache.nifi.bootstrap.RunNiFi Apa >>>> che NiFi now running and listening for Bootstrap requests on port 32977 >>>> </code> >>>> >>>> I changed couple of few things on the configuration mainly the memory to >>>> gain performance but nothing further.. I am no longer sure what is >>>> happening. >>>> onifi >>>> I have attached my log file , bootstrap & nifi configuration. >>>> >>>> on my configuration files.. >>>> i disable the # The G1GC is still considered experimental but has proven >>>> to be very advantageous in providing great >>>> # performance without significant "stop-the-world" delays. >>>> #java.arg.13=-XX:+UseG1GC >>>> >>>> I saw in one thread people complaining of this. >>>> >>>> I have lots of files which need to get processed but nifi is just >>>> standing there while it should be ingesting the file. >>>> My last change on the nifi.properties file was:: >>>> # nifi.flow.configuration.archive.max.storage=500 MB >>>> nifi.flow.configuration.archive.max.storage=50 GB >>>> Could this be the culprit..?? Because reverting back does seem to help >>>> neither..!! >>>> >>>> I apologies if I have failed to frame my question correctly. >>>> >>>> Please advice. >>>> -- >>>> NOEL ALEX MAKUMULI >>>> >>>> <bootstrap.txt><nifi.txt><nifi-app.log> >>>> >>>> >>> >>> >>> >>> -- >>> NOEL ALEX MAKUMULI >>> TANZANIA >>> >>> Mobile: +255 755 100 700 ( Active) >>> Mobile: +358 44 3077 817 ( Active) > >
