These scenarios are ripe for io race conditions between the sending process (the thing putting files in the directory) and the receiving process (nifi). It is vital to ensure the writer either changes the name when done writing (best case) and the reader only looks at that pattern OR you ensure the file is of sufficient age before grabbing (usually good but not foolproof).
thanks On Wed, Aug 12, 2020 at 7:18 AM Valentina Ivanova <valentina.ivan...@ri.se> wrote: > > > > > > > > > > > > > > > Hi Phillip, > > > > > > > > > > > > I am leaning to the solution you are proposing - I can't identify the > problem. > > > > > > > I am not using ListSFTP but executing a small script to obtain the list > with files using ExecuteStreamCommand. > > > > > > > > > > > > Thanks & all the best > > > > > > > > > > > > Valentina > > > > > > > ------------------------------ > > > *From:* Phillip Grenier <phillip@nifi.rocks> > > > *Sent:* Wednesday, 12 August 2020 16:02 > > > *To:* users@nifi.apache.org <users@nifi.apache.org> > > > *Subject:* Re: FetchSFTP Failed to fetch content for > StandardFlowFileRecord > > > > > > > > > > > > > Valentina, > > > > > > > > I have seen cases where the server will delete files after being touched > by the list(S)FTP processor, so if the fetch processor isn't fast enough it > gets deleted and causes that issue. If the file really does still exist on > the server then I would try a simple > > retry loop and see if a second attempt is successful. > > > > > > > > > > > > On Wed, Aug 12, 2020 at 8:08 AM Valentina Ivanova <valentina.ivan...@ri.se> > wrote: > > > > > > > > > > > Hi all! > > > > > > > > > > > > I am retrieving files thousands of files with FetchSFTP and from time to > time (say 1 per 10 min) I receive the following error. > > > > > > > > > > > > FetchSFTP[id=e19a32eb-f9c6-13a0-38b4-45a726ec4273] Failed to fetch content > for > StandardFlowFileRecord[uuid=82f316a2-f797-4d32-8aa8-3d1907b54e01,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1597232401901-271, > container=default, section=271], > > offset=515807, length=8690914],offset=8630853,name=1576278001264,size=109] > from filename [FILENAME] on remote host xxx.xxx.xxx.xxx:xxx due to > java.io.IOException: Failed to obtain file content for [FILENAME]; routing > to comms.failure: java.io.IOException: > > Failed to obtain file content for [FILENAME] > > > > > > > > > > > > The files are all there and are similar in size and content to the others > successfully fetched. Any idea what the problem could be? > > > > > > > > > > > > Thanks & best > > > > > > > > > > > > Valentina > > > > > > > > > > > > > > > > >
