Hello Tomislav, Are these processors running in a multi-node cluster? Is FetchFile downstream from a ListFile processor that is scheduled to run on all nodes versus Primary Node only? Is FetchFile's Completion Strategy set to "Move File" or "Delete File"? Typically, source processors should be scheduled to run on the primary node, otherwise when reading from the same source across multiple nodes, for example a shared network drive, each source processor might pull the same data. In a situation like this, the same file could be listed by each node, and the FetchFile processor on each node may attempt to fetch the same file.
If you set the source processor to run on Primary Node only, you can load-balance the connection between the source processor and FetchFile to distribute the load of fetching the files across the cluster. On Thu, Oct 3, 2019 at 2:32 AM Tomislav Novosel <to.novo...@gmail.com> wrote: > Hi all, > > I'm getting errors from FetchFile and UnpackContent processors. > I have pipeline where I fetch zip files as they come continuously on > shared network drive > with Minimum file age set to 30 sec to avoid fetching file before it is > written to disk completely. > > Sometimes I get this error from FetchFile: > > FetchFile[id=c741187c-1172-1166-e752-1f79197a8029] Could not fetch file > \\avl01\ATGRZ\TestFactory\02 Dep Service\01 > Processdata\Backup\dfs_atfexport\MANA38\ANA_12_BPE7347\ANA_12_BPE7347_TDL_HL_1\measurement_file.atf.zip > from file system for > StandardFlowFileRecord[uuid=e7a5e3c4-0981-4ff3-85ea-91e41f0c3c0e,claim=,offset=0,name=PEI_BPE7347_TDLHL1new_826_20191001161312.atf.zip,size=0] > because the existence of the file cannot be verified; routing to failure > > > And from UnpackContent sometimes I get this error: > > > UnpackContent[id=0164106c-d3b7-1e3f-c770-6e6e07f9259d] Unable to unpack > StandardFlowFileRecord[uuid=4a019d58-fe45-4276-a161-e46cd8b1667c,claim=StandardContentClaim > [resourceClaim=StandardResourceClaim[id=1570052741201-5000, > container=default, section=904], offset=1651, > length=28417768],offset=0,name=measurement.atf.zip,size=28417768] due to > IOException thrown from > UnpackContent[id=0164106c-d3b7-1e3f-c770-6e6e07f9259d]: > java.io.IOException: Truncated ZIP file; routing to failure: > > org.apache.nifi.processor.exception.ProcessException: IOException thrown > from UnpackContent[id=0164106c-d3b7-1e3f-c770-6e6e07f9259d]: > java.io.IOException: Truncated ZIP file > > > After getting this error from UnpackContent I tried to fetch file again > and to unpack it. It went well, without any errors. > So what does this errors mean? I spoke to colleagues who are using this > files on the source side and they said files are ok, not corrupted or > something. > > Please help or give advice. > > Thanks in advance. > Tom > > > > > > > > > > >
