Hi ,
I'm trying to fetch a set of files which have recently changed
in a "filesystem". Also I'm supposed to keep the original copy as it is.
For obtaining the latest files that have changed, I'm using a PutFile with
"replace" strategy piped to a GetFile with a minimum age of 5 sec, max file
age of 30 sec, Keep source file as true,
Also, running it in clustered mode. I'm seeing the below issues
- The queue starts growing if there's an error.
- Continuous errors with 'NoSuchFileException'
- Penalizing StandardFlowFileErrors
ERROR
0ab3b920-1f05-4f24-b861-4fded3d5d826
161.91.234.248:7087
GetFile[id=0ab3b920-1f05-4f24-b861-4fded3d5d826] Failed to retrieve files due
to org.apache.nifi.processor.exception.FlowFileAccessException: Failed to
import data from /nifi/UNZ/log201403230000.log for
StandardFlowFileRecord[uuid=f29bda59-8611-427c-b4d7-c921ee5e74b8,claim=,offset=0,name=6908587554457536,size=0]
due to java.nio.file.NoSuchFileException: /nifi/UNZ/log201403230000.log
18:45:56 IST
10:54:50 IST
ERROR
c552b5bc-f627-3cc3-b3d0-545c519eafd9
161.91.234.248:6087
PutFile[id=c552b5bc-f627-3cc3-b3d0-545c519eafd9] Penalizing
StandardFlowFileRecord[uuid=876e51f7-9a3d-4bf9-9d11-9073a5c950ad,claim=1430717088883-73580,offset=0,name=file1.log,size=29314779]
and transferring to failure due to
org.apache.nifi.processor.exception.ProcessException: Could not rename
/nifi/UNZ/.file1.log: org.apache.nifi.processor.exception.ProcessException:
Could not rename: /nifi/UNZ/.file1.log
10:54:56 IST
ERROR
60662bb3-490a-3b47-9371-e11c12cdfa1a
161.91.234.248:7087
PutFile[id=60662bb3-490a-3b47-9371-e11c12cdfa1a] Penalizing
StandardFlowFileRecord[uuid=522a2401-8269-4f0f-aff5-152d25cdcefa,claim=1430717094668-73059,offset=1533296,name=file2.log,size=28014262]
and transferring to failure due to
org.apache.nifi.processor.exception.ProcessException: Could not rename:
/data/softwares/RS/nifi/OUT/.file2.log:
org.apache.nifi.processor.exception.ProcessException: Could not rename:
/nifi/OUT/.file2.log
Do I have to tweak the Run schedule or keep the same minimum file age and
maximum file age to overcome this issue?
What might be an elegant solution in NiFi?
Thanks,
anup
________________________________
The information contained in this message may be confidential and legally
protected under applicable law. The message is intended solely for the
addressee(s). If you are not the intended recipient, you are hereby notified
that any use, forwarding, dissemination, or reproduction of this message is
strictly prohibited and may be unlawful. If you are not the intended recipient,
please contact the sender by return e-mail and destroy all copies of the
original message.
