[
https://issues.apache.org/jira/browse/NIFI-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16867597#comment-16867597
]
Joseph Witt commented on NIFI-6367:
-----------------------------------
Does the flowfile get routed to a failure relationship or is the session in
nifi rolled back?
If it is routed to failure then it is on the person designing the flow to pick
their poison in terms of whether to retry or not. Or, if the person has
insufficient failure data to make that decision we need to offer more context
(code change). Or, if it is rolledback we need to catch/look for this case in
particular and ensure it is routed to failure and/or some relationship making
it clear that the md5 doesn't match.
Is the case here that a ListS3 has given a flowfile with a file path and md5
but then during FetchS3 the md5 of the downloaded item doesn't match? Or
rather it is that the S3 client lib itself is getting a different md5 reported
as metadata which it finds doesn't match to the actual data?
> FetchS3Processor responds to md5 error on download by doing download again,
> again, and again
> --------------------------------------------------------------------------------------------
>
> Key: NIFI-6367
> URL: https://issues.apache.org/jira/browse/NIFI-6367
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.7.1
> Environment: NIFI (CentOS 7.2) with FetchS3Object running towards S3
> enviroment (non public). Enviroment / S3 had errors that introduced md5
> errors on sub 0.5% of downloads. Downloads with md5 errors accumulated in the
> input que of the processor.
> Reporter: Kefevs Pirkibo
> Assignee: Evan Reynolds
> Priority: Critical
>
> (6months old, but don't see changes in the relevant parts of the code, though
> I might be mistaken. This might be hard to replicate, so suggest a code
> wizard check if this is still a problem. )
> Case: NIFI running with FetchS3Object processor(s) towards S3 enviroment (non
> public). The enviroment and S3 had in combination hardware errors that
> resulted in sporadic md5 errors on the same files over and over again. Md5
> errors resulted in an unhandled AmazonClientException, and the file was
> downloaded yet again. (Reverted to the input que, first in line.) In our case
> this was identified after a number of days, with substantial bandwidth usage.
> It did not help that the FetchS3Objects where running with multiple
> instances, and after days accumulated the bad md5 checksum files for
> continuous download.
> Suggest: Someone code savy check what happens to files that are downloaded
> with bad md5, if they are reverted to the que due to uncought exception or
> other means, then this is still a potential problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
