Deadletter
I have a lot of processors in my flow, all of which can, and do, route
flowfiles to their failure relationships at some point.
In the first iteration of my flow, I routed every failure relationship to
an inactive DebugFlow but monitoring these was difficult, I wouldn't get
notifications when something started to fail and if the queue got filled up
it would apply backpressure and prevent new, good flowfiles from being
processed.
Not only was that just not a good way to handle failures, but my flow was
littered with all of these do-nothing processors and was an eye sore. So
then I tried routing processor failure relationships into themselves which
tidied up my flow but caused nifi to go berserk when a failure occurred
because the failure relationship is not penalized (nor should it be) and
most processors don't provide a 'Retry' relationship (InvokeHttp being a
notable exception). But really, most processors wouldn't conceivable
succeed if they were tried again. I mostly just wanted the flowfiles to sit
there until I had a chance to check out why they failed and fix them
manually.
This leads me to https://issues.apache.org/jira/browse/NIFI-3351. I think I
need a way to store failed flowfiles, fix them and reprocess them. The
process group I am currently considering implementing everywhere is:
Input Port [Failed Flowfile] --> PutS3 deadletter///${uuid} --> PutSlack
ListS3 deadletter/// --> FetchS3 ->
Output Port [Fixed]
This gives me storage of failed messages logically grouped and in a place
that wont block up my flow since s3 never goes down, err... wait.
Configurable process groups or template like
https://issues.apache.org/jira/browse/NIFI-1096 would make this easier to
reuse.
How do you manage failure relationships?
- Nick
Re: Deadletter
Nick
Since you’ve already designed Process Group (PG) that is specific to failed
flow files, I am not sure I understand your last question “. . .How do you
manage failure relationships?. . .”.
I am assuming that within your global flow all failure relationships are sent
to this PG, which essentially is a Dead Letter Storage.
Are you asking about how do you get more information from the failed Flow Files
(i.e., failure location, reason etc)?
Cheers
Oleg
On Mar 1, 2017, at 3:21 PM, Nick Carenza
mailto:nick.care...@thecontrolgroup.com>>
wrote:
I have a lot of processors in my flow, all of which can, and do, route
flowfiles to their failure relationships at some point.
In the first iteration of my flow, I routed every failure relationship to an
inactive DebugFlow but monitoring these was difficult, I wouldn't get
notifications when something started to fail and if the queue got filled up it
would apply backpressure and prevent new, good flowfiles from being processed.
Not only was that just not a good way to handle failures, but my flow was
littered with all of these do-nothing processors and was an eye sore. So then I
tried routing processor failure relationships into themselves which tidied up
my flow but caused nifi to go berserk when a failure occurred because the
failure relationship is not penalized (nor should it be) and most processors
don't provide a 'Retry' relationship (InvokeHttp being a notable exception).
But really, most processors wouldn't conceivable succeed if they were tried
again. I mostly just wanted the flowfiles to sit there until I had a chance to
check out why they failed and fix them manually.
This leads me to https://issues.apache.org/jira/browse/NIFI-3351. I think I
need a way to store failed flowfiles, fix them and reprocess them. The process
group I am currently considering implementing everywhere is:
Input Port [Failed Flowfile] --> PutS3 deadletter///${uuid} --> PutSlack
ListS3 deadletter/// --> FetchS3 -> Output
Port [Fixed]
This gives me storage of failed messages logically grouped and in a place that
wont block up my flow since s3 never goes down, err... wait. Configurable
process groups or template like https://issues.apache.org/jira/browse/NIFI-1096
would make this easier to reuse.
How do you manage failure relationships?
- Nick
Re: Deadletter
Sorry for the confusion, I meant to put emphasis on the _you_, as in 'you
all' or other users of nifi. I am looking to get insight into solutions
other have implemented to deal with failures.
- Nick
On Wed, Mar 1, 2017 at 12:29 PM, Oleg Zhurakousky <
ozhurakou...@hortonworks.com> wrote:
> Nick
>
> Since you’ve already designed Process Group (PG) that is specific to
> failed flow files, I am not sure I understand your last question “. . .How
> do you manage failure relationships?. . .”.
> I am assuming that within your global flow all failure relationships are
> sent to this PG, which essentially is a Dead Letter Storage.
>
> Are you asking about how do you get more information from the failed Flow
> Files (i.e., failure location, reason etc)?
>
> Cheers
> Oleg
>
> On Mar 1, 2017, at 3:21 PM, Nick Carenza
> wrote:
>
> I have a lot of processors in my flow, all of which can, and do, route
> flowfiles to their failure relationships at some point.
>
> In the first iteration of my flow, I routed every failure relationship to
> an inactive DebugFlow but monitoring these was difficult, I wouldn't get
> notifications when something started to fail and if the queue got filled up
> it would apply backpressure and prevent new, good flowfiles from being
> processed.
>
> Not only was that just not a good way to handle failures, but my flow was
> littered with all of these do-nothing processors and was an eye sore. So
> then I tried routing processor failure relationships into themselves which
> tidied up my flow but caused nifi to go berserk when a failure occurred
> because the failure relationship is not penalized (nor should it be) and
> most processors don't provide a 'Retry' relationship (InvokeHttp being a
> notable exception). But really, most processors wouldn't conceivable
> succeed if they were tried again. I mostly just wanted the flowfiles to sit
> there until I had a chance to check out why they failed and fix them
> manually.
>
> This leads me to https://issues.apache.org/jira/browse/NIFI-3351. I think
> I need a way to store failed flowfiles, fix them and reprocess them. The
> process group I am currently considering implementing everywhere is:
>
> Input Port [Failed Flowfile] --> PutS3 deadletter/ location>//${uuid} --> PutSlack
> ListS3 deadletter/// --> FetchS3 ->
> Output Port [Fixed]
>
> This gives me storage of failed messages logically grouped and in a place
> that wont block up my flow since s3 never goes down, err... wait.
> Configurable process groups or template like https://issues.apache.org/
> jira/browse/NIFI-1096 would make this easier to reuse.
>
> How do you manage failure relationships?
>
> - Nick
>
>
>
Re: Deadletter
No confusion, Nick. I hear you. In our case...
We attempt to reroute flowfile back through processors that, for
whatever reason, might bring them to a state in which they can be
successful (much to explain there, but...) and others we route to a
similar, "deadletter" store where their contents are examined by hand
and changes made (to the flow and processors used, to original document
contents, etc.), then readmitted to the flow later.
(Note: We have many custom processors doing very special things.)
I'm personally all ears on this thread--eager to hear what others will
say. Thanks for hosting it!
Russ
On 03/01/2017 01:38 PM, Nick Carenza wrote:
Sorry for the confusion, I meant to put emphasis on the _you_, as in
'you all' or other users of nifi. I am looking to get insight into
solutions other have implemented to deal with failures.
- Nick
On Wed, Mar 1, 2017 at 12:29 PM, Oleg Zhurakousky
mailto:ozhurakou...@hortonworks.com>>
wrote:
Nick
Since you’ve already designed Process Group (PG) that is specific
to failed flow files, I am not sure I understand your last
question “. . .How do you manage failure relationships?. . .”.
I am assuming that within your global flow all failure
relationships are sent to this PG, which essentially is a Dead
Letter Storage.
Are you asking about how do you get more information from the
failed Flow Files (i.e., failure location, reason etc)?
Cheers
Oleg
On Mar 1, 2017, at 3:21 PM, Nick Carenza
mailto:nick.care...@thecontrolgroup.com>> wrote:
I have a lot of processors in my flow, all of which can, and do,
route flowfiles to their failure relationships at some point.
In the first iteration of my flow, I routed every failure
relationship to an inactive DebugFlow but monitoring these was
difficult, I wouldn't get notifications when something started to
fail and if the queue got filled up it would apply backpressure
and prevent new, good flowfiles from being processed.
Not only was that just not a good way to handle failures, but my
flow was littered with all of these do-nothing processors and was
an eye sore. So then I tried routing processor failure
relationships into themselves which tidied up my flow but caused
nifi to go berserk when a failure occurred because the failure
relationship is not penalized (nor should it be) and most
processors don't provide a 'Retry' relationship (InvokeHttp being
a notable exception). But really, most processors wouldn't
conceivable succeed if they were tried again. I mostly just
wanted the flowfiles to sit there until I had a chance to check
out why they failed and fix them manually.
This leads me to https://issues.apache.org/jira/browse/NIFI-3351
<https://issues.apache.org/jira/browse/NIFI-3351>. I think I need
a way to store failed flowfiles, fix them and reprocess them. The
process group I am currently considering implementing everywhere is:
Input Port [Failed Flowfile] --> PutS3 deadletter///${uuid} --> PutSlack
ListS3 deadletter/// -->
FetchS3 -> Output Port [Fixed]
This gives me storage of failed messages logically grouped and in
a place that wont block up my flow since s3 never goes down,
err... wait. Configurable process groups or template like
https://issues.apache.org/jira/browse/NIFI-1096
<https://issues.apache.org/jira/browse/NIFI-1096> would make this
easier to reuse.
How do you manage failure relationships?
- Nick
Re: Deadletter
Well, in my experience Dead Letter is more of a global storage mechanism for
undeliverable units of “something” (i.e., messages). While I do see how
“failed” Flow Files may resemble such undeliverable/unprocessable units of
work, the nature of Data Flow makes it a bit different, since there is really
no one stop solution for that. Some may be satisfied with auto-termination
while others may try to reprocess or route to a different sub-flow which may be
the actual delegation to some storage where things could be reviewed. And for
that case as you’ve already noticed NiFi has provides all the moving parts. So
PutS3 or PutFile followed by PutSlack or SendEmail are just a different
implementations of the "store and notify” pattern that you essentially
presenting here.
Also, keep in mind that once failed there is no one-stop re-processing
solution. You may have certain FlowFiles that can be automatically fixed and
re-sent, while others may need manual intervention and for that you really need
to build and customize the Dead Latter component which may include some
internal splitting and routing of Flow Files that Could be fixed vs the ones
that can not be.
Cheers
Oleg
On Mar 1, 2017, at 3:38 PM, Nick Carenza
mailto:nick.care...@thecontrolgroup.com>>
wrote:
Sorry for the confusion, I meant to put emphasis on the _you_, as in 'you all'
or other users of nifi. I am looking to get insight into solutions other have
implemented to deal with failures.
- Nick
On Wed, Mar 1, 2017 at 12:29 PM, Oleg Zhurakousky
mailto:ozhurakou...@hortonworks.com>> wrote:
Nick
Since you’ve already designed Process Group (PG) that is specific to failed
flow files, I am not sure I understand your last question “. . .How do you
manage failure relationships?. . .”.
I am assuming that within your global flow all failure relationships are sent
to this PG, which essentially is a Dead Letter Storage.
Are you asking about how do you get more information from the failed Flow Files
(i.e., failure location, reason etc)?
Cheers
Oleg
On Mar 1, 2017, at 3:21 PM, Nick Carenza
mailto:nick.care...@thecontrolgroup.com>>
wrote:
I have a lot of processors in my flow, all of which can, and do, route
flowfiles to their failure relationships at some point.
In the first iteration of my flow, I routed every failure relationship to an
inactive DebugFlow but monitoring these was difficult, I wouldn't get
notifications when something started to fail and if the queue got filled up it
would apply backpressure and prevent new, good flowfiles from being processed.
Not only was that just not a good way to handle failures, but my flow was
littered with all of these do-nothing processors and was an eye sore. So then I
tried routing processor failure relationships into themselves which tidied up
my flow but caused nifi to go berserk when a failure occurred because the
failure relationship is not penalized (nor should it be) and most processors
don't provide a 'Retry' relationship (InvokeHttp being a notable exception).
But really, most processors wouldn't conceivable succeed if they were tried
again. I mostly just wanted the flowfiles to sit there until I had a chance to
check out why they failed and fix them manually.
This leads me to https://issues.apache.org/jira/browse/NIFI-3351. I think I
need a way to store failed flowfiles, fix them and reprocess them. The process
group I am currently considering implementing everywhere is:
Input Port [Failed Flowfile] --> PutS3 deadletter///${uuid} --> PutSlack
ListS3 deadletter/// --> FetchS3 -> Output
Port [Fixed]
This gives me storage of failed messages logically grouped and in a place that
wont block up my flow since s3 never goes down, err... wait. Configurable
process groups or template like https://issues.apache.org/jira/browse/NIFI-1096
would make this easier to reuse.
How do you manage failure relationships?
- Nick
