Matt thanks for your reply I guess what I am saying in that case - if there is an error in a FlowFile, then the processor that detects this cannot proceed so instead of calling an action to penalize the FlowFile it raises an exception OutOFServiceException or ProcessorException. You could have an exception cause PeanilisedFlowFileException for this case.
But within the processor other error causes may arise for an OutOFServiceException The point is that if the processor threw this exception then there can be a duration configuration - a time limit to keep this processor out of service and the connection to it and possibly any processors leading upto it - Naturally this will need to be indicated on the DFM - this will free resources and make the flow well behaved. Environmental failures will simply be a different category/cause of error that can be wrapped/captured also with a more general one With Kind Regards Michael de Courci mdecou...@gmail.com > On 28 Jan 2016, at 17:16, Matt Gilman <matt.c.gil...@gmail.com> wrote: > > Just to recap/level set... > > The distinct between yielding and penalization is important. Penalization > is an action taken on a FlowFile because the FlowFile cannot be processed > right now (like a naming conflict for instance). The Processor is > indicating that it cannot process that specific FlowFile at the moment but > may be able to process the next. Yielding is an indication that the > Processor is unable to work at all at the moment likely due to an > environmental issue (like the out of service comment). > > If the concept of penalization were moved to a connection, does it > automatically penalize all FlowFile transferred to it? We would lose some > granularity if a Processor wanted to penalize some FlowFile routed to a > given Relationship but not others. I'm not sure if this is done in practice > or not, just wanted to mention it. > > Outside of this minor concern, I like the idea. I especially like that it > would help with the consistency of Processor behavior and transparency > about what the data flow is actually doing. > > Matt > > > On Thu, Jan 28, 2016 at 12:00 PM, Michael de Courci < > mdecou...@googlemail.com> wrote: > >> Hi >> I think it would be better/simpler to have one “out of service” concept >> to replace penalizing and yielding and when a plugin throws an exception >> then the plugin is deemed out of service, for a duration and so the >> connection to that plugin is disabled for the out of service duration. >> >> When a plugin is out of service and the connection disabled - then >> resources that it uses will be freed(yielded). >> >> The question then is what the behaviour of the plugin before the disabled >> connection - should be. My thought is to tend towards stability and make >> sure resources are freed, so there may need to be a “domino effect”/cascade >> affect where all plugins before are gradually put out of service. >> >> >> With Kind Regards >> Michael de Courci >> mdecou...@gmail.com >> >> >> >> >>> On 28 Jan 2016, at 16:34, Mark Payne <marka...@hotmail.com> wrote: >>> >>> All, >>> >>> I've been thinking about how we handle the concept of penalizing >> FlowFiles. We've had a lot of questions >>> lately about how penalization works & the concept in general. Seems the >> following problems exist: >>> >>> - Confusion about difference between penalization & yielding >>> - DFM sees option to configure penalization period on all processors, >> even if they don't penalize FlowFiles. >>> - DFM cannot set penalty duration in 1 case and set a different value >> for a different case (different relationship, for example). >>> - Developers often forget to call penalize() >>> - Developer has to determine whether or not to penalize when building a >> processor. It is based on what the developer will >>> think may make sense, but in reality DFM's sometimes want to penalize >> things when the processor doesn't behave that way. >>> >>> I'm wondering if it doesn't make sense to remove the concept of >> penalization all together from Processors and instead >>> move the Penalty Duration so that it's a setting on the Connection. I >> think this would clear up the confusion and give the DFM >>> more control over when/how long to penalize. Could set to the default to >> 30 seconds for self-looping connections and no penalization >>> for other connections. >>> >>> Any thoughts? >>> >>> Thanks >>> -Mark >> >>
