Hi all, I'd like to add another option to Matt's list of solutions:
4) Add a processor property, 'Enable detailed error handling' (defaults to false), then toggle available list of relationships. This way, existing flows such as Peter's don't have to change, while he can opt-in new relationships. RouteOnAttribute can be a reference implementation. I like the idea of thinking relationships as potential exceptions. It can be better if relationships have hierarchy. Some users need more granular relationships while others don't. For NiFi 2.0 or later, supporting relationship hierarchy at framework can mitigate having additional property at each processor. Thanks, Koji On Fri, Oct 26, 2018 at 11:49 AM Matt Burgess <mattyb...@apache.org> wrote: > > Peter, > > Totally agree, RDBMS/JDBC is in a weird class as always, there is a > teaspoon of exception types for an ocean of causes. For NiFi 1.x, it > seems like we need to pick from a set of less-than-ideal solutions: > > 1) Add new relationships, but then your (possibly hundreds of) > processors are invalid > 2) Add new auto-terminated relationships, but then your > previously-handled errors are "lost" > 3) Add an attribute, but then each NiFi instance/release/flow is > responsible for parsing the error and handling it as desired. > > We could mitigate 1-2 with a tool that updates your flow/template by > sending all new failure relationships to the same target as the > existing one, but then the tool itself suffers from maintainability > issues (as does option #3). If we could recognize that the new > relationships are self-terminated and then send the errors out to the > original failure relationship, that could be quite confusing to the > user, especially as time goes on (how to suppress the "new" errors, > e.g.). > > IMHO I think we're between a rock and a hard place here, I guess with > great entropy comes great responsibility :P > > P.S. For your use case, is the workaround to just keep retrying? Or > are there other constraints at play? > > Regards, > Matt > > On Thu, Oct 25, 2018 at 10:27 PM Peter Wicks (pwicks) <pwi...@micron.com> > wrote: > > > > Matt, > > > > If I were to split an existing failure relationship into several > > relationships, I do not think I would want to auto-terminate in most cases. > > Specifically, I'm interested in a failure relationship for a database > > disconnect during SQL execution (database was online when the connection > > was verified in the DBCP pool, but went down during execution). If I were > > to find a way to separate this into its own relationship, I do not think > > most users would appreciate it being a condition silently not handled by > > the normal failure path. > > > > Thanks, > > Peter > > > > -----Original Message----- > > From: Matt Burgess [mailto:mattyb...@apache.org] > > Sent: Friday, October 26, 2018 10:18 AM > > To: dev@nifi.apache.org > > Subject: Re: [EXT] Re: New Standard Pattern - Put Exception that caused > > failure in an attribute > > > > NiFi (as of the last couple releases I think) has the ability to set > > auto-terminating relationships; this IMO is one of those use cases (for > > NiFi 1.x). If new relationships are added, they could default to > > auto-terminate; then the existing processors should remain valid. > > However we might want an "omnibus Jira" to capture those relationships we'd > > like to remove the auto-termination from in NiFi 2.0. > > > > Regards, > > Matt > > On Thu, Oct 25, 2018 at 10:12 PM Peter Wicks (pwicks) <pwi...@micron.com> > > wrote: > > > > > > Mark, > > > > > > I agree with you that this is the best option in general terms. After > > > thinking about it some more I think the biggest use case is for > > > troubleshooting. If a file routes to failure, you need to be watching the > > > UI to see what the exception was. An admin may have access to the NiFi > > > log files and could grep the error out, but a normal user who checks in > > > on the flow and sees a FlowFile in the error queue will not know what the > > > cause was; this is especially frustrating if retrying the file works > > > without failure the second time... Capturing the error message in an > > > attribute makes this easy to find. > > > > > > One thing I worry about too is adding new relationships to core > > > processors. After an upgrade, won't users need to go to each instance of > > > that processor and handle the new relationship? Right now I'd swagger we > > > have at least five thousand ExecuteSQL processors in our environment; and > > > while we have strong scripting skills in my NiFi team, I would not want > > > to encounter this without that. > > > > > > Thanks, > > > Peter > > > > > > -----Original Message----- > > > From: Mark Payne [mailto:marka...@hotmail.com] > > > Sent: Thursday, October 25, 2018 10:38 PM > > > To: dev@nifi.apache.org > > > Subject: [EXT] Re: New Standard Pattern - Put Exception that caused > > > failure in an attribute > > > > > > I agree - the notion of adding a "failure.reason" attribute is, in my > > > opinion, an anti-pattern that should be avoided. Relationships are not a > > > workaround but rather the preferred approach in this scenario - an > > > attribute I would consider a workaround. This is due to the fact that not > > > only is it brittle and complex to add processors that route on such > > > things, but there's no reason at all to assume that from release to > > > release (even bug fix/increment releases) that the Exception type or > > > message will be the same, so the flow could stop working at any time > > > after upgrading nifi. > > > Relationships offer a well-defined way to explicitly indicate "these are > > > the possible outcomes," > > > similar IMO to Java Exception classes vs. throwing Strings in C. > > > > > > > > > > On Oct 25, 2018, at 9:47 AM, Bryan Bende <bbe...@gmail.com> wrote: > > > > > > > > I think processors should really have well defined relationships for > > > > the error scenarios that need to be handled. Having the exception > > > > message is ok for a human who wants to see it, but in order to do > > > > anything with it in the flow you will have to have a bunch of > > > > parsing/interpreting of the message with a bunch of routing > > > > processors, which seems more brittle than just having the > > > > appropriate relationships. > > > > On Thu, Oct 25, 2018 at 1:36 AM Peter Wicks (pwicks) > > > > <pwi...@micron.com> wrote: > > > >> > > > >> When a FlowFile is routed to failure, frequently there is no clear > > > >> reason without looking into the actual error message. > > > >> Some processors work around this by creating many different > > > >> relationships, but even then frequently the generic Failure > > > >> relationship also provides little guidance. > > > >> > > > >> I've seen a few cases recently where processors are including the > > > >> exception message as an attribute on the FlowFile when routing to > > > >> failure (ExecuteStreamCommand, new PR for ExecuteSQL). Should this be > > > >> a standard pattern so that it's easier for users to route failures? > > > >> > > > >> --Peter > > >
