Re: [EXT] Re: New Standard Pattern - Put Exception that caused failure in an attribute
Apologies if I've missed this in the discussion so far - we use the InvokeHTTP processor a lot, and the invokehttp.java.exception.message attribute is really handy diving into why things have failed without having to match up logs with flow files (from a system with hundreds of processors making thousands of requests). We also route on invokehttp.status.code (e.g. to retry 403s due to a race hazard in an external system) but I don't imagine we'd route on invokehttp.java.exception.* since (as others have mentioned) it looks pretty fragile. -- James On Tue, 30 Oct 2018 at 16:44, Peter Wicks (pwicks) wrote: > > Sorry for the delayed response, I've been traveling. > > Responses in order: > > Matt, > Right now our work around is to keep retrying errors, usually with a penalty > or control rate processor. The problem is that we don't know why it failed, > and thus don't know if retry is the correct option. I have not found a way, > without code change, to be able to determine if retrying is the correct > option or not. > > Koji, > Detailed error handling would indeed be a good workaround to the problems > raised by myself and Matt. I have not see this on other processors, but that > does not mean we can't do it of course. I agree that having some kind of > hierarchy system for errors would be a much better solution. > > Pierre, > My primary use case is as you described, a user friendly way to see what > actually happened without going through the log files. But I while I know > it's fragile, routing on exception text stored in an attribute still feels > like a very legitimate use case. I know in many systems there are good > exception types that can be used to route FlowFile's to appropriate failure > relationships, but as Matt mentioned, JDBC has just a handful of exception > types for a very large number of possible error types. > > I think this is probably the same rational that was used to justify this > feature for Execute Stream Command's inclusion of this feature in the past. > To many possible failure conditions to handle with just a few failure > conditions. > > Uwe, > That is a fair question, but it doesn't feel like such a bad fit to me. It's > like extra metadata on the lineage, "We followed this path through the flow > because we had exception " " which caused the FlowFile to follow the > failure route". > > But I still prefer the attribute, it could be another option for Detailed > error handling; instead of, or in addition to, additional relationships for > failures, the exception text could be included in an attribute. > > Thanks, > Peter > > -Original Message- > From: u...@moosheimer.com [mailto:u...@moosheimer.com] > Sent: Saturday, October 27, 2018 10:46 AM > To: dev@nifi.apache.org > Subject: Re: [EXT] Re: New Standard Pattern - Put Exception that caused > failure in an attribute > > Do you really want to mix provenance and data lineage with logging/error > information? > > Writing exception information/logging information within an attribute is not > a bad idea in my opinion. > If a user wants to use this for routing, why not ... or whatever the user > wants to do. > > I could imagine that this can be switched on and off by a property via > config. E.g. in development on and on production off. > > Regards, > Uwe > > > Am 26.10.2018 um 09:26 schrieb Pierre Villard : > > > > Adding another option to the list. > > > > Peter - if I understand correctly and based on my own experience, the > > idea is not to have an 'exception' attribute to perform custom routing > > after the failure relationship but rather have a more user friendly > > way to see what happened without going through all the logs for a given > > flow file. > > > > If that's correct, then could we add this information somehow to the > > provenance event generated by the processor? Ideally adding a new > > field to a provenance event or using the existing 'details' field? > > > > Pierre > > > > > > Le ven. 26 oct. 2018 à 08:40, Koji Kawamura a > > écrit : > > > >> Hi all, > >> > >> I'd like to add another option to Matt's list of solutions: > >> > >> 4) Add a processor property, 'Enable detailed error handling' > >> (defaults to false), then toggle available list of relationships. > >> This way, existing flows such as Peter's don't have to change, while > >> he can opt-in new relationships. RouteOnAttribute can be a reference > >> implementation. > >> > >> I like the idea of thinking relationships as potential exceptions. It > >> can be better if relationships have hierarchy. > >> Some users need more granular relationships while others don't. > >> For NiFi 2.0 or later, supporting relationship hierarchy at framework > >> can mitigate having additional property at each processor. > >> > >> Thanks, > >> Koji > >> On Fri, Oct 26, 2018 at 11:49 AM Matt Burgess > >> wrote: > >>> > >>> Peter, > >>> > >>> Totally agree, RDBMS/JDBC is in a weird class as always, there is a > >>> teaspoon of
RE: [EXT] Re: New Standard Pattern - Put Exception that caused failure in an attribute
Sorry for the delayed response, I've been traveling. Responses in order: Matt, Right now our work around is to keep retrying errors, usually with a penalty or control rate processor. The problem is that we don't know why it failed, and thus don't know if retry is the correct option. I have not found a way, without code change, to be able to determine if retrying is the correct option or not. Koji, Detailed error handling would indeed be a good workaround to the problems raised by myself and Matt. I have not see this on other processors, but that does not mean we can't do it of course. I agree that having some kind of hierarchy system for errors would be a much better solution. Pierre, My primary use case is as you described, a user friendly way to see what actually happened without going through the log files. But I while I know it's fragile, routing on exception text stored in an attribute still feels like a very legitimate use case. I know in many systems there are good exception types that can be used to route FlowFile's to appropriate failure relationships, but as Matt mentioned, JDBC has just a handful of exception types for a very large number of possible error types. I think this is probably the same rational that was used to justify this feature for Execute Stream Command's inclusion of this feature in the past. To many possible failure conditions to handle with just a few failure conditions. Uwe, That is a fair question, but it doesn't feel like such a bad fit to me. It's like extra metadata on the lineage, "We followed this path through the flow because we had exception " " which caused the FlowFile to follow the failure route". But I still prefer the attribute, it could be another option for Detailed error handling; instead of, or in addition to, additional relationships for failures, the exception text could be included in an attribute. Thanks, Peter -Original Message- From: u...@moosheimer.com [mailto:u...@moosheimer.com] Sent: Saturday, October 27, 2018 10:46 AM To: dev@nifi.apache.org Subject: Re: [EXT] Re: New Standard Pattern - Put Exception that caused failure in an attribute Do you really want to mix provenance and data lineage with logging/error information? Writing exception information/logging information within an attribute is not a bad idea in my opinion. If a user wants to use this for routing, why not ... or whatever the user wants to do. I could imagine that this can be switched on and off by a property via config. E.g. in development on and on production off. Regards, Uwe > Am 26.10.2018 um 09:26 schrieb Pierre Villard : > > Adding another option to the list. > > Peter - if I understand correctly and based on my own experience, the > idea is not to have an 'exception' attribute to perform custom routing > after the failure relationship but rather have a more user friendly > way to see what happened without going through all the logs for a given flow > file. > > If that's correct, then could we add this information somehow to the > provenance event generated by the processor? Ideally adding a new > field to a provenance event or using the existing 'details' field? > > Pierre > > > Le ven. 26 oct. 2018 à 08:40, Koji Kawamura a > écrit : > >> Hi all, >> >> I'd like to add another option to Matt's list of solutions: >> >> 4) Add a processor property, 'Enable detailed error handling' >> (defaults to false), then toggle available list of relationships. >> This way, existing flows such as Peter's don't have to change, while >> he can opt-in new relationships. RouteOnAttribute can be a reference >> implementation. >> >> I like the idea of thinking relationships as potential exceptions. It >> can be better if relationships have hierarchy. >> Some users need more granular relationships while others don't. >> For NiFi 2.0 or later, supporting relationship hierarchy at framework >> can mitigate having additional property at each processor. >> >> Thanks, >> Koji >> On Fri, Oct 26, 2018 at 11:49 AM Matt Burgess >> wrote: >>> >>> Peter, >>> >>> Totally agree, RDBMS/JDBC is in a weird class as always, there is a >>> teaspoon of exception types for an ocean of causes. For NiFi 1.x, it >>> seems like we need to pick from a set of less-than-ideal solutions: >>> >>> 1) Add new relationships, but then your (possibly hundreds of) >>> processors are invalid >>> 2) Add new auto-terminated relationships, but then your >>> previously-handled errors are "lost" >>> 3) Add an attribute, but then each NiFi instance/release/flow is >>> responsible for parsing the error and handling it as desired. >>> >>> We could mitigate 1-2 with a tool that updates your flow/template by >>> sending all new failure relationships to the same target as the >>> existing one, but then the tool itself suffers from maintainability >>> issues (as does option #3). If we could recognize that the new >>> relationships are self-terminated
Re: Using record path API to remove field values
I'm not sure that removing a field from record path makes sense since
technically removing the field all together is a different schema.
You could easily make two versions of the schema, one with all fields,
and a second with a subset, then just use any record processor with a
reader using schema 1 and a writer using schema 2.
On Tue, Oct 30, 2018 at 9:28 AM Mike Thomsen wrote:
>
> Something more aggressive than that. I have a new ES bulk operation
> processor that uses the record API to specify operations. I want to be able
> to remove the record paths that relate to things like the index and type.
>
> In StandardFieldValue it also looks like we ignore nulls altogether:
>
> @Override
> public void updateValue(final Object newValue) {
> final Optional parentRecord = getParentRecord();
> if (!parentRecord.isPresent()) {
> if (value instanceof Record) {
> ((Record) value).setValue(getField().getFieldName(),
> newValue);
> return;
> } else if (value == null) {
> return; // value is null, nothing to update
> } else {
> throw new UnsupportedOperationException("Cannot update the
> field value because the value is not associated with any record");
> }
> }
>
> parentRecord.get().setValue(getField().getFieldName(), newValue);
> }
>
> Not sure what the rationale there is for not checking the value's field to
> see if isNullable() is true or not and then acting from what. If no one
> knows any gotchas, I can expand that out to allow null on nullable fields.
> Then I could implement null suppression in a similar way as the json
> writer, I suppose.
>
> Thanks,
>
> Mike
>
> On Tue, Oct 30, 2018 at 8:51 AM Pierre Villard
> wrote:
>
> > Hey Mike,
> >
> > In the JSON Record Set Writer you have a "Suppress Null Values" parameter
> > to do just that.
> > Are you thinking about something else?
> >
> > Pierre
> >
> > Le mar. 30 oct. 2018 à 13:44, Mike Thomsen a
> > écrit :
> >
> > > Is there an example of how to use the record path API to remove an
> > element
> > > from a record? Ideally, if the field is nullable, I'd like to be able to
> > > completely remove the value rather than send a null value (to Elastic in
> > > this case).
> > >
> > > Thanks,
> > >
> > > Mike
> > >
> >
Re: Hi
I mean ParseEvtx processor. On 10/30/18, 7:37 AM, "Milan Das" wrote: Did you try WinEventLogProcessor ? Thanks, Milan Das On 10/29/18, 11:17 PM, "find" <907611...@qq.com> wrote: Hi, Sorry to disturb you.I want to parse a .evtx(a window event log file) file using nifi(https://github.com/apache/nifi),but I just don't know how to use it(specific codes to implement),I wonder if you have a demo or not?Thanks in advance.
Re: Hi
https://community.hortonworks.com/articles/58493/parsing-evtx-files-with-apache-nifi.html On October 29, 2018 at 23:17:27, find (907611...@qq.com) wrote: Hi, Sorry to disturb you.I want to parse a .evtx(a window event log file) file using nifi(https://github.com/apache/nifi),but I just don't know how to use it(specific codes to implement),I wonder if you have a demo or not?Thanks in advance.
Re: Using record path API to remove field values
Hey Mike, In the JSON Record Set Writer you have a "Suppress Null Values" parameter to do just that. Are you thinking about something else? Pierre Le mar. 30 oct. 2018 à 13:44, Mike Thomsen a écrit : > Is there an example of how to use the record path API to remove an element > from a record? Ideally, if the field is nullable, I'd like to be able to > completely remove the value rather than send a null value (to Elastic in > this case). > > Thanks, > > Mike >
Using record path API to remove field values
Is there an example of how to use the record path API to remove an element from a record? Ideally, if the field is nullable, I'd like to be able to completely remove the value rather than send a null value (to Elastic in this case). Thanks, Mike
Re: Hi
Did you try WinEventLogProcessor ? Thanks, Milan Das On 10/29/18, 11:17 PM, "find" <907611...@qq.com> wrote: Hi, Sorry to disturb you.I want to parse a .evtx(a window event log file) file using nifi(https://github.com/apache/nifi),but I just don't know how to use it(specific codes to implement),I wonder if you have a demo or not?Thanks in advance.
