Re: [E] Re: User-supplied attributes for processors
I’ve created a pull request that hides the ‘+’ UI element as necessary. https://github.com/apache/nifi/pull/969 I’ll open a second Jira ticket to start the discussion around adding an additional tag to add user-supplied tags to flow files. - Kirk Tarou On 8/22/16, 6:06 PM, "Tarou, Kirk"wrote: >I created ticket a Jira ticket to remove the + icon: >https://issues.apache.org/jira/browse/NIFI-2629 > >The existence of a “DynamicProperty” annotation could be used to determine >whether to show the icon. > >I’m making an initial attempt to add a ‘hasDynamicProperty’ field to the >processor API response. > >- Kirk Tarou > > > >On 8/19/16, 6:41 PM, "Matt Burgess" wrote: > >>For #1, totally agree that if a processor doesn't support dynamic >>properties then the UI element could be hidden or disabled. >> >>For #2, I like the additional tab idea, it basically enables a decorator >>pattern on the processor to add attributes without any onus on any part >>of the flow to use them. >> >>Regards, >>Matt >> >>> On Aug 19, 2016, at 8:38 PM, Joe Witt wrote: >>> >>> Kirk, >>> >>> As Matt points out dynamic properties of processors have meaning >>> specific to those processors. So, we need to be careful to avoid >>> complicating that. >>> >>> You raise two other points there though that I'd like to further >>>discuss: >>> 1) Processors that don't really support dynamic properties should not >>> have the + icon present (or it should be disabled) >>> - I agree. Can you please raise a JIRA. >>> >>> 2) You'd like to be able to add a set of attributes to flow files that >>> exit a given processor and would like to be able to do this without >>> adding extra processors. >>> - I can see how this would be helpful and cleaner in some cases. I >>> envision that not as processor properties but rather a tab on >>> processor configuration that lets you tag flow file attributes on flow >>> files and which supports expression language. We should be really >>> clear where in the flow file lifecycle that tagging occurs such as on >>> session commit for example (so you know all other things in that >>> processor are done). Please raise a JIRA if this is consistent with >>> your idea. >>> >>> Thanks >>> Joe >>> >>> On Fri, Aug 19, 2016 at 12:14 PM, Tarou, Kirk >>> wrote: I think it would be beneficial to add dynamic properties to any processor for use further down the flow, not in the processor where the properties are added. For example, I may have a lot of ŒListSFTP¹ processors that feed into a single FetchSFTP, then a single PutFile. It would be nice to be able to add one or more attributes in ListSFTP to tell PutFile where to write the file or use RouteOnAttribute to send the file to the appropriate processor. I realize that this can be accomplished by adding an UpdateAttribute processor, but that would greatly increase the number of processors in my flow file. From a cursory scan of the 0.7.0 processors, it looks like UpdateAttribute, RouteOnAttribute, RouteOnContent, RouteText, HashAttribute and RouteHL7 are the only processors that actually allow you to add a property, yet all processors have the "+ New Property² widget on the ³Properties² page. If arbitrary properties can¹t be added, the ³+ New Property² widget should be hidden. - Kirk Tarou > On 8/18/16, 5:11 PM, "Matt Burgess" wrote: > > Kirk, > > The processors have to explicitly know about dynamic properties (and > their intent) in order to use them appropriately. For a processor >like > ListSFTP, it could be beneficial to have custom attributes (as > parameters to the SFTP session perhaps?) but the domain knowledge on > how they'd be used would have to be coded into the processor itself. > If you have a use case in mind, please feel free to file an > improvement Jira for ListSFTP to use dynamic properties. > > Regards, > Matt > > On Mon, Aug 8, 2016 at 4:50 PM, Tarou, Kirk > wrote: >> Is there some reason why Processors, like ListSFTP, don¹t allow >>custom >> attributes to be added? Relatedly, why do these processors allow >>users >> to add custom attributes & values in the UI even though it always >>throws >> the error: >> Œ[attribute]¹ validated against Œ[value]¹ is invalid because >> Œ[attribute]¹ is not a supported property >> >> - Kirk Tarou >
Re: [E] Re: User-supplied attributes for processors
I created ticket a Jira ticket to remove the + icon: https://issues.apache.org/jira/browse/NIFI-2629 The existence of a “DynamicProperty” annotation could be used to determine whether to show the icon. I’m making an initial attempt to add a ‘hasDynamicProperty’ field to the processor API response. - Kirk Tarou On 8/19/16, 6:41 PM, "Matt Burgess"wrote: >For #1, totally agree that if a processor doesn't support dynamic >properties then the UI element could be hidden or disabled. > >For #2, I like the additional tab idea, it basically enables a decorator >pattern on the processor to add attributes without any onus on any part >of the flow to use them. > >Regards, >Matt > >> On Aug 19, 2016, at 8:38 PM, Joe Witt wrote: >> >> Kirk, >> >> As Matt points out dynamic properties of processors have meaning >> specific to those processors. So, we need to be careful to avoid >> complicating that. >> >> You raise two other points there though that I'd like to further >>discuss: >> 1) Processors that don't really support dynamic properties should not >> have the + icon present (or it should be disabled) >> - I agree. Can you please raise a JIRA. >> >> 2) You'd like to be able to add a set of attributes to flow files that >> exit a given processor and would like to be able to do this without >> adding extra processors. >> - I can see how this would be helpful and cleaner in some cases. I >> envision that not as processor properties but rather a tab on >> processor configuration that lets you tag flow file attributes on flow >> files and which supports expression language. We should be really >> clear where in the flow file lifecycle that tagging occurs such as on >> session commit for example (so you know all other things in that >> processor are done). Please raise a JIRA if this is consistent with >> your idea. >> >> Thanks >> Joe >> >> On Fri, Aug 19, 2016 at 12:14 PM, Tarou, Kirk >> wrote: >>> I think it would be beneficial to add dynamic properties to any >>>processor >>> for use further down the flow, not in the processor where the >>>properties >>> are added. >>> >>> For example, I may have a lot of ŒListSFTP¹ processors that feed into a >>> single FetchSFTP, then a single PutFile. It would be nice to be able to >>> add one or more attributes in ListSFTP to tell PutFile where to write >>>the >>> file or use RouteOnAttribute to send the file to the appropriate >>>processor. >>> >>> I realize that this can be accomplished by adding an UpdateAttribute >>> processor, but that would greatly increase the number of processors in >>>my >>> flow file. >>> >>> From a cursory scan of the 0.7.0 processors, it looks like >>> UpdateAttribute, RouteOnAttribute, RouteOnContent, RouteText, >>> HashAttribute and RouteHL7 are the only processors that actually allow >>>you >>> to add a property, yet all processors have the "+ New Property² widget >>>on >>> the ³Properties² page. If arbitrary properties can¹t be added, the ³+ >>>New >>> Property² widget should be hidden. >>> >>> - Kirk Tarou >>> >>> >>> On 8/18/16, 5:11 PM, "Matt Burgess" wrote: Kirk, The processors have to explicitly know about dynamic properties (and their intent) in order to use them appropriately. For a processor like ListSFTP, it could be beneficial to have custom attributes (as parameters to the SFTP session perhaps?) but the domain knowledge on how they'd be used would have to be coded into the processor itself. If you have a use case in mind, please feel free to file an improvement Jira for ListSFTP to use dynamic properties. Regards, Matt On Mon, Aug 8, 2016 at 4:50 PM, Tarou, Kirk wrote: > Is there some reason why Processors, like ListSFTP, don¹t allow >custom > attributes to be added? Relatedly, why do these processors allow >users > to add custom attributes & values in the UI even though it always >throws > the error: > Œ[attribute]¹ validated against Œ[value]¹ is invalid because > Œ[attribute]¹ is not a supported property > > - Kirk Tarou >>>
Re: [E] Re: User-supplied attributes for processors
For #1, totally agree that if a processor doesn't support dynamic properties then the UI element could be hidden or disabled. For #2, I like the additional tab idea, it basically enables a decorator pattern on the processor to add attributes without any onus on any part of the flow to use them. Regards, Matt > On Aug 19, 2016, at 8:38 PM, Joe Wittwrote: > > Kirk, > > As Matt points out dynamic properties of processors have meaning > specific to those processors. So, we need to be careful to avoid > complicating that. > > You raise two other points there though that I'd like to further discuss: > 1) Processors that don't really support dynamic properties should not > have the + icon present (or it should be disabled) > - I agree. Can you please raise a JIRA. > > 2) You'd like to be able to add a set of attributes to flow files that > exit a given processor and would like to be able to do this without > adding extra processors. > - I can see how this would be helpful and cleaner in some cases. I > envision that not as processor properties but rather a tab on > processor configuration that lets you tag flow file attributes on flow > files and which supports expression language. We should be really > clear where in the flow file lifecycle that tagging occurs such as on > session commit for example (so you know all other things in that > processor are done). Please raise a JIRA if this is consistent with > your idea. > > Thanks > Joe > > On Fri, Aug 19, 2016 at 12:14 PM, Tarou, Kirk > wrote: >> I think it would be beneficial to add dynamic properties to any processor >> for use further down the flow, not in the processor where the properties >> are added. >> >> For example, I may have a lot of ŒListSFTP¹ processors that feed into a >> single FetchSFTP, then a single PutFile. It would be nice to be able to >> add one or more attributes in ListSFTP to tell PutFile where to write the >> file or use RouteOnAttribute to send the file to the appropriate processor. >> >> I realize that this can be accomplished by adding an UpdateAttribute >> processor, but that would greatly increase the number of processors in my >> flow file. >> >> From a cursory scan of the 0.7.0 processors, it looks like >> UpdateAttribute, RouteOnAttribute, RouteOnContent, RouteText, >> HashAttribute and RouteHL7 are the only processors that actually allow you >> to add a property, yet all processors have the "+ New Property² widget on >> the ³Properties² page. If arbitrary properties can¹t be added, the ³+ New >> Property² widget should be hidden. >> >> - Kirk Tarou >> >> >> >>> On 8/18/16, 5:11 PM, "Matt Burgess" wrote: >>> >>> Kirk, >>> >>> The processors have to explicitly know about dynamic properties (and >>> their intent) in order to use them appropriately. For a processor like >>> ListSFTP, it could be beneficial to have custom attributes (as >>> parameters to the SFTP session perhaps?) but the domain knowledge on >>> how they'd be used would have to be coded into the processor itself. >>> If you have a use case in mind, please feel free to file an >>> improvement Jira for ListSFTP to use dynamic properties. >>> >>> Regards, >>> Matt >>> >>> On Mon, Aug 8, 2016 at 4:50 PM, Tarou, Kirk >>> wrote: Is there some reason why Processors, like ListSFTP, don¹t allow custom attributes to be added? Relatedly, why do these processors allow users to add custom attributes & values in the UI even though it always throws the error: Œ[attribute]¹ validated against Œ[value]¹ is invalid because Œ[attribute]¹ is not a supported property - Kirk Tarou >>
Re: [E] Re: User-supplied attributes for processors
Kirk, As Matt points out dynamic properties of processors have meaning specific to those processors. So, we need to be careful to avoid complicating that. You raise two other points there though that I'd like to further discuss: 1) Processors that don't really support dynamic properties should not have the + icon present (or it should be disabled) - I agree. Can you please raise a JIRA. 2) You'd like to be able to add a set of attributes to flow files that exit a given processor and would like to be able to do this without adding extra processors. - I can see how this would be helpful and cleaner in some cases. I envision that not as processor properties but rather a tab on processor configuration that lets you tag flow file attributes on flow files and which supports expression language. We should be really clear where in the flow file lifecycle that tagging occurs such as on session commit for example (so you know all other things in that processor are done). Please raise a JIRA if this is consistent with your idea. Thanks Joe On Fri, Aug 19, 2016 at 12:14 PM, Tarou, Kirkwrote: > I think it would be beneficial to add dynamic properties to any processor > for use further down the flow, not in the processor where the properties > are added. > > For example, I may have a lot of ŒListSFTP¹ processors that feed into a > single FetchSFTP, then a single PutFile. It would be nice to be able to > add one or more attributes in ListSFTP to tell PutFile where to write the > file or use RouteOnAttribute to send the file to the appropriate processor. > > I realize that this can be accomplished by adding an UpdateAttribute > processor, but that would greatly increase the number of processors in my > flow file. > > From a cursory scan of the 0.7.0 processors, it looks like > UpdateAttribute, RouteOnAttribute, RouteOnContent, RouteText, > HashAttribute and RouteHL7 are the only processors that actually allow you > to add a property, yet all processors have the "+ New Property² widget on > the ³Properties² page. If arbitrary properties can¹t be added, the ³+ New > Property² widget should be hidden. > > - Kirk Tarou > > > > On 8/18/16, 5:11 PM, "Matt Burgess" wrote: > >>Kirk, >> >>The processors have to explicitly know about dynamic properties (and >>their intent) in order to use them appropriately. For a processor like >>ListSFTP, it could be beneficial to have custom attributes (as >>parameters to the SFTP session perhaps?) but the domain knowledge on >>how they'd be used would have to be coded into the processor itself. >>If you have a use case in mind, please feel free to file an >>improvement Jira for ListSFTP to use dynamic properties. >> >>Regards, >>Matt >> >>On Mon, Aug 8, 2016 at 4:50 PM, Tarou, Kirk >> wrote: >>> Is there some reason why Processors, like ListSFTP, don¹t allow custom >>>attributes to be added? Relatedly, why do these processors allow users >>>to add custom attributes & values in the UI even though it always throws >>>the error: >>> Œ[attribute]¹ validated against Œ[value]¹ is invalid because >>>Œ[attribute]¹ is not a supported property >>> >>> - Kirk Tarou >
Re: [E] Re: User-supplied attributes for processors
I think it would be beneficial to add dynamic properties to any processor for use further down the flow, not in the processor where the properties are added. For example, I may have a lot of ŒListSFTP¹ processors that feed into a single FetchSFTP, then a single PutFile. It would be nice to be able to add one or more attributes in ListSFTP to tell PutFile where to write the file or use RouteOnAttribute to send the file to the appropriate processor. I realize that this can be accomplished by adding an UpdateAttribute processor, but that would greatly increase the number of processors in my flow file. >From a cursory scan of the 0.7.0 processors, it looks like UpdateAttribute, RouteOnAttribute, RouteOnContent, RouteText, HashAttribute and RouteHL7 are the only processors that actually allow you to add a property, yet all processors have the "+ New Property² widget on the ³Properties² page. If arbitrary properties can¹t be added, the ³+ New Property² widget should be hidden. - Kirk Tarou On 8/18/16, 5:11 PM, "Matt Burgess"wrote: >Kirk, > >The processors have to explicitly know about dynamic properties (and >their intent) in order to use them appropriately. For a processor like >ListSFTP, it could be beneficial to have custom attributes (as >parameters to the SFTP session perhaps?) but the domain knowledge on >how they'd be used would have to be coded into the processor itself. >If you have a use case in mind, please feel free to file an >improvement Jira for ListSFTP to use dynamic properties. > >Regards, >Matt > >On Mon, Aug 8, 2016 at 4:50 PM, Tarou, Kirk > wrote: >> Is there some reason why Processors, like ListSFTP, don¹t allow custom >>attributes to be added? Relatedly, why do these processors allow users >>to add custom attributes & values in the UI even though it always throws >>the error: >> Œ[attribute]¹ validated against Œ[value]¹ is invalid because >>Œ[attribute]¹ is not a supported property >> >> - Kirk Tarou
Re: User-supplied attributes for processors
Kirk, The processors have to explicitly know about dynamic properties (and their intent) in order to use them appropriately. For a processor like ListSFTP, it could be beneficial to have custom attributes (as parameters to the SFTP session perhaps?) but the domain knowledge on how they'd be used would have to be coded into the processor itself. If you have a use case in mind, please feel free to file an improvement Jira for ListSFTP to use dynamic properties. Regards, Matt On Mon, Aug 8, 2016 at 4:50 PM, Tarou, Kirkwrote: > Is there some reason why Processors, like ListSFTP, don’t allow custom > attributes to be added? Relatedly, why do these processors allow users to add > custom attributes & values in the UI even though it always throws the error: > ‘[attribute]’ validated against ‘[value]’ is invalid because ‘[attribute]’ is > not a supported property > > - Kirk Tarou
User-supplied attributes for processors
Is there some reason why Processors, like ListSFTP, don’t allow custom attributes to be added? Relatedly, why do these processors allow users to add custom attributes & values in the UI even though it always throws the error: ‘[attribute]’ validated against ‘[value]’ is invalid because ‘[attribute]’ is not a supported property - Kirk Tarou