Re: Configuration Driven Property Updates
Mike, Just FYI, ControllerService's are configurable via the UI since 0.2.0 [1]. Also, I would also +1 for the second of Mark's options. Matt [1] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Controller_Services_and_Reporting_Tasks On Thu, Oct 22, 2015 at 6:42 AM, Michael D. Coon wrote: > Mark, > I went down the path of your "ControllerService" approach only to find > that the service configuration is tied to an XML file in the nifi > installation versus dynamically configurable via the UI (unless I'm missing > how to configure a controller service on the UI somewhere). It also means I > have a SINGLE instance of my module for the NiFi flow. This is not the case > when I have different modules needing multiple instances inside of my flow. > For these reasons I abandoned the idea of using a ControllerService for > this scenario. > I do think that option 2 would be good on its own even outside of this > particular scenario. And if option 2 were implemented, it would, at a > minimum, help support conditional properties and fix the issues I have > today with properties sticking around after I change the module to use at > runtime in the flow. > Mike > > > > On Thursday, October 22, 2015 9:16 AM, Jennifer Barnabee < > jennifer.barna...@gmail.com> wrote: > > > +1 for the second option. > > On Thu, Oct 22, 2015 at 9:08 AM, Mark Payne wrote: > > > I two thoughts on this concept. Firstly is the notion of Controller > > Services. Rather than having properties that > > change significantly depending on the strategy selected, each strategy > > could be implemented as a separate > > Controller Service. Then, in the Processor you'd just choose which > > Controller Service to use. This allows the > > properties that need to be configured for that strategy to move into the > > Controller Service itself. > > > > The second thought, is one that I had proposed a while back, but it never > > picked up any traction. The idea is > > to have properties "depend" on other properties. So, for instance, you > > could say Property B is only applicable > > if Property A is set to Allowable Value 1 or Allowable Value 2. If > > Property A is not set or is set to Allowable Value > > 3, for instance, then Property B is not even shown in the UI. > > > > The nice thing about the second option is that it would make the UI > > cleaner for existing Processors. For instance, > > we could indicate in CompressContent that the "Compression Level" > property > > should not even be shown if the > > mode is set to Decompress. Currently, we just document that "This > property > > is ignored unless mode is Compress." > > I would much prefer to not even show the property at all. > > > > Any thoughts? > > > > -Mark > > > > > > > > > On Oct 21, 2015, at 10:11 PM, Joe Witt wrote: > > > > > > Not to trivialize this but curious if we added an 'apply' type option > > > there would that be a sufficient user experience? The core point is > > > once you select a new strategy we need to send that fact to the server > > > to see what effect that has. That happens when you hit ok but as you > > > note that closes the dialogue and requires you to reopen it which is > > > just sort of awkward. So would an 'ok' option which applies the > > > changes and talks to the server and gets new info but doesn't close > > > the dialogue make this good enough? > > > > > > Thanks > > > Joe > > > > > > On Wed, Oct 21, 2015 at 6:09 PM, Michael D. Coon > > > wrote: > > >> Mike, > > >> Thanks, that is EXACTLY what I'm doing as well :)My current > > version just allows them to set the properties up once…if they need to > > change the selection, they unfortunately have to create a whole new > > processor configuration and delete the old one. Kinda stinks but, as you > > said, current UI doesn't support this particular use case. At least I'm > not > > alone! :) > > >> Mike > > >> > > >> > > >> > > >>On Wednesday, October 21, 2015 6:02 PM, Michael Moser < > > moser...@gmail.com> wrote: > > >> > > >> > > >> Mike, > > >> > > >> I have implemented a NiFi processor very similar to what you describe. > > A > > >> ServiceLoader discovers the plugins that are available to the > processor > > and > > >> populates the allowableValues of a PropertyDescriptor. > > >> > > >> In the end I don't think there is a way to resolve your > > questions/issues in > > >> the UI. After you select a plugin, the user will have to save the > > >> processor config, then go back into it to see the new properties that > > you > > >> made available. And if you change your plugin selection, then the > > >> processor will stay invalid until the user deletes the old properties > > that > > >> are no longer needed. The NiFi framework just doesn't support this > edge > > >> case. > > >> > > >> In the end, for my processor, I just made all properties from all > > plugins > > >> available all the time. I named the properties to begin with the > > plugin's > > >> name to help
Re: Configuration Driven Property Updates
Mark, I went down the path of your "ControllerService" approach only to find that the service configuration is tied to an XML file in the nifi installation versus dynamically configurable via the UI (unless I'm missing how to configure a controller service on the UI somewhere). It also means I have a SINGLE instance of my module for the NiFi flow. This is not the case when I have different modules needing multiple instances inside of my flow. For these reasons I abandoned the idea of using a ControllerService for this scenario. I do think that option 2 would be good on its own even outside of this particular scenario. And if option 2 were implemented, it would, at a minimum, help support conditional properties and fix the issues I have today with properties sticking around after I change the module to use at runtime in the flow. Mike On Thursday, October 22, 2015 9:16 AM, Jennifer Barnabee wrote: +1 for the second option. On Thu, Oct 22, 2015 at 9:08 AM, Mark Payne wrote: > I two thoughts on this concept. Firstly is the notion of Controller > Services. Rather than having properties that > change significantly depending on the strategy selected, each strategy > could be implemented as a separate > Controller Service. Then, in the Processor you'd just choose which > Controller Service to use. This allows the > properties that need to be configured for that strategy to move into the > Controller Service itself. > > The second thought, is one that I had proposed a while back, but it never > picked up any traction. The idea is > to have properties "depend" on other properties. So, for instance, you > could say Property B is only applicable > if Property A is set to Allowable Value 1 or Allowable Value 2. If > Property A is not set or is set to Allowable Value > 3, for instance, then Property B is not even shown in the UI. > > The nice thing about the second option is that it would make the UI > cleaner for existing Processors. For instance, > we could indicate in CompressContent that the "Compression Level" property > should not even be shown if the > mode is set to Decompress. Currently, we just document that "This property > is ignored unless mode is Compress." > I would much prefer to not even show the property at all. > > Any thoughts? > > -Mark > > > > > On Oct 21, 2015, at 10:11 PM, Joe Witt wrote: > > > > Not to trivialize this but curious if we added an 'apply' type option > > there would that be a sufficient user experience? The core point is > > once you select a new strategy we need to send that fact to the server > > to see what effect that has. That happens when you hit ok but as you > > note that closes the dialogue and requires you to reopen it which is > > just sort of awkward. So would an 'ok' option which applies the > > changes and talks to the server and gets new info but doesn't close > > the dialogue make this good enough? > > > > Thanks > > Joe > > > > On Wed, Oct 21, 2015 at 6:09 PM, Michael D. Coon > > wrote: > >> Mike, > >> Thanks, that is EXACTLY what I'm doing as well :) My current > version just allows them to set the properties up once…if they need to > change the selection, they unfortunately have to create a whole new > processor configuration and delete the old one. Kinda stinks but, as you > said, current UI doesn't support this particular use case. At least I'm not > alone! :) > >> Mike > >> > >> > >> > >> On Wednesday, October 21, 2015 6:02 PM, Michael Moser < > moser...@gmail.com> wrote: > >> > >> > >> Mike, > >> > >> I have implemented a NiFi processor very similar to what you describe. > A > >> ServiceLoader discovers the plugins that are available to the processor > and > >> populates the allowableValues of a PropertyDescriptor. > >> > >> In the end I don't think there is a way to resolve your > questions/issues in > >> the UI. After you select a plugin, the user will have to save the > >> processor config, then go back into it to see the new properties that > you > >> made available. And if you change your plugin selection, then the > >> processor will stay invalid until the user deletes the old properties > that > >> are no longer needed. The NiFi framework just doesn't support this edge > >> case. > >> > >> In the end, for my processor, I just made all properties from all > plugins > >> available all the time. I named the properties to begin with the > plugin's > >> name to help users keep track of which plugin they applied to. > >> > >> -- Mike > >> > >> > >> > >> On Wed, Oct 21, 2015 at 5:50 PM, Michael D. Coon > > >> wrote: > >> > >>> Hey Tony, > >>> I do have working code but not in a public repo. I am, in fact, > trying > >>> to use NiFi as a backing engine. I think the bottom line is that I > want to > >>> be able to add/remove some configuration properties based on the > settings > >>> of other configuration properties. For example, I might specify > "ClassA" > >>> for property "PluginClass" and the implementa
Re: Configuration Driven Property Updates
+1 for the second option. On Thu, Oct 22, 2015 at 9:08 AM, Mark Payne wrote: > I two thoughts on this concept. Firstly is the notion of Controller > Services. Rather than having properties that > change significantly depending on the strategy selected, each strategy > could be implemented as a separate > Controller Service. Then, in the Processor you'd just choose which > Controller Service to use. This allows the > properties that need to be configured for that strategy to move into the > Controller Service itself. > > The second thought, is one that I had proposed a while back, but it never > picked up any traction. The idea is > to have properties "depend" on other properties. So, for instance, you > could say Property B is only applicable > if Property A is set to Allowable Value 1 or Allowable Value 2. If > Property A is not set or is set to Allowable Value > 3, for instance, then Property B is not even shown in the UI. > > The nice thing about the second option is that it would make the UI > cleaner for existing Processors. For instance, > we could indicate in CompressContent that the "Compression Level" property > should not even be shown if the > mode is set to Decompress. Currently, we just document that "This property > is ignored unless mode is Compress." > I would much prefer to not even show the property at all. > > Any thoughts? > > -Mark > > > > > On Oct 21, 2015, at 10:11 PM, Joe Witt wrote: > > > > Not to trivialize this but curious if we added an 'apply' type option > > there would that be a sufficient user experience? The core point is > > once you select a new strategy we need to send that fact to the server > > to see what effect that has. That happens when you hit ok but as you > > note that closes the dialogue and requires you to reopen it which is > > just sort of awkward. So would an 'ok' option which applies the > > changes and talks to the server and gets new info but doesn't close > > the dialogue make this good enough? > > > > Thanks > > Joe > > > > On Wed, Oct 21, 2015 at 6:09 PM, Michael D. Coon > > wrote: > >> Mike, > >> Thanks, that is EXACTLY what I'm doing as well :)My current > version just allows them to set the properties up once…if they need to > change the selection, they unfortunately have to create a whole new > processor configuration and delete the old one. Kinda stinks but, as you > said, current UI doesn't support this particular use case. At least I'm not > alone! :) > >> Mike > >> > >> > >> > >> On Wednesday, October 21, 2015 6:02 PM, Michael Moser < > moser...@gmail.com> wrote: > >> > >> > >> Mike, > >> > >> I have implemented a NiFi processor very similar to what you describe. > A > >> ServiceLoader discovers the plugins that are available to the processor > and > >> populates the allowableValues of a PropertyDescriptor. > >> > >> In the end I don't think there is a way to resolve your > questions/issues in > >> the UI. After you select a plugin, the user will have to save the > >> processor config, then go back into it to see the new properties that > you > >> made available. And if you change your plugin selection, then the > >> processor will stay invalid until the user deletes the old properties > that > >> are no longer needed. The NiFi framework just doesn't support this edge > >> case. > >> > >> In the end, for my processor, I just made all properties from all > plugins > >> available all the time. I named the properties to begin with the > plugin's > >> name to help users keep track of which plugin they applied to. > >> > >> -- Mike > >> > >> > >> > >> On Wed, Oct 21, 2015 at 5:50 PM, Michael D. Coon > > >> wrote: > >> > >>> Hey Tony, > >>> I do have working code but not in a public repo. I am, in fact, > trying > >>> to use NiFi as a backing engine. I think the bottom line is that I > want to > >>> be able to add/remove some configuration properties based on the > settings > >>> of other configuration properties. For example, I might specify > "ClassA" > >>> for property "PluginClass" and the implementation "ClassA" has its own > set > >>> of parameters required to operate. I want to expose those on the UI. It > >>> works as I have it now, but if after setting up "ClassA" as my plugin I > >>> change the "PluginClass" setting to "ClassB", all of ClassA's > properties > >>> stick around even though they are no longer what I'm returning from > >>> "getSupportedProperties". It looks like, at least on the version I'm > >>> using, that NiFi is caching the properties internally and not syncing > them > >>> with the properties that I provide through the "getSupportedProperties" > >>> method. More specifically, if I actually set a value on any of the > "ClassA" > >>> required properties, they stick around forever. I would think that > whatever > >>> I return from the "getSupportedProperties" method should, in fact, be > the > >>> set of properties exposed on the UI. But they fall out of sync if I > change > >>> my settings and add/
Re: Configuration Driven Property Updates
I think the real issue is the treatment of Processor’s properties in general. In the currently discussed approach one of the discovered modules is itself a property of a Processor. On top of that a selected module's “property” presents a list of its own properties. IMHO the better way of solving it would be to introduce a notion of DynamicProcessor - a placeholder that itself would require further selection of a module and further configuration of its individual properties. Having said that it’s no different then selecting one of the processors from the processor selection (available today), but I guess the true benefit of the “dynamic” approach is the ability to change processor’s implementation without changing a processor in the flow and then reconnecting it with the rest of the components. Oleg > On Oct 22, 2015, at 6:32 AM, Mike Coon wrote: > > Joe, yes and no. Yes, sending the new configuration settings back to the > server for validation is what is needed without closing the dialogue. But it > also must refresh the dialogue with any newly added config info provided by > the getSupportedProperties method of the processor being configured. And as I > said in a previous message, when properties are removed or no longer returned > by the getSupportedProperties method, those must go away as well. > > Sent from my iPhone > >> On Oct 21, 2015, at 10:11 PM, Joe Witt wrote: >> >> Not to trivialize this but curious if we added an 'apply' type option >> there would that be a sufficient user experience? The core point is >> once you select a new strategy we need to send that fact to the server >> to see what effect that has. That happens when you hit ok but as you >> note that closes the dialogue and requires you to reopen it which is >> just sort of awkward. So would an 'ok' option which applies the >> changes and talks to the server and gets new info but doesn't close >> the dialogue make this good enough? >> >> Thanks >> Joe >> >> On Wed, Oct 21, 2015 at 6:09 PM, Michael D. Coon >> wrote: >>> Mike, >>> Thanks, that is EXACTLY what I'm doing as well :)My current version >>> just allows them to set the properties up once…if they need to change the >>> selection, they unfortunately have to create a whole new processor >>> configuration and delete the old one. Kinda stinks but, as you said, >>> current UI doesn't support this particular use case. At least I'm not >>> alone! :) >>> Mike >>> >>> >>> >>>On Wednesday, October 21, 2015 6:02 PM, Michael Moser >>> wrote: >>> >>> >>> Mike, >>> >>> I have implemented a NiFi processor very similar to what you describe. A >>> ServiceLoader discovers the plugins that are available to the processor and >>> populates the allowableValues of a PropertyDescriptor. >>> >>> In the end I don't think there is a way to resolve your questions/issues in >>> the UI. After you select a plugin, the user will have to save the >>> processor config, then go back into it to see the new properties that you >>> made available. And if you change your plugin selection, then the >>> processor will stay invalid until the user deletes the old properties that >>> are no longer needed. The NiFi framework just doesn't support this edge >>> case. >>> >>> In the end, for my processor, I just made all properties from all plugins >>> available all the time. I named the properties to begin with the plugin's >>> name to help users keep track of which plugin they applied to. >>> >>> -- Mike >>> >>> >>> >>> On Wed, Oct 21, 2015 at 5:50 PM, Michael D. Coon >>> wrote: >>> Hey Tony, I do have working code but not in a public repo. I am, in fact, trying to use NiFi as a backing engine. I think the bottom line is that I want to be able to add/remove some configuration properties based on the settings of other configuration properties. For example, I might specify "ClassA" for property "PluginClass" and the implementation "ClassA" has its own set of parameters required to operate. I want to expose those on the UI. It works as I have it now, but if after setting up "ClassA" as my plugin I change the "PluginClass" setting to "ClassB", all of ClassA's properties stick around even though they are no longer what I'm returning from "getSupportedProperties". It looks like, at least on the version I'm using, that NiFi is caching the properties internally and not syncing them with the properties that I provide through the "getSupportedProperties" method. More specifically, if I actually set a value on any of the "ClassA" required properties, they stick around forever. I would think that whatever I return from the "getSupportedProperties" method should, in fact, be the set of properties exposed on the UI. But they fall out of sync if I change my settings and add/remove properties. Mike On Wednesday, October 21, 2015 5:42 PM, Tony Kurc >>>
Re: Configuration Driven Property Updates
Joe, yes and no. Yes, sending the new configuration settings back to the server for validation is what is needed without closing the dialogue. But it also must refresh the dialogue with any newly added config info provided by the getSupportedProperties method of the processor being configured. And as I said in a previous message, when properties are removed or no longer returned by the getSupportedProperties method, those must go away as well. Sent from my iPhone > On Oct 21, 2015, at 10:11 PM, Joe Witt wrote: > > Not to trivialize this but curious if we added an 'apply' type option > there would that be a sufficient user experience? The core point is > once you select a new strategy we need to send that fact to the server > to see what effect that has. That happens when you hit ok but as you > note that closes the dialogue and requires you to reopen it which is > just sort of awkward. So would an 'ok' option which applies the > changes and talks to the server and gets new info but doesn't close > the dialogue make this good enough? > > Thanks > Joe > > On Wed, Oct 21, 2015 at 6:09 PM, Michael D. Coon > wrote: >> Mike, >> Thanks, that is EXACTLY what I'm doing as well :)My current version >> just allows them to set the properties up once…if they need to change the >> selection, they unfortunately have to create a whole new processor >> configuration and delete the old one. Kinda stinks but, as you said, current >> UI doesn't support this particular use case. At least I'm not alone! :) >> Mike >> >> >> >> On Wednesday, October 21, 2015 6:02 PM, Michael Moser >> wrote: >> >> >> Mike, >> >> I have implemented a NiFi processor very similar to what you describe. A >> ServiceLoader discovers the plugins that are available to the processor and >> populates the allowableValues of a PropertyDescriptor. >> >> In the end I don't think there is a way to resolve your questions/issues in >> the UI. After you select a plugin, the user will have to save the >> processor config, then go back into it to see the new properties that you >> made available. And if you change your plugin selection, then the >> processor will stay invalid until the user deletes the old properties that >> are no longer needed. The NiFi framework just doesn't support this edge >> case. >> >> In the end, for my processor, I just made all properties from all plugins >> available all the time. I named the properties to begin with the plugin's >> name to help users keep track of which plugin they applied to. >> >> -- Mike >> >> >> >> On Wed, Oct 21, 2015 at 5:50 PM, Michael D. Coon >> wrote: >> >>> Hey Tony, >>> I do have working code but not in a public repo. I am, in fact, trying >>> to use NiFi as a backing engine. I think the bottom line is that I want to >>> be able to add/remove some configuration properties based on the settings >>> of other configuration properties. For example, I might specify "ClassA" >>> for property "PluginClass" and the implementation "ClassA" has its own set >>> of parameters required to operate. I want to expose those on the UI. It >>> works as I have it now, but if after setting up "ClassA" as my plugin I >>> change the "PluginClass" setting to "ClassB", all of ClassA's properties >>> stick around even though they are no longer what I'm returning from >>> "getSupportedProperties". It looks like, at least on the version I'm >>> using, that NiFi is caching the properties internally and not syncing them >>> with the properties that I provide through the "getSupportedProperties" >>> method. More specifically, if I actually set a value on any of the "ClassA" >>> required properties, they stick around forever. I would think that whatever >>> I return from the "getSupportedProperties" method should, in fact, be the >>> set of properties exposed on the UI. But they fall out of sync if I change >>> my settings and add/remove properties. >>> Mike >>> >>> >>> >>> On Wednesday, October 21, 2015 5:42 PM, Tony Kurc >>> wrote: >>> >>> >>> Mike, >>> I'm not quite following what you're trying to accomplish. The use case you >>> described was a bit too meta for me to wrap my head around. It sounds like >>> a dataflow abstraction layer with nifi as a backing engine. It sounds like >>> you have some code, is it in a public repo so we could take a look at what >>> you have? >>> >>> Tony >>> On Oct 20, 2015 8:58 AM, "Michael D. Coon" >>> wrote: >>> All, I'm building a set of NiFi processors that allow data flow managers to dynamically configure reusable data processing modules wrapped inside >>> NiFi processors. Imagine that we might want to use NiFi in some cases and >>> Storm or Spark in others. The business logic in the processing modules would be the same regardless of what framework was feeding the data to the modules...it allows us to reuse capability in different data flow environments. I have this working to some
Re: Configuration Driven Property Updates
Not to trivialize this but curious if we added an 'apply' type option there would that be a sufficient user experience? The core point is once you select a new strategy we need to send that fact to the server to see what effect that has. That happens when you hit ok but as you note that closes the dialogue and requires you to reopen it which is just sort of awkward. So would an 'ok' option which applies the changes and talks to the server and gets new info but doesn't close the dialogue make this good enough? Thanks Joe On Wed, Oct 21, 2015 at 6:09 PM, Michael D. Coon wrote: > Mike, > Thanks, that is EXACTLY what I'm doing as well :)My current version > just allows them to set the properties up once…if they need to change the > selection, they unfortunately have to create a whole new processor > configuration and delete the old one. Kinda stinks but, as you said, current > UI doesn't support this particular use case. At least I'm not alone! :) > Mike > > > > On Wednesday, October 21, 2015 6:02 PM, Michael Moser > wrote: > > > Mike, > > I have implemented a NiFi processor very similar to what you describe. A > ServiceLoader discovers the plugins that are available to the processor and > populates the allowableValues of a PropertyDescriptor. > > In the end I don't think there is a way to resolve your questions/issues in > the UI. After you select a plugin, the user will have to save the > processor config, then go back into it to see the new properties that you > made available. And if you change your plugin selection, then the > processor will stay invalid until the user deletes the old properties that > are no longer needed. The NiFi framework just doesn't support this edge > case. > > In the end, for my processor, I just made all properties from all plugins > available all the time. I named the properties to begin with the plugin's > name to help users keep track of which plugin they applied to. > > -- Mike > > > > On Wed, Oct 21, 2015 at 5:50 PM, Michael D. Coon > wrote: > >> Hey Tony, >>I do have working code but not in a public repo. I am, in fact, trying >> to use NiFi as a backing engine. I think the bottom line is that I want to >> be able to add/remove some configuration properties based on the settings >> of other configuration properties. For example, I might specify "ClassA" >> for property "PluginClass" and the implementation "ClassA" has its own set >> of parameters required to operate. I want to expose those on the UI. It >> works as I have it now, but if after setting up "ClassA" as my plugin I >> change the "PluginClass" setting to "ClassB", all of ClassA's properties >> stick around even though they are no longer what I'm returning from >> "getSupportedProperties". It looks like, at least on the version I'm >> using, that NiFi is caching the properties internally and not syncing them >> with the properties that I provide through the "getSupportedProperties" >> method. More specifically, if I actually set a value on any of the "ClassA" >> required properties, they stick around forever. I would think that whatever >> I return from the "getSupportedProperties" method should, in fact, be the >> set of properties exposed on the UI. But they fall out of sync if I change >> my settings and add/remove properties. >> Mike >> >> >> >> On Wednesday, October 21, 2015 5:42 PM, Tony Kurc >> wrote: >> >> >> Mike, >> I'm not quite following what you're trying to accomplish. The use case you >> described was a bit too meta for me to wrap my head around. It sounds like >> a dataflow abstraction layer with nifi as a backing engine. It sounds like >> you have some code, is it in a public repo so we could take a look at what >> you have? >> >> Tony >> On Oct 20, 2015 8:58 AM, "Michael D. Coon" >> wrote: >> >> > All, >> >I'm building a set of NiFi processors that allow data flow managers to >> > dynamically configure reusable data processing modules wrapped inside >> NiFi >> > processors. Imagine that we might want to use NiFi in some cases and >> Storm >> > or Spark in others. The business logic in the processing modules would be >> > the same regardless of what framework was feeding the data to the >> > modules...it allows us to reuse capability in different data flow >> > environments. >> > >> >I have this working to some extent using ServiceLoader to discover the >> > plugin modules and making those "allowableValues" of a >> PropertyDescriptor. >> > Now it gets tricky. When the user selects one of the modules, the module >> > itself specifies its own set of configuration properties that I'd like to >> > expose to the data flow manager on the UI. That too works as I can grab >> the >> > properties from the module and expose them as NiFi PropertyDescriptors. >> So >> > far, so good. But when I change my mind and choose a different module >> from >> > the allowableValues set I created, the old properties remain set with an >> > error that they are not support
Re: Configuration Driven Property Updates
Mike, Thanks, that is EXACTLY what I'm doing as well :) My current version just allows them to set the properties up once…if they need to change the selection, they unfortunately have to create a whole new processor configuration and delete the old one. Kinda stinks but, as you said, current UI doesn't support this particular use case. At least I'm not alone! :) Mike On Wednesday, October 21, 2015 6:02 PM, Michael Moser wrote: Mike, I have implemented a NiFi processor very similar to what you describe. A ServiceLoader discovers the plugins that are available to the processor and populates the allowableValues of a PropertyDescriptor. In the end I don't think there is a way to resolve your questions/issues in the UI. After you select a plugin, the user will have to save the processor config, then go back into it to see the new properties that you made available. And if you change your plugin selection, then the processor will stay invalid until the user deletes the old properties that are no longer needed. The NiFi framework just doesn't support this edge case. In the end, for my processor, I just made all properties from all plugins available all the time. I named the properties to begin with the plugin's name to help users keep track of which plugin they applied to. -- Mike On Wed, Oct 21, 2015 at 5:50 PM, Michael D. Coon wrote: > Hey Tony, > I do have working code but not in a public repo. I am, in fact, trying > to use NiFi as a backing engine. I think the bottom line is that I want to > be able to add/remove some configuration properties based on the settings > of other configuration properties. For example, I might specify "ClassA" > for property "PluginClass" and the implementation "ClassA" has its own set > of parameters required to operate. I want to expose those on the UI. It > works as I have it now, but if after setting up "ClassA" as my plugin I > change the "PluginClass" setting to "ClassB", all of ClassA's properties > stick around even though they are no longer what I'm returning from > "getSupportedProperties". It looks like, at least on the version I'm > using, that NiFi is caching the properties internally and not syncing them > with the properties that I provide through the "getSupportedProperties" > method. More specifically, if I actually set a value on any of the "ClassA" > required properties, they stick around forever. I would think that whatever > I return from the "getSupportedProperties" method should, in fact, be the > set of properties exposed on the UI. But they fall out of sync if I change > my settings and add/remove properties. > Mike > > > > On Wednesday, October 21, 2015 5:42 PM, Tony Kurc > wrote: > > > Mike, > I'm not quite following what you're trying to accomplish. The use case you > described was a bit too meta for me to wrap my head around. It sounds like > a dataflow abstraction layer with nifi as a backing engine. It sounds like > you have some code, is it in a public repo so we could take a look at what > you have? > > Tony > On Oct 20, 2015 8:58 AM, "Michael D. Coon" > wrote: > > > All, > > I'm building a set of NiFi processors that allow data flow managers to > > dynamically configure reusable data processing modules wrapped inside > NiFi > > processors. Imagine that we might want to use NiFi in some cases and > Storm > > or Spark in others. The business logic in the processing modules would be > > the same regardless of what framework was feeding the data to the > > modules...it allows us to reuse capability in different data flow > > environments. > > > > I have this working to some extent using ServiceLoader to discover the > > plugin modules and making those "allowableValues" of a > PropertyDescriptor. > > Now it gets tricky. When the user selects one of the modules, the module > > itself specifies its own set of configuration properties that I'd like to > > expose to the data flow manager on the UI. That too works as I can grab > the > > properties from the module and expose them as NiFi PropertyDescriptors. > So > > far, so good. But when I change my mind and choose a different module > from > > the allowableValues set I created, the old properties remain set with an > > error that they are not supported properties. > > > > So how do I get rid of the old properties once a new module has been > > selected? Also, is there any way that the UI can dynamically update the > > required properties once I make a selection of which module I want to > use? > > As it stands, I have to close the settings window and reopen to get the > new > > properties list. > > Thanks for any advice! > > Mike > > > > > > > >
Re: Configuration Driven Property Updates
Mike, I have implemented a NiFi processor very similar to what you describe. A ServiceLoader discovers the plugins that are available to the processor and populates the allowableValues of a PropertyDescriptor. In the end I don't think there is a way to resolve your questions/issues in the UI. After you select a plugin, the user will have to save the processor config, then go back into it to see the new properties that you made available. And if you change your plugin selection, then the processor will stay invalid until the user deletes the old properties that are no longer needed. The NiFi framework just doesn't support this edge case. In the end, for my processor, I just made all properties from all plugins available all the time. I named the properties to begin with the plugin's name to help users keep track of which plugin they applied to. -- Mike On Wed, Oct 21, 2015 at 5:50 PM, Michael D. Coon wrote: > Hey Tony, >I do have working code but not in a public repo. I am, in fact, trying > to use NiFi as a backing engine. I think the bottom line is that I want to > be able to add/remove some configuration properties based on the settings > of other configuration properties. For example, I might specify "ClassA" > for property "PluginClass" and the implementation "ClassA" has its own set > of parameters required to operate. I want to expose those on the UI. It > works as I have it now, but if after setting up "ClassA" as my plugin I > change the "PluginClass" setting to "ClassB", all of ClassA's properties > stick around even though they are no longer what I'm returning from > "getSupportedProperties". It looks like, at least on the version I'm > using, that NiFi is caching the properties internally and not syncing them > with the properties that I provide through the "getSupportedProperties" > method. More specifically, if I actually set a value on any of the "ClassA" > required properties, they stick around forever. I would think that whatever > I return from the "getSupportedProperties" method should, in fact, be the > set of properties exposed on the UI. But they fall out of sync if I change > my settings and add/remove properties. > Mike > > > > On Wednesday, October 21, 2015 5:42 PM, Tony Kurc > wrote: > > > Mike, > I'm not quite following what you're trying to accomplish. The use case you > described was a bit too meta for me to wrap my head around. It sounds like > a dataflow abstraction layer with nifi as a backing engine. It sounds like > you have some code, is it in a public repo so we could take a look at what > you have? > > Tony > On Oct 20, 2015 8:58 AM, "Michael D. Coon" > wrote: > > > All, > >I'm building a set of NiFi processors that allow data flow managers to > > dynamically configure reusable data processing modules wrapped inside > NiFi > > processors. Imagine that we might want to use NiFi in some cases and > Storm > > or Spark in others. The business logic in the processing modules would be > > the same regardless of what framework was feeding the data to the > > modules...it allows us to reuse capability in different data flow > > environments. > > > >I have this working to some extent using ServiceLoader to discover the > > plugin modules and making those "allowableValues" of a > PropertyDescriptor. > > Now it gets tricky. When the user selects one of the modules, the module > > itself specifies its own set of configuration properties that I'd like to > > expose to the data flow manager on the UI. That too works as I can grab > the > > properties from the module and expose them as NiFi PropertyDescriptors. > So > > far, so good. But when I change my mind and choose a different module > from > > the allowableValues set I created, the old properties remain set with an > > error that they are not supported properties. > > > >So how do I get rid of the old properties once a new module has been > > selected? Also, is there any way that the UI can dynamically update the > > required properties once I make a selection of which module I want to > use? > > As it stands, I have to close the settings window and reopen to get the > new > > properties list. > >Thanks for any advice! > > Mike > > > > > > > >
Re: Configuration Driven Property Updates
Hey Tony, I do have working code but not in a public repo. I am, in fact, trying to use NiFi as a backing engine. I think the bottom line is that I want to be able to add/remove some configuration properties based on the settings of other configuration properties. For example, I might specify "ClassA" for property "PluginClass" and the implementation "ClassA" has its own set of parameters required to operate. I want to expose those on the UI. It works as I have it now, but if after setting up "ClassA" as my plugin I change the "PluginClass" setting to "ClassB", all of ClassA's properties stick around even though they are no longer what I'm returning from "getSupportedProperties". It looks like, at least on the version I'm using, that NiFi is caching the properties internally and not syncing them with the properties that I provide through the "getSupportedProperties" method. More specifically, if I actually set a value on any of the "ClassA" required properties, they stick around forever. I would think that whatever I return from the "getSupportedProperties" method should, in fact, be the set of properties exposed on the UI. But they fall out of sync if I change my settings and add/remove properties. Mike On Wednesday, October 21, 2015 5:42 PM, Tony Kurc wrote: Mike, I'm not quite following what you're trying to accomplish. The use case you described was a bit too meta for me to wrap my head around. It sounds like a dataflow abstraction layer with nifi as a backing engine. It sounds like you have some code, is it in a public repo so we could take a look at what you have? Tony On Oct 20, 2015 8:58 AM, "Michael D. Coon" wrote: > All, > I'm building a set of NiFi processors that allow data flow managers to > dynamically configure reusable data processing modules wrapped inside NiFi > processors. Imagine that we might want to use NiFi in some cases and Storm > or Spark in others. The business logic in the processing modules would be > the same regardless of what framework was feeding the data to the > modules...it allows us to reuse capability in different data flow > environments. > > I have this working to some extent using ServiceLoader to discover the > plugin modules and making those "allowableValues" of a PropertyDescriptor. > Now it gets tricky. When the user selects one of the modules, the module > itself specifies its own set of configuration properties that I'd like to > expose to the data flow manager on the UI. That too works as I can grab the > properties from the module and expose them as NiFi PropertyDescriptors. So > far, so good. But when I change my mind and choose a different module from > the allowableValues set I created, the old properties remain set with an > error that they are not supported properties. > > So how do I get rid of the old properties once a new module has been > selected? Also, is there any way that the UI can dynamically update the > required properties once I make a selection of which module I want to use? > As it stands, I have to close the settings window and reopen to get the new > properties list. > Thanks for any advice! > Mike > >
Re: Configuration Driven Property Updates
Mike, I'm not quite following what you're trying to accomplish. The use case you described was a bit too meta for me to wrap my head around. It sounds like a dataflow abstraction layer with nifi as a backing engine. It sounds like you have some code, is it in a public repo so we could take a look at what you have? Tony On Oct 20, 2015 8:58 AM, "Michael D. Coon" wrote: > All, >I'm building a set of NiFi processors that allow data flow managers to > dynamically configure reusable data processing modules wrapped inside NiFi > processors. Imagine that we might want to use NiFi in some cases and Storm > or Spark in others. The business logic in the processing modules would be > the same regardless of what framework was feeding the data to the > modules...it allows us to reuse capability in different data flow > environments. > >I have this working to some extent using ServiceLoader to discover the > plugin modules and making those "allowableValues" of a PropertyDescriptor. > Now it gets tricky. When the user selects one of the modules, the module > itself specifies its own set of configuration properties that I'd like to > expose to the data flow manager on the UI. That too works as I can grab the > properties from the module and expose them as NiFi PropertyDescriptors. So > far, so good. But when I change my mind and choose a different module from > the allowableValues set I created, the old properties remain set with an > error that they are not supported properties. > >So how do I get rid of the old properties once a new module has been > selected? Also, is there any way that the UI can dynamically update the > required properties once I make a selection of which module I want to use? > As it stands, I have to close the settings window and reopen to get the new > properties list. >Thanks for any advice! > Mike > >
