Re: Configuration Service
This original thread was from quite a long time ago, and there have been many improvements since then, one of which is a variable registry that is exposed directly through the UI [1] and does not require bouncing the NiFi instance. [1] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Variables_Window On Thu, Mar 14, 2019 at 6:51 AM Abhay Pasad wrote: > > One of the major issues with an external variable registry file which we > refer to in the nifi.properties file is that it needs NiFi instance to be > bounced. While this may serve your purpose in the lower environments in a > purpose, at an enterprise level cluster this might not be feasible in higher > environments like Production. What this also means is that due to lack of > common global variables/parameters, you may need to spin up a new controller > service every time. I think a Configuration service is a must and should be > worked upon. :) > > > > -- > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Configuration Service
One of the major issues with an external variable registry file which we refer to in the nifi.properties file is that it needs NiFi instance to be bounced. While this may serve your purpose in the lower environments in a purpose, at an enterprise level cluster this might not be feasible in higher environments like Production. What this also means is that due to lack of common global variables/parameters, you may need to spin up a new controller service every time. I think a Configuration service is a must and should be worked upon. :) -- Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Configuration Service
That certainly covers a part of it, but I take it since that is read at start time, you couldn't have a property change on a flow in progress. If we had something like the variable repository, we could fire property change events on updates, and change centralised properties at runtime. For example, failover of non load balanced endpoints, or just adjustments of other global variables that might feed a flow. Simon > On 27 Jan 2016, at 17:16, Bryan Bende wrote: > > FWIW you can externalize connection strings and other similar properties > into nifi.properties and have your processors references these environment > variables through expression language. This way you can move the > flow.xml.gz between two environments without changing anything after you > setup the nifi.properties for each environment. The variable registry would > just make this a much better user experience. > >> On Wed, Jan 27, 2016 at 12:07 PM, Joe Witt wrote: >> >> Matt, >> >> That is basically what we're talking about providing but it would be >> transparent through the expression language capability. This approach >> will make templates far more portable than they are today. >> >> Thanks >> Joe >> >> On Wed, Jan 27, 2016 at 11:41 AM, Angry Duck Studio >> wrote: >>> Why can't we just have a built-in persistent k-v store controller service >>> that any processor can use? Could even modify the expression language to >>> interact with it. Maybe something like ${props:} where 'props' >> refers >>> to the built-in store. >>> >>> -Matt R >>> >>> On Wed, Jan 27, 2016 at 7:16 AM, Matt Gilman >>> wrote: >>> Simon, One idea that's been thrown around is adding a 'variable registry' [1] where you could define variables at a group level that could be >> referenced by the encapsulated components. Additionally, this would help with the portability of templates when needing to define different values for different environments. Matt [1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry On Wed, Jan 27, 2016 at 4:30 AM, Simon Ball >> wrote: > Other thoughts: > > This will have implications for broadcasting OnPropertyChanged events, and > potentially locking processors around the changes in properties held >> by the > service. I’d love to hear if anyone can think of any other significant land > mines, or has had any thoughts on anything similar. > > Simon > > On 27 Jan 2016, at 09:26, Simon Ball sb...@hortonworks.com>> wrote: > > One of the problems with complex flows is repetition of common > configuration. Many people also want to be able to configure things >> like > connection strings into an environment specific location outside of >> the > Flow, and parameterise the flow. Things like the Kerberos|SSL|etc >> Context > service help in part with this, but the problem could be generalised. > > What do people think of the idea of a configuration provider >> controller > service, providing essentially the same concept as environment >> variables, > which holds essentially a key value store which processors can refer >> to? A > simple way to refer to contents could be an extension on expression > language providing a config(‘key’) function. > > What are your thoughts on this? Worth knocking up a quick prototype > implementation (I’m happy to do so if there is community interest)? > > Simon >>
Re: Configuration Service
FWIW you can externalize connection strings and other similar properties into nifi.properties and have your processors references these environment variables through expression language. This way you can move the flow.xml.gz between two environments without changing anything after you setup the nifi.properties for each environment. The variable registry would just make this a much better user experience. On Wed, Jan 27, 2016 at 12:07 PM, Joe Witt wrote: > Matt, > > That is basically what we're talking about providing but it would be > transparent through the expression language capability. This approach > will make templates far more portable than they are today. > > Thanks > Joe > > On Wed, Jan 27, 2016 at 11:41 AM, Angry Duck Studio > wrote: > > Why can't we just have a built-in persistent k-v store controller service > > that any processor can use? Could even modify the expression language to > > interact with it. Maybe something like ${props:} where 'props' > refers > > to the built-in store. > > > > -Matt R > > > > On Wed, Jan 27, 2016 at 7:16 AM, Matt Gilman > > wrote: > > > >> Simon, > >> > >> One idea that's been thrown around is adding a 'variable registry' [1] > >> where you could define variables at a group level that could be > referenced > >> by the encapsulated components. Additionally, this would help with the > >> portability of templates when needing to define different values for > >> different environments. > >> > >> Matt > >> > >> [1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry > >> > >> On Wed, Jan 27, 2016 at 4:30 AM, Simon Ball > wrote: > >> > >> > Other thoughts: > >> > > >> > This will have implications for broadcasting OnPropertyChanged events, > >> and > >> > potentially locking processors around the changes in properties held > by > >> the > >> > service. I’d love to hear if anyone can think of any other significant > >> land > >> > mines, or has had any thoughts on anything similar. > >> > > >> > Simon > >> > > >> > On 27 Jan 2016, at 09:26, Simon Ball >> > sb...@hortonworks.com>> wrote: > >> > > >> > One of the problems with complex flows is repetition of common > >> > configuration. Many people also want to be able to configure things > like > >> > connection strings into an environment specific location outside of > the > >> > Flow, and parameterise the flow. Things like the Kerberos|SSL|etc > Context > >> > service help in part with this, but the problem could be generalised. > >> > > >> > What do people think of the idea of a configuration provider > controller > >> > service, providing essentially the same concept as environment > variables, > >> > which holds essentially a key value store which processors can refer > to? > >> A > >> > simple way to refer to contents could be an extension on expression > >> > language providing a config(‘key’) function. > >> > > >> > What are your thoughts on this? Worth knocking up a quick prototype > >> > implementation (I’m happy to do so if there is community interest)? > >> > > >> > Simon > >> > > >> > > >> >
Re: Configuration Service
Matt, That is basically what we're talking about providing but it would be transparent through the expression language capability. This approach will make templates far more portable than they are today. Thanks Joe On Wed, Jan 27, 2016 at 11:41 AM, Angry Duck Studio wrote: > Why can't we just have a built-in persistent k-v store controller service > that any processor can use? Could even modify the expression language to > interact with it. Maybe something like ${props:} where 'props' refers > to the built-in store. > > -Matt R > > On Wed, Jan 27, 2016 at 7:16 AM, Matt Gilman > wrote: > >> Simon, >> >> One idea that's been thrown around is adding a 'variable registry' [1] >> where you could define variables at a group level that could be referenced >> by the encapsulated components. Additionally, this would help with the >> portability of templates when needing to define different values for >> different environments. >> >> Matt >> >> [1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry >> >> On Wed, Jan 27, 2016 at 4:30 AM, Simon Ball wrote: >> >> > Other thoughts: >> > >> > This will have implications for broadcasting OnPropertyChanged events, >> and >> > potentially locking processors around the changes in properties held by >> the >> > service. I’d love to hear if anyone can think of any other significant >> land >> > mines, or has had any thoughts on anything similar. >> > >> > Simon >> > >> > On 27 Jan 2016, at 09:26, Simon Ball > > sb...@hortonworks.com>> wrote: >> > >> > One of the problems with complex flows is repetition of common >> > configuration. Many people also want to be able to configure things like >> > connection strings into an environment specific location outside of the >> > Flow, and parameterise the flow. Things like the Kerberos|SSL|etc Context >> > service help in part with this, but the problem could be generalised. >> > >> > What do people think of the idea of a configuration provider controller >> > service, providing essentially the same concept as environment variables, >> > which holds essentially a key value store which processors can refer to? >> A >> > simple way to refer to contents could be an extension on expression >> > language providing a config(‘key’) function. >> > >> > What are your thoughts on this? Worth knocking up a quick prototype >> > implementation (I’m happy to do so if there is community interest)? >> > >> > Simon >> > >> > >>
Re: Configuration Service
Why can't we just have a built-in persistent k-v store controller service that any processor can use? Could even modify the expression language to interact with it. Maybe something like ${props:} where 'props' refers to the built-in store. -Matt R On Wed, Jan 27, 2016 at 7:16 AM, Matt Gilman wrote: > Simon, > > One idea that's been thrown around is adding a 'variable registry' [1] > where you could define variables at a group level that could be referenced > by the encapsulated components. Additionally, this would help with the > portability of templates when needing to define different values for > different environments. > > Matt > > [1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry > > On Wed, Jan 27, 2016 at 4:30 AM, Simon Ball wrote: > > > Other thoughts: > > > > This will have implications for broadcasting OnPropertyChanged events, > and > > potentially locking processors around the changes in properties held by > the > > service. I’d love to hear if anyone can think of any other significant > land > > mines, or has had any thoughts on anything similar. > > > > Simon > > > > On 27 Jan 2016, at 09:26, Simon Ball > sb...@hortonworks.com>> wrote: > > > > One of the problems with complex flows is repetition of common > > configuration. Many people also want to be able to configure things like > > connection strings into an environment specific location outside of the > > Flow, and parameterise the flow. Things like the Kerberos|SSL|etc Context > > service help in part with this, but the problem could be generalised. > > > > What do people think of the idea of a configuration provider controller > > service, providing essentially the same concept as environment variables, > > which holds essentially a key value store which processors can refer to? > A > > simple way to refer to contents could be an extension on expression > > language providing a config(‘key’) function. > > > > What are your thoughts on this? Worth knocking up a quick prototype > > implementation (I’m happy to do so if there is community interest)? > > > > Simon > > > > >
Re: Configuration Service
Simon, One idea that's been thrown around is adding a 'variable registry' [1] where you could define variables at a group level that could be referenced by the encapsulated components. Additionally, this would help with the portability of templates when needing to define different values for different environments. Matt [1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry On Wed, Jan 27, 2016 at 4:30 AM, Simon Ball wrote: > Other thoughts: > > This will have implications for broadcasting OnPropertyChanged events, and > potentially locking processors around the changes in properties held by the > service. I’d love to hear if anyone can think of any other significant land > mines, or has had any thoughts on anything similar. > > Simon > > On 27 Jan 2016, at 09:26, Simon Ball sb...@hortonworks.com>> wrote: > > One of the problems with complex flows is repetition of common > configuration. Many people also want to be able to configure things like > connection strings into an environment specific location outside of the > Flow, and parameterise the flow. Things like the Kerberos|SSL|etc Context > service help in part with this, but the problem could be generalised. > > What do people think of the idea of a configuration provider controller > service, providing essentially the same concept as environment variables, > which holds essentially a key value store which processors can refer to? A > simple way to refer to contents could be an extension on expression > language providing a config(‘key’) function. > > What are your thoughts on this? Worth knocking up a quick prototype > implementation (I’m happy to do so if there is community interest)? > > Simon > >
Re: Configuration Service
Other thoughts: This will have implications for broadcasting OnPropertyChanged events, and potentially locking processors around the changes in properties held by the service. I’d love to hear if anyone can think of any other significant land mines, or has had any thoughts on anything similar. Simon On 27 Jan 2016, at 09:26, Simon Ball mailto:sb...@hortonworks.com>> wrote: One of the problems with complex flows is repetition of common configuration. Many people also want to be able to configure things like connection strings into an environment specific location outside of the Flow, and parameterise the flow. Things like the Kerberos|SSL|etc Context service help in part with this, but the problem could be generalised. What do people think of the idea of a configuration provider controller service, providing essentially the same concept as environment variables, which holds essentially a key value store which processors can refer to? A simple way to refer to contents could be an extension on expression language providing a config(‘key’) function. What are your thoughts on this? Worth knocking up a quick prototype implementation (I’m happy to do so if there is community interest)? Simon
Configuration Service
One of the problems with complex flows is repetition of common configuration. Many people also want to be able to configure things like connection strings into an environment specific location outside of the Flow, and parameterise the flow. Things like the Kerberos|SSL|etc Context service help in part with this, but the problem could be generalised. What do people think of the idea of a configuration provider controller service, providing essentially the same concept as environment variables, which holds essentially a key value store which processors can refer to? A simple way to refer to contents could be an extension on expression language providing a config(‘key’) function. What are your thoughts on this? Worth knocking up a quick prototype implementation (I’m happy to do so if there is community interest)? Simon