Re: Configuration Service

[Bryan Bende]

This original thread was from quite a long time ago, and there have
been many improvements since then, one of which is a variable registry
that is exposed directly through the UI [1] and does not require
bouncing the NiFi instance.

[1] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Variables_Window

On Thu, Mar 14, 2019 at 6:51 AM Abhay Pasad  wrote:
>
> One of the major issues with an external variable registry file which we
> refer to in the nifi.properties file is that it needs NiFi instance to be
> bounced. While this may serve your purpose in the lower environments in a
> purpose, at an enterprise level cluster this might not be feasible in higher
> environments like Production. What this also means is that due to lack of
> common global variables/parameters, you may need to spin up a new controller
> service every time. I think a Configuration service is a must and should be
> worked upon. :)
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Configuration Service

[Abhay Pasad]

One of the major issues with an external variable registry file which we
refer to in the nifi.properties file is that it needs NiFi instance to be
bounced. While this may serve your purpose in the lower environments in a
purpose, at an enterprise level cluster this might not be feasible in higher
environments like Production. What this also means is that due to lack of
common global variables/parameters, you may need to spin up a new controller
service every time. I think a Configuration service is a must and should be
worked upon. :)



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Configuration Service

[Joe Witt]

Matt,

That is basically what we're talking about providing but it would be
transparent through the expression language capability.  This approach
will make templates far more portable than they are today.

Thanks
Joe

On Wed, Jan 27, 2016 at 11:41 AM, Angry Duck Studio
 wrote:
> Why can't we just have a built-in persistent k-v store controller service
> that any processor can use? Could even modify the expression language to
> interact with it. Maybe something like ${props:} where 'props' refers
> to the built-in store.
>
> -Matt R
>
> On Wed, Jan 27, 2016 at 7:16 AM, Matt Gilman 
> wrote:
>
>> Simon,
>>
>> One idea that's been thrown around is adding a 'variable registry' [1]
>> where you could define variables at a group level that could be referenced
>> by the encapsulated components. Additionally, this would help with the
>> portability of templates when needing to define different values for
>> different environments.
>>
>> Matt
>>
>> [1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>>
>> On Wed, Jan 27, 2016 at 4:30 AM, Simon Ball  wrote:
>>
>> > Other thoughts:
>> >
>> > This will have implications for broadcasting OnPropertyChanged events,
>> and
>> > potentially locking processors around the changes in properties held by
>> the
>> > service. I’d love to hear if anyone can think of any other significant
>> land
>> > mines, or has had any thoughts on anything similar.
>> >
>> > Simon
>> >
>> > On 27 Jan 2016, at 09:26, Simon Ball  > sb...@hortonworks.com>> wrote:
>> >
>> > One of the problems with complex flows is repetition of common
>> > configuration. Many people also want to be able to configure things like
>> > connection strings into an environment specific location outside of the
>> > Flow, and parameterise the flow. Things like the Kerberos|SSL|etc Context
>> > service help in part with this, but the problem could be generalised.
>> >
>> > What do people think of the idea of a configuration provider controller
>> > service, providing essentially the same concept as environment variables,
>> > which holds essentially a key value store which processors can refer to?
>> A
>> > simple way to refer to contents could be an extension on expression
>> > language providing a config(‘key’) function.
>> >
>> > What are your thoughts on this? Worth knocking up a quick prototype
>> > implementation (I’m happy to do so if there is community interest)?
>> >
>> > Simon
>> >
>> >
>>

Re: Configuration Service

[Simon Ball]

That certainly covers a part of it, but I take it since that is read at start 
time, you couldn't have a property change on a flow in progress. If we had 
something like the variable repository, we could fire property change events on 
updates, and change centralised properties at runtime. For example, failover of 
non load balanced endpoints, or just adjustments of other global variables that 
might feed a flow.

Simon


> On 27 Jan 2016, at 17:16, Bryan Bende  wrote:
> 
> FWIW you can externalize connection strings and other similar properties
> into nifi.properties and have your processors references these environment
> variables through expression language. This way you can move the
> flow.xml.gz between two environments without changing anything after you
> setup the nifi.properties for each environment. The variable registry would
> just make this a much better user experience.
> 
>> On Wed, Jan 27, 2016 at 12:07 PM, Joe Witt  wrote:
>> 
>> Matt,
>> 
>> That is basically what we're talking about providing but it would be
>> transparent through the expression language capability.  This approach
>> will make templates far more portable than they are today.
>> 
>> Thanks
>> Joe
>> 
>> On Wed, Jan 27, 2016 at 11:41 AM, Angry Duck Studio
>>  wrote:
>>> Why can't we just have a built-in persistent k-v store controller service
>>> that any processor can use? Could even modify the expression language to
>>> interact with it. Maybe something like ${props:} where 'props'
>> refers
>>> to the built-in store.
>>> 
>>> -Matt R
>>> 
>>> On Wed, Jan 27, 2016 at 7:16 AM, Matt Gilman 
>>> wrote:
>>> 
 Simon,
 
 One idea that's been thrown around is adding a 'variable registry' [1]
 where you could define variables at a group level that could be
>> referenced
 by the encapsulated components. Additionally, this would help with the
 portability of templates when needing to define different values for
 different environments.
 
 Matt
 
 [1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
 
 On Wed, Jan 27, 2016 at 4:30 AM, Simon Ball 
>> wrote:
 
> Other thoughts:
> 
> This will have implications for broadcasting OnPropertyChanged events,
 and
> potentially locking processors around the changes in properties held
>> by
 the
> service. I’d love to hear if anyone can think of any other significant
 land
> mines, or has had any thoughts on anything similar.
> 
> Simon
> 
> On 27 Jan 2016, at 09:26, Simon Ball > wrote:
> 
> One of the problems with complex flows is repetition of common
> configuration. Many people also want to be able to configure things
>> like
> connection strings into an environment specific location outside of
>> the
> Flow, and parameterise the flow. Things like the Kerberos|SSL|etc
>> Context
> service help in part with this, but the problem could be generalised.
> 
> What do people think of the idea of a configuration provider
>> controller
> service, providing essentially the same concept as environment
>> variables,
> which holds essentially a key value store which processors can refer
>> to?
 A
> simple way to refer to contents could be an extension on expression
> language providing a config(‘key’) function.
> 
> What are your thoughts on this? Worth knocking up a quick prototype
> implementation (I’m happy to do so if there is community interest)?
> 
> Simon
>> 

Re: Configuration Service

[Simon Ball]

Other thoughts:

This will have implications for broadcasting OnPropertyChanged events, and 
potentially locking processors around the changes in properties held by the 
service. I’d love to hear if anyone can think of any other significant land 
mines, or has had any thoughts on anything similar.

Simon

On 27 Jan 2016, at 09:26, Simon Ball 
> wrote:

One of the problems with complex flows is repetition of common configuration. 
Many people also want to be able to configure things like connection strings 
into an environment specific location outside of the Flow, and parameterise the 
flow. Things like the Kerberos|SSL|etc Context service help in part with this, 
but the problem could be generalised.

What do people think of the idea of a configuration provider controller 
service, providing essentially the same concept as environment variables, which 
holds essentially a key value store which processors can refer to? A simple way 
to refer to contents could be an extension on expression language providing a 
config(‘key’) function.

What are your thoughts on this? Worth knocking up a quick prototype 
implementation (I’m happy to do so if there is community interest)?

Simon

Re: Configuration Service

[Matt Gilman]

Simon,

One idea that's been thrown around is adding a 'variable registry' [1]
where you could define variables at a group level that could be referenced
by the encapsulated components. Additionally, this would help with the
portability of templates when needing to define different values for
different environments.

Matt

[1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry

On Wed, Jan 27, 2016 at 4:30 AM, Simon Ball  wrote:

> Other thoughts:
>
> This will have implications for broadcasting OnPropertyChanged events, and
> potentially locking processors around the changes in properties held by the
> service. I’d love to hear if anyone can think of any other significant land
> mines, or has had any thoughts on anything similar.
>
> Simon
>
> On 27 Jan 2016, at 09:26, Simon Ball > wrote:
>
> One of the problems with complex flows is repetition of common
> configuration. Many people also want to be able to configure things like
> connection strings into an environment specific location outside of the
> Flow, and parameterise the flow. Things like the Kerberos|SSL|etc Context
> service help in part with this, but the problem could be generalised.
>
> What do people think of the idea of a configuration provider controller
> service, providing essentially the same concept as environment variables,
> which holds essentially a key value store which processors can refer to? A
> simple way to refer to contents could be an extension on expression
> language providing a config(‘key’) function.
>
> What are your thoughts on this? Worth knocking up a quick prototype
> implementation (I’m happy to do so if there is community interest)?
>
> Simon
>
>