Re: [VOTE] Release Apache NiFi Registry 0.8.0

[Pierre Villard]

+1 binding

Went through the usual steps for this release.

Le sam. 17 oct. 2020 à 22:03, Bryan Bende  a écrit :

> +1 (binding)
>
> - Verified everything in the standard release helper
> - Ran the test-all-dbs profile, created one minor jira [1]
> - Ran a secure registry with secure nifi
> - Test basic save and import
>
> Thanks for RM'ing!
>
> [1] https://issues.apache.org/jira/browse/NIFIREG-426
>
> On Fri, Oct 16, 2020 at 11:56 AM Andrew Lim 
> wrote:
> >
> > +1 (binding)
> >
> > -Ran full clean install on OS X (Catalina 10.15.2)
> > -Tested secure NiFi Registry 0.8.0 with secure NiFi 1.12.1
> > -Tested basic flow, bucket and user functionality
> > -Tested common versioned flow use cases
> > -Reviewed documentation updates
> >
> > Drew
> >
> > > On Oct 14, 2020, at 11:25 AM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > > Registry 0.8.0.
> > >
> > > The source zip, including signatures, digests, etc. can be found at:
> > > https://repository.apache.org/content/repositories/orgapachenifi-1174
> > >
> > > The source being voted upon and the convenience binaries can be found
> at:
> > >
> https://dist.apache.org/repos/dist/dev/nifi/nifi-registry/nifi-registry-0.8.0/
> > >
> > > A helpful reminder on how the release candidate verification process
> works:
> > >
> https://cwiki.apache.org/confluence/display/NIFI/How+to+help+verify+an+Apache+NiFi+release+candidate
> > >
> > > The Git tag is nifi-registry-0.8.0-RC1
> > > The Git commit ID is 74c7435386b1118ed0b545efea7343deec6f4b29
> > >
> https://gitbox.apache.org/repos/asf?p=nifi-registry.git;a=commit;h=74c7435386b1118ed0b545efea7343deec6f4b29
> > >
> > > Checksums of nifi-registry-0.8.0-source-release.zip:
> > > SHA256:
> afa4409a6b82aabd25fece787ba5eba1496313a3370bdf503c29d24f1d0b5a2c
> > > SHA512:
> > >
> 9c6e516f24b61b11dd889a5ea14950e7498801b7a51bdc14e4a8c091e9b834db213968bce8ee360882da9ee1c29aa8062eb6b9d9ec99c62b2f332c84aefe110f
> > >
> > > Release artifacts are signed with the key F92A93B30C07C6D5. The KEYS
> file
> > > is available here:
> > > https://dist.apache.org/repos/dist/release/nifi/KEYS
> > >
> > > 19 issues were closed/resolved for this release:
> > >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12348583&projectId=12320920
> > >
> > > Release note highlights can be found here:
> > >
> https://cwiki.apache.org/confluence/display/NIFIREG/Release+Notes#ReleaseNotes-NiFiRegistry0.8.0
> > >
> > > The vote will be open for 72 hours.
> > > Please download the release candidate and evaluate the necessary items
> > > including checking hashes, signatures, build from source, and test.
> Then
> > > please vote:
> > >
> > > [ ] +1 Release this package as nifi-registry-0.8.0
> > > [ ] +0 no opinion
> > > [ ] -1 Do not release this package because...
> >
>

[RESULT][VOTE] Release Apache NiFi Registry 0.8.0

[Pierre Villard]

Apache NiFi Community,

I am pleased to announce that the nifi-registry-0.8.0 release of Apache
NiFi Registry passes with:
4 +1 (binding) votes
1 +1 (non-binding) votes
0 0 votes
0 -1 votes

Thanks to all who helped make this release possible.

Here is the PMC vote thread:
https://mail-archives.apache.org/mod_mbox/nifi-dev/202010.mbox/%3ccabgfv27fskwoa2vfeql-h7+3ekqhetj08fhjrleb5lqdgfr...@mail.gmail.com%3e

Parameter Contexts Expression Language scopes

[Chad Zobrisky]

Hello,

I was configuring an SSL Context Controller Service today and had the
keystores and passwords passed into the container via environment
variables. I thought it would be nice to be able to reference these from
the parameter context. Maybe either giving Parameter Context values the
VARIABLE_REGISTRY scope in the Expression Language, or a new scope for
references external to nifi?

I think for refreshing the Parameter Context on those external changes, it
would require an edit/re-apply just as it does now, and would have to make
sure it is well documented.

I'd be interested in creating a PR for this if the idea makes sense and is
acceptable.

Thanks,
Chad

question about AttributeRollingWindow

[Di Giulio Sara]

Hello,

I am trying to use attributeRollingWindow processor, but I do not understand 
why the average value is not the average of the mapped field of the input queue 
for the time I set.
May you send me an example where the processor is configured?

Thank you in advanced,

Sara Di Giulio

Data Scientist ICT
Brembo S.p.A
+39 035 605 2114
+39 344 019 5211
Sara Di Giulio
ICT
sara_digiu...@brembo.it




[https://www.brembo.com/signature/brembo.jpg] 
[https://www.brembo.com/signature/facebook.jpg] 
  
[https://www.brembo.com/signature/instagram.jpg] 
  
[https://www.brembo.com/signature/linkedin.jpg] 
  
[https://www.brembo.com/signature/twitter.jpg] 
  
[https://www.brembo.com/signature/youtube.jpg] 



This e-mail and any attachments is confidential and intended for the 
addressee(s) only and may contain information that is privileged (under 
applicable law as for example client attorney communication), or otherwise 
legally protected. Access to this email by anybody else is unauthorized. If you 
are not the intended recipient, please delete this message and any attachments 
and advise the sender by return e-mail.
Whilst Brembo Group companies take reasonable care to ensure that any 
attachment to this e-mail does not contain software viruses, this cannot be 
guaranteed and you should therefore carry out your own virus checks before 
opening any attachment.
Brembo Group companies accept no responsibility or liability for any damage 
that you suffer as a result of software viruses.
For Brembo S.p.A. and other EU subsidiaries only: Brembo S.p.A. and/or its EU 
affiliates, depending on the entity you are communicating with, will process 
personal data collected in the context of the exchange of this thread of 
e-mails in compliance with the EU Regulation 679/2016 (GDPR), as independent 
data controller.
You can find the privacy policy of your data controller, as applicable, at 
https://www.brembo.com/en/company/about/personal-data-protection

Re: Parameter Contexts Expression Language scopes

[Andy LoPresto]

Hi Chad,

Parameters were introduced as a way to deprecate (NiFi) variables entirely. I’m 
not sure that introducing a dependency between the two is a positive step 
forward. I think there is a separate conversation to be had about allowing 
parameters access to environment variables, but I think this could introduce 
problems as parameters are designed for flexibility and portability, and moving 
from a system where a parameter was actually a pass-through to an environment 
variable would cause unexpected problems on the destination system. 

I think the pros and cons of this need to be clearly enumerated and discussed 
here. Thanks for bringing this up. 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Oct 19, 2020, at 9:43 AM, Chad Zobrisky  wrote:
> 
> Hello,
> 
> I was configuring an SSL Context Controller Service today and had the
> keystores and passwords passed into the container via environment
> variables. I thought it would be nice to be able to reference these from
> the parameter context. Maybe either giving Parameter Context values the
> VARIABLE_REGISTRY scope in the Expression Language, or a new scope for
> references external to nifi?
> 
> I think for refreshing the Parameter Context on those external changes, it
> would require an edit/re-apply just as it does now, and would have to make
> sure it is well documented.
> 
> I'd be interested in creating a PR for this if the idea makes sense and is
> acceptable.
> 
> Thanks,
> Chad

Re: Parameter Contexts Expression Language scopes

[Chad Zobrisky]

Andy,

Thanks for the response!

When I was thinking through this the deprecation of variables was
definitely on my mind but the fact that it already had direct access to
environment variables was the simplest path. I think it does make more
sense to add access to environment variables to the parameter context, or
allowing a specific scope just for environment variables in the
expression language.

I think giving access to environment variables actually allows more
portability between environments, eg dev, test, prod. Defining those once
and allowing for nifi to pull them in makes sense to me and I think is
common in container environments.

Looking forward to discussing more and better approaches.
Chad

On Mon, Oct 19, 2020 at 1:46 PM Andy LoPresto  wrote:

> Hi Chad,
>
> Parameters were introduced as a way to deprecate (NiFi) variables
> entirely. I’m not sure that introducing a dependency between the two is a
> positive step forward. I think there is a separate conversation to be had
> about allowing parameters access to environment variables, but I think this
> could introduce problems as parameters are designed for flexibility and
> portability, and moving from a system where a parameter was actually a
> pass-through to an environment variable would cause unexpected problems on
> the destination system.
>
> I think the pros and cons of this need to be clearly enumerated and
> discussed here. Thanks for bringing this up.
>
>
> Andy LoPresto
> alopre...@apache.org
> alopresto.apa...@gmail.com
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On Oct 19, 2020, at 9:43 AM, Chad Zobrisky  wrote:
> >
> > Hello,
> >
> > I was configuring an SSL Context Controller Service today and had the
> > keystores and passwords passed into the container via environment
> > variables. I thought it would be nice to be able to reference these from
> > the parameter context. Maybe either giving Parameter Context values the
> > VARIABLE_REGISTRY scope in the Expression Language, or a new scope for
> > references external to nifi?
> >
> > I think for refreshing the Parameter Context on those external changes,
> it
> > would require an edit/re-apply just as it does now, and would have to
> make
> > sure it is well documented.
> >
> > I'd be interested in creating a PR for this if the idea makes sense and
> is
> > acceptable.
> >
> > Thanks,
> > Chad
>
>

Re: Parameter Contexts Expression Language scopes

[u...@moosheimer.com]

Chad,

So far I thought that only the NiFi variables are deprecated and access to 
environment variables will still be possible.

If this is not the case, then I agree with you. It should definitely be 
possible to access environment variables. Otherwise I can't imagine how to 
refer to the hostname or the current JAVA path or ... or ... or on each node?!

Mit freundlichen Grüßen / best regards
Kay-Uwe Moosheimer

> Am 19.10.2020 um 20:00 schrieb Chad Zobrisky :
> 
> Andy,
> 
> Thanks for the response!
> 
> When I was thinking through this the deprecation of variables was
> definitely on my mind but the fact that it already had direct access to
> environment variables was the simplest path. I think it does make more
> sense to add access to environment variables to the parameter context, or
> allowing a specific scope just for environment variables in the
> expression language.
> 
> I think giving access to environment variables actually allows more
> portability between environments, eg dev, test, prod. Defining those once
> and allowing for nifi to pull them in makes sense to me and I think is
> common in container environments.
> 
> Looking forward to discussing more and better approaches.
> Chad
> 
>> On Mon, Oct 19, 2020 at 1:46 PM Andy LoPresto  wrote:
>> 
>> Hi Chad,
>> 
>> Parameters were introduced as a way to deprecate (NiFi) variables
>> entirely. I’m not sure that introducing a dependency between the two is a
>> positive step forward. I think there is a separate conversation to be had
>> about allowing parameters access to environment variables, but I think this
>> could introduce problems as parameters are designed for flexibility and
>> portability, and moving from a system where a parameter was actually a
>> pass-through to an environment variable would cause unexpected problems on
>> the destination system.
>> 
>> I think the pros and cons of this need to be clearly enumerated and
>> discussed here. Thanks for bringing this up.
>> 
>> 
>> Andy LoPresto
>> alopre...@apache.org
>> alopresto.apa...@gmail.com
>> He/Him
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>> 
 On Oct 19, 2020, at 9:43 AM, Chad Zobrisky  wrote:
>>> 
>>> Hello,
>>> 
>>> I was configuring an SSL Context Controller Service today and had the
>>> keystores and passwords passed into the container via environment
>>> variables. I thought it would be nice to be able to reference these from
>>> the parameter context. Maybe either giving Parameter Context values the
>>> VARIABLE_REGISTRY scope in the Expression Language, or a new scope for
>>> references external to nifi?
>>> 
>>> I think for refreshing the Parameter Context on those external changes,
>> it
>>> would require an edit/re-apply just as it does now, and would have to
>> make
>>> sure it is well documented.
>>> 
>>> I'd be interested in creating a PR for this if the idea makes sense and
>> is
>>> acceptable.
>>> 
>>> Thanks,
>>> Chad
>> 
>> 

Re: Parameter Contexts Expression Language scopes

[Bryan Bende]

Access to environment variables directly from expression language is
not being removed.

The discussion is about whether a parameter value should be able to
use expression language to reference an environment variable.

For example, processor property has #{keystore.password} -> parameter
"keystore.password" has value "${KEYSTORE_PASSWORD}" which then gets
the password from an environment variable.



On Mon, Oct 19, 2020 at 2:14 PM u...@moosheimer.com  wrote:
>
> Chad,
>
> So far I thought that only the NiFi variables are deprecated and access to 
> environment variables will still be possible.
>
> If this is not the case, then I agree with you. It should definitely be 
> possible to access environment variables. Otherwise I can't imagine how to 
> refer to the hostname or the current JAVA path or ... or ... or on each node?!
>
> Mit freundlichen Grüßen / best regards
> Kay-Uwe Moosheimer
>
> > Am 19.10.2020 um 20:00 schrieb Chad Zobrisky :
> >
> > Andy,
> >
> > Thanks for the response!
> >
> > When I was thinking through this the deprecation of variables was
> > definitely on my mind but the fact that it already had direct access to
> > environment variables was the simplest path. I think it does make more
> > sense to add access to environment variables to the parameter context, or
> > allowing a specific scope just for environment variables in the
> > expression language.
> >
> > I think giving access to environment variables actually allows more
> > portability between environments, eg dev, test, prod. Defining those once
> > and allowing for nifi to pull them in makes sense to me and I think is
> > common in container environments.
> >
> > Looking forward to discussing more and better approaches.
> > Chad
> >
> >> On Mon, Oct 19, 2020 at 1:46 PM Andy LoPresto  wrote:
> >>
> >> Hi Chad,
> >>
> >> Parameters were introduced as a way to deprecate (NiFi) variables
> >> entirely. I’m not sure that introducing a dependency between the two is a
> >> positive step forward. I think there is a separate conversation to be had
> >> about allowing parameters access to environment variables, but I think this
> >> could introduce problems as parameters are designed for flexibility and
> >> portability, and moving from a system where a parameter was actually a
> >> pass-through to an environment variable would cause unexpected problems on
> >> the destination system.
> >>
> >> I think the pros and cons of this need to be clearly enumerated and
> >> discussed here. Thanks for bringing this up.
> >>
> >>
> >> Andy LoPresto
> >> alopre...@apache.org
> >> alopresto.apa...@gmail.com
> >> He/Him
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>
>  On Oct 19, 2020, at 9:43 AM, Chad Zobrisky  wrote:
> >>>
> >>> Hello,
> >>>
> >>> I was configuring an SSL Context Controller Service today and had the
> >>> keystores and passwords passed into the container via environment
> >>> variables. I thought it would be nice to be able to reference these from
> >>> the parameter context. Maybe either giving Parameter Context values the
> >>> VARIABLE_REGISTRY scope in the Expression Language, or a new scope for
> >>> references external to nifi?
> >>>
> >>> I think for refreshing the Parameter Context on those external changes,
> >> it
> >>> would require an edit/re-apply just as it does now, and would have to
> >> make
> >>> sure it is well documented.
> >>>
> >>> I'd be interested in creating a PR for this if the idea makes sense and
> >> is
> >>> acceptable.
> >>>
> >>> Thanks,
> >>> Chad
> >>
> >>
>

Re: Parameter Contexts Expression Language scopes

[Chris Sampson]

Being based primarily in Docker containers and having experience with both
Kubernetes (where secrets such as KESTORE_PASSWORD can be injected as
environment variables or files) and Docker Swarm (which only handles
secrets as environment variables), I'd have definitely been wanting this
before moving from Variables to Parameters if I was still in Swarm (or
Docker Compose/straight up Docker).

It's certainly possible to script creating/updating Parameters via the
Toolkit/NiPyAPI, but in Docker Swarm that's not so easy (whereas it's
possible as a Job in Kubernetes, for example). So environment variables
could save the day in that instance.

I guess one likely problem (but no different to how I guess the Variable
Registry uses env vars) would be how NiFi will handle changes to the env
vars - does it:

   - ignore them until instance restart, which could lead to maintainer
   confusion (I've changed KEYSTORE_PASSWORD in the env but things are still
   failing in NiFi)
   - alert the maintainer to the fact that the env var has changed and a
   Parameter needs updating, with the new value being used after all
   associated processors/controllers have been restarted
   - automatically attempt to update the parameters by restarting all
   associated processors/controllers, which I'd guess would be a bit dangerous
   for interrupting in-flow data, etc.


---
*Chris Sampson*
IT Consultant
chris.samp...@naimuri.com



On Mon, 19 Oct 2020 at 19:35, Bryan Bende  wrote:

> Access to environment variables directly from expression language is
> not being removed.
>
> The discussion is about whether a parameter value should be able to
> use expression language to reference an environment variable.
>
> For example, processor property has #{keystore.password} -> parameter
> "keystore.password" has value "${KEYSTORE_PASSWORD}" which then gets
> the password from an environment variable.
>
>
>
> On Mon, Oct 19, 2020 at 2:14 PM u...@moosheimer.com 
> wrote:
> >
> > Chad,
> >
> > So far I thought that only the NiFi variables are deprecated and access
> to environment variables will still be possible.
> >
> > If this is not the case, then I agree with you. It should definitely be
> possible to access environment variables. Otherwise I can't imagine how to
> refer to the hostname or the current JAVA path or ... or ... or on each
> node?!
> >
> > Mit freundlichen Grüßen / best regards
> > Kay-Uwe Moosheimer
> >
> > > Am 19.10.2020 um 20:00 schrieb Chad Zobrisky :
> > >
> > > Andy,
> > >
> > > Thanks for the response!
> > >
> > > When I was thinking through this the deprecation of variables was
> > > definitely on my mind but the fact that it already had direct access to
> > > environment variables was the simplest path. I think it does make more
> > > sense to add access to environment variables to the parameter context,
> or
> > > allowing a specific scope just for environment variables in the
> > > expression language.
> > >
> > > I think giving access to environment variables actually allows more
> > > portability between environments, eg dev, test, prod. Defining those
> once
> > > and allowing for nifi to pull them in makes sense to me and I think is
> > > common in container environments.
> > >
> > > Looking forward to discussing more and better approaches.
> > > Chad
> > >
> > >> On Mon, Oct 19, 2020 at 1:46 PM Andy LoPresto 
> wrote:
> > >>
> > >> Hi Chad,
> > >>
> > >> Parameters were introduced as a way to deprecate (NiFi) variables
> > >> entirely. I’m not sure that introducing a dependency between the two
> is a
> > >> positive step forward. I think there is a separate conversation to be
> had
> > >> about allowing parameters access to environment variables, but I
> think this
> > >> could introduce problems as parameters are designed for flexibility
> and
> > >> portability, and moving from a system where a parameter was actually a
> > >> pass-through to an environment variable would cause unexpected
> problems on
> > >> the destination system.
> > >>
> > >> I think the pros and cons of this need to be clearly enumerated and
> > >> discussed here. Thanks for bringing this up.
> > >>
> > >>
> > >> Andy LoPresto
> > >> alopre...@apache.org
> > >> alopresto.apa...@gmail.com
> > >> He/Him
> > >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > >>
> >  On Oct 19, 2020, at 9:43 AM, Chad Zobrisky 
> wrote:
> > >>>
> > >>> Hello,
> > >>>
> > >>> I was configuring an SSL Context Controller Service today and had the
> > >>> keystores and passwords passed into the container via environment
> > >>> variables. I thought it would be nice to be able to reference these
> from
> > >>> the parameter context. Maybe either giving Parameter Context values
> the
> > >>> VARIABLE_REGISTRY scope in the Expression Language, or a new scope
> for
> > >>> references external to nifi?
> > >>>
> > >>> I think for refreshing the Parameter Context on those external
> changes,
> > >> it
> > >>> woul

Re: Parameter Contexts Expression Language scopes

[u...@moosheimer.com]

Then I must have misunderstood it.
Thanks Bryan for clarification.

However, the idea of Chad makes sense for me.

Mit freundlichen Grüßen / best regards
Kay-Uwe Moosheimer

> Am 19.10.2020 um 20:35 schrieb Bryan Bende :
> 
> Access to environment variables directly from expression language is
> not being removed.
> 
> The discussion is about whether a parameter value should be able to
> use expression language to reference an environment variable.
> 
> For example, processor property has #{keystore.password} -> parameter
> "keystore.password" has value "${KEYSTORE_PASSWORD}" which then gets
> the password from an environment variable.
> 
> 
> 
>> On Mon, Oct 19, 2020 at 2:14 PM u...@moosheimer.com  
>> wrote:
>> 
>> Chad,
>> 
>> So far I thought that only the NiFi variables are deprecated and access to 
>> environment variables will still be possible.
>> 
>> If this is not the case, then I agree with you. It should definitely be 
>> possible to access environment variables. Otherwise I can't imagine how to 
>> refer to the hostname or the current JAVA path or ... or ... or on each 
>> node?!
>> 
>> Mit freundlichen Grüßen / best regards
>> Kay-Uwe Moosheimer
>> 
 Am 19.10.2020 um 20:00 schrieb Chad Zobrisky :
>>> 
>>> Andy,
>>> 
>>> Thanks for the response!
>>> 
>>> When I was thinking through this the deprecation of variables was
>>> definitely on my mind but the fact that it already had direct access to
>>> environment variables was the simplest path. I think it does make more
>>> sense to add access to environment variables to the parameter context, or
>>> allowing a specific scope just for environment variables in the
>>> expression language.
>>> 
>>> I think giving access to environment variables actually allows more
>>> portability between environments, eg dev, test, prod. Defining those once
>>> and allowing for nifi to pull them in makes sense to me and I think is
>>> common in container environments.
>>> 
>>> Looking forward to discussing more and better approaches.
>>> Chad
>>> 
 On Mon, Oct 19, 2020 at 1:46 PM Andy LoPresto  wrote:
 
 Hi Chad,
 
 Parameters were introduced as a way to deprecate (NiFi) variables
 entirely. I’m not sure that introducing a dependency between the two is a
 positive step forward. I think there is a separate conversation to be had
 about allowing parameters access to environment variables, but I think this
 could introduce problems as parameters are designed for flexibility and
 portability, and moving from a system where a parameter was actually a
 pass-through to an environment variable would cause unexpected problems on
 the destination system.
 
 I think the pros and cons of this need to be clearly enumerated and
 discussed here. Thanks for bringing this up.
 
 
 Andy LoPresto
 alopre...@apache.org
 alopresto.apa...@gmail.com
 He/Him
 PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
 
>> On Oct 19, 2020, at 9:43 AM, Chad Zobrisky  wrote:
> 
> Hello,
> 
> I was configuring an SSL Context Controller Service today and had the
> keystores and passwords passed into the container via environment
> variables. I thought it would be nice to be able to reference these from
> the parameter context. Maybe either giving Parameter Context values the
> VARIABLE_REGISTRY scope in the Expression Language, or a new scope for
> references external to nifi?
> 
> I think for refreshing the Parameter Context on those external changes,
 it
> would require an edit/re-apply just as it does now, and would have to
 make
> sure it is well documented.
> 
> I'd be interested in creating a PR for this if the idea makes sense and
 is
> acceptable.
> 
> Thanks,
> Chad
 
 
>> 

Re: Parameter Contexts Expression Language scopes

[Bryan Bende]

I think there is some confusion based on terminology...

A given property has an expression language scope defined, which can
be "flow file attributes" or "variable registry only". This was
created mostly for documentation purposes so that users could look at
a property in the docs and see "expression language supported: true"
and know which values they could reference. What it really comes down
to in the code is the difference between
"someProperty.evaluateAttributeExpressions()" and
"someProperty.evaluateAttributeExpressions(flowFile)"... meaning can I
reference a value from a flow file or not.

In the case of "variable registry only", the value can come from any
of the following...
a) system properties
b) expression language
c) process-group variable registry
d) file-based variable registry

When people have stated that "variables are being deprecated in favor
of parameters", they are referring to the last two items (c & d).

The reason being that parameters solved several short-comings of those
two options...

- the ability to store sensitive values encrypted
- the ability to reference them from any property using a new syntax
#{...}, not long dependent on component developer saying
"expressionLanguageSupported(true)" on the descriptor
- the ability to create policies for which users/groups could
reference a set of parameters
- the hierarchical ambiguity of what "${foo}" actually resolves to

If you just want to use environment variables, why use parameter
contexts? Expression language has always offered access to env vars
and still will going forward.

The one  argument I can see is that not all properties support
expression language, so using parameters gives you a way around that.





On Mon, Oct 19, 2020 at 3:58 PM Chris Sampson
 wrote:
>
> Being based primarily in Docker containers and having experience with both
> Kubernetes (where secrets such as KESTORE_PASSWORD can be injected as
> environment variables or files) and Docker Swarm (which only handles
> secrets as environment variables), I'd have definitely been wanting this
> before moving from Variables to Parameters if I was still in Swarm (or
> Docker Compose/straight up Docker).
>
> It's certainly possible to script creating/updating Parameters via the
> Toolkit/NiPyAPI, but in Docker Swarm that's not so easy (whereas it's
> possible as a Job in Kubernetes, for example). So environment variables
> could save the day in that instance.
>
> I guess one likely problem (but no different to how I guess the Variable
> Registry uses env vars) would be how NiFi will handle changes to the env
> vars - does it:
>
>- ignore them until instance restart, which could lead to maintainer
>confusion (I've changed KEYSTORE_PASSWORD in the env but things are still
>failing in NiFi)
>- alert the maintainer to the fact that the env var has changed and a
>Parameter needs updating, with the new value being used after all
>associated processors/controllers have been restarted
>- automatically attempt to update the parameters by restarting all
>associated processors/controllers, which I'd guess would be a bit dangerous
>for interrupting in-flow data, etc.
>
>
> ---
> *Chris Sampson*
> IT Consultant
> chris.samp...@naimuri.com
> 
>
>
> On Mon, 19 Oct 2020 at 19:35, Bryan Bende  wrote:
>
> > Access to environment variables directly from expression language is
> > not being removed.
> >
> > The discussion is about whether a parameter value should be able to
> > use expression language to reference an environment variable.
> >
> > For example, processor property has #{keystore.password} -> parameter
> > "keystore.password" has value "${KEYSTORE_PASSWORD}" which then gets
> > the password from an environment variable.
> >
> >
> >
> > On Mon, Oct 19, 2020 at 2:14 PM u...@moosheimer.com 
> > wrote:
> > >
> > > Chad,
> > >
> > > So far I thought that only the NiFi variables are deprecated and access
> > to environment variables will still be possible.
> > >
> > > If this is not the case, then I agree with you. It should definitely be
> > possible to access environment variables. Otherwise I can't imagine how to
> > refer to the hostname or the current JAVA path or ... or ... or on each
> > node?!
> > >
> > > Mit freundlichen Grüßen / best regards
> > > Kay-Uwe Moosheimer
> > >
> > > > Am 19.10.2020 um 20:00 schrieb Chad Zobrisky :
> > > >
> > > > Andy,
> > > >
> > > > Thanks for the response!
> > > >
> > > > When I was thinking through this the deprecation of variables was
> > > > definitely on my mind but the fact that it already had direct access to
> > > > environment variables was the simplest path. I think it does make more
> > > > sense to add access to environment variables to the parameter context,
> > or
> > > > allowing a specific scope just for environment variables in the
> > > > expression language.
> > > >
> > > > I think giving access to environment variables actually allows more
> > > > portabil

Re: Parameter Contexts Expression Language scopes

[Bryan Bende]

Sorry, the one list was meant to be...

a) system properties
b) environment variables
c) process-group variable registry
d) file-based variable registry

On Mon, Oct 19, 2020 at 4:35 PM Bryan Bende  wrote:
>
> I think there is some confusion based on terminology...
>
> A given property has an expression language scope defined, which can
> be "flow file attributes" or "variable registry only". This was
> created mostly for documentation purposes so that users could look at
> a property in the docs and see "expression language supported: true"
> and know which values they could reference. What it really comes down
> to in the code is the difference between
> "someProperty.evaluateAttributeExpressions()" and
> "someProperty.evaluateAttributeExpressions(flowFile)"... meaning can I
> reference a value from a flow file or not.
>
> In the case of "variable registry only", the value can come from any
> of the following...
> a) system properties
> b) expression language
> c) process-group variable registry
> d) file-based variable registry
>
> When people have stated that "variables are being deprecated in favor
> of parameters", they are referring to the last two items (c & d).
>
> The reason being that parameters solved several short-comings of those
> two options...
>
> - the ability to store sensitive values encrypted
> - the ability to reference them from any property using a new syntax
> #{...}, not long dependent on component developer saying
> "expressionLanguageSupported(true)" on the descriptor
> - the ability to create policies for which users/groups could
> reference a set of parameters
> - the hierarchical ambiguity of what "${foo}" actually resolves to
>
> If you just want to use environment variables, why use parameter
> contexts? Expression language has always offered access to env vars
> and still will going forward.
>
> The one  argument I can see is that not all properties support
> expression language, so using parameters gives you a way around that.
>
>
>
>
>
> On Mon, Oct 19, 2020 at 3:58 PM Chris Sampson
>  wrote:
> >
> > Being based primarily in Docker containers and having experience with both
> > Kubernetes (where secrets such as KESTORE_PASSWORD can be injected as
> > environment variables or files) and Docker Swarm (which only handles
> > secrets as environment variables), I'd have definitely been wanting this
> > before moving from Variables to Parameters if I was still in Swarm (or
> > Docker Compose/straight up Docker).
> >
> > It's certainly possible to script creating/updating Parameters via the
> > Toolkit/NiPyAPI, but in Docker Swarm that's not so easy (whereas it's
> > possible as a Job in Kubernetes, for example). So environment variables
> > could save the day in that instance.
> >
> > I guess one likely problem (but no different to how I guess the Variable
> > Registry uses env vars) would be how NiFi will handle changes to the env
> > vars - does it:
> >
> >- ignore them until instance restart, which could lead to maintainer
> >confusion (I've changed KEYSTORE_PASSWORD in the env but things are still
> >failing in NiFi)
> >- alert the maintainer to the fact that the env var has changed and a
> >Parameter needs updating, with the new value being used after all
> >associated processors/controllers have been restarted
> >- automatically attempt to update the parameters by restarting all
> >associated processors/controllers, which I'd guess would be a bit 
> > dangerous
> >for interrupting in-flow data, etc.
> >
> >
> > ---
> > *Chris Sampson*
> > IT Consultant
> > chris.samp...@naimuri.com
> > 
> >
> >
> > On Mon, 19 Oct 2020 at 19:35, Bryan Bende  wrote:
> >
> > > Access to environment variables directly from expression language is
> > > not being removed.
> > >
> > > The discussion is about whether a parameter value should be able to
> > > use expression language to reference an environment variable.
> > >
> > > For example, processor property has #{keystore.password} -> parameter
> > > "keystore.password" has value "${KEYSTORE_PASSWORD}" which then gets
> > > the password from an environment variable.
> > >
> > >
> > >
> > > On Mon, Oct 19, 2020 at 2:14 PM u...@moosheimer.com 
> > > wrote:
> > > >
> > > > Chad,
> > > >
> > > > So far I thought that only the NiFi variables are deprecated and access
> > > to environment variables will still be possible.
> > > >
> > > > If this is not the case, then I agree with you. It should definitely be
> > > possible to access environment variables. Otherwise I can't imagine how to
> > > refer to the hostname or the current JAVA path or ... or ... or on each
> > > node?!
> > > >
> > > > Mit freundlichen Grüßen / best regards
> > > > Kay-Uwe Moosheimer
> > > >
> > > > > Am 19.10.2020 um 20:00 schrieb Chad Zobrisky :
> > > > >
> > > > > Andy,
> > > > >
> > > > > Thanks for the response!
> > > > >
> > > > > When I was thinking through this the deprecation of variables was
> > > > >

Re: Parameter Contexts Expression Language scopes

[u...@moosheimer.com]

If parameters make it possible to use environment variables, then the system 
can be configured in one place. You would not separate between expression 
language and parameters.

It is also possible that an environment variable should be replaced by a 
parameter or vice versa.
It makes sense to configure this in a central place.

Therefore I think this is a good suggestion.

Mit freundlichen Grüßen / best regards
Kay-Uwe Moosheimer

> Am 19.10.2020 um 23:18 schrieb Bryan Bende :
> 
> I think there is some confusion based on terminology...
> 
> A given property has an expression language scope defined, which can
> be "flow file attributes" or "variable registry only". This was
> created mostly for documentation purposes so that users could look at
> a property in the docs and see "expression language supported: true"
> and know which values they could reference. What it really comes down
> to in the code is the difference between
> "someProperty.evaluateAttributeExpressions()" and
> "someProperty.evaluateAttributeExpressions(flowFile)"... meaning can I
> reference a value from a flow file or not.
> 
> In the case of "variable registry only", the value can come from any
> of the following...
> a) system properties
> b) expression language
> c) process-group variable registry
> d) file-based variable registry
> 
> When people have stated that "variables are being deprecated in favor
> of parameters", they are referring to the last two items (c & d).
> 
> The reason being that parameters solved several short-comings of those
> two options...
> 
> - the ability to store sensitive values encrypted
> - the ability to reference them from any property using a new syntax
> #{...}, not long dependent on component developer saying
> "expressionLanguageSupported(true)" on the descriptor
> - the ability to create policies for which users/groups could
> reference a set of parameters
> - the hierarchical ambiguity of what "${foo}" actually resolves to
> 
> If you just want to use environment variables, why use parameter
> contexts? Expression language has always offered access to env vars
> and still will going forward.
> 
> The one  argument I can see is that not all properties support
> expression language, so using parameters gives you a way around that.
> 
> 
> 
> 
> 
>> On Mon, Oct 19, 2020 at 3:58 PM Chris Sampson
>>  wrote:
>> 
>> Being based primarily in Docker containers and having experience with both
>> Kubernetes (where secrets such as KESTORE_PASSWORD can be injected as
>> environment variables or files) and Docker Swarm (which only handles
>> secrets as environment variables), I'd have definitely been wanting this
>> before moving from Variables to Parameters if I was still in Swarm (or
>> Docker Compose/straight up Docker).
>> 
>> It's certainly possible to script creating/updating Parameters via the
>> Toolkit/NiPyAPI, but in Docker Swarm that's not so easy (whereas it's
>> possible as a Job in Kubernetes, for example). So environment variables
>> could save the day in that instance.
>> 
>> I guess one likely problem (but no different to how I guess the Variable
>> Registry uses env vars) would be how NiFi will handle changes to the env
>> vars - does it:
>> 
>>   - ignore them until instance restart, which could lead to maintainer
>>   confusion (I've changed KEYSTORE_PASSWORD in the env but things are still
>>   failing in NiFi)
>>   - alert the maintainer to the fact that the env var has changed and a
>>   Parameter needs updating, with the new value being used after all
>>   associated processors/controllers have been restarted
>>   - automatically attempt to update the parameters by restarting all
>>   associated processors/controllers, which I'd guess would be a bit dangerous
>>   for interrupting in-flow data, etc.
>> 
>> 
>> ---
>> *Chris Sampson*
>> IT Consultant
>> chris.samp...@naimuri.com
>> 
>> 
>> 
>>> On Mon, 19 Oct 2020 at 19:35, Bryan Bende  wrote:
>>> 
>>> Access to environment variables directly from expression language is
>>> not being removed.
>>> 
>>> The discussion is about whether a parameter value should be able to
>>> use expression language to reference an environment variable.
>>> 
>>> For example, processor property has #{keystore.password} -> parameter
>>> "keystore.password" has value "${KEYSTORE_PASSWORD}" which then gets
>>> the password from an environment variable.
>>> 
>>> 
>>> 
>>> On Mon, Oct 19, 2020 at 2:14 PM u...@moosheimer.com 
>>> wrote:
 
 Chad,
 
 So far I thought that only the NiFi variables are deprecated and access
>>> to environment variables will still be possible.
 
 If this is not the case, then I agree with you. It should definitely be
>>> possible to access environment variables. Otherwise I can't imagine how to
>>> refer to the hostname or the current JAVA path or ... or ... or on each
>>> node?!
 
 Mit freundlichen Grüßen / best regards
 Kay-Uwe Moosheimer
 
> Am 19.10.2020 um 20: