[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428756#comment-16428756
 ] 

ASF subversion and git services commented on NIFI-4149:
---

Commit 7cc77937d8636feb65d708d14c3b22a6c71a1ed7 in nifi's branch 
refs/heads/master from [~pvillard]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=7cc7793 ]

NIFI-4149 - fixed contrib-check errors


> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.7.0
>
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428581#comment-16428581
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2205
  
@pvillard31 thanks for the update! I made a couple of very minor tweaks to 
verbiage (expanded the term "registry" to "variable registry" in the docs for 
clarification, and expanded the method name on the dto from "scopeEl" to 
"ExpressionLanguageScope" for getter/setter). Otherwise, all looks great. Many 
thanks for tackling this task! It's far from trivial and everything works 
great! Code looks great, definitely a very welcome improvement to the 
usability! +1 merged to master.


> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
> Fix For: 1.7.0
>
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428572#comment-16428572
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2205


> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428570#comment-16428570
 ] 

ASF subversion and git services commented on NIFI-4149:
---

Commit 644133dc353e8ca3b66658e768b1199c4820da06 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=644133d ]

NIFI-4149: Minor tweaks to verbiage

This closes #2205.


> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412954#comment-16412954
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2205
  
@markap14 - thanks for the feedback, completely agree. I pushed a commit 
that should address your comments and I rebased against master.


> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409875#comment-16409875
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2205
  
Hey @pvillard31 I definitely where this is going. I agree that another 
PR/JIRA can be issued for the more elaborate indication of processor-specific 
variables in the EL. The only issue that I have with this as-is, is that if we 
have processor that still indicates `.expressionLanguageSupported(true)`, in 
the UI all that it says is "Expression Language Scope: NONE" and that believes 
me to believe that Expression Language is not supported. I think if the 
processor still indicates `true` instead of the Scope, that we still show in 
the UI: "Expression Language Supported: true" or whatever it is that we show 
currently.

The only other note, which is very minor, is that in the UI I would avoid 
showing the Scope as NONE, VARIABLE_REGISTRY, FLOWFILE_ATTRIBUTES and instead 
use human-friendly syntax: "Not Supported", "Variable Registry Only" and 
"Variable Registry and FlowFile Attributes" - perhaps just update 
ExpressionLanguageScope enum to contain a "description" or something and then 
populate the DTO with that? Thoughts on that?


> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403550#comment-16403550
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2205
  
@joewitt @markap14 
I finally found time to get back to this one. I updated every component to 
use the new method with the EL scope, I updated the mock framwork to fail in 
case scope is not correctly set, I updated the documentation, changed a bit the 
UI to add the scope in both the processor documentation and the properties 
tooltip.

A bit like the recent change with the definition of dependencies in poms, 
this PR is quite impacting and possibly/probably conflicting with opened PRs. 
Let me know if you have feedbacks/comments.

Note - @markap14, I didn't address your comment to document the variables 
made available in some processors like in RouteText. I can do it in this PR or 
in a follow-up JIRA. Just wanted to get the main work done asap, took me quite 
some time :)


> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Major
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319042#comment-16319042
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2205
  
@pvillard31 regarding the mock framework: nope, that looks good! I must 
have just missed it the first time around. Thanks!


> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215372#comment-16215372
 ] 

Mark Payne commented on NIFI-4149:
--

[~pvillard] - ah I missed that change to MockPropertyValue. I think that logic 
actually needs to go into its ensureExpressionsEvaluated() method though, no?

> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215249#comment-16215249
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2205
  
Thanks for the feedback @markap14! I'll have a look for variables at 
processor level (never noticed it to be honest). Regarding the mock framework, 
I added some changes 
[here](https://github.com/apache/nifi/pull/2205/files#diff-870890bcfd73493b10d56ae9f7e1d3b1R192).
 Are you expecting something else?


> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215137#comment-16215137
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/2205
  
@pvillard31 one note to consider is that a Processor is able to also 
provide its own set of 'variables' to the EL when evaluated. For example, when 
RouteText is used with a "Matching Strategy" of "Satisfies Expression", the 
variables "line" and "lineNo" are also available in EL to make routing 
decisions. I think it would be nice for this to be better reflected here as 
well, such as :

```
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.expressionLanguageVariable("line", "The line of text to evaluate")
.expressionLanguageVariable("lineNo", "The 1-based number that represents 
which line in the incoming FlowFile this line is")
```

Additionally, the Mock Framework currently will throw an Exception if a 
Property indicates that EL is supported but does not evaluate the Expression 
Language (or vice versa). It probably makes sense to update that to also ensure 
that the proper scope was used (if VARIABLE_REGISTRY is specified but FlowFile 
attributes are evaluated, that should fail). This could potentially be done as 
a separate JIRA, though.



> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214847#comment-16214847
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2205#discussion_r146207102
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/expression/ExpressionLanguageScope.java 
---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.expression;
+
+/**
+ * Indicates the scope of expression language on a property descriptor
+ */
+public enum ExpressionLanguageScope {
+
+/**
+ * Expression language is disabled
+ */
+NONE,
+
+/**
+ * Expression language is evaluated against variables in registry
+ */
+VARIABLE_REGISTRY_ONLY,
--- End diff --

Done


> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16214846#comment-16214846
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2205#discussion_r146207015
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/expression/ExpressionLanguageScope.java 
---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.expression;
+
+/**
+ * Indicates the scope of expression language on a property descriptor
--- End diff --

Right, I added some comments to reflect how scope are hierarchically used.


> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207014#comment-16207014
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2205#discussion_r145028735
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/expression/ExpressionLanguageScope.java 
---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.expression;
+
+/**
+ * Indicates the scope of expression language on a property descriptor
--- End diff --

would be good to add comments here and/or with each scope but they're 
effectively hierarchical.

NONE -> VARIABLE_REGISTRY -> FLOWFILE

If something supports flowfile it supports both flowfile attributes and 
variable registry.  If something supports variable registry then it does not 
consider flowfile attributes.  Further, when supporting the variable registry 
there is an implied scope of 'variables defined in the closest process group 
and then up to the higher group and so on until the root group and finally the 
nifi.properties variables are evaluated and then environment variables/etc..'


> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207011#comment-16207011
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2205#discussion_r145028535
  
--- Diff: 
nifi-api/src/main/java/org/apache/nifi/expression/ExpressionLanguageScope.java 
---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.expression;
+
+/**
+ * Indicates the scope of expression language on a property descriptor
+ */
+public enum ExpressionLanguageScope {
+
+/**
+ * Expression language is disabled
+ */
+NONE,
+
+/**
+ * Expression language is evaluated against variables in registry
+ */
+VARIABLE_REGISTRY_ONLY,
--- End diff --

probably makes sense to just name the scope as 'VARIABLE_REGISTRY'


> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198496#comment-16198496
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2205
  
Note to myself: improve the documentation to take care of the case exposed 
in the screenshot. In case the processor does not accept incoming connection, 
the documentation should not say "evaluated per flow file" even though the 
scope is saying otherwise. That's because a property can be shared between two 
processors with different annotations for incoming connections.


> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198422#comment-16198422
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

GitHub user pvillard31 opened a pull request:

https://github.com/apache/nifi/pull/2205

NIFI-4149 - [WIP] - Indicate if EL is evaluated against FFs or not

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pvillard31/nifi NIFI-4149

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2205.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2205


commit 0b6e38308bb434f769b5977bfe6c3c5d26936328
Author: Pierre Villard 
Date:   2017-09-19T07:27:46Z

NIFI-4149 - [WIP] - Indicate if EL is evaluated against FFs or not




> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198426#comment-16198426
 ] 

ASF GitHub Bot commented on NIFI-4149:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/2205
  
This PR is work in progress.

For now, the idea is to provide a way to specify how is evaluated the 
expression language: against attributes of incoming flow files (+ registries) 
or only the variables in registries. The current PR also deprecates the 
existing ``.expressionLanguageSupported(boolean)`` method.

This PR updates the documentation generation to display the information in 
the documentation of each processor. Example:
![screenshot-2017-10-10 
nifi](https://user-images.githubusercontent.com/11541012/31379809-ac25752c-adaf-11e7-8127-c4ead12e6523.png)

Remaining work:
- update all the existing processors to switch to the correct method / scope
- add this information in the UI (in the tooltip of the properties)

The objective of this current PR is to get feedbacks from the community 
regarding the current approach before updating all the existing processors.

Thanks!



> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[Pierre Villard (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093442#comment-16093442
 ] 

Pierre Villard commented on NIFI-4149:
--

The only problematic situation I can think of is when a property is expecting a 
regular expression. IIRC we have properties accepting both but it could require 
specific handling if we are using the combination of regex and EL. To be 
confirmed.

I guess we could deprecate the {{evaluateAttributeExpressions()}} and directly 
make the call in {{getValue()}} (and similar methods). For the UX, I'd say that 
we could deprecate {{supportsExpressionLanguage()}} in the builders and add a 
method like you suggest: {{supportsFlowFileExpressions()}}. I agree that the 
tooltip should also be updated to reflect the change.

Also agree regarding lifecycle.




> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (NIFI-4149) Indicate if EL is evaluated against FFs or not

[Joey Frazee (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084405#comment-16084405
 ] 

Joey Frazee commented on NIFI-4149:
---

It'd definitely be useful to enable EL on any property by default with the 
assumption/clear indication that what it's eval'ing is java properties, 
variable registry properties or the environment. Are there any circumstances 
where you'd want to prohibit EL from these sources? I can't think of any 
(assuming NIFI-2767 never goes in).

So the important thing then is how to clearly indicate the behavior in the UI 
for the user and walk back all the supportsExpressionLanguage() methods on the 
builders. For the former, I assume it'd be a tooltip update. For the latter, we 
could either say no harm, no foul or deprecate the existing method and add 
something like supportsFlowFileExpressions() so people will go clean stuff up.

Since I mentioned NIFI-2767, I'll add that the observation's been made that 
what counts as a valid attribute expression is also entangled with the 
processor lifecycle. So a more general solution would include a way to indicate 
at what parts of the lifecycle the expressions can be evaluated against which 
sources.

> Indicate if EL is evaluated against FFs or not
> --
>
> Key: NIFI-4149
> URL: https://issues.apache.org/jira/browse/NIFI-4149
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Pierre Villard
>
> With the addition of EL in a lot of places to improve SDLC and workflow 
> staging, it becomes important to indicate to users if the expression language 
> enabled on a property will be evaluated against the attributes of incoming 
> flow files or if it will only be evaluated against various variable stores 
> (env variables, variable registry, etc).
> Actually, the expression language (without evaluation against flow files) 
> could be allowed on any property by default, and evaluation against flow 
> files would be what is actually indicated in the UI as we are doing today. 
> Adopting this approach could solve a lot of JIRA/PRs we are seeing to add EL 
> on some specific properties (without evaluation against FFs).
> Having expression language to access external values could make sense on any 
> property for any user. However evaluating the expression language against FFs 
> is clearly a more complex challenge when it comes to session management and 
> such.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)