[jira] [Commented] (NIFI-4199) NiFi processors should be able to share proxy settings

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480182#comment-16480182
 ] 

ASF GitHub Bot commented on NIFI-4199:
--

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2704#discussion_r189164964
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
 ---
@@ -92,6 +94,23 @@
 .addValidator(StandardValidators.PORT_VALIDATOR)
 .build();
 
+public static final PropertyDescriptor PROXY_USERNAME = new 
PropertyDescriptor.Builder()
+.name("Proxy Username")
--- End diff --

I recommend a separate `name` vs `displayName` for PROXY_USERNAME.


> NiFi processors should be able to share proxy settings
> --
>
> Key: NIFI-4199
> URL: https://issues.apache.org/jira/browse/NIFI-4199
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre F de Miranda
>Assignee: Koji Kawamura
>Priority: Major
>
> Currently, configuring proxy settings for NiFi processors may be a tedious 
> process that requires the DFM to set proxy settings on individual processors. 
> This leads to:
> * Duplication of work
> * Management overhead (as password must be changed on multiple locations)
> * Lower security (as proxy credentials must be known by "n" DFMs)
> Ideally, NiFi should offer a way to minimise duplication of work by offering 
> a something similar to the Standard SSL Context services. This way, the DFM 
> could set the proxy settings once an all authorised users could tap into 
> those settings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4199) NiFi processors should be able to share proxy settings

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480181#comment-16480181
 ] 

ASF GitHub Bot commented on NIFI-4199:
--

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2704#discussion_r189164994
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
 ---
@@ -92,6 +94,23 @@
 .addValidator(StandardValidators.PORT_VALIDATOR)
 .build();
 
+public static final PropertyDescriptor PROXY_USERNAME = new 
PropertyDescriptor.Builder()
+.name("Proxy Username")
+.description("Proxy username")
+.expressionLanguageSupported(true)
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PROXY_PASSWORD = new 
PropertyDescriptor.Builder()
+.name("Proxy Password")
--- End diff --

I recommend a separate `name` vs `displayName` for PROXY_PASSWORD.


> NiFi processors should be able to share proxy settings
> --
>
> Key: NIFI-4199
> URL: https://issues.apache.org/jira/browse/NIFI-4199
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre F de Miranda
>Assignee: Koji Kawamura
>Priority: Major
>
> Currently, configuring proxy settings for NiFi processors may be a tedious 
> process that requires the DFM to set proxy settings on individual processors. 
> This leads to:
> * Duplication of work
> * Management overhead (as password must be changed on multiple locations)
> * Lower security (as proxy credentials must be known by "n" DFMs)
> Ideally, NiFi should offer a way to minimise duplication of work by offering 
> a something similar to the Standard SSL Context services. This way, the DFM 
> could set the proxy settings once an all authorised users could tap into 
> those settings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4199) NiFi processors should be able to share proxy settings

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480180#comment-16480180
 ] 

ASF GitHub Bot commented on NIFI-4199:
--

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2704#discussion_r189164826
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestListS3.java
 ---
@@ -311,5 +312,10 @@ public void testGetPropertyDescriptors() throws 
Exception {
 assertTrue(pd.contains(ListS3.PREFIX));
 assertTrue(pd.contains(ListS3.USE_VERSIONS));
 assertTrue(pd.contains(ListS3.MIN_AGE));
+
assertTrue(pd.contains(ProxyConfigurationService.PROXY_CONFIGURATION_SERVICE));
+assertTrue(pd.contains(ListS3.PROXY_HOST));
--- End diff --

Minor tweak: The check for PROXY_HOST and PROXY_HOST_PORT duplicates checks 
above on lines 309-310.  I believe this is why we add 5 lines of new 
assertions, but the count of property descriptors only goes up by 3 from 17 to 
20.  It doesn't make any difference, really, but the math was bothering me.


> NiFi processors should be able to share proxy settings
> --
>
> Key: NIFI-4199
> URL: https://issues.apache.org/jira/browse/NIFI-4199
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Andre F de Miranda
>Assignee: Koji Kawamura
>Priority: Major
>
> Currently, configuring proxy settings for NiFi processors may be a tedious 
> process that requires the DFM to set proxy settings on individual processors. 
> This leads to:
> * Duplication of work
> * Management overhead (as password must be changed on multiple locations)
> * Lower security (as proxy credentials must be known by "n" DFMs)
> Ideally, NiFi should offer a way to minimise duplication of work by offering 
> a something similar to the Standard SSL Context services. This way, the DFM 
> could set the proxy settings once an all authorised users could tap into 
> those settings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2704: NIFI-4199: Consistent proxy support across componen...

[jvwing]

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2704#discussion_r189164826
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestListS3.java
 ---
@@ -311,5 +312,10 @@ public void testGetPropertyDescriptors() throws 
Exception {
 assertTrue(pd.contains(ListS3.PREFIX));
 assertTrue(pd.contains(ListS3.USE_VERSIONS));
 assertTrue(pd.contains(ListS3.MIN_AGE));
+
assertTrue(pd.contains(ProxyConfigurationService.PROXY_CONFIGURATION_SERVICE));
+assertTrue(pd.contains(ListS3.PROXY_HOST));
--- End diff --

Minor tweak: The check for PROXY_HOST and PROXY_HOST_PORT duplicates checks 
above on lines 309-310.  I believe this is why we add 5 lines of new 
assertions, but the count of property descriptors only goes up by 3 from 17 to 
20.  It doesn't make any difference, really, but the math was bothering me.


---

[GitHub] nifi pull request #2704: NIFI-4199: Consistent proxy support across componen...

[jvwing]

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2704#discussion_r189164994
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
 ---
@@ -92,6 +94,23 @@
 .addValidator(StandardValidators.PORT_VALIDATOR)
 .build();
 
+public static final PropertyDescriptor PROXY_USERNAME = new 
PropertyDescriptor.Builder()
+.name("Proxy Username")
+.description("Proxy username")
+.expressionLanguageSupported(true)
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
+.build();
+
+public static final PropertyDescriptor PROXY_PASSWORD = new 
PropertyDescriptor.Builder()
+.name("Proxy Password")
--- End diff --

I recommend a separate `name` vs `displayName` for PROXY_PASSWORD.


---

[GitHub] nifi pull request #2704: NIFI-4199: Consistent proxy support across componen...

[jvwing]

Github user jvwing commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2704#discussion_r189164964
  
--- Diff: 
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/AbstractAWSProcessor.java
 ---
@@ -92,6 +94,23 @@
 .addValidator(StandardValidators.PORT_VALIDATOR)
 .build();
 
+public static final PropertyDescriptor PROXY_USERNAME = new 
PropertyDescriptor.Builder()
+.name("Proxy Username")
--- End diff --

I recommend a separate `name` vs `displayName` for PROXY_USERNAME.


---

[jira] [Updated] (NIFI-5171) YandexTranslate Processor Improvement

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-5171:
---
Affects Version/s: (was: 1.6.0)
   Status: Patch Available  (was: Open)

> YandexTranslate Processor Improvement
> -
>
> Key: NIFI-5171
> URL: https://issues.apache.org/jira/browse/NIFI-5171
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Henry Sowell
>Priority: Minor
>
> I'm currently working to improve the YandexTranslate processor to 
> automatically detect the input language. Here are the core items to change:
>  * Default input language blank
>  * Receive back detected language from API
> If default language set, current behavior would remain as-is. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2715: Nifi-5171 - fixed Yandex Jersey issues by adding de...

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2715#discussion_r189161793
  
--- Diff: 
nifi-nar-bundles/nifi-language-translation-bundle/nifi-yandex-processors/pom.xml
 ---
@@ -53,7 +53,12 @@
 jersey-media-json-jackson
 ${jersey.version}
 
-
+
+org.glassfish.jersey.inject
+jersey-hk2
--- End diff --

I think this is licensed under CDDL 1.1? If so it should be included in the 
NAR's NOTICE file along with the other Jersey stuff.


---

[jira] [Updated] (NIFI-5213) Allow AvroReader with explicit schema to read files with embedded schema

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-5213:
---
Status: Patch Available  (was: In Progress)

> Allow AvroReader with explicit schema to read files with embedded schema
> 
>
> Key: NIFI-5213
> URL: https://issues.apache.org/jira/browse/NIFI-5213
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Minor
>
> AvroReader allows the choice of schema access strategy from such options as 
> Use Embedded Schema, Use Schema Name, Use Schema Text, etc. If the incoming 
> Avro files will have embedded schemas, then Use Embedded Schema is best 
> practice for the Avro Reader. However it is not intuitive that if the same 
> schema that is embedded in the file is specified by name (using a schema 
> registry) or explicitly via Schema Text, that errors can occur. This has been 
> noticed in QueryRecord for example, and the error is also not intuitive or 
> descriptive (it is often an ArrayIndexOutOfBoundsException).
> To provide a better user experience, it would be an improvement for 
> AvroReader to be able to successfully process Avro files with embedded 
> schemas, even when the Schema Access Strategy is not "Use Embedded Schema". 
> Of course, the explicit schema would have to match the embedded schema, or an 
> error would be reported (and rightfully so).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5213) Allow AvroReader with explicit schema to read files with embedded schema

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480125#comment-16480125
 ] 

ASF GitHub Bot commented on NIFI-5213:
--

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/2718

NIFI-5213: Allow AvroReader to process files w embedded schema even when 
the access strategy is explicit schema

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-5213

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2718.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2718


commit 40b9e110abadf802780e2d697aeab336b50094b9
Author: Matthew Burgess 
Date:   2018-05-18T04:29:52Z

NIFI-5213: Allow AvroReader to process files w embedded schema even when 
the access strategy is explicit schema




> Allow AvroReader with explicit schema to read files with embedded schema
> 
>
> Key: NIFI-5213
> URL: https://issues.apache.org/jira/browse/NIFI-5213
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Minor
>
> AvroReader allows the choice of schema access strategy from such options as 
> Use Embedded Schema, Use Schema Name, Use Schema Text, etc. If the incoming 
> Avro files will have embedded schemas, then Use Embedded Schema is best 
> practice for the Avro Reader. However it is not intuitive that if the same 
> schema that is embedded in the file is specified by name (using a schema 
> registry) or explicitly via Schema Text, that errors can occur. This has been 
> noticed in QueryRecord for example, and the error is also not intuitive or 
> descriptive (it is often an ArrayIndexOutOfBoundsException).
> To provide a better user experience, it would be an improvement for 
> AvroReader to be able to successfully process Avro files with embedded 
> schemas, even when the Schema Access Strategy is not "Use Embedded Schema". 
> Of course, the explicit schema would have to match the embedded schema, or an 
> error would be reported (and rightfully so).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2718: NIFI-5213: Allow AvroReader to process files w embe...

[mattyb149]

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/2718

NIFI-5213: Allow AvroReader to process files w embedded schema even when 
the access strategy is explicit schema

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-5213

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2718.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2718


commit 40b9e110abadf802780e2d697aeab336b50094b9
Author: Matthew Burgess 
Date:   2018-05-18T04:29:52Z

NIFI-5213: Allow AvroReader to process files w embedded schema even when 
the access strategy is explicit schema




---

[jira] [Assigned] (NIFI-5213) Allow AvroReader with explicit schema to read files with embedded schema

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess reassigned NIFI-5213:
--

Assignee: Matt Burgess

> Allow AvroReader with explicit schema to read files with embedded schema
> 
>
> Key: NIFI-5213
> URL: https://issues.apache.org/jira/browse/NIFI-5213
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Minor
>
> AvroReader allows the choice of schema access strategy from such options as 
> Use Embedded Schema, Use Schema Name, Use Schema Text, etc. If the incoming 
> Avro files will have embedded schemas, then Use Embedded Schema is best 
> practice for the Avro Reader. However it is not intuitive that if the same 
> schema that is embedded in the file is specified by name (using a schema 
> registry) or explicitly via Schema Text, that errors can occur. This has been 
> noticed in QueryRecord for example, and the error is also not intuitive or 
> descriptive (it is often an ArrayIndexOutOfBoundsException).
> To provide a better user experience, it would be an improvement for 
> AvroReader to be able to successfully process Avro files with embedded 
> schemas, even when the Schema Access Strategy is not "Use Embedded Schema". 
> Of course, the explicit schema would have to match the embedded schema, or an 
> error would be reported (and rightfully so).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5213) Allow AvroReader with explicit schema to read files with embedded schema

[Matt Burgess (JIRA)]

Matt Burgess created NIFI-5213:
--

 Summary: Allow AvroReader with explicit schema to read files with 
embedded schema
 Key: NIFI-5213
 URL: https://issues.apache.org/jira/browse/NIFI-5213
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Reporter: Matt Burgess


AvroReader allows the choice of schema access strategy from such options as Use 
Embedded Schema, Use Schema Name, Use Schema Text, etc. If the incoming Avro 
files will have embedded schemas, then Use Embedded Schema is best practice for 
the Avro Reader. However it is not intuitive that if the same schema that is 
embedded in the file is specified by name (using a schema registry) or 
explicitly via Schema Text, that errors can occur. This has been noticed in 
QueryRecord for example, and the error is also not intuitive or descriptive (it 
is often an ArrayIndexOutOfBoundsException).

To provide a better user experience, it would be an improvement for AvroReader 
to be able to successfully process Avro files with embedded schemas, even when 
the Schema Access Strategy is not "Use Embedded Schema". Of course, the 
explicit schema would have to match the embedded schema, or an error would be 
reported (and rightfully so).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479985#comment-16479985
 ] 

ASF GitHub Bot commented on NIFI-5145:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2717


> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.7.0
>
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-5145:
---
Status: Patch Available  (was: Reopened)

> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.7.0
>
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2717: NIFI-5145: Fixed MockPropertyValue to call correct ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2717


---

[jira] [Updated] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-5145:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.7.0
>
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479984#comment-16479984
 ] 

ASF subversion and git services commented on NIFI-5145:
---

Commit 29e96ed0e0b63bc2f5b0f07751fe7efbc10f18ab in nifi's branch 
refs/heads/master from [~ca9mbu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=29e96ed ]

NIFI-5145: Fixed MockPropertyValue to call the correct 
evaluateAttributeExpressions when FF is null

Signed-off-by: Matthew Burgess 

This closes #2717


> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.7.0
>
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479983#comment-16479983
 ] 

ASF GitHub Bot commented on NIFI-5145:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2717
  
Self-merging since Travis passed and the change is to the test framework.


> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.7.0
>
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2717: NIFI-5145: Fixed MockPropertyValue to call correct evaluat...

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2717
  
Self-merging since Travis passed and the change is to the test framework.


---

[GitHub] nifi pull request #2717: NIFI-5145: Fixed MockPropertyValue to call correct ...

[mattyb149]

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/2717

NIFI-5145: Fixed MockPropertyValue to call correct 
evaluateAttributeExpressions when FF is null

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-5145

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2717.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2717


commit 80377819678d373951e047ca5f6a5f591eb4efc2
Author: Matthew Burgess 
Date:   2018-05-18T00:48:36Z

NIFI-5145: Fixed MockPropertyValue to call the correct 
evaluateAttributeExpressions when FF is null




---

[jira] [Commented] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479956#comment-16479956
 ] 

ASF GitHub Bot commented on NIFI-5145:
--

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/2717

NIFI-5145: Fixed MockPropertyValue to call correct 
evaluateAttributeExpressions when FF is null

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-5145

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2717.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2717


commit 80377819678d373951e047ca5f6a5f591eb4efc2
Author: Matthew Burgess 
Date:   2018-05-18T00:48:36Z

NIFI-5145: Fixed MockPropertyValue to call the correct 
evaluateAttributeExpressions when FF is null




> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.7.0
>
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[Matt Burgess (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479945#comment-16479945
 ] 

Matt Burgess commented on NIFI-5145:


PR 2714 was not a complete fix, it caused some unit test failures. I am looking 
into it and hope to have a PR up soon.

> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.7.0
>
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479925#comment-16479925
 ] 

ASF GitHub Bot commented on NIFI-5145:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2714
  
I think I spoke too soon, there are unit test failures. I'll look into it



> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.7.0
>
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2714: NIFI-5145 Changed the override in MockPropertyValue to use...

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2714
  
I think I spoke too soon, there are unit test failures. I'll look into it



---

[jira] [Commented] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479916#comment-16479916
 ] 

ASF GitHub Bot commented on NIFI-5145:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2714


> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.7.0
>
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479914#comment-16479914
 ] 

ASF subversion and git services commented on NIFI-5145:
---

Commit 6bb43bd4716dda44738d4754518cb60467b62ea6 in nifi's branch 
refs/heads/master from [~mike.thomsen]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=6bb43bd ]

NIFI-5145 Changed the override in MockPropertyValue to use a better way of 
passing over null flowfiles in evaluateExpressionLanguage.

Signed-off-by: Matthew Burgess 

This closes #2714


> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.7.0
>
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2714: NIFI-5145 Changed the override in MockPropertyValue...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2714


---

[jira] [Commented] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479913#comment-16479913
 ] 

ASF GitHub Bot commented on NIFI-5145:
--

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2714
  
+1 LGTM, thanks! (and sorry for my confusion) Merging to master


> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.7.0
>
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2714: NIFI-5145 Changed the override in MockPropertyValue to use...

[mattyb149]

Github user mattyb149 commented on the issue:

https://github.com/apache/nifi/pull/2714
  
+1 LGTM, thanks! (and sorry for my confusion) Merging to master


---

[jira] [Updated] (NIFI-5211) Create JSON reader, writer, signer, and verifier

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-5211:

Fix Version/s: (was: 1.7.0)

> Create JSON reader, writer, signer, and verifier
> 
>
> Key: NIFI-5211
> URL: https://issues.apache.org/jira/browse/NIFI-5211
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Andy LoPresto
>Priority: Minor
>  Labels: security, tls
>
> This piece should accept a TLS configuration object and serialize/deserialize 
> it to/from JSON, and should provide a signature generation and verification 
> service to ensure it is trusted and has not been manipulated. 
> * JSON de/serialization
> * Read arbitrary JSON and verify an HMAC/SHA-512 signature before parsing
> * Generate a signature over JSON and persist in place
> * Derive the signature key from the master key in {{bootstrap.conf}} with a 
> one-way transformation (i.e. {{HMAC/SHA-512("JSON TLS key", MK) -> TLSK}})



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5212) Configure JettyServer with custom TLS protocols and cipher suites

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-5212:

Fix Version/s: (was: 1.7.0)

> Configure JettyServer with custom TLS protocols and cipher suites
> -
>
> Key: NIFI-5212
> URL: https://issues.apache.org/jira/browse/NIFI-5212
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Andy LoPresto
>Priority: Minor
>  Labels: jetty, security, tls
>
> This piece should read a TLS configuration and configure the {{JettyServer}} 
> and underlying {{SslContextFactory}} with the provided protocol versions and 
> cipher suites, performing basic validation (to ensure at least one valid 
> cipher suite and protocol, verifying against {{java.tls.disabledAlgorithms}}, 
> etc.). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5210) Create service to retrieve TLS configurations from remote endpoint

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-5210:

Fix Version/s: (was: 1.7.0)

> Create service to retrieve TLS configurations from remote endpoint
> --
>
> Key: NIFI-5210
> URL: https://issues.apache.org/jira/browse/NIFI-5210
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Extensions
>Affects Versions: 1.6.0
>Reporter: Andy LoPresto
>Priority: Minor
>  Labels: security, tls
>
> One component of this system will be to retrieve the configurations from a 
> remote service (if the admin opts in). This piece should:
> * communicate over HTTPS only
> ** have a custom truststore?
> * have a configurable URL
> ** have a secondary URL?
> * have a configurable polling interval



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5205) NiFi SysAdmin Guide max.appendable.size Default Value

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479622#comment-16479622
 ] 

ASF subversion and git services commented on NIFI-5205:
---

Commit 8aaa51af311b25a7479062de3e4c558769144363 in nifi's branch 
refs/heads/master from [~andrewmlim]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=8aaa51a ]

NIFI-5205 Updated default value for nifi.content.claim.max.appendable.size in 
Admin Guide

This closes #2716

Signed-off-by: Scott Aslan 


> NiFi SysAdmin Guide max.appendable.size Default Value
> -
>
> Key: NIFI-5205
> URL: https://issues.apache.org/jira/browse/NIFI-5205
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation  Website
>Affects Versions: 1.6.0
>Reporter: Mike Albanese
>Assignee: Andrew Lim
>Priority: Minor
> Fix For: 1.6.0
>
> Attachments: Screen Shot 2018-05-16 at 3.56.48 PM.png
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Documentation for nifi.content.claim.max.appendable.size does not match 
> default value of 1 MB in nifi.properties for 1.6.0.
> Keep in mind that I am making the assumption that NiFi defaults to this same 
> value of 1 MB when it starts up. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5205) NiFi SysAdmin Guide max.appendable.size Default Value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479624#comment-16479624
 ] 

ASF GitHub Bot commented on NIFI-5205:
--

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi/pull/2716
  
Thanks @andrewmlim this has been merged to master.


> NiFi SysAdmin Guide max.appendable.size Default Value
> -
>
> Key: NIFI-5205
> URL: https://issues.apache.org/jira/browse/NIFI-5205
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation  Website
>Affects Versions: 1.6.0
>Reporter: Mike Albanese
>Assignee: Andrew Lim
>Priority: Minor
> Fix For: 1.6.0
>
> Attachments: Screen Shot 2018-05-16 at 3.56.48 PM.png
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Documentation for nifi.content.claim.max.appendable.size does not match 
> default value of 1 MB in nifi.properties for 1.6.0.
> Keep in mind that I am making the assumption that NiFi defaults to this same 
> value of 1 MB when it starts up. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5205) NiFi SysAdmin Guide max.appendable.size Default Value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479623#comment-16479623
 ] 

ASF GitHub Bot commented on NIFI-5205:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2716


> NiFi SysAdmin Guide max.appendable.size Default Value
> -
>
> Key: NIFI-5205
> URL: https://issues.apache.org/jira/browse/NIFI-5205
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation  Website
>Affects Versions: 1.6.0
>Reporter: Mike Albanese
>Assignee: Andrew Lim
>Priority: Minor
> Fix For: 1.6.0
>
> Attachments: Screen Shot 2018-05-16 at 3.56.48 PM.png
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Documentation for nifi.content.claim.max.appendable.size does not match 
> default value of 1 MB in nifi.properties for 1.6.0.
> Keep in mind that I am making the assumption that NiFi defaults to this same 
> value of 1 MB when it starts up. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2716: NIFI-5205 Updated default value for nifi.content.claim.max...

[scottyaslan]

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi/pull/2716
  
Thanks @andrewmlim this has been merged to master.


---

[GitHub] nifi pull request #2716: NIFI-5205 Updated default value for nifi.content.cl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2716


---

[jira] [Reopened] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess reopened NIFI-5145:


Reopening due to minor change needed for mock framework

> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.7.0
>
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5044) SelectHiveQL accept only one statement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479592#comment-16479592
 ] 

ASF GitHub Bot commented on NIFI-5044:
--

Github user bdesert commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2695#discussion_r189075604
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -311,7 +340,15 @@ private void onTrigger(final ProcessContext context, 
final ProcessSession sessio
 try (final Connection con = 
dbcpService.getConnection(fileToProcess == null ? Collections.emptyMap() : 
fileToProcess.getAttributes());
  final Statement st = (flowbased ? 
con.prepareStatement(selectQuery) : con.createStatement())
 ) {
-
+Pair failure = 
executeConfigStatements(con, preQueries);
+if (failure != null) {
+// In case of failure, assigning config query to 
"selectQuery" var will allow to avoid major changes in error handling (catch 
block),
--- End diff --

@pvillard31 , I think if I just rename this var into "hqlStatement" - that 
will remove confusions and make reusing it for pre- and post- queries natural. 
Going to push updates for this.


> SelectHiveQL accept only one statement
> --
>
> Key: NIFI-5044
> URL: https://issues.apache.org/jira/browse/NIFI-5044
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0
>Reporter: Davide Isoardi
>Assignee: Ed Berezitsky
>Priority: Critical
>  Labels: features, patch, pull-request-available
> Attachments: 
> 0001-NIFI-5044-SelectHiveQL-accept-only-one-statement.patch
>
>
> In [this 
> |[https://github.com/apache/nifi/commit/bbc714e73ba245de7bc32fd9958667c847101f7d]
>  ] commit claims to add support to running multiple statements both on 
> SelectHiveQL and PutHiveQL; instead, it adds only the support to PutHiveQL, 
> so SelectHiveQL still lacks this important feature. @Matt Burgess, I saw that 
> you worked on that, is there any reason for this? If not, can we support it?
> If I try to execute this query:
> {quote}set hive.vectorized.execution.enabled = false; SELECT * FROM table_name
> {quote}
> I have this error:
>  
> {quote}2018-04-05 13:35:40,572 ERROR [Timer-Driven Process Thread-146] 
> o.a.nifi.processors.hive.SelectHiveQL 
> SelectHiveQL[id=243d4c17-b1fe-14af--ee8ce15e] Unable to execute 
> HiveQL select query set hive.vectorized.execution.enabled = false; SELECT * 
> FROM table_name for 
> StandardFlowFileRecord[uuid=0e035558-07ce-473b-b0d4-ac00b8b1df93,claim=StandardContentClaim
>  [resourceClaim=StandardResourceClaim[id=1522824912161-2753, 
> container=default, section=705], offset=838441, 
> length=25],offset=0,name=cliente_attributi.csv,size=25] due to 
> org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!; routing to failure: {}
>  org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL$2.process(SelectHiveQL.java:305)
>  at 
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2529)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:275)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$0(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> 

[GitHub] nifi pull request #2695: NIFI-5044 SelectHiveQL accept only one statement

[bdesert]

Github user bdesert commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2695#discussion_r189075604
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -311,7 +340,15 @@ private void onTrigger(final ProcessContext context, 
final ProcessSession sessio
 try (final Connection con = 
dbcpService.getConnection(fileToProcess == null ? Collections.emptyMap() : 
fileToProcess.getAttributes());
  final Statement st = (flowbased ? 
con.prepareStatement(selectQuery) : con.createStatement())
 ) {
-
+Pair failure = 
executeConfigStatements(con, preQueries);
+if (failure != null) {
+// In case of failure, assigning config query to 
"selectQuery" var will allow to avoid major changes in error handling (catch 
block),
--- End diff --

@pvillard31 , I think if I just rename this var into "hqlStatement" - that 
will remove confusions and make reusing it for pre- and post- queries natural. 
Going to push updates for this.


---

[jira] [Commented] (NIFI-4585) Document nifi.cluster.node.max.concurrent.requests property

[Mike Albanese (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479575#comment-16479575
 ] 

Mike Albanese commented on NIFI-4585:
-

Noticed this as well. Seems the documentation hasn't yet been updated for 1.6, 
seeing as there are other inconsistencies such as NIFI-5205. Definitely a 
bigger pain point here considering it isn't fully outlined how this affects 
load balancing among a cluster.

> Document nifi.cluster.node.max.concurrent.requests property
> ---
>
> Key: NIFI-4585
> URL: https://issues.apache.org/jira/browse/NIFI-4585
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation  Website
>Reporter: Pierre Villard
>Priority: Minor
>
> The property:
> {noformat}
> nifi.cluster.node.max.concurrent.requests
> {noformat}
> needs to be documented in:
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#cluster-node-properties



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5205) NiFi SysAdmin Guide max.appendable.size Default Value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479570#comment-16479570
 ] 

ASF GitHub Bot commented on NIFI-5205:
--

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi/pull/2716
  
Reviewing...


> NiFi SysAdmin Guide max.appendable.size Default Value
> -
>
> Key: NIFI-5205
> URL: https://issues.apache.org/jira/browse/NIFI-5205
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation  Website
>Affects Versions: 1.6.0
>Reporter: Mike Albanese
>Assignee: Andrew Lim
>Priority: Minor
> Fix For: 1.6.0
>
> Attachments: Screen Shot 2018-05-16 at 3.56.48 PM.png
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Documentation for nifi.content.claim.max.appendable.size does not match 
> default value of 1 MB in nifi.properties for 1.6.0.
> Keep in mind that I am making the assumption that NiFi defaults to this same 
> value of 1 MB when it starts up. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2716: NIFI-5205 Updated default value for nifi.content.claim.max...

[scottyaslan]

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi/pull/2716
  
Reviewing...


---

[jira] [Commented] (NIFI-5205) NiFi SysAdmin Guide max.appendable.size Default Value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479567#comment-16479567
 ] 

ASF GitHub Bot commented on NIFI-5205:
--

GitHub user andrewmlim opened a pull request:

https://github.com/apache/nifi/pull/2716

NIFI-5205 Updated default value for nifi.content.claim.max.appendable…

….size in Admin Guide

Also fixed the formatting of some default values while I was in there.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewmlim/nifi NIFI-5205

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2716.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2716


commit 5fafedc07742af5cd71afca073b0062d3202c4cf
Author: Andrew Lim 
Date:   2018-05-17T19:00:04Z

NIFI-5205 Updated default value for nifi.content.claim.max.appendable.size 
in Admin Guide




> NiFi SysAdmin Guide max.appendable.size Default Value
> -
>
> Key: NIFI-5205
> URL: https://issues.apache.org/jira/browse/NIFI-5205
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation  Website
>Affects Versions: 1.6.0
>Reporter: Mike Albanese
>Assignee: Andrew Lim
>Priority: Minor
> Fix For: 1.6.0
>
> Attachments: Screen Shot 2018-05-16 at 3.56.48 PM.png
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Documentation for nifi.content.claim.max.appendable.size does not match 
> default value of 1 MB in nifi.properties for 1.6.0.
> Keep in mind that I am making the assumption that NiFi defaults to this same 
> value of 1 MB when it starts up. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2716: NIFI-5205 Updated default value for nifi.content.cl...

[andrewmlim]

GitHub user andrewmlim opened a pull request:

https://github.com/apache/nifi/pull/2716

NIFI-5205 Updated default value for nifi.content.claim.max.appendable…

….size in Admin Guide

Also fixed the formatting of some default values while I was in there.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewmlim/nifi NIFI-5205

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2716.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2716


commit 5fafedc07742af5cd71afca073b0062d3202c4cf
Author: Andrew Lim 
Date:   2018-05-17T19:00:04Z

NIFI-5205 Updated default value for nifi.content.claim.max.appendable.size 
in Admin Guide




---

[jira] [Commented] (NIFI-5205) NiFi SysAdmin Guide max.appendable.size Default Value

[Mike Albanese (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479564#comment-16479564
 ] 

Mike Albanese commented on NIFI-5205:
-

Thanks for the updates Andrew. Yeah was also going to mention NIFI-4585 as an 
issue but good to see that someone else opened up a story for it.

> NiFi SysAdmin Guide max.appendable.size Default Value
> -
>
> Key: NIFI-5205
> URL: https://issues.apache.org/jira/browse/NIFI-5205
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation  Website
>Affects Versions: 1.6.0
>Reporter: Mike Albanese
>Assignee: Andrew Lim
>Priority: Minor
> Fix For: 1.6.0
>
> Attachments: Screen Shot 2018-05-16 at 3.56.48 PM.png
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Documentation for nifi.content.claim.max.appendable.size does not match 
> default value of 1 MB in nifi.properties for 1.6.0.
> Keep in mind that I am making the assumption that NiFi defaults to this same 
> value of 1 MB when it starts up. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5205) NiFi SysAdmin Guide max.appendable.size Default Value

[Andrew Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479561#comment-16479561
 ] 

Andrew Lim commented on NIFI-5205:
--

Yes, I will make changes to the Admin Guide so the default value there is 1 MB 
also.

I did a quick review of the default values in nifi.properties with what is 
stated in documentation and for the most part it looks consistent.  But yes, 
there are discrepancies.  Will file new Jiras as needed.

NIFI-4585 covers a missing property.  Will likely tackle that after I get a PR 
in for this one.

> NiFi SysAdmin Guide max.appendable.size Default Value
> -
>
> Key: NIFI-5205
> URL: https://issues.apache.org/jira/browse/NIFI-5205
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation  Website
>Affects Versions: 1.6.0
>Reporter: Mike Albanese
>Assignee: Andrew Lim
>Priority: Minor
> Fix For: 1.6.0
>
> Attachments: Screen Shot 2018-05-16 at 3.56.48 PM.png
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Documentation for nifi.content.claim.max.appendable.size does not match 
> default value of 1 MB in nifi.properties for 1.6.0.
> Keep in mind that I am making the assumption that NiFi defaults to this same 
> value of 1 MB when it starts up. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5212) Configure JettyServer with custom TLS protocols and cipher suites

[Andy LoPresto (JIRA)]

Andy LoPresto created NIFI-5212:
---

 Summary: Configure JettyServer with custom TLS protocols and 
cipher suites
 Key: NIFI-5212
 URL: https://issues.apache.org/jira/browse/NIFI-5212
 Project: Apache NiFi
  Issue Type: Sub-task
  Components: Core Framework
Affects Versions: 1.6.0
Reporter: Andy LoPresto
 Fix For: 1.7.0


This piece should read a TLS configuration and configure the {{JettyServer}} 
and underlying {{SslContextFactory}} with the provided protocol versions and 
cipher suites, performing basic validation (to ensure at least one valid cipher 
suite and protocol, verifying against {{java.tls.disabledAlgorithms}}, etc.). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5211) Create JSON reader, writer, signer, and verifier

[Andy LoPresto (JIRA)]

Andy LoPresto created NIFI-5211:
---

 Summary: Create JSON reader, writer, signer, and verifier
 Key: NIFI-5211
 URL: https://issues.apache.org/jira/browse/NIFI-5211
 Project: Apache NiFi
  Issue Type: Sub-task
  Components: Extensions
Affects Versions: 1.6.0
Reporter: Andy LoPresto
 Fix For: 1.7.0


This piece should accept a TLS configuration object and serialize/deserialize 
it to/from JSON, and should provide a signature generation and verification 
service to ensure it is trusted and has not been manipulated. 

* JSON de/serialization
* Read arbitrary JSON and verify an HMAC/SHA-512 signature before parsing
* Generate a signature over JSON and persist in place
* Derive the signature key from the master key in {{bootstrap.conf}} with a 
one-way transformation (i.e. {{HMAC/SHA-512("JSON TLS key", MK) -> TLSK}})



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5210) Create service to retrieve TLS configurations from remote endpoint

[Andy LoPresto (JIRA)]

Andy LoPresto created NIFI-5210:
---

 Summary: Create service to retrieve TLS configurations from remote 
endpoint
 Key: NIFI-5210
 URL: https://issues.apache.org/jira/browse/NIFI-5210
 Project: Apache NiFi
  Issue Type: Sub-task
  Components: Extensions
Affects Versions: 1.6.0
Reporter: Andy LoPresto
 Fix For: 1.7.0


One component of this system will be to retrieve the configurations from a 
remote service (if the admin opts in). This piece should:
* communicate over HTTPS only
** have a custom truststore?
* have a configurable URL
** have a secondary URL?
* have a configurable polling interval



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5205) NiFi SysAdmin Guide max.appendable.size Default Value

[Randy Bovay (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479471#comment-16479471
 ] 

Randy Bovay commented on NIFI-5205:
---

It appears this was 10MB before, so it's been decreased to 1MB.  Interesting.

I think the 'bug' is that the website is not up to date with the 1.6 
properties.  I don't believe that's restricted to just this 1 property either.  

> NiFi SysAdmin Guide max.appendable.size Default Value
> -
>
> Key: NIFI-5205
> URL: https://issues.apache.org/jira/browse/NIFI-5205
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation  Website
>Affects Versions: 1.6.0
>Reporter: Michael
>Assignee: Andrew Lim
>Priority: Minor
> Fix For: 1.6.0
>
> Attachments: Screen Shot 2018-05-16 at 3.56.48 PM.png
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Documentation for nifi.content.claim.max.appendable.size does not match 
> default value of 1 MB in nifi.properties for 1.6.0.
> Keep in mind that I am making the assumption that NiFi defaults to this same 
> value of 1 MB when it starts up. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5204) When node joins cluster, if a processor is stopping but cluster says the state is disabled, node ends up in inconsistent state

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-5204:
--
   Resolution: Fixed
Fix Version/s: 1.7.0
   Status: Resolved  (was: Patch Available)

> When node joins cluster, if a processor is stopping but cluster says the 
> state is disabled, node ends up in inconsistent state
> --
>
> Key: NIFI-5204
> URL: https://issues.apache.org/jira/browse/NIFI-5204
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.7.0
>
>
> In order to make this "easy" to replicate, I did the following:
> 1) Create a 2-node cluster.
>  2) On both nodes, update nifi.properties to set 
> "nifi.variable.registry.properties" to "1.properties"
>  3) On both nodes, create 1.properties in $NIFI_HOME. For first node, set 
> "sleep=2 mins" and for second node, set "sleep=0 millis"
>  4) Update DebugFlow to support expression language for the "@OnStopped Pause 
> Time"
>  5) Configure flow with a DebugFlow processor. Can auto-terminate 
> relationships and set run period to "10 secs."Set "@OnStopped Pause time" to 
> "${sleep}"
>  6) Disable DebugFlow processor.
>  7) Disconnected Node 1.
>  8) Go to Node 1 in browser and Start DebugFlow.
>  9) Stop DebugFlow.
>  10) While processor is still "stopping", go back Node 2 in browser and 
> request that Node 1 re-join the cluster.
> Now, when Node 1 re-joins the cluster, it will attempt to disable the 
> processor but won't be able to because the processor is still stopping. The 
> following will be in the logs:
> {code:java}
> 2018-05-16 15:21:50,986 WARN [Reconnect to Cluster] 
> org.apache.nifi.controller.ProcessorNode Processor cannot be disabled because 
> its state is set to STOPPING{code}
> So we now have a node in an inconsistent state.
> Additionally, if we now go to Node 1 in our browser and unselect all 
> components, and attempt to STOP the process group, the request that is 
> replicated attempts to stop the DebugFlow processor. Node 2 will now fail to 
> stop the processor because the processor is disabled. As a result, Node 2 
> will now be kicked out of the cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5204) When node joins cluster, if a processor is stopping but cluster says the state is disabled, node ends up in inconsistent state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479447#comment-16479447
 ] 

ASF GitHub Bot commented on NIFI-5204:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2713


> When node joins cluster, if a processor is stopping but cluster says the 
> state is disabled, node ends up in inconsistent state
> --
>
> Key: NIFI-5204
> URL: https://issues.apache.org/jira/browse/NIFI-5204
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.7.0
>
>
> In order to make this "easy" to replicate, I did the following:
> 1) Create a 2-node cluster.
>  2) On both nodes, update nifi.properties to set 
> "nifi.variable.registry.properties" to "1.properties"
>  3) On both nodes, create 1.properties in $NIFI_HOME. For first node, set 
> "sleep=2 mins" and for second node, set "sleep=0 millis"
>  4) Update DebugFlow to support expression language for the "@OnStopped Pause 
> Time"
>  5) Configure flow with a DebugFlow processor. Can auto-terminate 
> relationships and set run period to "10 secs."Set "@OnStopped Pause time" to 
> "${sleep}"
>  6) Disable DebugFlow processor.
>  7) Disconnected Node 1.
>  8) Go to Node 1 in browser and Start DebugFlow.
>  9) Stop DebugFlow.
>  10) While processor is still "stopping", go back Node 2 in browser and 
> request that Node 1 re-join the cluster.
> Now, when Node 1 re-joins the cluster, it will attempt to disable the 
> processor but won't be able to because the processor is still stopping. The 
> following will be in the logs:
> {code:java}
> 2018-05-16 15:21:50,986 WARN [Reconnect to Cluster] 
> org.apache.nifi.controller.ProcessorNode Processor cannot be disabled because 
> its state is set to STOPPING{code}
> So we now have a node in an inconsistent state.
> Additionally, if we now go to Node 1 in our browser and unselect all 
> components, and attempt to STOP the process group, the request that is 
> replicated attempts to stop the DebugFlow processor. Node 2 will now fail to 
> stop the processor because the processor is disabled. As a result, Node 2 
> will now be kicked out of the cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2713: NIFI-5204: If processor joins cluster and inherits ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2713


---

[jira] [Commented] (NIFI-5204) When node joins cluster, if a processor is stopping but cluster says the state is disabled, node ends up in inconsistent state

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479443#comment-16479443
 ] 

ASF subversion and git services commented on NIFI-5204:
---

Commit 2afbf963812bccdec8c997e6c63a93c644016769 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=2afbf96 ]

NIFI-5204: Ensure that verifyCanStop throws ISE if component is disabled
NIFI-5204: If processor joins cluster and inherits 'disabled' state but is 
still stopping, ensure that the state becomes disabled when the processor 
finishes stopping
This closes #2713


> When node joins cluster, if a processor is stopping but cluster says the 
> state is disabled, node ends up in inconsistent state
> --
>
> Key: NIFI-5204
> URL: https://issues.apache.org/jira/browse/NIFI-5204
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
>
> In order to make this "easy" to replicate, I did the following:
> 1) Create a 2-node cluster.
>  2) On both nodes, update nifi.properties to set 
> "nifi.variable.registry.properties" to "1.properties"
>  3) On both nodes, create 1.properties in $NIFI_HOME. For first node, set 
> "sleep=2 mins" and for second node, set "sleep=0 millis"
>  4) Update DebugFlow to support expression language for the "@OnStopped Pause 
> Time"
>  5) Configure flow with a DebugFlow processor. Can auto-terminate 
> relationships and set run period to "10 secs."Set "@OnStopped Pause time" to 
> "${sleep}"
>  6) Disable DebugFlow processor.
>  7) Disconnected Node 1.
>  8) Go to Node 1 in browser and Start DebugFlow.
>  9) Stop DebugFlow.
>  10) While processor is still "stopping", go back Node 2 in browser and 
> request that Node 1 re-join the cluster.
> Now, when Node 1 re-joins the cluster, it will attempt to disable the 
> processor but won't be able to because the processor is still stopping. The 
> following will be in the logs:
> {code:java}
> 2018-05-16 15:21:50,986 WARN [Reconnect to Cluster] 
> org.apache.nifi.controller.ProcessorNode Processor cannot be disabled because 
> its state is set to STOPPING{code}
> So we now have a node in an inconsistent state.
> Additionally, if we now go to Node 1 in our browser and unselect all 
> components, and attempt to STOP the process group, the request that is 
> replicated attempts to stop the DebugFlow processor. Node 2 will now fail to 
> stop the processor because the processor is disabled. As a result, Node 2 
> will now be kicked out of the cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5204) When node joins cluster, if a processor is stopping but cluster says the state is disabled, node ends up in inconsistent state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479446#comment-16479446
 ] 

ASF GitHub Bot commented on NIFI-5204:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2713
  
Thanks @markap14! This has been merged to master.


> When node joins cluster, if a processor is stopping but cluster says the 
> state is disabled, node ends up in inconsistent state
> --
>
> Key: NIFI-5204
> URL: https://issues.apache.org/jira/browse/NIFI-5204
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
>
> In order to make this "easy" to replicate, I did the following:
> 1) Create a 2-node cluster.
>  2) On both nodes, update nifi.properties to set 
> "nifi.variable.registry.properties" to "1.properties"
>  3) On both nodes, create 1.properties in $NIFI_HOME. For first node, set 
> "sleep=2 mins" and for second node, set "sleep=0 millis"
>  4) Update DebugFlow to support expression language for the "@OnStopped Pause 
> Time"
>  5) Configure flow with a DebugFlow processor. Can auto-terminate 
> relationships and set run period to "10 secs."Set "@OnStopped Pause time" to 
> "${sleep}"
>  6) Disable DebugFlow processor.
>  7) Disconnected Node 1.
>  8) Go to Node 1 in browser and Start DebugFlow.
>  9) Stop DebugFlow.
>  10) While processor is still "stopping", go back Node 2 in browser and 
> request that Node 1 re-join the cluster.
> Now, when Node 1 re-joins the cluster, it will attempt to disable the 
> processor but won't be able to because the processor is still stopping. The 
> following will be in the logs:
> {code:java}
> 2018-05-16 15:21:50,986 WARN [Reconnect to Cluster] 
> org.apache.nifi.controller.ProcessorNode Processor cannot be disabled because 
> its state is set to STOPPING{code}
> So we now have a node in an inconsistent state.
> Additionally, if we now go to Node 1 in our browser and unselect all 
> components, and attempt to STOP the process group, the request that is 
> replicated attempts to stop the DebugFlow processor. Node 2 will now fail to 
> stop the processor because the processor is disabled. As a result, Node 2 
> will now be kicked out of the cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5204) When node joins cluster, if a processor is stopping but cluster says the state is disabled, node ends up in inconsistent state

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479442#comment-16479442
 ] 

ASF subversion and git services commented on NIFI-5204:
---

Commit 2afbf963812bccdec8c997e6c63a93c644016769 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=2afbf96 ]

NIFI-5204: Ensure that verifyCanStop throws ISE if component is disabled
NIFI-5204: If processor joins cluster and inherits 'disabled' state but is 
still stopping, ensure that the state becomes disabled when the processor 
finishes stopping
This closes #2713


> When node joins cluster, if a processor is stopping but cluster says the 
> state is disabled, node ends up in inconsistent state
> --
>
> Key: NIFI-5204
> URL: https://issues.apache.org/jira/browse/NIFI-5204
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
>
> In order to make this "easy" to replicate, I did the following:
> 1) Create a 2-node cluster.
>  2) On both nodes, update nifi.properties to set 
> "nifi.variable.registry.properties" to "1.properties"
>  3) On both nodes, create 1.properties in $NIFI_HOME. For first node, set 
> "sleep=2 mins" and for second node, set "sleep=0 millis"
>  4) Update DebugFlow to support expression language for the "@OnStopped Pause 
> Time"
>  5) Configure flow with a DebugFlow processor. Can auto-terminate 
> relationships and set run period to "10 secs."Set "@OnStopped Pause time" to 
> "${sleep}"
>  6) Disable DebugFlow processor.
>  7) Disconnected Node 1.
>  8) Go to Node 1 in browser and Start DebugFlow.
>  9) Stop DebugFlow.
>  10) While processor is still "stopping", go back Node 2 in browser and 
> request that Node 1 re-join the cluster.
> Now, when Node 1 re-joins the cluster, it will attempt to disable the 
> processor but won't be able to because the processor is still stopping. The 
> following will be in the logs:
> {code:java}
> 2018-05-16 15:21:50,986 WARN [Reconnect to Cluster] 
> org.apache.nifi.controller.ProcessorNode Processor cannot be disabled because 
> its state is set to STOPPING{code}
> So we now have a node in an inconsistent state.
> Additionally, if we now go to Node 1 in our browser and unselect all 
> components, and attempt to STOP the process group, the request that is 
> replicated attempts to stop the DebugFlow processor. Node 2 will now fail to 
> stop the processor because the processor is disabled. As a result, Node 2 
> will now be kicked out of the cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2713: NIFI-5204: If processor joins cluster and inherits 'disabl...

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2713
  
Thanks @markap14! This has been merged to master.


---

[jira] [Comment Edited] (NIFI-5171) YandexTranslate Processor Improvement

[Henry Sowell (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479406#comment-16479406
 ] 

Henry Sowell edited comment on NIFI-5171 at 5/17/18 5:20 PM:
-

I created a pull request that adds this functionality.

https://github.com/apache/nifi/pull/2715


was (Author: veteranbv):
I created a pull request that adds this functionality.

> YandexTranslate Processor Improvement
> -
>
> Key: NIFI-5171
> URL: https://issues.apache.org/jira/browse/NIFI-5171
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Henry Sowell
>Priority: Minor
>
> I'm currently working to improve the YandexTranslate processor to 
> automatically detect the input language. Here are the core items to change:
>  * Default input language blank
>  * Receive back detected language from API
> If default language set, current behavior would remain as-is. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5205) NiFi SysAdmin Guide max.appendable.size Default Value

[Andrew Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479415#comment-16479415
 ] 

Andrew Lim commented on NIFI-5205:
--

Built off of master and confirmed the default value for this property:

nifi.content.claim.max.appendable.size=1 MB

> NiFi SysAdmin Guide max.appendable.size Default Value
> -
>
> Key: NIFI-5205
> URL: https://issues.apache.org/jira/browse/NIFI-5205
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation  Website
>Affects Versions: 1.6.0
>Reporter: Michael
>Assignee: Andrew Lim
>Priority: Minor
> Fix For: 1.6.0
>
> Attachments: Screen Shot 2018-05-16 at 3.56.48 PM.png
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Documentation for nifi.content.claim.max.appendable.size does not match 
> default value of 1 MB in nifi.properties for 1.6.0.
> Keep in mind that I am making the assumption that NiFi defaults to this same 
> value of 1 MB when it starts up. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5171) YandexTranslate Processor Improvement

[Henry Sowell (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479406#comment-16479406
 ] 

Henry Sowell commented on NIFI-5171:


I created a pull request that adds this functionality.

> YandexTranslate Processor Improvement
> -
>
> Key: NIFI-5171
> URL: https://issues.apache.org/jira/browse/NIFI-5171
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.6.0
>Reporter: Henry Sowell
>Priority: Minor
>
> I'm currently working to improve the YandexTranslate processor to 
> automatically detect the input language. Here are the core items to change:
>  * Default input language blank
>  * Receive back detected language from API
> If default language set, current behavior would remain as-is. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-4369) ConfigEcryptionTool should stream flow contents

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-4369:

Labels: memory performance security toolkit  (was: )

> ConfigEcryptionTool should stream flow contents
> ---
>
> Key: NIFI-4369
> URL: https://issues.apache.org/jira/browse/NIFI-4369
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Affects Versions: 1.3.0
>Reporter: Jeff Storck
>Priority: Major
>  Labels: memory, performance, security, toolkit
>
> ConfigEncryptionTool loads the entire flow xml into memory before operating 
> on it.  Rather, it should stream the flow contents to avoid requiring large 
> amounts of memory to process.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (NIFI-5100) Toolkit encrypt-config.sh should expose an option to specify where to store secure_hash.key

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto resolved NIFI-5100.
-
   Resolution: Not A Problem
 Assignee: Andy LoPresto
Fix Version/s: 1.7.0

> Toolkit encrypt-config.sh should expose an option to specify where to store 
> secure_hash.key
> ---
>
> Key: NIFI-5100
> URL: https://issues.apache.org/jira/browse/NIFI-5100
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Reporter: Koji Kawamura
>Assignee: Andy LoPresto
>Priority: Major
> Fix For: 1.7.0
>
>
> After NIFI-4942 is merged, master branch has been failing due to following 
> RAT check error:
> {code:java}
> [INFO] --- apache-rat-plugin:0.12:check (default) @ 
> nifi-toolkit-encrypt-config ---
> [INFO] Enabled default license matchers.
> [INFO] Will parse SCM ignores for exclusions...
> [INFO] Finished adding exclusions from SCM ignore files.
> [INFO] 61 implicit excludes (use -debug for more details).
> [INFO] Exclude: nb-configuration.xml
> [INFO] Exclude: nbactions.xml
> [INFO] Exclude: DEPENDENCIES
> [INFO] Exclude: .github/PULL_REQUEST_TEMPLATE.md
> [INFO] Exclude: src/test/resources/scrypt.py
> [INFO] Exclude: src/test/resources/secure_hash.key
> [INFO] Exclude: src/test/resources/secure_hash_128.key
> [INFO] 92 resources included (use -debug for more details)
> [INFO] Rat check: Summary over all files. Unapproved: 1, unknown: 1, 
> generated: 0, approved: 88 licenses.
> {code}
> The project RAT check passes successfully if I run the contrib-check locally. 
> However, there's a difference in the output. Here is what I get when I run it 
> locally:
> {code:java}
> [INFO] 91 resources included (use -debug for more details)
> [INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, 
> generated: 0, approved: 88 licenses.
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> {code}
> There is one more resources being checked when the test runs on Travis. I did 
> the test on Ubuntu then confirmed following file is created, and it produces 
> the RAT check error.
> {code:java}
> nifi-toolkit/nifi-toolkit-encrypt-config/secure_hash.key
> {code}
> Debugged further, I found following test creates the file:
> {code:java}
> mvn --projects nifi-toolkit/nifi-toolkit-encrypt-config -Pcontrib-check 
> -Dtest=org.apache.nifi.toolkit.encryptconfig.EncryptConfigMainTest#testShouldPerformFullOperationForNiFiPropertiesAndLoginIdentityProvidersAndAuthorizers
>  test
> {code}
> The test class should NOT create such file.
> Actually the file is created by ConfigEncryptionTool.groovy. And the 
> destination of the file can not be changed from current directory.
> https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy#L71
> The 'secure_hash.key' is always created at current directory. The tool should 
> expose where to write the file.
> {code}
>  ./bin/encrypt-config.sh -n /tmp/enc-test/nifi.properties -o 
> /tmp/enc-test/nifi-enc.properties -b /tmp/enc-test/bootstrap.conf --verbose
> $ ll
> total 60
> drwxrwxr-x 6 nifi nifi  4096 Apr 20 02:42 ./
> drwxrwxr-x 3 nifi nifi  4096 Apr 19 03:57 ../
> drwxr-xr-x 2 nifi nifi  4096 Apr 19 01:45 bin/
> drwxr-xr-x 3 nifi nifi  4096 Apr 19 01:45 classpath/
> drwxr-xr-x 2 nifi nifi  4096 Apr 19 01:45 conf/
> drwxrwxr-x 2 nifi nifi 12288 Apr 19 03:57 lib/
> -rw-r--r-- 1 nifi nifi 15986 Apr 19 01:45 LICENSE
> -rw-r--r-- 1 nifi nifi  5473 Apr 19 01:45 NOTICE
> -rw--- 1 nifi nifi91 Apr 20 02:42 secure_hash.key
>  {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5209) Remove toolkit migration without password functionality

[Andy LoPresto (JIRA)]

Andy LoPresto created NIFI-5209:
---

 Summary: Remove toolkit migration without password functionality
 Key: NIFI-5209
 URL: https://issues.apache.org/jira/browse/NIFI-5209
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Tools and Build
Affects Versions: 1.7.0
Reporter: Andy LoPresto
Assignee: Andy LoPresto


In NIFI-4942, new functionality was added to allow Ambari clients to perform 
the encrypted configuration migration without providing the original password 
or key by using a secure hash of the original credential to demonstrate 
knowledge of that value. The Ambari team found another way on their end to 
perform this action, and rather than allow the {{./secure_hash.key}} behavior 
to be released and then removed at a later time, complicating our security 
posture and potentially creating difficult support cases, it is better to 
remove it completely before the 1.7.0 release. 

However, it is not as simple as just backing out a few commits, as necessary 
refactoring of the tool code also occurred at that time. I will remove this 
feature while maintaining the improvements made to the toolkit. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5100) Toolkit encrypt-config.sh should expose an option to specify where to store secure_hash.key

[Andy LoPresto (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479367#comment-16479367
 ] 

Andy LoPresto commented on NIFI-5100:
-

[~ijokarumawak] I am planning to remove the {{./secure_hash.key}} functionality 
because Ambari was able to find a different way to achieve the objectives in 
NIFI-4942, but the reason that file location was hard-coded was because if a 
user could change the location, they could pass in a self-generated file 
containing a hashed value they controlled, and "authenticate" by passing the 
same value in the command-line invocation, thus bypassing the actual check for 
knowledge of the original password/key. I am closing this ticket but I will 
link it to the ticket to remove that functionality. Please let me know if you 
have any concerns. 

> Toolkit encrypt-config.sh should expose an option to specify where to store 
> secure_hash.key
> ---
>
> Key: NIFI-5100
> URL: https://issues.apache.org/jira/browse/NIFI-5100
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Tools and Build
>Reporter: Koji Kawamura
>Priority: Major
>
> After NIFI-4942 is merged, master branch has been failing due to following 
> RAT check error:
> {code:java}
> [INFO] --- apache-rat-plugin:0.12:check (default) @ 
> nifi-toolkit-encrypt-config ---
> [INFO] Enabled default license matchers.
> [INFO] Will parse SCM ignores for exclusions...
> [INFO] Finished adding exclusions from SCM ignore files.
> [INFO] 61 implicit excludes (use -debug for more details).
> [INFO] Exclude: nb-configuration.xml
> [INFO] Exclude: nbactions.xml
> [INFO] Exclude: DEPENDENCIES
> [INFO] Exclude: .github/PULL_REQUEST_TEMPLATE.md
> [INFO] Exclude: src/test/resources/scrypt.py
> [INFO] Exclude: src/test/resources/secure_hash.key
> [INFO] Exclude: src/test/resources/secure_hash_128.key
> [INFO] 92 resources included (use -debug for more details)
> [INFO] Rat check: Summary over all files. Unapproved: 1, unknown: 1, 
> generated: 0, approved: 88 licenses.
> {code}
> The project RAT check passes successfully if I run the contrib-check locally. 
> However, there's a difference in the output. Here is what I get when I run it 
> locally:
> {code:java}
> [INFO] 91 resources included (use -debug for more details)
> [INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, 
> generated: 0, approved: 88 licenses.
> [INFO] 
> 
> [INFO] BUILD SUCCESS
> [INFO] 
> 
> {code}
> There is one more resources being checked when the test runs on Travis. I did 
> the test on Ubuntu then confirmed following file is created, and it produces 
> the RAT check error.
> {code:java}
> nifi-toolkit/nifi-toolkit-encrypt-config/secure_hash.key
> {code}
> Debugged further, I found following test creates the file:
> {code:java}
> mvn --projects nifi-toolkit/nifi-toolkit-encrypt-config -Pcontrib-check 
> -Dtest=org.apache.nifi.toolkit.encryptconfig.EncryptConfigMainTest#testShouldPerformFullOperationForNiFiPropertiesAndLoginIdentityProvidersAndAuthorizers
>  test
> {code}
> The test class should NOT create such file.
> Actually the file is created by ConfigEncryptionTool.groovy. And the 
> destination of the file can not be changed from current directory.
> https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy#L71
> The 'secure_hash.key' is always created at current directory. The tool should 
> expose where to write the file.
> {code}
>  ./bin/encrypt-config.sh -n /tmp/enc-test/nifi.properties -o 
> /tmp/enc-test/nifi-enc.properties -b /tmp/enc-test/bootstrap.conf --verbose
> $ ll
> total 60
> drwxrwxr-x 6 nifi nifi  4096 Apr 20 02:42 ./
> drwxrwxr-x 3 nifi nifi  4096 Apr 19 03:57 ../
> drwxr-xr-x 2 nifi nifi  4096 Apr 19 01:45 bin/
> drwxr-xr-x 3 nifi nifi  4096 Apr 19 01:45 classpath/
> drwxr-xr-x 2 nifi nifi  4096 Apr 19 01:45 conf/
> drwxrwxr-x 2 nifi nifi 12288 Apr 19 03:57 lib/
> -rw-r--r-- 1 nifi nifi 15986 Apr 19 01:45 LICENSE
> -rw-r--r-- 1 nifi nifi  5473 Apr 19 01:45 NOTICE
> -rw--- 1 nifi nifi91 Apr 20 02:42 secure_hash.key
>  {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2715: Nifi-5171 - fixed Yandex Jersey issues by adding de...

[veteranbv]

GitHub user veteranbv reopened a pull request:

https://github.com/apache/nifi/pull/2715

Nifi-5171 - fixed Yandex Jersey issues by adding dependency to POM and 
modified A…

…PI call to now detect for languages

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [x] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [x] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/veteranbv/nifi nifi-5171

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2715.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2715


commit 3d797dcbfb96219ecc651eb505c6b3f49f9e9249
Author: veteranbv 
Date:   2018-05-17T16:47:19Z

fixed Yandex Jersey issues by adding dependency to POM and modified API 
call to now detect for languages




---

[GitHub] nifi pull request #2715: Nifi-5171 - fixed Yandex Jersey issues by adding de...

[veteranbv]

Github user veteranbv closed the pull request at:

https://github.com/apache/nifi/pull/2715


---

[GitHub] nifi pull request #2715: Nifi-5171 - fixed Yandex Jersey issues by adding de...

[veteranbv]

GitHub user veteranbv opened a pull request:

https://github.com/apache/nifi/pull/2715

Nifi-5171 - fixed Yandex Jersey issues by adding dependency to POM and 
modified A…

…PI call to now detect for languages

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/veteranbv/nifi nifi-5171

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2715.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2715


commit 3d797dcbfb96219ecc651eb505c6b3f49f9e9249
Author: veteranbv 
Date:   2018-05-17T16:47:19Z

fixed Yandex Jersey issues by adding dependency to POM and modified API 
call to now detect for languages




---

[jira] [Commented] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479360#comment-16479360
 ] 

ASF GitHub Bot commented on NIFI-5145:
--

GitHub user MikeThomsen opened a pull request:

https://github.com/apache/nifi/pull/2714

NIFI-5145 Changed the override in MockPropertyValue to use a better w…

…ay of passing over null flowfiles in evaluateExpressionLanguage.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MikeThomsen/nifi NIFI-5145

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2714.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2714


commit 35db02cd771be74779e50a7f419c28904bc671d8
Author: Mike Thomsen 
Date:   2018-05-17T16:47:53Z

NIFI-5145 Changed the override in MockPropertyValue to use a better way of 
passing over null flowfiles in evaluateExpressionLanguage.




> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.7.0
>
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2714: NIFI-5145 Changed the override in MockPropertyValue to use...

[MikeThomsen]

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2714
  
@mattyb149 Updated it per our email thread about this.


---

[jira] [Commented] (NIFI-5145) MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null inputs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479361#comment-16479361
 ] 

ASF GitHub Bot commented on NIFI-5145:
--

Github user MikeThomsen commented on the issue:

https://github.com/apache/nifi/pull/2714
  
@mattyb149 Updated it per our email thread about this.


> MockPropertyValue.evaluateExpressionLanguage(FlowFile) cannot handle null 
> inputs
> 
>
> Key: NIFI-5145
> URL: https://issues.apache.org/jira/browse/NIFI-5145
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Major
> Fix For: 1.7.0
>
>
> The method mentioned in the title line cannot handle null inputs, even though 
> the main NiFi execution classes can handle that scenario. This forces hack to 
> pass testing with nulls that looks like this:
> String val = flowFile != null ? 
> context.getProperty(PROP).evaluateExpressionLanguage(flowfile).getValue() : 
> context.getProperty(PROP).evaluateExpressionLanguage(new 
> HashMap()).getValue();



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #2714: NIFI-5145 Changed the override in MockPropertyValue...

[MikeThomsen]

GitHub user MikeThomsen opened a pull request:

https://github.com/apache/nifi/pull/2714

NIFI-5145 Changed the override in MockPropertyValue to use a better w…

…ay of passing over null flowfiles in evaluateExpressionLanguage.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/MikeThomsen/nifi NIFI-5145

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/2714.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2714


commit 35db02cd771be74779e50a7f419c28904bc671d8
Author: Mike Thomsen 
Date:   2018-05-17T16:47:53Z

NIFI-5145 Changed the override in MockPropertyValue to use a better way of 
passing over null flowfiles in evaluateExpressionLanguage.




---

[jira] [Updated] (NIFI-5208) Editing a flow on a Disconnected Node

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-5208:
--
Description: 
Currently, editing on a flow on a disconnected node is allowed. This feature is 
useful when needing to debug a node-specific environmental issue prior to 
re-joining the cluster. However, this can also lead to issues when the user 
doesn't realize that the node they are viewing is disconnected. This was never 
an issue in our 0.x baseline as viewing the disconnected node required the user 
to manually direct their browser away from the NCM and towards the disconnected 
node.

In 1.x, this can happen transparently without the need for the user to redirect 
their browser. There is a label at the top to indicate that the node is 
disconnected but this is not sufficient. If the user continues with their 
edits, it will make it difficult to re-join the cluster without manual 
inventions to retain their changes.

There is a dialog that should inform the user that the cluster connection state 
has changed. However, there appears to be a regression that is preventing that 
dialog from showing. We should restore this dialog and make it confirm the 
users intent to make changes in a disconnected state. Furthermore, changes 
should be prevented without this confirmation. Confirmation should happen 
anytime the cluster connected state changes.

  was:
Currently, editing on a flow on a disconnected node is allowed. This feature is 
useful when needing to debug a node-specific environmental issue prior to 
re-joining the cluster. However, this can also lead to issues when the user 
doesn't realize that the node they are viewing is disconnected. This was never 
an issue in our 0.x baseline as viewing the disconnected node required the user 
to manually direct their browser away from the NCM and towards the disconnected 
node.

In 1.x, this can happen transparently with the need for the user to redirect 
their browser. There is a label at the top to indicate that the node is 
disconnected but this is not sufficient. If the user continues to with their 
edits, it will make it difficult to re-join the cluster without manual 
inventions to retain their changes.

There is a dialog that should inform the user that the cluster connection state 
has changed. However, there appears to be a regression that is preventing that 
dialog from showing. We should restore this dialog and make it confirm the 
users intent to make changes in a disconnected state.


> Editing a flow on a Disconnected Node
> -
>
> Key: NIFI-5208
> URL: https://issues.apache.org/jira/browse/NIFI-5208
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework, Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Critical
>
> Currently, editing on a flow on a disconnected node is allowed. This feature 
> is useful when needing to debug a node-specific environmental issue prior to 
> re-joining the cluster. However, this can also lead to issues when the user 
> doesn't realize that the node they are viewing is disconnected. This was 
> never an issue in our 0.x baseline as viewing the disconnected node required 
> the user to manually direct their browser away from the NCM and towards the 
> disconnected node.
> In 1.x, this can happen transparently without the need for the user to 
> redirect their browser. There is a label at the top to indicate that the node 
> is disconnected but this is not sufficient. If the user continues with their 
> edits, it will make it difficult to re-join the cluster without manual 
> inventions to retain their changes.
> There is a dialog that should inform the user that the cluster connection 
> state has changed. However, there appears to be a regression that is 
> preventing that dialog from showing. We should restore this dialog and make 
> it confirm the users intent to make changes in a disconnected state. 
> Furthermore, changes should be prevented without this confirmation. 
> Confirmation should happen anytime the cluster connected state changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5208) Editing a flow on a Disconnected Node

[Matt Gilman (JIRA)]

Matt Gilman created NIFI-5208:
-

 Summary: Editing a flow on a Disconnected Node
 Key: NIFI-5208
 URL: https://issues.apache.org/jira/browse/NIFI-5208
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework, Core UI
Reporter: Matt Gilman
Assignee: Matt Gilman


Currently, editing on a flow on a disconnected node is allowed. This feature is 
useful when needing to debug a node-specific environmental issue prior to 
re-joining the cluster. However, this can also lead to issues when the user 
doesn't realize that the node they are viewing is disconnected. This was never 
an issue in our 0.x baseline as viewing the disconnected node required the user 
to manually direct their browser away from the NCM and towards the disconnected 
node.

In 1.x, this can happen transparently with the need for the user to redirect 
their browser. There is a label at the top to indicate that the node is 
disconnected but this is not sufficient. If the user continues to with their 
edits, it will make it difficult to re-join the cluster without manual 
inventions to retain their changes.

There is a dialog that should inform the user that the cluster connection state 
has changed. However, there appears to be a regression that is preventing that 
dialog from showing. We should restore this dialog and make it confirm the 
users intent to make changes in a disconnected state.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5204) When node joins cluster, if a processor is stopping but cluster says the state is disabled, node ends up in inconsistent state

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479251#comment-16479251
 ] 

ASF GitHub Bot commented on NIFI-5204:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2713
  
Will review...


> When node joins cluster, if a processor is stopping but cluster says the 
> state is disabled, node ends up in inconsistent state
> --
>
> Key: NIFI-5204
> URL: https://issues.apache.org/jira/browse/NIFI-5204
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Critical
>
> In order to make this "easy" to replicate, I did the following:
> 1) Create a 2-node cluster.
>  2) On both nodes, update nifi.properties to set 
> "nifi.variable.registry.properties" to "1.properties"
>  3) On both nodes, create 1.properties in $NIFI_HOME. For first node, set 
> "sleep=2 mins" and for second node, set "sleep=0 millis"
>  4) Update DebugFlow to support expression language for the "@OnStopped Pause 
> Time"
>  5) Configure flow with a DebugFlow processor. Can auto-terminate 
> relationships and set run period to "10 secs."Set "@OnStopped Pause time" to 
> "${sleep}"
>  6) Disable DebugFlow processor.
>  7) Disconnected Node 1.
>  8) Go to Node 1 in browser and Start DebugFlow.
>  9) Stop DebugFlow.
>  10) While processor is still "stopping", go back Node 2 in browser and 
> request that Node 1 re-join the cluster.
> Now, when Node 1 re-joins the cluster, it will attempt to disable the 
> processor but won't be able to because the processor is still stopping. The 
> following will be in the logs:
> {code:java}
> 2018-05-16 15:21:50,986 WARN [Reconnect to Cluster] 
> org.apache.nifi.controller.ProcessorNode Processor cannot be disabled because 
> its state is set to STOPPING{code}
> So we now have a node in an inconsistent state.
> Additionally, if we now go to Node 1 in our browser and unselect all 
> components, and attempt to STOP the process group, the request that is 
> replicated attempts to stop the DebugFlow processor. Node 2 will now fail to 
> stop the processor because the processor is disabled. As a result, Node 2 
> will now be kicked out of the cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2713: NIFI-5204: If processor joins cluster and inherits 'disabl...

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/2713
  
Will review...


---

[GitHub] nifi pull request #2695: NIFI-5044 SelectHiveQL accept only one statement

[bdesert]

Github user bdesert commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2695#discussion_r189008316
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -113,6 +126,17 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor HIVEQL_POST_QUERY = new 
PropertyDescriptor.Builder()
+.name("hive-post-query")
+.displayName("HiveQL Post-Query")
+.description("HiveQL post-query to execute. 
Semicolon-delimited list of queries. "
++ "Example: 'set tez.queue.name=default; set 
hive.exec.orc.split.strategy=HYBRID; set 
hive.exec.reducers.bytes.per.reducer=258998272'. "
++ "Note, the results/outputs of these queries will be 
suppressed if successful executed.")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
--- End diff --

Well, would be interesting to have a parser here, but then it should be 
global, and include also main query ("select" statement). I believe it can be 
done, but out of scope for this change. We can have another Improvement request 
in Jira to have new HiveQL validator for all Hive related processors.


---

[jira] [Commented] (NIFI-5044) SelectHiveQL accept only one statement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479244#comment-16479244
 ] 

ASF GitHub Bot commented on NIFI-5044:
--

Github user bdesert commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2695#discussion_r189008316
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -113,6 +126,17 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor HIVEQL_POST_QUERY = new 
PropertyDescriptor.Builder()
+.name("hive-post-query")
+.displayName("HiveQL Post-Query")
+.description("HiveQL post-query to execute. 
Semicolon-delimited list of queries. "
++ "Example: 'set tez.queue.name=default; set 
hive.exec.orc.split.strategy=HYBRID; set 
hive.exec.reducers.bytes.per.reducer=258998272'. "
++ "Note, the results/outputs of these queries will be 
suppressed if successful executed.")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
--- End diff --

Well, would be interesting to have a parser here, but then it should be 
global, and include also main query ("select" statement). I believe it can be 
done, but out of scope for this change. We can have another Improvement request 
in Jira to have new HiveQL validator for all Hive related processors.


> SelectHiveQL accept only one statement
> --
>
> Key: NIFI-5044
> URL: https://issues.apache.org/jira/browse/NIFI-5044
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0
>Reporter: Davide Isoardi
>Assignee: Ed Berezitsky
>Priority: Critical
>  Labels: features, patch, pull-request-available
> Attachments: 
> 0001-NIFI-5044-SelectHiveQL-accept-only-one-statement.patch
>
>
> In [this 
> |[https://github.com/apache/nifi/commit/bbc714e73ba245de7bc32fd9958667c847101f7d]
>  ] commit claims to add support to running multiple statements both on 
> SelectHiveQL and PutHiveQL; instead, it adds only the support to PutHiveQL, 
> so SelectHiveQL still lacks this important feature. @Matt Burgess, I saw that 
> you worked on that, is there any reason for this? If not, can we support it?
> If I try to execute this query:
> {quote}set hive.vectorized.execution.enabled = false; SELECT * FROM table_name
> {quote}
> I have this error:
>  
> {quote}2018-04-05 13:35:40,572 ERROR [Timer-Driven Process Thread-146] 
> o.a.nifi.processors.hive.SelectHiveQL 
> SelectHiveQL[id=243d4c17-b1fe-14af--ee8ce15e] Unable to execute 
> HiveQL select query set hive.vectorized.execution.enabled = false; SELECT * 
> FROM table_name for 
> StandardFlowFileRecord[uuid=0e035558-07ce-473b-b0d4-ac00b8b1df93,claim=StandardContentClaim
>  [resourceClaim=StandardResourceClaim[id=1522824912161-2753, 
> container=default, section=705], offset=838441, 
> length=25],offset=0,name=cliente_attributi.csv,size=25] due to 
> org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!; routing to failure: {}
>  org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL$2.process(SelectHiveQL.java:305)
>  at 
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2529)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:275)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$0(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>  at 
> 

[jira] [Assigned] (NIFI-5205) NiFi SysAdmin Guide max.appendable.size Default Value

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim reassigned NIFI-5205:


Assignee: Andrew Lim

> NiFi SysAdmin Guide max.appendable.size Default Value
> -
>
> Key: NIFI-5205
> URL: https://issues.apache.org/jira/browse/NIFI-5205
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation  Website
>Affects Versions: 1.6.0
>Reporter: Michael
>Assignee: Andrew Lim
>Priority: Minor
> Fix For: 1.6.0
>
> Attachments: Screen Shot 2018-05-16 at 3.56.48 PM.png
>
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Documentation for nifi.content.claim.max.appendable.size does not match 
> default value of 1 MB in nifi.properties for 1.6.0.
> Keep in mind that I am making the assumption that NiFi defaults to this same 
> value of 1 MB when it starts up. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (NIFI-2966) Add policy resource descriptors to Admin Guide listing of policies

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim resolved NIFI-2966.
--
   Resolution: Fixed
Fix Version/s: 1.7.0

> Add policy resource descriptors to Admin Guide listing of policies
> --
>
> Key: NIFI-2966
> URL: https://issues.apache.org/jira/browse/NIFI-2966
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation  Website
>Affects Versions: 1.0.0
>Reporter: Andy LoPresto
>Assignee: Andrew Lim
>Priority: Minor
>  Labels: authorization, beginner, documentation, policy
> Fix For: 1.7.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> While ideally a user/admin would never have to examine the 
> {{$NIFI_HOME/conf/users.xml}} or {{$NIFI_HOME/conf/authorizations.xml}} 
> files, in my development and debugging, I have had to on more than one 
> occasion. It would be helpful to list the actual policy resource descriptor 
> (e.g. {{/provenance}}, {{/flow}}, {{/tenants}}) in the table in the [Admin 
> Guide|https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#global-access-policies]
>  listing these policies. When the provenance policy was missing for both of 
> my defined users, I could not add it through the UI, so I had to manually 
> edit the {{authorizations.xml}} file, but I did not know the resource 
> descriptor a priori, so I had to search the code base for it. 
> I envision simply adding an additional column to the table and adding the 
> descriptor, including a placeholder if necessary (e.g. 
> {{/data/process-groups/}}_root-process-group-id_). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5044) SelectHiveQL accept only one statement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479227#comment-16479227
 ] 

ASF GitHub Bot commented on NIFI-5044:
--

Github user bdesert commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2695#discussion_r189005203
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -113,6 +126,17 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor HIVEQL_POST_QUERY = new 
PropertyDescriptor.Builder()
+.name("hive-post-query")
+.displayName("HiveQL Post-Query")
+.description("HiveQL post-query to execute. 
Semicolon-delimited list of queries. "
++ "Example: 'set tez.queue.name=default; set 
hive.exec.orc.split.strategy=HYBRID; set 
hive.exec.reducers.bytes.per.reducer=258998272'. "
--- End diff --

In my opinion - there shouldn't be "post" queries at all, as after "select" 
it doesn't make sense to run anything. But based on discussion, there is a 
demand for "post-queries", that's why we implement.  I'll be happy to get any 
suggestion on what could be a good example for post-query :)


> SelectHiveQL accept only one statement
> --
>
> Key: NIFI-5044
> URL: https://issues.apache.org/jira/browse/NIFI-5044
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0
>Reporter: Davide Isoardi
>Assignee: Ed Berezitsky
>Priority: Critical
>  Labels: features, patch, pull-request-available
> Attachments: 
> 0001-NIFI-5044-SelectHiveQL-accept-only-one-statement.patch
>
>
> In [this 
> |[https://github.com/apache/nifi/commit/bbc714e73ba245de7bc32fd9958667c847101f7d]
>  ] commit claims to add support to running multiple statements both on 
> SelectHiveQL and PutHiveQL; instead, it adds only the support to PutHiveQL, 
> so SelectHiveQL still lacks this important feature. @Matt Burgess, I saw that 
> you worked on that, is there any reason for this? If not, can we support it?
> If I try to execute this query:
> {quote}set hive.vectorized.execution.enabled = false; SELECT * FROM table_name
> {quote}
> I have this error:
>  
> {quote}2018-04-05 13:35:40,572 ERROR [Timer-Driven Process Thread-146] 
> o.a.nifi.processors.hive.SelectHiveQL 
> SelectHiveQL[id=243d4c17-b1fe-14af--ee8ce15e] Unable to execute 
> HiveQL select query set hive.vectorized.execution.enabled = false; SELECT * 
> FROM table_name for 
> StandardFlowFileRecord[uuid=0e035558-07ce-473b-b0d4-ac00b8b1df93,claim=StandardContentClaim
>  [resourceClaim=StandardResourceClaim[id=1522824912161-2753, 
> container=default, section=705], offset=838441, 
> length=25],offset=0,name=cliente_attributi.csv,size=25] due to 
> org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!; routing to failure: {}
>  org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL$2.process(SelectHiveQL.java:305)
>  at 
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2529)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:275)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$0(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  at 
> 

[GitHub] nifi pull request #2695: NIFI-5044 SelectHiveQL accept only one statement

[bdesert]

Github user bdesert commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2695#discussion_r189005203
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -113,6 +126,17 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor HIVEQL_POST_QUERY = new 
PropertyDescriptor.Builder()
+.name("hive-post-query")
+.displayName("HiveQL Post-Query")
+.description("HiveQL post-query to execute. 
Semicolon-delimited list of queries. "
++ "Example: 'set tez.queue.name=default; set 
hive.exec.orc.split.strategy=HYBRID; set 
hive.exec.reducers.bytes.per.reducer=258998272'. "
--- End diff --

In my opinion - there shouldn't be "post" queries at all, as after "select" 
it doesn't make sense to run anything. But based on discussion, there is a 
demand for "post-queries", that's why we implement.  I'll be happy to get any 
suggestion on what could be a good example for post-query :)


---

[jira] [Updated] (NIFI-4508) AMQP Processor that uses basicConsume

[Randy Bovay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-4508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randy Bovay updated NIFI-4508:
--
Affects Version/s: 1.7.0
   1.4.0
   1.5.0
   1.6.0

> AMQP Processor that uses basicConsume
> -
>
> Key: NIFI-4508
> URL: https://issues.apache.org/jira/browse/NIFI-4508
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0
>Reporter: Randy Bovay
>Priority: Major
>
> Due to poor performance of the AMQP Processor, we need to be able to have a 
> basicConsume based interface to RabbitMQ.
> https://community.hortonworks.com/questions/66799/consumeamqp-performance-issue-less-than-50-msgs-se.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-1741) Help Guide should Say what RELEASE capabilities are available for.

[Randy Bovay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randy Bovay updated NIFI-1741:
--
Affects Version/s: 1.7.0
   1.6.0

> Help Guide should Say what RELEASE capabilities are available for.
> --
>
> Key: NIFI-1741
> URL: https://issues.apache.org/jira/browse/NIFI-1741
> Project: Apache NiFi
>  Issue Type: Wish
>  Components: Documentation  Website
>Affects Versions: 0.6.0, 0.5.1, 0.6.1, 1.6.0, 1.7.0
>Reporter: Randy Bovay
>Priority: Minor
>
> As new capabilities are released, and people are using older versions, it 
> would be good to see if the version of NiFi being used supports that 
> particular function.
> i.e, getDelimitedFields is not available until 0.6.  The website doesn't 
> reflect that, so someone may start to develop against that not realizing they 
> were on 0.4, or 0.5.
> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5044) SelectHiveQL accept only one statement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479172#comment-16479172
 ] 

ASF GitHub Bot commented on NIFI-5044:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2695#discussion_r18896
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -113,6 +126,17 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor HIVEQL_POST_QUERY = new 
PropertyDescriptor.Builder()
+.name("hive-post-query")
+.displayName("HiveQL Post-Query")
+.description("HiveQL post-query to execute. 
Semicolon-delimited list of queries. "
++ "Example: 'set tez.queue.name=default; set 
hive.exec.orc.split.strategy=HYBRID; set 
hive.exec.reducers.bytes.per.reducer=258998272'. "
++ "Note, the results/outputs of these queries will be 
suppressed if successful executed.")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
--- End diff --

Could it make sense to have a custom validator here just to be sure the 
statements will be accepted by the driver? (would need to check if EL is used 
or not first). Not sure it's worth the trouble though...


> SelectHiveQL accept only one statement
> --
>
> Key: NIFI-5044
> URL: https://issues.apache.org/jira/browse/NIFI-5044
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0
>Reporter: Davide Isoardi
>Assignee: Ed Berezitsky
>Priority: Critical
>  Labels: features, patch, pull-request-available
> Attachments: 
> 0001-NIFI-5044-SelectHiveQL-accept-only-one-statement.patch
>
>
> In [this 
> |[https://github.com/apache/nifi/commit/bbc714e73ba245de7bc32fd9958667c847101f7d]
>  ] commit claims to add support to running multiple statements both on 
> SelectHiveQL and PutHiveQL; instead, it adds only the support to PutHiveQL, 
> so SelectHiveQL still lacks this important feature. @Matt Burgess, I saw that 
> you worked on that, is there any reason for this? If not, can we support it?
> If I try to execute this query:
> {quote}set hive.vectorized.execution.enabled = false; SELECT * FROM table_name
> {quote}
> I have this error:
>  
> {quote}2018-04-05 13:35:40,572 ERROR [Timer-Driven Process Thread-146] 
> o.a.nifi.processors.hive.SelectHiveQL 
> SelectHiveQL[id=243d4c17-b1fe-14af--ee8ce15e] Unable to execute 
> HiveQL select query set hive.vectorized.execution.enabled = false; SELECT * 
> FROM table_name for 
> StandardFlowFileRecord[uuid=0e035558-07ce-473b-b0d4-ac00b8b1df93,claim=StandardContentClaim
>  [resourceClaim=StandardResourceClaim[id=1522824912161-2753, 
> container=default, section=705], offset=838441, 
> length=25],offset=0,name=cliente_attributi.csv,size=25] due to 
> org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!; routing to failure: {}
>  org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL$2.process(SelectHiveQL.java:305)
>  at 
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2529)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:275)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$0(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  at 
> 

[jira] [Commented] (NIFI-5044) SelectHiveQL accept only one statement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479170#comment-16479170
 ] 

ASF GitHub Bot commented on NIFI-5044:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2695#discussion_r188985931
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -113,6 +126,17 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor HIVEQL_POST_QUERY = new 
PropertyDescriptor.Builder()
+.name("hive-post-query")
+.displayName("HiveQL Post-Query")
+.description("HiveQL post-query to execute. 
Semicolon-delimited list of queries. "
++ "Example: 'set tez.queue.name=default; set 
hive.exec.orc.split.strategy=HYBRID; set 
hive.exec.reducers.bytes.per.reducer=258998272'. "
--- End diff --

I'm clearly not a Hive expert but does it make sense to give this kind of 
examples for post-queries?


> SelectHiveQL accept only one statement
> --
>
> Key: NIFI-5044
> URL: https://issues.apache.org/jira/browse/NIFI-5044
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0
>Reporter: Davide Isoardi
>Assignee: Ed Berezitsky
>Priority: Critical
>  Labels: features, patch, pull-request-available
> Attachments: 
> 0001-NIFI-5044-SelectHiveQL-accept-only-one-statement.patch
>
>
> In [this 
> |[https://github.com/apache/nifi/commit/bbc714e73ba245de7bc32fd9958667c847101f7d]
>  ] commit claims to add support to running multiple statements both on 
> SelectHiveQL and PutHiveQL; instead, it adds only the support to PutHiveQL, 
> so SelectHiveQL still lacks this important feature. @Matt Burgess, I saw that 
> you worked on that, is there any reason for this? If not, can we support it?
> If I try to execute this query:
> {quote}set hive.vectorized.execution.enabled = false; SELECT * FROM table_name
> {quote}
> I have this error:
>  
> {quote}2018-04-05 13:35:40,572 ERROR [Timer-Driven Process Thread-146] 
> o.a.nifi.processors.hive.SelectHiveQL 
> SelectHiveQL[id=243d4c17-b1fe-14af--ee8ce15e] Unable to execute 
> HiveQL select query set hive.vectorized.execution.enabled = false; SELECT * 
> FROM table_name for 
> StandardFlowFileRecord[uuid=0e035558-07ce-473b-b0d4-ac00b8b1df93,claim=StandardContentClaim
>  [resourceClaim=StandardResourceClaim[id=1522824912161-2753, 
> container=default, section=705], offset=838441, 
> length=25],offset=0,name=cliente_attributi.csv,size=25] due to 
> org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!; routing to failure: {}
>  org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL$2.process(SelectHiveQL.java:305)
>  at 
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2529)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:275)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$0(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
>  Caused by: java.sql.SQLException: The 

[jira] [Commented] (NIFI-5044) SelectHiveQL accept only one statement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479171#comment-16479171
 ] 

ASF GitHub Bot commented on NIFI-5044:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2695#discussion_r188990055
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -311,7 +340,15 @@ private void onTrigger(final ProcessContext context, 
final ProcessSession sessio
 try (final Connection con = 
dbcpService.getConnection(fileToProcess == null ? Collections.emptyMap() : 
fileToProcess.getAttributes());
  final Statement st = (flowbased ? 
con.prepareStatement(selectQuery) : con.createStatement())
 ) {
-
+Pair failure = 
executeConfigStatements(con, preQueries);
+if (failure != null) {
+// In case of failure, assigning config query to 
"selectQuery" var will allow to avoid major changes in error handling (catch 
block),
--- End diff --

I understand the intent here, just wondering if it's a good idea on the 
long term for code maintainability/clarity. @mattyb149 what are your thoughts?


> SelectHiveQL accept only one statement
> --
>
> Key: NIFI-5044
> URL: https://issues.apache.org/jira/browse/NIFI-5044
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0
>Reporter: Davide Isoardi
>Assignee: Ed Berezitsky
>Priority: Critical
>  Labels: features, patch, pull-request-available
> Attachments: 
> 0001-NIFI-5044-SelectHiveQL-accept-only-one-statement.patch
>
>
> In [this 
> |[https://github.com/apache/nifi/commit/bbc714e73ba245de7bc32fd9958667c847101f7d]
>  ] commit claims to add support to running multiple statements both on 
> SelectHiveQL and PutHiveQL; instead, it adds only the support to PutHiveQL, 
> so SelectHiveQL still lacks this important feature. @Matt Burgess, I saw that 
> you worked on that, is there any reason for this? If not, can we support it?
> If I try to execute this query:
> {quote}set hive.vectorized.execution.enabled = false; SELECT * FROM table_name
> {quote}
> I have this error:
>  
> {quote}2018-04-05 13:35:40,572 ERROR [Timer-Driven Process Thread-146] 
> o.a.nifi.processors.hive.SelectHiveQL 
> SelectHiveQL[id=243d4c17-b1fe-14af--ee8ce15e] Unable to execute 
> HiveQL select query set hive.vectorized.execution.enabled = false; SELECT * 
> FROM table_name for 
> StandardFlowFileRecord[uuid=0e035558-07ce-473b-b0d4-ac00b8b1df93,claim=StandardContentClaim
>  [resourceClaim=StandardResourceClaim[id=1522824912161-2753, 
> container=default, section=705], offset=838441, 
> length=25],offset=0,name=cliente_attributi.csv,size=25] due to 
> org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!; routing to failure: {}
>  org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: 
> The query did not generate a result set!
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL$2.process(SelectHiveQL.java:305)
>  at 
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2529)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:275)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$0(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
>  at 
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
>  at 
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:215)
>  at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>  at 
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>  at 
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
>  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> 

[GitHub] nifi pull request #2695: NIFI-5044 SelectHiveQL accept only one statement

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2695#discussion_r188990055
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -311,7 +340,15 @@ private void onTrigger(final ProcessContext context, 
final ProcessSession sessio
 try (final Connection con = 
dbcpService.getConnection(fileToProcess == null ? Collections.emptyMap() : 
fileToProcess.getAttributes());
  final Statement st = (flowbased ? 
con.prepareStatement(selectQuery) : con.createStatement())
 ) {
-
+Pair failure = 
executeConfigStatements(con, preQueries);
+if (failure != null) {
+// In case of failure, assigning config query to 
"selectQuery" var will allow to avoid major changes in error handling (catch 
block),
--- End diff --

I understand the intent here, just wondering if it's a good idea on the 
long term for code maintainability/clarity. @mattyb149 what are your thoughts?


---

[GitHub] nifi pull request #2695: NIFI-5044 SelectHiveQL accept only one statement

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2695#discussion_r18896
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -113,6 +126,17 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor HIVEQL_POST_QUERY = new 
PropertyDescriptor.Builder()
+.name("hive-post-query")
+.displayName("HiveQL Post-Query")
+.description("HiveQL post-query to execute. 
Semicolon-delimited list of queries. "
++ "Example: 'set tez.queue.name=default; set 
hive.exec.orc.split.strategy=HYBRID; set 
hive.exec.reducers.bytes.per.reducer=258998272'. "
++ "Note, the results/outputs of these queries will be 
suppressed if successful executed.")
+.required(false)
+.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
--- End diff --

Could it make sense to have a custom validator here just to be sure the 
statements will be accepted by the driver? (would need to check if EL is used 
or not first). Not sure it's worth the trouble though...


---

[GitHub] nifi pull request #2695: NIFI-5044 SelectHiveQL accept only one statement

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/2695#discussion_r188985931
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
 ---
@@ -113,6 +126,17 @@
 
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
 .build();
 
+public static final PropertyDescriptor HIVEQL_POST_QUERY = new 
PropertyDescriptor.Builder()
+.name("hive-post-query")
+.displayName("HiveQL Post-Query")
+.description("HiveQL post-query to execute. 
Semicolon-delimited list of queries. "
++ "Example: 'set tez.queue.name=default; set 
hive.exec.orc.split.strategy=HYBRID; set 
hive.exec.reducers.bytes.per.reducer=258998272'. "
--- End diff --

I'm clearly not a Hive expert but does it make sense to give this kind of 
examples for post-queries?


---

[jira] [Updated] (NIFI-5197) Fix invalid expression language scope declarations

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-5197:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Fix invalid expression language scope declarations
> --
>
> Key: NIFI-5197
> URL: https://issues.apache.org/jira/browse/NIFI-5197
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
> Fix For: 1.7.0
>
>
> Some of the expression language scope declarations are wrong, such as 
> declaring VARIABLE_REGISTRY and then passing in a flowfile as the argument in 
> the processor. These need to be hunted down and cleaned up in one single 
> commit. Each set of unit and integration tests should be run to verify where 
> problems reside.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (NIFI-5079) Add support for logical types to Hive DDL creation in PutORC

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess closed NIFI-5079.
--

> Add support for logical types to Hive DDL creation in PutORC
> 
>
> Key: NIFI-5079
> URL: https://issues.apache.org/jira/browse/NIFI-5079
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> While other processors have been updated to support Avro logical types (such 
> as decimal, timestamp, etc.), ConvertAvroToORC does not currently support 
> them. We should update ConvertAvroToORC to handle Avro logical types, not 
> just in the conversion to ORC but also in the generated Hive DDL.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (NIFI-5079) Add support for logical types to Hive DDL creation in PutORC

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess resolved NIFI-5079.

Resolution: Won't Fix

> Add support for logical types to Hive DDL creation in PutORC
> 
>
> Key: NIFI-5079
> URL: https://issues.apache.org/jira/browse/NIFI-5079
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> While other processors have been updated to support Avro logical types (such 
> as decimal, timestamp, etc.), ConvertAvroToORC does not currently support 
> them. We should update ConvertAvroToORC to handle Avro logical types, not 
> just in the conversion to ORC but also in the generated Hive DDL.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5079) Add support for logical types to Hive DDL creation in PutORC

[Matt Burgess (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479113#comment-16479113
 ] 

Matt Burgess commented on NIFI-5079:


Closing as absorbed by NIFI-4963 which includes PutORC

> Add support for logical types to Hive DDL creation in PutORC
> 
>
> Key: NIFI-5079
> URL: https://issues.apache.org/jira/browse/NIFI-5079
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> While other processors have been updated to support Avro logical types (such 
> as decimal, timestamp, etc.), ConvertAvroToORC does not currently support 
> them. We should update ConvertAvroToORC to handle Avro logical types, not 
> just in the conversion to ORC but also in the generated Hive DDL.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #2702: Added Apache Pulsar processors

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi/pull/2702
  
@MikeThomsen I've only spent a couple of minutes looking at this, but I'm 
not sure it can work out as nicely as you are hoping... the controller service 
API here is heavily dependent on the actual pulsar client API. 

The only way you can get the transparent swapping between processors and CS 
impls is if the client API is hidden behind the CS impl. For example with 
HBase, we have...

- HBase processors (no dependency on hbase-client, dependency on hbase CS 
api)
- HBase CS API (no dependency on hbase-client)
- HBase CS 1.1.2 impl (dependency on hbase-client 1.1.2)

So because the processors and CS API do not know about hbase-client, then 
we can transparently provide new implementations without changing the 
processors.

In this case we have...
- Pulsar processors (depends on Pulsar client 1.21.0-incubating)
- Pulsar CS API (depends on Pulsar client 1.21.0-incubating)
- Pulsar CS Impls (depends on Pulsar client 1.21.0-incubating)

I'm not saying the current setup is bad, just mentioning that it won't work 
out the way the hbase setup works. 

The trade-off is that in order to achieve the hbase setup you essentially 
need to recreate parts of their client API and depending how much you have to 
recreate, it may not be worth it if you are recreating the entire pulsar client 
API just to shield the processors.

Also, something to think about is whether Pulsar's client will work well 
across different broker versions. For example, when Pulsar 2.x comes out, will 
the 1.x client work well against a 2.x broker? or vice versa?

In Kafka land their client has had issues across versions, like 0.8 client 
against 0.9 broker did not perform as well as 0.9 client against 0.9 broker, so 
for this reason you really need to use the corresponding client that goes with 
the broker.

If Pulsar's client doesn't have this problem, maybe we don't need to worry 
at all about this versioning stuff.


---

[jira] [Updated] (NIFI-5079) Add support for logical types to Hive DDL creation in PutORC

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-5079:
---
Summary: Add support for logical types to Hive DDL creation in PutORC  
(was: Add support for logical types to Hive DDL creation in ConvertAvroToORC)

> Add support for logical types to Hive DDL creation in PutORC
> 
>
> Key: NIFI-5079
> URL: https://issues.apache.org/jira/browse/NIFI-5079
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> While other processors have been updated to support Avro logical types (such 
> as decimal, timestamp, etc.), ConvertAvroToORC does not currently support 
> them. We should update ConvertAvroToORC to handle Avro logical types, not 
> just in the conversion to ORC but also in the generated Hive DDL.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5197) Fix invalid expression language scope declarations

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479099#comment-16479099
 ] 

ASF GitHub Bot commented on NIFI-5197:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/2712


> Fix invalid expression language scope declarations
> --
>
> Key: NIFI-5197
> URL: https://issues.apache.org/jira/browse/NIFI-5197
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
> Fix For: 1.7.0
>
>
> Some of the expression language scope declarations are wrong, such as 
> declaring VARIABLE_REGISTRY and then passing in a flowfile as the argument in 
> the processor. These need to be hunted down and cleaned up in one single 
> commit. Each set of unit and integration tests should be run to verify where 
> problems reside.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5197) Fix invalid expression language scope declarations

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-5197:
---
Status: Patch Available  (was: Open)

> Fix invalid expression language scope declarations
> --
>
> Key: NIFI-5197
> URL: https://issues.apache.org/jira/browse/NIFI-5197
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
>
> Some of the expression language scope declarations are wrong, such as 
> declaring VARIABLE_REGISTRY and then passing in a flowfile as the argument in 
> the processor. These need to be hunted down and cleaned up in one single 
> commit. Each set of unit and integration tests should be run to verify where 
> problems reside.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5197) Fix invalid expression language scope declarations

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-5197:
---
Fix Version/s: 1.7.0

> Fix invalid expression language scope declarations
> --
>
> Key: NIFI-5197
> URL: https://issues.apache.org/jira/browse/NIFI-5197
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Mike Thomsen
>Assignee: Mike Thomsen
>Priority: Minor
> Fix For: 1.7.0
>
>
> Some of the expression language scope declarations are wrong, such as 
> declaring VARIABLE_REGISTRY and then passing in a flowfile as the argument in 
> the processor. These need to be hunted down and cleaned up in one single 
> commit. Each set of unit and integration tests should be run to verify where 
> problems reside.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)