[jira] [Commented] (NIFI-5826) UpdateRecord processor throwing PatternSyntaxException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704345#comment-16704345
 ] 

ASF GitHub Bot commented on NIFI-5826:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3183#discussion_r237764346
  
--- Diff: 
nifi-commons/nifi-record-path/src/main/java/org/apache/nifi/record/path/util/RecordPathUtils.java
 ---
@@ -39,4 +39,52 @@ public static String getFirstStringValue(final 
RecordPathSegment segment, final
 
 return stringValue;
 }
+
+/**
+ * This method handles backslash sequences after ANTLR parser converts 
all backslash into double ones
+ * with exception for \t, \r and \n. See
+ * org/apache/nifi/record/path/RecordPathParser.g
--- End diff --

@bdesert Thanks for the explanation. I read through your comment over and 
over, and tested different input string. However, that differs from what what 
I'm seeing.

I debugged how Lexer works, too, and found that:

- The `ESC` fragment handles an escaped special character in String 
representation. I.e. String `\t` will be converted to actual tab character.
- The string values user input from NiFi UI are passed to 
`RecordPath.compile` method as it is. E.g. the input string 
`replaceRegex(/name, '\[', '')` is passed to as is, then the single back-slash 
is converted to double back-slash by the ESC fragment line 155.
- Since the escaped characters such as `\n` are already unescaped by this 
Lexer, the `RecordPathUtils.unescapeBackslash` method added by this PR will not 
see any **escaped** characters other than the doubled back-slash `//`. And the 
doubled back-slash is converted back to a single back-slash by 
`RecordPathUtils.unescapeBackslash`.
- I believe the line 153-156 is aimed to preserve escaped characters as it 
is, because such escape doesn't mean anything for the RecordPath/AttrExpLang 
spec. And those should be unescaped later by underlying syntaxes such as RegEx.
- And current line 155 does it wrongly. It should append a single 
back-slash..
- Other Lexers (AttributeExpressionLexer.g and HL7QueryLexer.g) have 
the same issue.
- So, I think we should fix all Lexers instead of adding another conversion.

Here is the [Lexer 
code](https://github.com/apache/nifi/blob/master/nifi-commons/nifi-record-path/src/main/antlr3/org/apache/nifi/record/path/RecordPathLexer.g#L143)
 for reference:
```
143 fragment
144 ESC
145   :  '\\'
146 (
147 '"'{ setText("\""); }
148   |  '\''  { setText("\'"); }
149   |  'r'   { setText("\r"); }
150   |  'n'   { setText("\n"); }
151   |  't'   { setText("\t"); }
152   |  '\\'  { setText(""); }
153   |  nextChar = ~('"' | '\'' | 'r' | 'n' | 't' | '\\')
154{
155  StringBuilder lBuf = new StringBuilder(); 
lBuf.append("").appendCodePoint(nextChar); setText(lBuf.toString());
156}
157)
158  ;
```

Can share any input String value that a user would set into a processor 
property text box (or passed to RecordPath.compile method), that will go the 
proposed `RecordPathUtils.unescapeBackslash` method lines of code for `\n`, 
`\r` or `\t`? Current test case does have regex test cases using those escaped 
characters, but coverage shows those branches are not used:

![image](https://user-images.githubusercontent.com/1107620/49273985-18162900-f4ba-11e8-8ac7-2600f8f8c698.png)



> UpdateRecord processor throwing PatternSyntaxException
> --
>
> Key: NIFI-5826
> URL: https://issues.apache.org/jira/browse/NIFI-5826
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.7.1
> Environment: Nifi in docker container
>Reporter: ravi kargam
>Assignee: Ed Berezitsky
>Priority: Minor
> Attachments: NIFI-5826_PR-3183.patch, 
> UpdateRecord_Config_Exception.JPG
>
>
> with replacement value strategy as Record Path Value,
> I am trying to replace square bracket symbol *[* with parenthesis symbol *(* 
> in my employeeName column in my csvReader structure with below syntax
> replaceRegex(/employeeName, "[\[]", "(")
> Processor is throwing following exception.
> RecordPathException: java.util.regex.PatternSyntaxException: Unclosed 
> character class near index 4 [\\[]
> It worked fine with other special characters such as \{, }, <, >, ;, _, "
> For double qoute ("), i had to use single escape character, for above listed 
> other characters, worked fine without any escape character. Other folks in 
> Nifi Slack tried \s, \d, \w, \. 
> looks like non of them 

[GitHub] nifi pull request #3183: NIFI-5826 Fix to escaped backslash

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3183#discussion_r237764346
  
--- Diff: 
nifi-commons/nifi-record-path/src/main/java/org/apache/nifi/record/path/util/RecordPathUtils.java
 ---
@@ -39,4 +39,52 @@ public static String getFirstStringValue(final 
RecordPathSegment segment, final
 
 return stringValue;
 }
+
+/**
+ * This method handles backslash sequences after ANTLR parser converts 
all backslash into double ones
+ * with exception for \t, \r and \n. See
+ * org/apache/nifi/record/path/RecordPathParser.g
--- End diff --

@bdesert Thanks for the explanation. I read through your comment over and 
over, and tested different input string. However, that differs from what what 
I'm seeing.

I debugged how Lexer works, too, and found that:

- The `ESC` fragment handles an escaped special character in String 
representation. I.e. String `\t` will be converted to actual tab character.
- The string values user input from NiFi UI are passed to 
`RecordPath.compile` method as it is. E.g. the input string 
`replaceRegex(/name, '\[', '')` is passed to as is, then the single back-slash 
is converted to double back-slash by the ESC fragment line 155.
- Since the escaped characters such as `\n` are already unescaped by this 
Lexer, the `RecordPathUtils.unescapeBackslash` method added by this PR will not 
see any **escaped** characters other than the doubled back-slash `//`. And the 
doubled back-slash is converted back to a single back-slash by 
`RecordPathUtils.unescapeBackslash`.
- I believe the line 153-156 is aimed to preserve escaped characters as it 
is, because such escape doesn't mean anything for the RecordPath/AttrExpLang 
spec. And those should be unescaped later by underlying syntaxes such as RegEx.
- And current line 155 does it wrongly. It should append a single 
back-slash..
- Other Lexers (AttributeExpressionLexer.g and HL7QueryLexer.g) have 
the same issue.
- So, I think we should fix all Lexers instead of adding another conversion.

Here is the [Lexer 
code](https://github.com/apache/nifi/blob/master/nifi-commons/nifi-record-path/src/main/antlr3/org/apache/nifi/record/path/RecordPathLexer.g#L143)
 for reference:
```
143 fragment
144 ESC
145   :  '\\'
146 (
147 '"'{ setText("\""); }
148   |  '\''  { setText("\'"); }
149   |  'r'   { setText("\r"); }
150   |  'n'   { setText("\n"); }
151   |  't'   { setText("\t"); }
152   |  '\\'  { setText(""); }
153   |  nextChar = ~('"' | '\'' | 'r' | 'n' | 't' | '\\')
154{
155  StringBuilder lBuf = new StringBuilder(); 
lBuf.append("").appendCodePoint(nextChar); setText(lBuf.toString());
156}
157)
158  ;
```

Can share any input String value that a user would set into a processor 
property text box (or passed to RecordPath.compile method), that will go the 
proposed `RecordPathUtils.unescapeBackslash` method lines of code for `\n`, 
`\r` or `\t`? Current test case does have regex test cases using those escaped 
characters, but coverage shows those branches are not used:

![image](https://user-images.githubusercontent.com/1107620/49273985-18162900-f4ba-11e8-8ac7-2600f8f8c698.png)



---

[jira] [Created] (NIFI-5857) Non deterministic behaviour in Kubernetes by trying to inject custom properties

[dirkjkb (JIRA)]

dirkjkb created NIFI-5857:
-

 Summary: Non deterministic behaviour in Kubernetes by trying to 
inject custom properties
 Key: NIFI-5857
 URL: https://issues.apache.org/jira/browse/NIFI-5857
 Project: Apache NiFi
  Issue Type: Bug
  Components: Docker
Affects Versions: 1.8.0
 Environment: Kubernetes, Docker
Reporter: dirkjkb


I want to override some config files in Nifi via Kubernetes. In order to do so 
I am trying to replace the files after the start. It appears that the docker 
file is started through a start.sh script which calls several other scripts. 
This implementation Leeds to a non deterministic state, since the replacement 
time can differ from the start.sh runtime. Furthermore, after restarting a pod, 
the replacing command will be run each time again what also leeds to a fuzzy 
state. 

My proposal would be instead of injecting and running some sh files who will 
set some variables the customized config files should just be copy replaced in 
the building step. The run command can then be replaced through the ENTRYPOINT 
["bin/nifi.sh", "run"] Command. 

In order to get the logging output to the console, a logback-test.xml file 
should be created and configured so that all the meaningful information will be 
piped to stdout. 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5856) Add capability to assign available matching controller services to processors during import from registry

[Ed Berezitsky (JIRA)]

Ed Berezitsky created NIFI-5856:
---

 Summary: Add capability to assign available matching controller 
services to processors during import from registry
 Key: NIFI-5856
 URL: https://issues.apache.org/jira/browse/NIFI-5856
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Affects Versions: 1.8.0
Reporter: Ed Berezitsky


As a user I would like to reduce manual configuration of components after 
importing flows from NIFI registry.

Use cases:
 * a component uses controller service(s) defined in a scope of *parent* (or 
higher) level (e.g. record-based processors, DB pools, etc) can have 
controllers assigned by default, if ID registered is not available (versioned 
from another NIFI instance)
 * a controller service that is in a scope of imported flow uses another 
controller in a scope  of *parent* (or higher) level (e.g. Record 
readers/writer using schema registry).

Current state:
 * a lookup for a controller service is done by ID. If ID is not found, a 
controller won't be assigned and property of a processor/controller will stay 
blank and will require manual configuration/selection

Specifications/Requirements:
 * Change current behavior to enable default assignment of controller services 
to processor/controller property in case desired controller service cannot be 
found by ID.
 * in order to reduce wrong automatic assignments, both type and name of a 
controller service should be considered. 
 * Since names aren't unique, have a NIFI property to specify strict and 
nonstrict policy for multi-match:
 ** strict mode will restrict automatic assignment of controller service, and 
property in the processor/controller will stay blank (as per current 
specification).
 ** nonstrict mode will allow any of matching controllers to be assigned (first 
found).

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #3187: NIFI-5847 Added GovCloud-East region to enum list.

[zenfenan]

Github user zenfenan commented on the issue:

https://github.com/apache/nifi/pull/3187
  
@jmlogan It is an enum on the AWS SDK side so whatever new regions that are 
added to AWS cloud services, the team managing the SDK project update the enum 
on their side. 


---

[jira] [Commented] (NIFI-5847) AWS Region List Missing New Region

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704264#comment-16704264
 ] 

ASF GitHub Bot commented on NIFI-5847:
--

Github user zenfenan commented on the issue:

https://github.com/apache/nifi/pull/3187
  
@jmlogan It is an enum on the AWS SDK side so whatever new regions that are 
added to AWS cloud services, the team managing the SDK project update the enum 
on their side. 


> AWS Region List Missing New Region
> --
>
> Key: NIFI-5847
> URL: https://issues.apache.org/jira/browse/NIFI-5847
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Jon logan
>Priority: Major
>
> A new region was added to AWS, the enum needs to be updated accordingly.
>  
> https://aws.amazon.com/blogs/aws/aws-govcloud-us-east-now-open/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5847) AWS Region List Missing New Region

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704257#comment-16704257
 ] 

ASF GitHub Bot commented on NIFI-5847:
--

Github user jmlogan commented on the issue:

https://github.com/apache/nifi/pull/3187
  
I can close it but I was wondering how the SDK derives that list ...it
looks like it's just an enum on their side -- what about regions that
aren't in that enum? It seems like this still might be a problem for these
regions. I would imagine this is a problem that users have ran into, and
isn't being addressed in either PR.

On Thu, Nov 29, 2018 at 4:35 AM Sivaprasanna 
wrote:

> *@zenfenan* commented on this pull request.
>
> As discussed in the original user mails thread, this custom enum on the
> NiFi side has to go. I raised the PR #3190
>  addressing the same. With
> that, we don't have to update our enum every time there is a change.
>
> @jmlogan  Mind closing this one? If the
> intention is to have the new region added then we can update the SDK
> version in the POM which would bring the latest available regions
> automatically without even having the enum.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
> 

> .
>



> AWS Region List Missing New Region
> --
>
> Key: NIFI-5847
> URL: https://issues.apache.org/jira/browse/NIFI-5847
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Jon logan
>Priority: Major
>
> A new region was added to AWS, the enum needs to be updated accordingly.
>  
> https://aws.amazon.com/blogs/aws/aws-govcloud-us-east-now-open/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #3187: NIFI-5847 Added GovCloud-East region to enum list.

[jmlogan]

Github user jmlogan commented on the issue:

https://github.com/apache/nifi/pull/3187
  
I can close it but I was wondering how the SDK derives that list ...it
looks like it's just an enum on their side -- what about regions that
aren't in that enum? It seems like this still might be a problem for these
regions. I would imagine this is a problem that users have ran into, and
isn't being addressed in either PR.

On Thu, Nov 29, 2018 at 4:35 AM Sivaprasanna 
wrote:

> *@zenfenan* commented on this pull request.
>
> As discussed in the original user mails thread, this custom enum on the
> NiFi side has to go. I raised the PR #3190
>  addressing the same. With
> that, we don't have to update our enum every time there is a change.
>
> @jmlogan  Mind closing this one? If the
> intention is to have the new region added then we can update the SDK
> version in the POM which would bring the latest available regions
> automatically without even having the enum.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
> 

> .
>



---

[jira] [Commented] (NIFI-5835) Add support to use SQL Server rowversion as maximum value column

[Matt Burgess (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704198#comment-16704198
 ] 

Matt Burgess commented on NIFI-5835:


I suspect the driver returns the type ROWID or something similar, we'd just 
need to support that type in the code as well. For these cases it's kind of an 
"add support when you run into it" kind of thing, it's fairly difficult to 
handle all the types generically as no DB seems to do things the same way :)

> Add support to use SQL Server rowversion as maximum value column
> 
>
> Key: NIFI-5835
> URL: https://issues.apache.org/jira/browse/NIFI-5835
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Affects Versions: 1.8.0
>Reporter: Charlie Meyer
>Priority: Major
>
> I'm attempting to do incremental fetch from a Microsoft SQL Server database 
> and would like to use rowversion [1] as my maximum value column. When I 
> configured the processor to use that column, it threw an exception [2].
>  
> [1] 
> [https://docs.microsoft.com/en-us/sql/t-sql/data-types/rowversion-transact-sql?view=sql-server-2017]
>  
> [2] 
> [https://github.com/apache/nifi/blob/d8d220ccb86d1797f56f34649d70a1acff278eb5/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractDatabaseFetchProcessor.java#L456]
>  
> [~alopresto]'s reply on the mailing list:
>  
> Looking at this issue briefly, it seems that the NiFi code explicitly lists 
> the accepted datatypes which can be used, and rowversion is not enumerated. 
> Therefore it throws an exception. I suggest you open a feature request on our 
> Jira page to support this. While it seems proprietary to Microsoft SQL 
> versions, it says on the documentation page:
>  
> Is a data type that exposes automatically generated, unique binary numbers 
> within a database. rowversion is generally used as a mechanism for 
> version-stamping table rows. The storage size is 8 bytes. The rowversion data 
> type is just an incrementing number and does not preserve a date or a time.
>  
> I think we could handle this datatype the same way we handle INTEGER, 
> SMALLINT, TINYINT (or TIMESTAMP, as that is the functional equivalent from MS 
> SQL which is now deprecated) in that switch statement, as it is simply an 
> incrementing 8 byte natural number. However, I would welcome input from 
> someone like [~mattyb149] to see if maybe there is a translation that can be 
> done in the Microsoft-specific driver to a generic integer datatype before it 
> reaches this logic. I would expect 
> SQLServerResultSetMetaData#getColumnType(int column) to perform this 
> translation; perhaps the version of the driver needs to be updated?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5846) Redirect URL is incorrect after logout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16704063#comment-16704063
 ] 

ASF GitHub Bot commented on NIFI-5846:
--

Github user kotarot commented on the issue:

https://github.com/apache/nifi/pull/3185
  
Thank you for reviewing and merging it!


> Redirect URL is incorrect after logout
> --
>
> Key: NIFI-5846
> URL: https://issues.apache.org/jira/browse/NIFI-5846
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Kotaro Terada
>Assignee: Kotaro Terada
>Priority: Major
> Attachments: login-incorrect-url.png
>
>
> When we click the logout button on the Web UI, it currently redirects to 
> "/login" instead of "/nifi/login" after logging out, which causes the error 
> page shown in the attached screenshot.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #3185: NIFI-5846: Redirect URL is incorrect after logout

[kotarot]

Github user kotarot commented on the issue:

https://github.com/apache/nifi/pull/3185
  
Thank you for reviewing and merging it!


---

[jira] [Updated] (NIFI-5855) Optimize GenerateTableFetch to remove unnecessary ORDER BY clause

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess updated NIFI-5855:
---
Status: Patch Available  (was: In Progress)

> Optimize GenerateTableFetch to remove unnecessary ORDER BY clause
> -
>
> Key: NIFI-5855
> URL: https://issues.apache.org/jira/browse/NIFI-5855
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
> column/table information removes an unnecessary COUNT(*) clause when 
> Partition Size = 0, the ORDER BY clause can also be removed. This is because 
> there will be a single generated query that will fetch all rows that match 
> the filters. The ORDER BY is necessary during pagination (when Partition Size 
> > 0) so it can fetch from Row X to Row Y based on the row number, which is 
> only constant if the column is ordered.
> This Jira proposes to remove the ORDER BY clause from the generated query if 
> Partition Size is set to 0. Note that this can cause the output of the 
> executed query to not guaranteed to be ordered; it is a possible change in 
> behavior, but we don't document any guarantees about row ordering anyway. If 
> the user needs the rows ordered, they can do it with QueryRecord, or they can 
> add the ORDER BY clause manually via ReplaceText.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5855) Optimize GenerateTableFetch to remove unnecessary ORDER BY clause

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703929#comment-16703929
 ] 

ASF GitHub Bot commented on NIFI-5855:
--

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/3191

NIFI-5855: Remove unnecessary ORDER BY clause in GenerateTableFetch when 
Partition Size is zero

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-5855

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/3191.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3191


commit 78a3c9badf4724691022081413bbb0a6de2f960e
Author: Matthew Burgess 
Date:   2018-11-29T22:46:24Z

NIFI-5855: Remove unnecessary ORDER BY clause in GenerateTableFetch when 
Partition Size is zero




> Optimize GenerateTableFetch to remove unnecessary ORDER BY clause
> -
>
> Key: NIFI-5855
> URL: https://issues.apache.org/jira/browse/NIFI-5855
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
> column/table information removes an unnecessary COUNT(*) clause when 
> Partition Size = 0, the ORDER BY clause can also be removed. This is because 
> there will be a single generated query that will fetch all rows that match 
> the filters. The ORDER BY is necessary during pagination (when Partition Size 
> > 0) so it can fetch from Row X to Row Y based on the row number, which is 
> only constant if the column is ordered.
> This Jira proposes to remove the ORDER BY clause from the generated query if 
> Partition Size is set to 0. Note that this can cause the output of the 
> executed query to not guaranteed to be ordered; it is a possible change in 
> behavior, but we don't document any guarantees about row ordering anyway. If 
> the user needs the rows ordered, they can do it with QueryRecord, or they can 
> add the ORDER BY clause manually via ReplaceText.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #3191: NIFI-5855: Remove unnecessary ORDER BY clause in Ge...

[mattyb149]

GitHub user mattyb149 opened a pull request:

https://github.com/apache/nifi/pull/3191

NIFI-5855: Remove unnecessary ORDER BY clause in GenerateTableFetch when 
Partition Size is zero

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mattyb149/nifi NIFI-5855

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/3191.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3191


commit 78a3c9badf4724691022081413bbb0a6de2f960e
Author: Matthew Burgess 
Date:   2018-11-29T22:46:24Z

NIFI-5855: Remove unnecessary ORDER BY clause in GenerateTableFetch when 
Partition Size is zero




---

[jira] [Assigned] (NIFI-5855) Optimize GenerateTableFetch to remove unnecessary ORDER BY clause

[Matt Burgess (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Burgess reassigned NIFI-5855:
--

Assignee: Matt Burgess

> Optimize GenerateTableFetch to remove unnecessary ORDER BY clause
> -
>
> Key: NIFI-5855
> URL: https://issues.apache.org/jira/browse/NIFI-5855
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
> column/table information removes an unnecessary COUNT(*) clause when 
> Partition Size = 0, the ORDER BY clause can also be removed. This is because 
> there will be a single generated query that will fetch all rows that match 
> the filters. The ORDER BY is necessary during pagination (when Partition Size 
> > 0) so it can fetch from Row X to Row Y based on the row number, which is 
> only constant if the column is ordered.
> This Jira proposes to remove the ORDER BY clause from the generated query if 
> Partition Size is set to 0. Note that this can cause the output of the 
> executed query to not guaranteed to be ordered; it is a possible change in 
> behavior, but we don't document any guarantees about row ordering anyway. If 
> the user needs the rows ordered, they can do it with QueryRecord, or they can 
> add the ORDER BY clause manually via ReplaceText.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5855) Optimize GenerateTableFetch to remove unnecessary ORDER BY clause

[Matt Burgess (JIRA)]

Matt Burgess created NIFI-5855:
--

 Summary: Optimize GenerateTableFetch to remove unnecessary ORDER 
BY clause
 Key: NIFI-5855
 URL: https://issues.apache.org/jira/browse/NIFI-5855
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Reporter: Matt Burgess


Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
column/table information removes an unnecessary COUNT(*) clause when Partition 
Size = 0, the ORDER BY clause can also be removed. This is because there will 
be a single generated query that will fetch all rows that match the filters. 
The ORDER BY is necessary during pagination (when Partition Size > 0) so it can 
fetch from Row X to Row Y based on the row number, which is only constant if 
the column is ordered.

This Jira proposes to remove the ORDER BY clause from the generated query if 
Partition Size is set to 0. Note that this can cause the output of the executed 
query to not guaranteed to be ordered; it is a possible change in behavior, but 
we don't document any guarantees about row ordering anyway. If the user needs 
the rows ordered, they can do it with QueryRecord, or they can add the ORDER BY 
clause manually via ReplaceText.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5855) Optimize GenerateTableFetch to remove unnecessary ORDER BY clause

[Matt Burgess (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703928#comment-16703928
 ] 

Matt Burgess commented on NIFI-5855:


In addition to the check in NIFI-5727, we should remove the COUNT(*) when the 
partition size is zero, for the same logical consequence (i.e. row count is not 
used)

> Optimize GenerateTableFetch to remove unnecessary ORDER BY clause
> -
>
> Key: NIFI-5855
> URL: https://issues.apache.org/jira/browse/NIFI-5855
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>Priority: Major
>
> Related to NIFI-5727, which for the query that GenerateTableFetch uses to get 
> column/table information removes an unnecessary COUNT(*) clause when 
> Partition Size = 0, the ORDER BY clause can also be removed. This is because 
> there will be a single generated query that will fetch all rows that match 
> the filters. The ORDER BY is necessary during pagination (when Partition Size 
> > 0) so it can fetch from Row X to Row Y based on the row number, which is 
> only constant if the column is ordered.
> This Jira proposes to remove the ORDER BY clause from the generated query if 
> Partition Size is set to 0. Note that this can cause the output of the 
> executed query to not guaranteed to be ordered; it is a possible change in 
> behavior, but we don't document any guarantees about row ordering anyway. If 
> the user needs the rows ordered, they can do it with QueryRecord, or they can 
> add the ORDER BY clause manually via ReplaceText.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703881#comment-16703881
 ] 

ASF GitHub Bot commented on NIFIREG-211:


Github user bbende commented on the issue:

https://github.com/apache/nifi-registry/pull/148
  
Just pushed a new commit that allows the client to specify a SHA-256 when 
uploading the bundle. If they specify one and it doesn't match what the server 
calculates, then it is rejected. If they don't specify one then we accept the 
bundle, but we mark a flag called "sha256Supplied" as false, so consumers later 
will know if the bundle was uploaded with or without a client provided checksum.

Since the upload is already a multi-part request, I thought it made the 
most sense for the SHA-256 to be another field in the multi-part form data. As 
an example of a curl command to upload with the checksum:

`curl -v -F file=@./nifi-example-processors-nar-1.0.0.nar 
-Fsha256=93d2c6537142497ca3f908452d3c3c2bcae928fdc07e1b48fd3ba95ec22bc639 
"http://localhost:18080/nifi-registry-api/buckets/72f897d9-5645-4168-852f-4885cc7c9cfe/extensions/bundles/nifi-nar;
`


> Add extension bundles as a type of versioned item
> -
>
> Key: NIFIREG-211
> URL: https://issues.apache.org/jira/browse/NIFIREG-211
> Project: NiFi Registry
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 0.4.0
>
>
> This ticket is to capture the work for adding extension bundles to NiFi 
> Registry.
> This work may require several follow on tickets, but at a high-level will 
> include some of the following:
> - Add a new type of item called an extension bundle, where each bundle
>  can contain one ore extensions or APIs
>  
>  - Support bundles for traditional NiFi (aka NARs) and also bundles for
>  MiNiFi CPP
>  
>  - Ability to upload the binary artifact for a bundle and extract the
>  metadata about the bundle, and metadata about the extensions contained
>  in the bundle (more on this later)
>  
>  - Provide a pluggable storage provider for saving the content of each
>  extension bundle so that we can have different implementations like
>  local fileysystem, S3, and other object stores
>  
>  - Provide a REST API for listing and retrieving available bundles,
>  integrate this into the registry Java client and NiFi CLI
> - Security considerations such as checksums and cryptographic signatures for 
> bundles



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-registry issue #148: NIFIREG-211 Initial work for adding extenion bundl...

[bbende]

Github user bbende commented on the issue:

https://github.com/apache/nifi-registry/pull/148
  
Just pushed a new commit that allows the client to specify a SHA-256 when 
uploading the bundle. If they specify one and it doesn't match what the server 
calculates, then it is rejected. If they don't specify one then we accept the 
bundle, but we mark a flag called "sha256Supplied" as false, so consumers later 
will know if the bundle was uploaded with or without a client provided checksum.

Since the upload is already a multi-part request, I thought it made the 
most sense for the SHA-256 to be another field in the multi-part form data. As 
an example of a curl command to upload with the checksum:

`curl -v -F file=@./nifi-example-processors-nar-1.0.0.nar 
-Fsha256=93d2c6537142497ca3f908452d3c3c2bcae928fdc07e1b48fd3ba95ec22bc639 
"http://localhost:18080/nifi-registry-api/buckets/72f897d9-5645-4168-852f-4885cc7c9cfe/extensions/bundles/nifi-nar;
`


---

[jira] [Created] (NIFI-5854) Enhance time unit features

[Andy LoPresto (JIRA)]

Andy LoPresto created NIFI-5854:
---

 Summary: Enhance time unit features
 Key: NIFI-5854
 URL: https://issues.apache.org/jira/browse/NIFI-5854
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework
Affects Versions: 1.8.0
Reporter: Andy LoPresto
Assignee: Andy LoPresto


There is some ambiguity with time units (specifically around processor 
properties). Two features which I think should be added:

* Currently only whole numbers are parsed correctly. For example, {{10 
milliseconds}} and {{0.010 seconds}} are functionally equivalent, but only the 
former will be parsed. This is due to the regex used in 
{{StandardValidators.TIME_PERIOD_VALIDATOR}} which relies on 
{{FormatUtils.TIME_DURATION_REGEX}} (see below). Decimal amounts should be 
parsed
* The enumerated time units are *nanoseconds, milliseconds, seconds, minutes, 
hours, days, weeks*. While I don't intend to extend this to "millennia", etc. 
as every unit including and above *months* would be ambiguous, *microseconds* 
seems like a valid and missing unit

*Definition of {{FormatUtils.TIME_DURATION_REGEX}}:*
{code}
public static final String TIME_DURATION_REGEX = "(\\d+)\\s*(" + 
VALID_TIME_UNITS + ")";
public static final Pattern TIME_DURATION_PATTERN = 
Pattern.compile(TIME_DURATION_REGEX);
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4532) OpenID Connect User Authentication

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703811#comment-16703811
 ] 

ASF GitHub Bot commented on NIFI-4532:
--

Github user mosermw commented on the issue:

https://github.com/apache/nifi/pull/3009
  
@SarthakSahu Recommend sticking with /bin/sh, as Alpine base images don't 
usually contain /bin/bash.


> OpenID Connect User Authentication
> --
>
> Key: NIFI-4532
> URL: https://issues.apache.org/jira/browse/NIFI-4532
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Docker
>Reporter: Aldrin Piri
>Assignee: Sarthak
>Priority: Major
>
> Provide configuration for OpenID Connect user authentication in Docker images



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #3009: NIFI-4532 OpenID Connect User Authentication

[mosermw]

Github user mosermw commented on the issue:

https://github.com/apache/nifi/pull/3009
  
@SarthakSahu Recommend sticking with /bin/sh, as Alpine base images don't 
usually contain /bin/bash.


---

[jira] [Created] (NIFI-5853) Support ZooKeeper as a backend for DistributedMapCacheServer

[Boris Tyukin (JIRA)]

Boris Tyukin created NIFI-5853:
--

 Summary: Support ZooKeeper as a backend for 
DistributedMapCacheServer
 Key: NIFI-5853
 URL: https://issues.apache.org/jira/browse/NIFI-5853
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Extensions
Affects Versions: 1.8.0
Reporter: Boris Tyukin


ZooKeeper is readily available on most big data clusters, already used by NiFi 
cluster to store state, highly reliable and scalable. It would be great to 
support it as a backend for DistributedMapCacheServer



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFIREG-211) Add extension bundles as a type of versioned item

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFIREG-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703480#comment-16703480
 ] 

ASF GitHub Bot commented on NIFIREG-211:


Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/148#discussion_r237568373
  
--- Diff: 
nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/service/extension/StandardExtensionService.java
 ---
@@ -188,13 +188,14 @@ public ExtensionBundleVersion 
createExtensionBundleVersion(final String bucketId
 
 // create the bundle version in the metadata db
 final String userIdentity = 
NiFiUserUtils.getNiFiUserIdentity();
+final long bundleCreatedTime = 
extensionBundle.getCreated().getTime();
 
 final ExtensionBundleVersionMetadata versionMetadata = new 
ExtensionBundleVersionMetadata();
 versionMetadata.setId(UUID.randomUUID().toString());
 versionMetadata.setExtensionBundleId(extensionBundle.getId());
 versionMetadata.setBucketId(bucketIdentifier);
 versionMetadata.setVersion(version);
-versionMetadata.setTimestamp(System.currentTimeMillis());
+versionMetadata.setTimestamp(bundleCreatedTime);
--- End diff --

This works great when the bundle is being created with the first version 
uploaded. But I noticed when subsequent versions are uploaded they are also 
getting tagged with the same creation time. Given that we don't know what was 
done in getOrCreateExtensionBundle, we might need to refactor the code above, 
or... it might be better to just revert this change to always use currentTime 
for the bundle version. I'm good with either


> Add extension bundles as a type of versioned item
> -
>
> Key: NIFIREG-211
> URL: https://issues.apache.org/jira/browse/NIFIREG-211
> Project: NiFi Registry
>  Issue Type: Improvement
>Reporter: Bryan Bende
>Assignee: Bryan Bende
>Priority: Major
> Fix For: 0.4.0
>
>
> This ticket is to capture the work for adding extension bundles to NiFi 
> Registry.
> This work may require several follow on tickets, but at a high-level will 
> include some of the following:
> - Add a new type of item called an extension bundle, where each bundle
>  can contain one ore extensions or APIs
>  
>  - Support bundles for traditional NiFi (aka NARs) and also bundles for
>  MiNiFi CPP
>  
>  - Ability to upload the binary artifact for a bundle and extract the
>  metadata about the bundle, and metadata about the extensions contained
>  in the bundle (more on this later)
>  
>  - Provide a pluggable storage provider for saving the content of each
>  extension bundle so that we can have different implementations like
>  local fileysystem, S3, and other object stores
>  
>  - Provide a REST API for listing and retrieving available bundles,
>  integrate this into the registry Java client and NiFi CLI
> - Security considerations such as checksums and cryptographic signatures for 
> bundles



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-registry pull request #148: NIFIREG-211 Initial work for adding extenio...

[kevdoran]

Github user kevdoran commented on a diff in the pull request:

https://github.com/apache/nifi-registry/pull/148#discussion_r237568373
  
--- Diff: 
nifi-registry-core/nifi-registry-framework/src/main/java/org/apache/nifi/registry/service/extension/StandardExtensionService.java
 ---
@@ -188,13 +188,14 @@ public ExtensionBundleVersion 
createExtensionBundleVersion(final String bucketId
 
 // create the bundle version in the metadata db
 final String userIdentity = 
NiFiUserUtils.getNiFiUserIdentity();
+final long bundleCreatedTime = 
extensionBundle.getCreated().getTime();
 
 final ExtensionBundleVersionMetadata versionMetadata = new 
ExtensionBundleVersionMetadata();
 versionMetadata.setId(UUID.randomUUID().toString());
 versionMetadata.setExtensionBundleId(extensionBundle.getId());
 versionMetadata.setBucketId(bucketIdentifier);
 versionMetadata.setVersion(version);
-versionMetadata.setTimestamp(System.currentTimeMillis());
+versionMetadata.setTimestamp(bundleCreatedTime);
--- End diff --

This works great when the bundle is being created with the first version 
uploaded. But I noticed when subsequent versions are uploaded they are also 
getting tagged with the same creation time. Given that we don't know what was 
done in getOrCreateExtensionBundle, we might need to refactor the code above, 
or... it might be better to just revert this change to always use currentTime 
for the bundle version. I'm good with either


---

[jira] [Commented] (NIFI-5846) Redirect URL is incorrect after logout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703335#comment-16703335
 ] 

ASF GitHub Bot commented on NIFI-5846:
--

Github user patricker commented on the issue:

https://github.com/apache/nifi/pull/3185
  
+1 LG.


> Redirect URL is incorrect after logout
> --
>
> Key: NIFI-5846
> URL: https://issues.apache.org/jira/browse/NIFI-5846
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Kotaro Terada
>Assignee: Kotaro Terada
>Priority: Major
> Attachments: login-incorrect-url.png
>
>
> When we click the logout button on the Web UI, it currently redirects to 
> "/login" instead of "/nifi/login" after logging out, which causes the error 
> page shown in the attached screenshot.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5829) Create Lookup Controller Services for RecordSetWriter and RecordReader

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703346#comment-16703346
 ] 

ASF GitHub Bot commented on NIFI-5829:
--

Github user patricker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3188#discussion_r237530330
  
--- Diff: 
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/record/script/ScriptedRecordSetWriter.java
 ---
@@ -60,12 +61,16 @@ public void onEnabled(final ConfigurationContext 
context) {
 super.onEnabled(context);
 }
 
-
 @Override
 public RecordSetWriter createWriter(ComponentLog logger, RecordSchema 
schema, OutputStream out) throws SchemaNotFoundException, IOException {
+return createWriter(new HashMap<>(), logger, schema, out);
--- End diff --

Thanks, Updated.


> Create Lookup Controller Services for RecordSetWriter and RecordReader
> --
>
> Key: NIFI-5829
> URL: https://issues.apache.org/jira/browse/NIFI-5829
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Peter Wicks
>Assignee: Peter Wicks
>Priority: Minor
>
> In the same way as the new DBCP Lookup Service, create lookup services for 
> RecordSetWriter and RecordReader.
> This will help to make flows much more generic, and allow for some very 
> flexible processors.
> Example: A processor like PutDatabaseRecord will be able to push to any 
> database in any readable record format from a single processor by configuring 
> everything through the lookup services.
> Example: ConvertRecord will be able to convert from any number of various 
> input formats to a constant output format by using the RecordReader Lookup, 
> and a fixed RecordSetWriter.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #3188: NIFI-5829 Create Lookup Controller Services for Rec...

[patricker]

Github user patricker commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3188#discussion_r237530330
  
--- Diff: 
nifi-nar-bundles/nifi-scripting-bundle/nifi-scripting-processors/src/main/java/org/apache/nifi/record/script/ScriptedRecordSetWriter.java
 ---
@@ -60,12 +61,16 @@ public void onEnabled(final ConfigurationContext 
context) {
 super.onEnabled(context);
 }
 
-
 @Override
 public RecordSetWriter createWriter(ComponentLog logger, RecordSchema 
schema, OutputStream out) throws SchemaNotFoundException, IOException {
+return createWriter(new HashMap<>(), logger, schema, out);
--- End diff --

Thanks, Updated.


---

[jira] [Commented] (NIFI-5826) UpdateRecord processor throwing PatternSyntaxException

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703340#comment-16703340
 ] 

ASF GitHub Bot commented on NIFI-5826:
--

Github user bdesert commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3183#discussion_r237529258
  
--- Diff: 
nifi-commons/nifi-record-path/src/main/java/org/apache/nifi/record/path/util/RecordPathUtils.java
 ---
@@ -39,4 +39,52 @@ public static String getFirstStringValue(final 
RecordPathSegment segment, final
 
 return stringValue;
 }
+
+/**
+ * This method handles backslash sequences after ANTLR parser converts 
all backslash into double ones
+ * with exception for \t, \r and \n. See
+ * org/apache/nifi/record/path/RecordPathParser.g
--- End diff --

Yeah, that was supposed to be Lexer (I'll update PR later just in case I'll 
need to make more changes).
Regarding your question on Lexer.
When I started working on this issue, I also first started looking into 
Lexer, trying to understand all these escapes for backslash.
After that I took a look at Expression Language Lexer and found the same 
code  there:

https://github.com/apache/nifi/blob/master/nifi-commons/nifi-expression-language/src/main/antlr3/org/apache/nifi/attribute/expression/language/antlr/AttributeExpressionLexer.g#L235
 

Then I started digging into actual code in Java and understood the reasons.
ANTLR parses the code coming from a textbox. All special cases for 
backslash sequences (\r,\n,\t) should be supported and escaped to be able to 
capture actual new lines and tab chars. But then, if you support backslash 
sequences, you would also need to escape "\" itself,  so there is another 
escape for backslash on a line 152. And then, if you escape single backslash, 
as special character, to avoid confusion with \r,\n,\t,\\, the rest of the 
characters are being escaped with double backslash + character (line 155)-  to 
escape actual two char sequence of "\t" or "\r" , etc...
Changing Lexer would change default behavior for backslash sequences not 
only for regex functions, but for all record based input. That would create 
backward compatibility issue, and that is the main reason I decided not to 
change Lexer. 
And also, as I mentioned in already above, current "escape" policy is 
consistent with EL, so i preferred to keep consistency and just fix a bug of 
not escaping back for regular expressions.

Let me know if that makes sense, or if you have better ideas.


> UpdateRecord processor throwing PatternSyntaxException
> --
>
> Key: NIFI-5826
> URL: https://issues.apache.org/jira/browse/NIFI-5826
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.7.1
> Environment: Nifi in docker container
>Reporter: ravi kargam
>Assignee: Ed Berezitsky
>Priority: Minor
> Attachments: NIFI-5826_PR-3183.patch, 
> UpdateRecord_Config_Exception.JPG
>
>
> with replacement value strategy as Record Path Value,
> I am trying to replace square bracket symbol *[* with parenthesis symbol *(* 
> in my employeeName column in my csvReader structure with below syntax
> replaceRegex(/employeeName, "[\[]", "(")
> Processor is throwing following exception.
> RecordPathException: java.util.regex.PatternSyntaxException: Unclosed 
> character class near index 4 [\\[]
> It worked fine with other special characters such as \{, }, <, >, ;, _, "
> For double qoute ("), i had to use single escape character, for above listed 
> other characters, worked fine without any escape character. Other folks in 
> Nifi Slack tried \s, \d, \w, \. 
> looks like non of them worked.
> replace function worked for replacing [ and ]characters. didn't test any 
> other characters.
> Please address resolve the issue.
> Regards,
> Ravi
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #3183: NIFI-5826 Fix to escaped backslash

[bdesert]

Github user bdesert commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3183#discussion_r237529258
  
--- Diff: 
nifi-commons/nifi-record-path/src/main/java/org/apache/nifi/record/path/util/RecordPathUtils.java
 ---
@@ -39,4 +39,52 @@ public static String getFirstStringValue(final 
RecordPathSegment segment, final
 
 return stringValue;
 }
+
+/**
+ * This method handles backslash sequences after ANTLR parser converts 
all backslash into double ones
+ * with exception for \t, \r and \n. See
+ * org/apache/nifi/record/path/RecordPathParser.g
--- End diff --

Yeah, that was supposed to be Lexer (I'll update PR later just in case I'll 
need to make more changes).
Regarding your question on Lexer.
When I started working on this issue, I also first started looking into 
Lexer, trying to understand all these escapes for backslash.
After that I took a look at Expression Language Lexer and found the same 
code  there:

https://github.com/apache/nifi/blob/master/nifi-commons/nifi-expression-language/src/main/antlr3/org/apache/nifi/attribute/expression/language/antlr/AttributeExpressionLexer.g#L235
 

Then I started digging into actual code in Java and understood the reasons.
ANTLR parses the code coming from a textbox. All special cases for 
backslash sequences (\r,\n,\t) should be supported and escaped to be able to 
capture actual new lines and tab chars. But then, if you support backslash 
sequences, you would also need to escape "\" itself,  so there is another 
escape for backslash on a line 152. And then, if you escape single backslash, 
as special character, to avoid confusion with \r,\n,\t,\\, the rest of the 
characters are being escaped with double backslash + character (line 155)-  to 
escape actual two char sequence of "\t" or "\r" , etc...
Changing Lexer would change default behavior for backslash sequences not 
only for regex functions, but for all record based input. That would create 
backward compatibility issue, and that is the main reason I decided not to 
change Lexer. 
And also, as I mentioned in already above, current "escape" policy is 
consistent with EL, so i preferred to keep consistency and just fix a bug of 
not escaping back for regular expressions.

Let me know if that makes sense, or if you have better ideas.


---

[jira] [Commented] (NIFI-5846) Redirect URL is incorrect after logout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703336#comment-16703336
 ] 

ASF GitHub Bot commented on NIFI-5846:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/3185


> Redirect URL is incorrect after logout
> --
>
> Key: NIFI-5846
> URL: https://issues.apache.org/jira/browse/NIFI-5846
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Kotaro Terada
>Assignee: Kotaro Terada
>Priority: Major
> Attachments: login-incorrect-url.png
>
>
> When we click the logout button on the Web UI, it currently redirects to 
> "/login" instead of "/nifi/login" after logging out, which causes the error 
> page shown in the attached screenshot.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5846) Redirect URL is incorrect after logout

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703334#comment-16703334
 ] 

ASF subversion and git services commented on NIFI-5846:
---

Commit 72662f0b2c5ec5f088aade94337f7edce665ee7e in nifi's branch 
refs/heads/master from [~kotarot]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=72662f0 ]

NIFI-5846: Redirect URL is incorrect after logout

This closes #3185.

Signed-off-by: Peter Wicks 


> Redirect URL is incorrect after logout
> --
>
> Key: NIFI-5846
> URL: https://issues.apache.org/jira/browse/NIFI-5846
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Kotaro Terada
>Assignee: Kotaro Terada
>Priority: Major
> Attachments: login-incorrect-url.png
>
>
> When we click the logout button on the Web UI, it currently redirects to 
> "/login" instead of "/nifi/login" after logging out, which causes the error 
> page shown in the attached screenshot.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #3185: NIFI-5846: Redirect URL is incorrect after logout

[patricker]

Github user patricker commented on the issue:

https://github.com/apache/nifi/pull/3185
  
+1 LG.


---

[GitHub] nifi pull request #3185: NIFI-5846: Redirect URL is incorrect after logout

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/3185


---

[jira] [Resolved] (NIFI-764) Add support for global variables

[Peter Wicks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Wicks resolved NIFI-764.
--
Resolution: Duplicate

Already implemented with other tickets.

> Add support for global variables
> 
>
> Key: NIFI-764
> URL: https://issues.apache.org/jira/browse/NIFI-764
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Core UI
>Reporter: Matt Gilman
>Priority: Minor
>
> Add support for global variables (possibly scoped at a group level). These 
> could be used within NiFi expression language providing support for more 
> portable templates and bulk configuration changes.
> The general idea is that the user will be able to define a variable name and 
> value and then use it within expression language. At this point they would be 
> able to export a template with components referencing the variables. When 
> imported into another NiFi instance with the same variables defined, it 
> provides an easy way to move a flow between different environment and update 
> it's configuration in a single action (like when values for development and 
> production differ per instance).
> Another benefit of an approach like this, it that it could allow for bunk 
> edits. If multiple components reference the same variable (e.g. 
> .post.uri), it allows the user to make that change in a single location 
> rather than having to identify each instance of that value. Will obviously 
> need to consider the mechanics when modifying a variable as it will require 
> all referencing components to be stopped.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-4621) Allow inputs to ListSFTP

[Peter Wicks (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703310#comment-16703310
 ] 

Peter Wicks commented on NIFI-4621:
---

[~kislayom] How's this going?

> Allow inputs to ListSFTP
> 
>
> Key: NIFI-4621
> URL: https://issues.apache.org/jira/browse/NIFI-4621
> Project: Apache NiFi
>  Issue Type: Improvement
>Affects Versions: 1.4.0
>Reporter: Soumya Shanta Ghosh
>Assignee: Kislay Kumar
>Priority: Critical
>
> ListSFTP supports listing of the supplied directory (Remote Path) 
> out-of-the-box on the supplied "Hostname" using the 'Username" and 'Password" 
> / "Private Key Passphrase". 
> The password can change at a regular interval (depending on organization 
> policy) or the Hostname or the Remote Path can change based on some other 
> requirement.
> This is a case to allow ListSFTP to leverage the use of Nifi Expression 
> language so that the values of Hostname, Password and/or Remote Path can be 
> set based on the attributes of an incoming flow file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5852) ExecuteSQL Pre/Post SQL Don't Allow Semicolons in String Literals

[Peter Wicks (JIRA)]

Peter Wicks created NIFI-5852:
-

 Summary: ExecuteSQL Pre/Post SQL Don't Allow Semicolons in String 
Literals
 Key: NIFI-5852
 URL: https://issues.apache.org/jira/browse/NIFI-5852
 Project: Apache NiFi
  Issue Type: Improvement
Reporter: Peter Wicks


In NIFI-5780 pre/post SQL statements were added to ExecuteSQL. If the SQL 
statement contains a string constant, like `WHERE field='some;value'` then the 
code breaks because it splits on semicolons no matter where they appear.

Some form of smarter string parsing would allow for semicolons to appear inside 
of strings.

[~deonashh]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (MINIFICPP-646) Re-evaluate passing attributes. Potentially deprecate attribute usage functions

[Mr TheSegfault (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mr TheSegfault updated MINIFICPP-646:
-
Labels: nanofi  (was: CAPI)

> Re-evaluate passing attributes. Potentially deprecate attribute usage 
> functions
> ---
>
> Key: MINIFICPP-646
> URL: https://issues.apache.org/jira/browse/MINIFICPP-646
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Mr TheSegfault
>Priority: Blocker
>  Labels: nanofi
> Fix For: 0.6.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (MINIFICPP-688) Fix spelling in PYTHON.md

[Mr TheSegfault (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mr TheSegfault resolved MINIFICPP-688.
--
Resolution: Fixed

> Fix spelling in PYTHON.md
> -
>
> Key: MINIFICPP-688
> URL: https://issues.apache.org/jira/browse/MINIFICPP-688
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
> Fix For: 0.6.0
>
>
> Fix small spelling error



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (MINIFICPP-688) Fix spelling in PYTHON.md

[Mr TheSegfault (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mr TheSegfault updated MINIFICPP-688:
-
Fix Version/s: 0.6.0

> Fix spelling in PYTHON.md
> -
>
> Key: MINIFICPP-688
> URL: https://issues.apache.org/jira/browse/MINIFICPP-688
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
> Fix For: 0.6.0
>
>
> Fix small spelling error



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-688) Fix spelling in PYTHON.md

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703280#comment-16703280
 ] 

ASF GitHub Bot commented on MINIFICPP-688:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/451


> Fix spelling in PYTHON.md
> -
>
> Key: MINIFICPP-688
> URL: https://issues.apache.org/jira/browse/MINIFICPP-688
> Project: NiFi MiNiFi C++
>  Issue Type: Bug
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Fix small spelling error



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5846) Redirect URL is incorrect after logout

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703282#comment-16703282
 ] 

ASF GitHub Bot commented on NIFI-5846:
--

Github user patricker commented on the issue:

https://github.com/apache/nifi/pull/3185
  
Building and Testing. I only have a Kerberos test environment, but I think 
it will work the same for all of the user/pass logins.


> Redirect URL is incorrect after logout
> --
>
> Key: NIFI-5846
> URL: https://issues.apache.org/jira/browse/NIFI-5846
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Kotaro Terada
>Assignee: Kotaro Terada
>Priority: Major
> Attachments: login-incorrect-url.png
>
>
> When we click the logout button on the Web UI, it currently redirects to 
> "/login" instead of "/nifi/login" after logging out, which causes the error 
> page shown in the attached screenshot.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi issue #3185: NIFI-5846: Redirect URL is incorrect after logout

[patricker]

Github user patricker commented on the issue:

https://github.com/apache/nifi/pull/3185
  
Building and Testing. I only have a Kerberos test environment, but I think 
it will work the same for all of the user/pass logins.


---

[GitHub] nifi-minifi-cpp pull request #451: MINIFICPP-688 fix small spelling error an...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/451


---

[jira] [Commented] (MINIFICPP-681) Add content hash processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703278#comment-16703278
 ] 

ASF GitHub Bot commented on MINIFICPP-681:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/452


> Add content hash processor
> --
>
> Key: MINIFICPP-681
> URL: https://issues.apache.org/jira/browse/MINIFICPP-681
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Major
> Fix For: 0.6.0
>
>
> Add a new processor that supports hashing content and add the checksum to the 
> flowfile as an attribute.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp pull request #452: MINIFICPP-681 - Minor fixes and improveme...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/452


---

[jira] [Commented] (MINIFICPP-479) Incorporate property validation information into manifest

[Mr TheSegfault (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703214#comment-16703214
 ] 

Mr TheSegfault commented on MINIFICPP-479:
--

As I mentioned to [~aboda] there is work that can better inform the servers 
what validators to use. Since the servers follow an open standard, we should 
provide some guarantees regarding validators that follow those of NiFi . I've 
taken the ticket as I previously did some work on this in June and can work 
with Arpad to merge those ideas. 

> Incorporate property validation information into manifest
> -
>
> Key: MINIFICPP-479
> URL: https://issues.apache.org/jira/browse/MINIFICPP-479
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Mr TheSegfault
>Priority: Major
>
> High-level intent is to avoid round-trip to c2 to know that flow is valid 
> (or, invalid in common/trivial ways).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-479) Incorporate property validation information into manifest

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703210#comment-16703210
 ] 

ASF GitHub Bot commented on MINIFICPP-479:
--

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/453
  
Thanks for submitting this. I think there is definite overlap between work 
I previously had with MINIFICPP-479 
(https://github.com/apache/nifi-minifi-cpp/compare/master...phrocker:MINIFICPP-479?expand=1)
 I collected commits from across computers and squashed them, but has the side 
effect that it does validation at flow file load time ( something we've 
desperately needed) -- giving us a much earlier failure. a bit different 
implementation -- and it informs c2 of java validators to be used.

what is the motivation of this PR beyond that work that we discussed 
offline? I think the same "pros" exist for both works. That work has been 
around since June -- so it's probably stale in its ideas, so I'll take a look 
at this at some point. Thanks again. 


> Incorporate property validation information into manifest
> -
>
> Key: MINIFICPP-479
> URL: https://issues.apache.org/jira/browse/MINIFICPP-479
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Major
>
> High-level intent is to avoid round-trip to c2 to know that flow is valid 
> (or, invalid in common/trivial ways).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (MINIFICPP-479) Incorporate property validation information into manifest

[Mr TheSegfault (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MINIFICPP-479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mr TheSegfault reassigned MINIFICPP-479:


Assignee: Mr TheSegfault  (was: Andrew Christianson)

> Incorporate property validation information into manifest
> -
>
> Key: MINIFICPP-479
> URL: https://issues.apache.org/jira/browse/MINIFICPP-479
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Andrew Christianson
>Assignee: Mr TheSegfault
>Priority: Major
>
> High-level intent is to avoid round-trip to c2 to know that flow is valid 
> (or, invalid in common/trivial ways).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp issue #453: POF: static property improvements

[phrocker]

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/453
  
Thanks for submitting this. I think there is definite overlap between work 
I previously had with MINIFICPP-479 
(https://github.com/apache/nifi-minifi-cpp/compare/master...phrocker:MINIFICPP-479?expand=1)
 I collected commits from across computers and squashed them, but has the side 
effect that it does validation at flow file load time ( something we've 
desperately needed) -- giving us a much earlier failure. a bit different 
implementation -- and it informs c2 of java validators to be used.

what is the motivation of this PR beyond that work that we discussed 
offline? I think the same "pros" exist for both works. That work has been 
around since June -- so it's probably stale in its ideas, so I'll take a look 
at this at some point. Thanks again. 


---

[GitHub] nifi-minifi-cpp issue #453: POF: static property improvements

[arpadboda]

Github user arpadboda commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/453
  
Overview:
-Please read tests first. They should give an idea of the motivation of the 
change
-This is POF, some parts of the code might leave space for improvement, 
some are not yet implemented.

Pros of suggested solution:
-Fully backward-compatible, no change required in processors. 
-All property types can be assigned to Property (string property) without 
loosing the validator.
-Values are still stored in string, so no change in overhead of getter
-Easily extendable with further validators,  properties
-Validators can provide data in any format for server-side validation, no 
duplication of data

Cons:
-Templates, templates, templates everywhere... Added some static asserts to 
help development with meaningful error messages. 


---

[GitHub] nifi-minifi-cpp pull request #453: POF: static property improvements

[arpadboda]

GitHub user arpadboda opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/453

POF: static property improvements

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arpadboda/nifi-minifi-cpp 
Static_property_improvements

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/453.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #453


commit d2ffd85546ea3dd927e86234f1eb45bc08cf20d0
Author: Arpad Boda 
Date:   2018-11-29T09:36:09Z

POF: static property improvements




---

[jira] [Commented] (MINIFICPP-681) Add content hash processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703097#comment-16703097
 ] 

ASF GitHub Bot commented on MINIFICPP-681:
--

GitHub user arpadboda opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/452

MINIFICPP-681 - Minor fixes and improvements of HashContent proc. and…

… related docs

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arpadboda/nifi-minifi-cpp MINIFICPP-681

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/452.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #452


commit 0947c38d1c9c2aa28da9fc0b1b57dbd2ee6f6cda
Author: Arpad Boda 
Date:   2018-11-29T12:13:26Z

MINIFICPP-681 - Minor fixes and improvements of HashContent proc. and 
related docs




> Add content hash processor
> --
>
> Key: MINIFICPP-681
> URL: https://issues.apache.org/jira/browse/MINIFICPP-681
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Major
> Fix For: 0.6.0
>
>
> Add a new processor that supports hashing content and add the checksum to the 
> flowfile as an attribute.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi-minifi-cpp pull request #452: MINIFICPP-681 - Minor fixes and improveme...

[arpadboda]

GitHub user arpadboda opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/452

MINIFICPP-681 - Minor fixes and improvements of HashContent proc. and…

… related docs

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [ ] Does your PR title start with MINIFICPP- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arpadboda/nifi-minifi-cpp MINIFICPP-681

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/452.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #452


commit 0947c38d1c9c2aa28da9fc0b1b57dbd2ee6f6cda
Author: Arpad Boda 
Date:   2018-11-29T12:13:26Z

MINIFICPP-681 - Minor fixes and improvements of HashContent proc. and 
related docs




---

[GitHub] nifi-minifi-cpp pull request #445: MINIFICPP-681 - Add content hash processo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/445


---

[jira] [Commented] (MINIFICPP-681) Add content hash processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703083#comment-16703083
 ] 

ASF GitHub Bot commented on MINIFICPP-681:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/445


> Add content hash processor
> --
>
> Key: MINIFICPP-681
> URL: https://issues.apache.org/jira/browse/MINIFICPP-681
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Major
> Fix For: 0.6.0
>
>
> Add a new processor that supports hashing content and add the checksum to the 
> flowfile as an attribute.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5849) ListSFTP, ListFTP processors lose Cluster state on restart.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702924#comment-16702924
 ] 

ASF GitHub Bot commented on NIFI-5849:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/3189


> ListSFTP, ListFTP processors lose Cluster state on restart.
> ---
>
> Key: NIFI-5849
> URL: https://issues.apache.org/jira/browse/NIFI-5849
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.8.0
>Reporter: Truong Duc Kien
>Assignee: Koji Kawamura
>Priority: Critical
> Fix For: 1.9.0
>
>
> In a cluster environment, restarting a processor that extends 
> {{AbstractListProcessor}} could lead to the erasure of that processor's 
> existing cluster states. 
> This bug was introduced in commit 
> [https://github.com/apache/nifi/commit/8b9d4461185848fd552a639ac14b7926164d5426]
>  .
> Before that commit, the {{resetState}} variable is changed to false after the 
> state is clear, inside {{updateState}} method. However, the commit moved this 
> line to inside the {{onTrigger}} method. This is problematic, because 
> {{onTrigger}} method is only called on the primary node. As a result, the 
> {{resetState}} variable is only reset to false on primary node, but not on 
> the other nodes. When the user restarts the processor,  every nodes will 
> re-execute the {{updateState}} method and the Cluster state will be cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-681) Add content hash processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702937#comment-16702937
 ] 

ASF GitHub Bot commented on MINIFICPP-681:
--

Github user arpadboda commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/445#discussion_r237412934
  
--- Diff: libminifi/src/processors/HashContent.cpp ---
@@ -0,0 +1,100 @@
+/**
+ * @file HashContent.cpp
+ * HashContent class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifdef OPENSSL_SUPPORT
+
+#include 
+#include 
+#include 
+#include 
+#include "processors/HashContent.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "core/FlowFile.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property HashContent::HashAttribute("Hash Attribute", "Attribute to 
store checksum to", "Checksum");
+core::Property HashContent::HashAlgorithm("Hash Algorithm", "Name of the 
algorithm used to generate checksum", "MD5");
+core::Relationship HashContent::Success("success", "success operational on 
the flow record");
+
+void HashContent::initialize() {
+  //! Set the supported properties
+  std::set properties;
+  properties.insert(HashAttribute);
+  properties.insert(HashAlgorithm);
+  setSupportedProperties(properties);
+  //! Set the supported relationships
+  std::set relationships;
+  relationships.insert(Success);
+  setSupportedRelationships(relationships);
+}
+
+void HashContent::onSchedule(core::ProcessContext *context, 
core::ProcessSessionFactory *sessionFactory) {
+  std::string value;
+
+  attrKey_ = (context->getProperty(HashAttribute.getName(), value)) ? 
value : "Checksum";
+  algoName_ = (context->getProperty(HashAlgorithm.getName(), value)) ? 
value : "MD5";
--- End diff --

Sure, changed. 


> Add content hash processor
> --
>
> Key: MINIFICPP-681
> URL: https://issues.apache.org/jira/browse/MINIFICPP-681
> Project: NiFi MiNiFi C++
>  Issue Type: Improvement
>Reporter: Arpad Boda
>Assignee: Arpad Boda
>Priority: Major
> Fix For: 0.6.0
>
>
> Add a new processor that supports hashing content and add the checksum to the 
> flowfile as an attribute.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MINIFICPP-681) Add content hash processor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/MINIFICPP-681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702936#comment-16702936
 ] 

ASF GitHub Bot commented on MINIFICPP-681:
--

Github user arpadboda commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/445#discussion_r237412844
  
--- Diff: libminifi/include/processors/HashContent.h ---
@@ -0,0 +1,196 @@
+/**
+ * @file HashContent.h
+ * HashContent class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef NIFI_MINIFI_CPP_HashContent_H
+#define NIFI_MINIFI_CPP_HashContent_H
+
+#ifdef OPENSSL_SUPPORT
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include 
+#include 
+
+#include "FlowFileRecord.h"
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/Resource.h"
+#include "io/BaseStream.h"
+
+using HashReturnType = std::pair;
+
+namespace {
+#define HASH_BUFFER_SIZE 16384
+
+  std::string digestToString(const unsigned char * const digest, size_t 
size) {
+std::stringstream ss;
+for(int i = 0; i < size; i++)
+{
+  ss << std::uppercase << std::hex << std::setw(2) << 
std::setfill('0') << (int)digest[i];
+}
+return ss.str();
+  }
+
+  HashReturnType MD5Hash(const 
std::shared_ptr& stream) {
+HashReturnType ret_val;
+ret_val.second = 0;
+uint8_t buffer[HASH_BUFFER_SIZE];
+MD5_CTX context;
+MD5_Init();
+
+size_t ret = 0;
+do {
+  ret = stream->readData(buffer, HASH_BUFFER_SIZE);
+  if(ret > 0) {
+MD5_Update(, buffer, ret);
+ret_val.second += ret;
+  }
+} while(ret > 0);
+
+if (ret_val.second > 0) {
+  unsigned char digest[MD5_DIGEST_LENGTH];
+  MD5_Final(digest, );
+  ret_val.first = digestToString(digest, MD5_DIGEST_LENGTH);
+}
+return ret_val;
+  }
+
+  HashReturnType SHA1Hash(const 
std::shared_ptr& stream) {
+HashReturnType ret_val;
+ret_val.second = 0;
+uint8_t buffer[HASH_BUFFER_SIZE];
+SHA_CTX context;
+SHA1_Init();
+
+size_t ret = 0;
+do {
+  ret = stream->readData(buffer, HASH_BUFFER_SIZE);
+  if(ret > 0) {
+SHA1_Update(, buffer, ret);
+ret_val.second += ret;
+  }
+} while(ret > 0);
+
+if (ret_val.second > 0) {
+  unsigned char digest[SHA_DIGEST_LENGTH];
+  SHA1_Final(digest, );
+  ret_val.first = digestToString(digest, SHA_DIGEST_LENGTH);
+}
+return ret_val;
+  }
+
+  HashReturnType SHA256Hash(const 
std::shared_ptr& stream) {
+HashReturnType ret_val;
+ret_val.second = 0;
+uint8_t buffer[HASH_BUFFER_SIZE];
+SHA256_CTX context;
+SHA256_Init();
+
+size_t ret ;
+do {
+  ret = stream->readData(buffer, HASH_BUFFER_SIZE);
+  if(ret > 0) {
+SHA256_Update(, buffer, ret);
+ret_val.second += ret;
+  }
+} while(ret > 0);
+
+if (ret_val.second > 0) {
+  unsigned char digest[SHA256_DIGEST_LENGTH];
+  SHA256_Final(digest, );
+  ret_val.first = digestToString(digest, SHA256_DIGEST_LENGTH);
+}
+return ret_val;
+  }
+}
+
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+static const std::map&)>> 
HashAlgos =
+  { {"MD5",  MD5Hash}, {"SHA1", SHA1Hash}, {"SHA256", SHA256Hash} };
+
+//! HashContent Class
+class HashContent : public core::Processor {
+ public:
+  //! Constructor
+  /*!
+  * Create a new processor
+  */
+  explicit HashContent(std::string name,  utils::Identifier uuid = 
utils::Identifier())
+  : 

[GitHub] nifi-minifi-cpp pull request #445: MINIFICPP-681 - Add content hash processo...

[arpadboda]

Github user arpadboda commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/445#discussion_r237412934
  
--- Diff: libminifi/src/processors/HashContent.cpp ---
@@ -0,0 +1,100 @@
+/**
+ * @file HashContent.cpp
+ * HashContent class implementation
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifdef OPENSSL_SUPPORT
+
+#include 
+#include 
+#include 
+#include 
+#include "processors/HashContent.h"
+#include "core/ProcessContext.h"
+#include "core/ProcessSession.h"
+#include "core/FlowFile.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+core::Property HashContent::HashAttribute("Hash Attribute", "Attribute to 
store checksum to", "Checksum");
+core::Property HashContent::HashAlgorithm("Hash Algorithm", "Name of the 
algorithm used to generate checksum", "MD5");
+core::Relationship HashContent::Success("success", "success operational on 
the flow record");
+
+void HashContent::initialize() {
+  //! Set the supported properties
+  std::set properties;
+  properties.insert(HashAttribute);
+  properties.insert(HashAlgorithm);
+  setSupportedProperties(properties);
+  //! Set the supported relationships
+  std::set relationships;
+  relationships.insert(Success);
+  setSupportedRelationships(relationships);
+}
+
+void HashContent::onSchedule(core::ProcessContext *context, 
core::ProcessSessionFactory *sessionFactory) {
+  std::string value;
+
+  attrKey_ = (context->getProperty(HashAttribute.getName(), value)) ? 
value : "Checksum";
+  algoName_ = (context->getProperty(HashAlgorithm.getName(), value)) ? 
value : "MD5";
--- End diff --

Sure, changed. 


---

[GitHub] nifi-minifi-cpp pull request #445: MINIFICPP-681 - Add content hash processo...

[arpadboda]

Github user arpadboda commented on a diff in the pull request:

https://github.com/apache/nifi-minifi-cpp/pull/445#discussion_r237412844
  
--- Diff: libminifi/include/processors/HashContent.h ---
@@ -0,0 +1,196 @@
+/**
+ * @file HashContent.h
+ * HashContent class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef NIFI_MINIFI_CPP_HashContent_H
+#define NIFI_MINIFI_CPP_HashContent_H
+
+#ifdef OPENSSL_SUPPORT
+
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#include 
+#include 
+
+#include "FlowFileRecord.h"
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/Resource.h"
+#include "io/BaseStream.h"
+
+using HashReturnType = std::pair;
+
+namespace {
+#define HASH_BUFFER_SIZE 16384
+
+  std::string digestToString(const unsigned char * const digest, size_t 
size) {
+std::stringstream ss;
+for(int i = 0; i < size; i++)
+{
+  ss << std::uppercase << std::hex << std::setw(2) << 
std::setfill('0') << (int)digest[i];
+}
+return ss.str();
+  }
+
+  HashReturnType MD5Hash(const 
std::shared_ptr& stream) {
+HashReturnType ret_val;
+ret_val.second = 0;
+uint8_t buffer[HASH_BUFFER_SIZE];
+MD5_CTX context;
+MD5_Init();
+
+size_t ret = 0;
+do {
+  ret = stream->readData(buffer, HASH_BUFFER_SIZE);
+  if(ret > 0) {
+MD5_Update(, buffer, ret);
+ret_val.second += ret;
+  }
+} while(ret > 0);
+
+if (ret_val.second > 0) {
+  unsigned char digest[MD5_DIGEST_LENGTH];
+  MD5_Final(digest, );
+  ret_val.first = digestToString(digest, MD5_DIGEST_LENGTH);
+}
+return ret_val;
+  }
+
+  HashReturnType SHA1Hash(const 
std::shared_ptr& stream) {
+HashReturnType ret_val;
+ret_val.second = 0;
+uint8_t buffer[HASH_BUFFER_SIZE];
+SHA_CTX context;
+SHA1_Init();
+
+size_t ret = 0;
+do {
+  ret = stream->readData(buffer, HASH_BUFFER_SIZE);
+  if(ret > 0) {
+SHA1_Update(, buffer, ret);
+ret_val.second += ret;
+  }
+} while(ret > 0);
+
+if (ret_val.second > 0) {
+  unsigned char digest[SHA_DIGEST_LENGTH];
+  SHA1_Final(digest, );
+  ret_val.first = digestToString(digest, SHA_DIGEST_LENGTH);
+}
+return ret_val;
+  }
+
+  HashReturnType SHA256Hash(const 
std::shared_ptr& stream) {
+HashReturnType ret_val;
+ret_val.second = 0;
+uint8_t buffer[HASH_BUFFER_SIZE];
+SHA256_CTX context;
+SHA256_Init();
+
+size_t ret ;
+do {
+  ret = stream->readData(buffer, HASH_BUFFER_SIZE);
+  if(ret > 0) {
+SHA256_Update(, buffer, ret);
+ret_val.second += ret;
+  }
+} while(ret > 0);
+
+if (ret_val.second > 0) {
+  unsigned char digest[SHA256_DIGEST_LENGTH];
+  SHA256_Final(digest, );
+  ret_val.first = digestToString(digest, SHA256_DIGEST_LENGTH);
+}
+return ret_val;
+  }
+}
+
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace processors {
+
+static const std::map&)>> 
HashAlgos =
+  { {"MD5",  MD5Hash}, {"SHA1", SHA1Hash}, {"SHA256", SHA256Hash} };
+
+//! HashContent Class
+class HashContent : public core::Processor {
+ public:
+  //! Constructor
+  /*!
+  * Create a new processor
+  */
+  explicit HashContent(std::string name,  utils::Identifier uuid = 
utils::Identifier())
+  : Processor(name, uuid)
+  {
+logger_ = logging::LoggerFactory::getLogger();
+  }
+  //! Processor Name
+  static constexpr char const* ProcessorName = "HashContent";
+  //! Supported Properties
+  static core::Property 

[jira] [Updated] (NIFI-5849) ListSFTP, ListFTP processors lose Cluster state on restart.

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-5849:
-
Component/s: Extensions

> ListSFTP, ListFTP processors lose Cluster state on restart.
> ---
>
> Key: NIFI-5849
> URL: https://issues.apache.org/jira/browse/NIFI-5849
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.8.0
>Reporter: Truong Duc Kien
>Assignee: Koji Kawamura
>Priority: Critical
> Fix For: 1.9.0
>
>
> In a cluster environment, restarting a processor that extends 
> {{AbstractListProcessor}} could lead to the erasure of that processor's 
> existing cluster states. 
> This bug was introduced in commit 
> [https://github.com/apache/nifi/commit/8b9d4461185848fd552a639ac14b7926164d5426]
>  .
> Before that commit, the {{resetState}} variable is changed to false after the 
> state is clear, inside {{updateState}} method. However, the commit moved this 
> line to inside the {{onTrigger}} method. This is problematic, because 
> {{onTrigger}} method is only called on the primary node. As a result, the 
> {{resetState}} variable is only reset to false on primary node, but not on 
> the other nodes. When the user restarts the processor,  every nodes will 
> re-execute the {{updateState}} method and the Cluster state will be cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5849) ListSFTP, ListFTP processors lose Cluster state on restart.

[Pierre Villard (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pierre Villard updated NIFI-5849:
-
   Resolution: Fixed
Fix Version/s: 1.9.0
   Status: Resolved  (was: Patch Available)

> ListSFTP, ListFTP processors lose Cluster state on restart.
> ---
>
> Key: NIFI-5849
> URL: https://issues.apache.org/jira/browse/NIFI-5849
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.8.0
>Reporter: Truong Duc Kien
>Assignee: Koji Kawamura
>Priority: Critical
> Fix For: 1.9.0
>
>
> In a cluster environment, restarting a processor that extends 
> {{AbstractListProcessor}} could lead to the erasure of that processor's 
> existing cluster states. 
> This bug was introduced in commit 
> [https://github.com/apache/nifi/commit/8b9d4461185848fd552a639ac14b7926164d5426]
>  .
> Before that commit, the {{resetState}} variable is changed to false after the 
> state is clear, inside {{updateState}} method. However, the commit moved this 
> line to inside the {{onTrigger}} method. This is problematic, because 
> {{onTrigger}} method is only called on the primary node. As a result, the 
> {{resetState}} variable is only reset to false on primary node, but not on 
> the other nodes. When the user restarts the processor,  every nodes will 
> re-execute the {{updateState}} method and the Cluster state will be cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5406) Add new listing strategy by tracking listed entities to ListXXXX processors

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702923#comment-16702923
 ] 

ASF subversion and git services commented on NIFI-5406:
---

Commit 30f2f4205121113c26bb00ac5a8697dffaeb8206 in nifi's branch 
refs/heads/master from [~ijokarumawak]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=30f2f42 ]

NIFI-5849: ListXXX can lose cluster state on processor restart

NIFI-5406 introduced the issue by trying to use the resetState variable for
different purposes. AbstractListProcessor should have had a different variable
to control whether to clear state for tracking entity strategy.

Signed-off-by: Pierre Villard 

This closes #3189.


> Add new listing strategy by tracking listed entities to List processors
> ---
>
> Key: NIFI-5406
> URL: https://issues.apache.org/jira/browse/NIFI-5406
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
> Fix For: 1.8.0
>
>
> Current List processors (ListFile, ListFTP, ListSFTP ... etc) implementation 
> relies on file last modified timestamp to pick new or updated files. This 
> approach is efficient and lightweight in terms of state management, because 
> it only tracks latest modified timestamp and last executed timestamp. 
> However, timestamps do not work as expected in some file systems, causing 
> List processors missing files periodically. See NIFI-3332 comments for 
> details.
> In order to pick every entity that has not seen before or has been updated 
> since it had seen last time, we need another set of processors using 
> different approach, that is by tracking listed entities:
>  * Add new abstract processor AbstractWatchEntries similar to 
> AbstractListProcessor but uses different approach
>  * Target entities have: name (path), size and last-modified-timestamp
>  * Implementation Processors have following properties:
>  ** 'Watch Time Window' to limit the maximum time period to hold the already 
> listed entries. E.g. if set as '30min', the processor keeps entities listed 
> in the last 30 mins.
>  ** 'Minimum File Age' to defer listing entities potentially being written
>  * Any entity added but not listed ever having last-modified-timestamp older 
> than configured 'Watch Time Window' will not be listed. If user needs to pick 
> these items, they have to make 'Watch Time Window' longer. It also increases 
> the size of data the processor has to persist in the K/V store. Efficiency vs 
> reliability trade-off.
>  * The already-listed entities are persisted into one of supported K/V store 
> through DistributedMapCacheClient service. User can chose what KVS to use 
> from HBase, Redis, Couchbase and File (DistributedMapCacheServer with 
> persistence file).
>  * The reason to use KVS instead of ManagedState is, to avoid hammering 
> Zookeeper too much with frequently updating Zk node with large amount of 
> data. The number of already-listed entries can be huge depending on 
> use-cases. Also, we can compress entities with DistributedMapCacheClient as 
> it supports putting byte array, while ManagedState only supports Map String>.
>  * On each onTrigger:
>  ** Processor performs listing. Listed entries meeting any of the following 
> condition will be written to the 'success' output FlowFile:
>  *** Not exists in the already-listed entities
>  *** Having newer last-modified-timestamp
>  *** Having different size
>  ** Already listed entries those are old enough compared to 'Watch Time 
> Window' are discarded from the already-listed entries.
>  * Initial supporting target is Local file system, FTP and SFTP



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5849) ListSFTP, ListFTP processors lose Cluster state on restart.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702922#comment-16702922
 ] 

ASF subversion and git services commented on NIFI-5849:
---

Commit 30f2f4205121113c26bb00ac5a8697dffaeb8206 in nifi's branch 
refs/heads/master from [~ijokarumawak]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=30f2f42 ]

NIFI-5849: ListXXX can lose cluster state on processor restart

NIFI-5406 introduced the issue by trying to use the resetState variable for
different purposes. AbstractListProcessor should have had a different variable
to control whether to clear state for tracking entity strategy.

Signed-off-by: Pierre Villard 

This closes #3189.


> ListSFTP, ListFTP processors lose Cluster state on restart.
> ---
>
> Key: NIFI-5849
> URL: https://issues.apache.org/jira/browse/NIFI-5849
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Truong Duc Kien
>Assignee: Koji Kawamura
>Priority: Critical
>
> In a cluster environment, restarting a processor that extends 
> {{AbstractListProcessor}} could lead to the erasure of that processor's 
> existing cluster states. 
> This bug was introduced in commit 
> [https://github.com/apache/nifi/commit/8b9d4461185848fd552a639ac14b7926164d5426]
>  .
> Before that commit, the {{resetState}} variable is changed to false after the 
> state is clear, inside {{updateState}} method. However, the commit moved this 
> line to inside the {{onTrigger}} method. This is problematic, because 
> {{onTrigger}} method is only called on the primary node. As a result, the 
> {{resetState}} variable is only reset to false on primary node, but not on 
> the other nodes. When the user restarts the processor,  every nodes will 
> re-execute the {{updateState}} method and the Cluster state will be cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #3189: NIFI-5849: ListXXX can lose cluster state on proces...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/3189


---

[jira] [Commented] (NIFI-5850) Replace custom AWS Region Enums with AWS SDK Region enums

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702919#comment-16702919
 ] 

ASF GitHub Bot commented on NIFI-5850:
--

GitHub user zenfenan opened a pull request:

https://github.com/apache/nifi/pull/3190

NIFI-5850: Replaced custom AWS regions enum with the one from AWS Jav…

…a SDK

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zenfenan/nifi NIFI-5850

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/3190.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3190


commit 0688425f971d14e258a4955cd2d2ed7046c28466
Author: zenfenan 
Date:   2018-11-29T09:29:23Z

NIFI-5850: Replaced custom AWS regions enum with the one from AWS Java SDK




> Replace custom AWS Region Enums with AWS SDK Region enums
> -
>
> Key: NIFI-5850
> URL: https://issues.apache.org/jira/browse/NIFI-5850
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.6.0, 1.7.0, 1.8.0, 1.7.1
>Reporter: Sivaprasanna Sethuraman
>Assignee: Sivaprasanna Sethuraman
>Priority: Major
>
> With NIFI-5129, we introduced a custom Enum for displaying readable AWS 
> regions. This readable names have now been added to the official AWS Java SDK 
> so ideally this custom AWS Regions enum have to removed and the region names 
> should be read from the AWS SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5849) ListSFTP, ListFTP processors lose Cluster state on restart.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702918#comment-16702918
 ] 

ASF GitHub Bot commented on NIFI-5849:
--

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/3189
  
+1, code change LGTM, merging to master, thanks @ijokarumawak 


> ListSFTP, ListFTP processors lose Cluster state on restart.
> ---
>
> Key: NIFI-5849
> URL: https://issues.apache.org/jira/browse/NIFI-5849
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Truong Duc Kien
>Assignee: Koji Kawamura
>Priority: Critical
>
> In a cluster environment, restarting a processor that extends 
> {{AbstractListProcessor}} could lead to the erasure of that processor's 
> existing cluster states. 
> This bug was introduced in commit 
> [https://github.com/apache/nifi/commit/8b9d4461185848fd552a639ac14b7926164d5426]
>  .
> Before that commit, the {{resetState}} variable is changed to false after the 
> state is clear, inside {{updateState}} method. However, the commit moved this 
> line to inside the {{onTrigger}} method. This is problematic, because 
> {{onTrigger}} method is only called on the primary node. As a result, the 
> {{resetState}} variable is only reset to false on primary node, but not on 
> the other nodes. When the user restarts the processor,  every nodes will 
> re-execute the {{updateState}} method and the Cluster state will be cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #3190: NIFI-5850: Replaced custom AWS regions enum with th...

[zenfenan]

GitHub user zenfenan opened a pull request:

https://github.com/apache/nifi/pull/3190

NIFI-5850: Replaced custom AWS regions enum with the one from AWS Jav…

…a SDK

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zenfenan/nifi NIFI-5850

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/3190.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3190


commit 0688425f971d14e258a4955cd2d2ed7046c28466
Author: zenfenan 
Date:   2018-11-29T09:29:23Z

NIFI-5850: Replaced custom AWS regions enum with the one from AWS Java SDK




---

[GitHub] nifi issue #3189: NIFI-5849: ListXXX can lose cluster state on processor res...

[pvillard31]

Github user pvillard31 commented on the issue:

https://github.com/apache/nifi/pull/3189
  
+1, code change LGTM, merging to master, thanks @ijokarumawak 


---

[jira] [Commented] (NIFI-5838) Misconfigured Kite processors can block NiFi

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702911#comment-16702911
 ] 

ASF GitHub Bot commented on NIFI-5838:
--

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3182#discussion_r237407789
  
--- Diff: 
nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/AbstractKiteProcessor.java
 ---
@@ -101,38 +100,38 @@ protected static Schema getSchema(String 
uriOrLiteral, Configuration conf) {
 return parseSchema(uriOrLiteral);
 }
 
+if(uri.getScheme() == null) {
+throw new SchemaNotFoundException("If the schema is not a JSON 
string, a scheme must be specified in the URI "
++ "(ex: dataset:, view:, resource:, file:, hdfs:, 
etc).");
+}
+
 try {
 if ("dataset".equals(uri.getScheme()) || 
"view".equals(uri.getScheme())) {
 return 
Datasets.load(uri).getDataset().getDescriptor().getSchema();
 } else if ("resource".equals(uri.getScheme())) {
-try (InputStream in = 
Resources.getResource(uri.getSchemeSpecificPart())
-.openStream()) {
+try (InputStream in = 
Resources.getResource(uri.getSchemeSpecificPart()).openStream()) {
 return parseSchema(uri, in);
 }
 } else {
 // try to open the file
 Path schemaPath = new Path(uri);
-FileSystem fs = schemaPath.getFileSystem(conf);
-try (InputStream in = fs.open(schemaPath)) {
+try (InputStream in = 
schemaPath.getFileSystem(conf).open(schemaPath)) {
--- End diff --

Done - thanks @ijokarumawak 


> Misconfigured Kite processors can block NiFi
> 
>
> Key: NIFI-5838
> URL: https://issues.apache.org/jira/browse/NIFI-5838
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Reporter: Pierre Villard
>Assignee: Pierre Villard
>Priority: Critical
>
> I faced the following situation today: no way to access the NiFi UI (it was 
> just hanging forever or throwing timeouts), no change in case of NiFi 
> restart, even with disabling the auto resume feature.
> By looking at the thread dump, I found a lot of:
> {noformat}
> "NiFi Web Server-142" Id=142 BLOCKED  on 
> org.apache.hadoop.ipc.Client$Connection@3843e7a6
>     at org.apache.hadoop.ipc.Client$Connection.addCall(Client.java:463)
>     at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:375)
>     at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522)
>     at org.apache.hadoop.ipc.Client.call(Client.java:1451)
>     at org.apache.hadoop.ipc.Client.call(Client.java:1412)
>     at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
>     at com.sun.proxy.$Proxy348.getBlockLocations(Unknown Source)
>     at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:255)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
>     at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
>     at com.sun.proxy.$Proxy349.getBlockLocations(Unknown Source)
>     at 
> org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1226)
>     at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1213)
>     at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1201)
>     at 
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:306)
>     at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:272)
>     - waiting on java.lang.Object@46d84263
>     at org.apache.hadoop.hdfs.DFSInputStream.(DFSInputStream.java:264)
>     at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1526)
>     at 
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
>     at 
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:299)
>     at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>     at 
> 

[GitHub] nifi pull request #3182: NIFI-5838 - Improve the schema validation method in...

[pvillard31]

Github user pvillard31 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/3182#discussion_r237407789
  
--- Diff: 
nifi-nar-bundles/nifi-kite-bundle/nifi-kite-processors/src/main/java/org/apache/nifi/processors/kite/AbstractKiteProcessor.java
 ---
@@ -101,38 +100,38 @@ protected static Schema getSchema(String 
uriOrLiteral, Configuration conf) {
 return parseSchema(uriOrLiteral);
 }
 
+if(uri.getScheme() == null) {
+throw new SchemaNotFoundException("If the schema is not a JSON 
string, a scheme must be specified in the URI "
++ "(ex: dataset:, view:, resource:, file:, hdfs:, 
etc).");
+}
+
 try {
 if ("dataset".equals(uri.getScheme()) || 
"view".equals(uri.getScheme())) {
 return 
Datasets.load(uri).getDataset().getDescriptor().getSchema();
 } else if ("resource".equals(uri.getScheme())) {
-try (InputStream in = 
Resources.getResource(uri.getSchemeSpecificPart())
-.openStream()) {
+try (InputStream in = 
Resources.getResource(uri.getSchemeSpecificPart()).openStream()) {
 return parseSchema(uri, in);
 }
 } else {
 // try to open the file
 Path schemaPath = new Path(uri);
-FileSystem fs = schemaPath.getFileSystem(conf);
-try (InputStream in = fs.open(schemaPath)) {
+try (InputStream in = 
schemaPath.getFileSystem(conf).open(schemaPath)) {
--- End diff --

Done - thanks @ijokarumawak 


---

[GitHub] nifi issue #3009: NIFI-4532 OpenID Connect User Authentication

[SarthakSahu]

Github user SarthakSahu commented on the issue:

https://github.com/apache/nifi/pull/3009
  
@apiri could you please delegate this to somebody for review.


---

[jira] [Updated] (NIFI-5849) ListSFTP, ListFTP processors lose Cluster state on restart.

[Koji Kawamura (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura updated NIFI-5849:

Status: Patch Available  (was: In Progress)

> ListSFTP, ListFTP processors lose Cluster state on restart.
> ---
>
> Key: NIFI-5849
> URL: https://issues.apache.org/jira/browse/NIFI-5849
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Truong Duc Kien
>Assignee: Koji Kawamura
>Priority: Critical
>
> In a cluster environment, restarting a processor that extends 
> {{AbstractListProcessor}} could lead to the erasure of that processor's 
> existing cluster states. 
> This bug was introduced in commit 
> [https://github.com/apache/nifi/commit/8b9d4461185848fd552a639ac14b7926164d5426]
>  .
> Before that commit, the {{resetState}} variable is changed to false after the 
> state is clear, inside {{updateState}} method. However, the commit moved this 
> line to inside the {{onTrigger}} method. This is problematic, because 
> {{onTrigger}} method is only called on the primary node. As a result, the 
> {{resetState}} variable is only reset to false on primary node, but not on 
> the other nodes. When the user restarts the processor,  every nodes will 
> re-execute the {{updateState}} method and the Cluster state will be cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (NIFI-5849) ListSFTP, ListFTP processors lose Cluster state on restart.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702872#comment-16702872
 ] 

ASF GitHub Bot commented on NIFI-5849:
--

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/3189

NIFI-5849: ListXXX can lose cluster state on processor restart

NIFI-5406 introduced the issue by trying to use the resetState variable for
different purposes. AbstractListProcessor should have had a different 
variable
to control whether to clear state for tracking entity strategy.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-5849

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/3189.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3189


commit 6dac106dc1c8c946db9dce7a5be9b7769a7ea6c8
Author: Koji Kawamura 
Date:   2018-11-29T08:44:44Z

NIFI-5849: ListXXX can lose cluster state on processor restart

NIFI-5406 introduced the issue by trying to use the resetState variable for
different purposes. AbstractListProcessor should have had a different 
variable
to control whether to clear state for tracking entity strategy.




> ListSFTP, ListFTP processors lose Cluster state on restart.
> ---
>
> Key: NIFI-5849
> URL: https://issues.apache.org/jira/browse/NIFI-5849
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Truong Duc Kien
>Assignee: Koji Kawamura
>Priority: Critical
>
> In a cluster environment, restarting a processor that extends 
> {{AbstractListProcessor}} could lead to the erasure of that processor's 
> existing cluster states. 
> This bug was introduced in commit 
> [https://github.com/apache/nifi/commit/8b9d4461185848fd552a639ac14b7926164d5426]
>  .
> Before that commit, the {{resetState}} variable is changed to false after the 
> state is clear, inside {{updateState}} method. However, the commit moved this 
> line to inside the {{onTrigger}} method. This is problematic, because 
> {{onTrigger}} method is only called on the primary node. As a result, the 
> {{resetState}} variable is only reset to false on primary node, but not on 
> the other nodes. When the user restarts the processor,  every nodes will 
> re-execute the {{updateState}} method and the Cluster state will be cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] nifi pull request #3189: NIFI-5849: ListXXX can lose cluster state on proces...

[ijokarumawak]

GitHub user ijokarumawak opened a pull request:

https://github.com/apache/nifi/pull/3189

NIFI-5849: ListXXX can lose cluster state on processor restart

NIFI-5406 introduced the issue by trying to use the resetState variable for
different purposes. AbstractListProcessor should have had a different 
variable
to control whether to clear state for tracking entity strategy.

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [x] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [x] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijokarumawak/nifi nifi-5849

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/3189.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3189


commit 6dac106dc1c8c946db9dce7a5be9b7769a7ea6c8
Author: Koji Kawamura 
Date:   2018-11-29T08:44:44Z

NIFI-5849: ListXXX can lose cluster state on processor restart

NIFI-5406 introduced the issue by trying to use the resetState variable for
different purposes. AbstractListProcessor should have had a different 
variable
to control whether to clear state for tracking entity strategy.




---

[jira] [Commented] (NIFI-5849) ListSFTP, ListFTP processors lose Cluster state on restart.

[Koji Kawamura (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702851#comment-16702851
 ] 

Koji Kawamura commented on NIFI-5849:
-

[~kien_truong] Thank you very much for finding this. I'll fix this shortly.

> ListSFTP, ListFTP processors lose Cluster state on restart.
> ---
>
> Key: NIFI-5849
> URL: https://issues.apache.org/jira/browse/NIFI-5849
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Truong Duc Kien
>Assignee: Koji Kawamura
>Priority: Critical
>
> In a cluster environment, restarting a processor that extends 
> {{AbstractListProcessor}} could lead to the erasure of that processor's 
> existing cluster states. 
> This bug was introduced in commit 
> [https://github.com/apache/nifi/commit/8b9d4461185848fd552a639ac14b7926164d5426]
>  .
> Before that commit, the {{resetState}} variable is changed to false after the 
> state is clear, inside {{updateState}} method. However, the commit moved this 
> line to inside the {{onTrigger}} method. This is problematic, because 
> {{onTrigger}} method is only called on the primary node. As a result, the 
> {{resetState}} variable is only reset to false on primary node, but not on 
> the other nodes. When the user restarts the processor,  every nodes will 
> re-execute the {{updateState}} method and the Cluster state will be cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (NIFI-5850) Replace custom AWS Region Enums with AWS SDK Region enums

[Sivaprasanna Sethuraman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sivaprasanna Sethuraman updated NIFI-5850:
--
Description: With NIFI-5129, we introduced a custom Enum for displaying 
readable AWS regions. This readable names have now been added to the official 
AWS Java SDK so ideally this custom AWS Regions enum have to removed and the 
region names should be read from the AWS SDK.  (was: With NIFI-5129, we 
introduced a custom Enum for displaying readable AWS regions. This readable 
names have now been added to the official AWS Java SDK so ideally this custom 
AWS REgions Enum have to removed and the region names should be read from the 
AWS SDK.)

> Replace custom AWS Region Enums with AWS SDK Region enums
> -
>
> Key: NIFI-5850
> URL: https://issues.apache.org/jira/browse/NIFI-5850
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 1.6.0, 1.7.0, 1.8.0, 1.7.1
>Reporter: Sivaprasanna Sethuraman
>Assignee: Sivaprasanna Sethuraman
>Priority: Major
>
> With NIFI-5129, we introduced a custom Enum for displaying readable AWS 
> regions. This readable names have now been added to the official AWS Java SDK 
> so ideally this custom AWS Regions enum have to removed and the region names 
> should be read from the AWS SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (NIFI-5849) ListSFTP, ListFTP processors lose Cluster state on restart.

[Koji Kawamura (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-5849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Kawamura reassigned NIFI-5849:
---

Assignee: Koji Kawamura

> ListSFTP, ListFTP processors lose Cluster state on restart.
> ---
>
> Key: NIFI-5849
> URL: https://issues.apache.org/jira/browse/NIFI-5849
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Truong Duc Kien
>Assignee: Koji Kawamura
>Priority: Critical
>
> In a cluster environment, restarting a processor that extends 
> {{AbstractListProcessor}} could lead to the erasure of that processor's 
> existing cluster states. 
> This bug was introduced in commit 
> [https://github.com/apache/nifi/commit/8b9d4461185848fd552a639ac14b7926164d5426]
>  .
> Before that commit, the {{resetState}} variable is changed to false after the 
> state is clear, inside {{updateState}} method. However, the commit moved this 
> line to inside the {{onTrigger}} method. This is problematic, because 
> {{onTrigger}} method is only called on the primary node. As a result, the 
> {{resetState}} variable is only reset to false on primary node, but not on 
> the other nodes. When the user restarts the processor,  every nodes will 
> re-execute the {{updateState}} method and the Cluster state will be cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (NIFI-5851) Add MFA to *S3Object Processors

[Lukas Heusser (JIRA)]

Lukas Heusser created NIFI-5851:
---

 Summary: Add MFA to *S3Object Processors
 Key: NIFI-5851
 URL: https://issues.apache.org/jira/browse/NIFI-5851
 Project: Apache NiFi
  Issue Type: New Feature
Reporter: Lukas Heusser


Add the possibility to add MFA to the *S3Object processors. Due to security 
this is a quiet important feature for customers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)