[jira] [Commented] (NIFI-3026) S2S initial connection behavior enhancement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678709#comment-15678709
 ] 

ASF GitHub Bot commented on NIFI-3026:
--

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/1208
  
@markap14 I'm still not 100% sure if it addresses your concern.. but I've 
updated the code. 

RemoteProcessGroup has `getTargetUri` method that returns a String. I 
wondered if we should do the same thing with this, by adding `getTargetUris`, 
but I left it as it is. Because if we do, we may want to change 
RemoteProcessGroupDTO as well, it will change NiFi Web REST API data format..

Local contrib check has passed, and now I'm testing it with running NiFi 
environments. It will take a while to go through a variety of S2S tests. But I 
think it's ready for code review.

Would you take a look and give me feedback?


> S2S initial connection behavior enhancement
> ---
>
> Key: NIFI-3026
> URL: https://issues.apache.org/jira/browse/NIFI-3026
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework, Core UI
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>
> s2s client behavior and initial connection improvement is needed.
> Current experience is this: I, as a client (e.g. minifi), connect to a nifi 
> cluster of e.g. 10 nodes. but i need to specify 1 node URL to establish this 
> connection. this node may not be available 100% and go down, in which case my 
> initial connection won't work.
> Once S2S makes the first connection, it then has a list of all nodes, and can 
> check their status. But first connection failure would be a concern if the 
> specified URL is somehow not working. Usually for these problems, the client 
> should be able to specify multiple urls (according to multiple target cluster 
> nodes), comma-separated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1208: NIFI-3026: Support multiple remote target URLs

[ijokarumawak]

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/1208
  
@markap14 I'm still not 100% sure if it addresses your concern.. but I've 
updated the code. 

RemoteProcessGroup has `getTargetUri` method that returns a String. I 
wondered if we should do the same thing with this, by adding `getTargetUris`, 
but I left it as it is. Because if we do, we may want to change 
RemoteProcessGroupDTO as well, it will change NiFi Web REST API data format..

Local contrib check has passed, and now I'm testing it with running NiFi 
environments. It will take a while to go through a variety of S2S tests. But I 
think it's ready for code review.

Would you take a look and give me feedback?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1236: LDAP - Configurable strategy to identify users

[ijokarumawak]

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/1236
  
@mcgilman Thanks for updating the docs! Exception handling at shutdown 
looks good. +1
Would you squash the commits to get it merged?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-949) Allow configuration of cleanup resources

[Ben Icore (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678495#comment-15678495
 ] 

Ben Icore commented on NIFI-949:


This is still very much an issue even with 0.7.1  if the system has been 
running under any kind of load for even a few hours, there is a significant 
amount of repo cleanup that happens when the system is restarted.

> Allow configuration of cleanup resources
> 
>
> Key: NIFI-949
> URL: https://issues.apache.org/jira/browse/NIFI-949
> Project: Apache NiFi
>  Issue Type: Improvement
>Reporter: Brandon DeVries
>Priority: Minor
>
> Allow the allocation of resource to the cleanup of the content and provenance 
> repositories to be configurable.   There are times at which the resources 
> spent performing cleanup activities can degrade the operation of the flow 
> sufficiently to cause problems.  Sometimes this requires removing the 
> offending node from the cluster to allow it to clean up without taking on new 
> data (and thus backing up).  Making the cleanup resources configurable could 
> prevent such a situation by allowing them to be reduced under times of heavy 
> load to keep the node useable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3069) Error clearing content_repository causes app to fail when configured securely

[Andy LoPresto (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678468#comment-15678468
 ] 

Andy LoPresto commented on NIFI-3069:
-

We discussed and [~JDP10101] was running on an existing cluster. He'll try to 
reproduce by Monday. 

> Error clearing content_repository causes app to fail when configured securely
> -
>
> Key: NIFI-3069
> URL: https://issues.apache.org/jira/browse/NIFI-3069
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Andy LoPresto
>Priority: Blocker
>  Labels: content-repository, logging
> Fix For: 1.1.0
>
>
> An issue was introduced into {{master}} (I believe commit 
> [1be08714731f01347ac1f98e18047fe7d9ab8afd|https://github.com/apache/nifi/commit/1be08714731f01347ac1f98e18047fe7d9ab8afd]
>  from [PR 1202|https://github.com/apache/nifi/pull/1202] to resolve 
> [NIFI-2854]) which causes the following line to be repeated in 
> {{logs/nifi-app.log}} infinitely. In a normal instance, the application can 
> still start up successfully, but in a secured instance, this prevents the 
> application from running. 
> {code}
> hw12203:...space/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT
>  (master) alopresto
> 🔓 100s @ 17:32:36 $ tail -f logs/nifi-app.log
> 2016-11-18 17:39:03,012 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:04,016 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:05,020 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:06,023 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:07,027 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:08,031 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:09,035 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:10,040 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:11,045 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:12,047 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:13,050 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/

[jira] [Commented] (NIFI-3069) Error clearing content_repository causes app to fail when configured securely

[Andy LoPresto (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678434#comment-15678434
 ] 

Andy LoPresto commented on NIFI-3069:
-

Correct, if I try to start up a secure instance (even built completely clean), 
I cannot access the UI, and the {{./bin/nifi.sh status}} command indicates that 
the application isn't running. 

> Error clearing content_repository causes app to fail when configured securely
> -
>
> Key: NIFI-3069
> URL: https://issues.apache.org/jira/browse/NIFI-3069
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Andy LoPresto
>Priority: Blocker
>  Labels: content-repository, logging
> Fix For: 1.1.0
>
>
> An issue was introduced into {{master}} (I believe commit 
> [1be08714731f01347ac1f98e18047fe7d9ab8afd|https://github.com/apache/nifi/commit/1be08714731f01347ac1f98e18047fe7d9ab8afd]
>  from [PR 1202|https://github.com/apache/nifi/pull/1202] to resolve 
> [NIFI-2854]) which causes the following line to be repeated in 
> {{logs/nifi-app.log}} infinitely. In a normal instance, the application can 
> still start up successfully, but in a secured instance, this prevents the 
> application from running. 
> {code}
> hw12203:...space/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT
>  (master) alopresto
> 🔓 100s @ 17:32:36 $ tail -f logs/nifi-app.log
> 2016-11-18 17:39:03,012 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:04,016 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:05,020 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:06,023 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:07,027 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:08,031 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:09,035 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:10,040 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:11,045 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:12,047 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:13,050 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to

[jira] [Commented] (NIFI-3069) Error clearing content_repository causes app to fail when configured securely

[Joseph Percivall (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678422#comment-15678422
 ] 

Joseph Percivall commented on NIFI-3069:


[~alopresto] so you're not able to start up a secure instance at all? While 
testing NIFI-2854 I was able to start up a secure cluster without these issues.

> Error clearing content_repository causes app to fail when configured securely
> -
>
> Key: NIFI-3069
> URL: https://issues.apache.org/jira/browse/NIFI-3069
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Andy LoPresto
>Priority: Blocker
>  Labels: content-repository, logging
> Fix For: 1.1.0
>
>
> An issue was introduced into {{master}} (I believe commit 
> [1be08714731f01347ac1f98e18047fe7d9ab8afd|https://github.com/apache/nifi/commit/1be08714731f01347ac1f98e18047fe7d9ab8afd]
>  from [PR 1202|https://github.com/apache/nifi/pull/1202] to resolve 
> [NIFI-2854]) which causes the following line to be repeated in 
> {{logs/nifi-app.log}} infinitely. In a normal instance, the application can 
> still start up successfully, but in a secured instance, this prevents the 
> application from running. 
> {code}
> hw12203:...space/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT
>  (master) alopresto
> 🔓 100s @ 17:32:36 $ tail -f logs/nifi-app.log
> 2016-11-18 17:39:03,012 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:04,016 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:05,020 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:06,023 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:07,027 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:08,031 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:09,035 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:10,040 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:11,045 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:12,047 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:13,050 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileExcep

[jira] [Commented] (NIFI-3064) ExecuteSQL convert to Avro fails when a MySQL query returns two columns with the same name even with different aliases

[Karthik Narayanan (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678391#comment-15678391
 ] 

Karthik Narayanan commented on NIFI-3064:
-

Create a pull request to fix this issue.

https://github.com/apache/nifi/pull/1249

> ExecuteSQL convert to Avro fails when a MySQL query returns two columns with 
> the same name even with different aliases
> --
>
> Key: NIFI-3064
> URL: https://issues.apache.org/jira/browse/NIFI-3064
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0, 0.6.1
>Reporter: Karthik Narayanan
>Assignee: Karthik Narayanan
> Fix For: 1.0.0
>
>
> For the following query:
> select A.host as host1,B.host as host from `hosts` A  join host2 B on 
> B.hostid=A.hostid;
> A.host and B.host are columns that have same name, but come from different 
> tables, and have a different alias name. But looks like Avro conversion is 
> ignoring the alias and just looking at the column name. This causes the query 
> to fail with this error
> Duplicate Field Name in any.data.hosts ; name type UNION pos:1 and name type 
> UNION pos:0.
> I have attached a screenshot for the same. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-3064) ExecuteSQL convert to Avro fails when a MySQL query returns two columns with the same name even with different aliases

[Karthik Narayanan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Narayanan updated NIFI-3064:

Fix Version/s: 1.0.0
   Status: Patch Available  (was: In Progress)

https://github.com/apache/nifi/pull/1249

> ExecuteSQL convert to Avro fails when a MySQL query returns two columns with 
> the same name even with different aliases
> --
>
> Key: NIFI-3064
> URL: https://issues.apache.org/jira/browse/NIFI-3064
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 0.6.1, 0.7.0, 1.0.0
>Reporter: Karthik Narayanan
>Assignee: Karthik Narayanan
> Fix For: 1.0.0
>
>
> For the following query:
> select A.host as host1,B.host as host from `hosts` A  join host2 B on 
> B.hostid=A.hostid;
> A.host and B.host are columns that have same name, but come from different 
> tables, and have a different alias name. But looks like Avro conversion is 
> ignoring the alias and just looking at the column name. This causes the query 
> to fail with this error
> Duplicate Field Name in any.data.hosts ; name type UNION pos:1 and name type 
> UNION pos:0.
> I have attached a screenshot for the same. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1249: NiFI-3604 onvert to Avro fails when a MySQL query r...

[rkarthik29]

GitHub user rkarthik29 opened a pull request:

https://github.com/apache/nifi/pull/1249

NiFI-3604 onvert to Avro fails when a MySQL query returns two columns with 
the same name even with different aliases

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rkarthik29/nifi-1 master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1249.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1249


commit 4a8d58d7b2a1989f072163b6fd670566ef5ac1f6
Author: Kathik Narayanan 
Date:   2016-11-18T22:24:54Z

fix for nifi 3064

commit 47ba71d4574727027fdb35083a780f804c11ff78
Author: Kathik Narayanan 
Date:   2016-11-18T22:27:04Z

Merge branch 'master' of https://github.com/apache/nifi

merged to apache master

commit 0647259941e67f3947248ebf71ff26432abbdf68
Author: Kathik Narayanan 
Date:   2016-11-18T23:35:10Z

fixed test case was failing because it was expecting column label to be 
returned.

commit 8ce3d1ee2c18cd418cdf06f4c4ac704f6530f405
Author: Kathik Narayanan 
Date:   2016-11-18T23:36:38Z

Revert "fixed test case was failing because it was expecting column label 
to be returned."

This reverts commit 0647259941e67f3947248ebf71ff26432abbdf68.

commit 9b122155e0d7aad550be040976230c855e851c6d
Author: Kathik Narayanan 
Date:   2016-11-18T23:38:05Z

NIFI-3604 : Test case failing, updated test case to be consistent with new 
code behavior

commit 368e310e77f8dbe3a61669052d8e64f1a4242edf
Author: Kathik Narayanan 
Date:   2016-11-19T02:09:29Z

NIFI-3604 coding style check fix.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3069) Error clearing content_repository causes app to fail when configured securely

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678363#comment-15678363
 ] 

ASF GitHub Bot commented on NIFI-3069:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/1247
  
There is an issue in `master` where the `nifi-app.log` is spammed with the 
following lines:

```
2016-11-18 17:39:14,053 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
2016-11-18 17:39:15,055 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
```

Running an unsecured instance has this message but still loads the app 
fine, but with keystores configured, the application fails to start. 

```

hw12203:...space/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT
 (pr1247) alopresto
🔓 404s @ 17:39:21 $ ./bin/nifi.sh status

Java home: /Users/alopresto/.jenv/versions/1.8
NiFi home: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT

Bootstrap Config File: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/conf/bootstrap.conf

2016-11-18 17:40:05,146 INFO [main] org.apache.nifi.bootstrap.Command 
Apache NiFi is not running


hw12203:...space/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT
 (pr1247) alopresto
🔓 47s @ 17:40:08 $
```

At this point I can't verify the restricted processor changes because it 
requires a secured instance with multiple user accounts. The bug needs to be 
fixed in `master` first. I have filed 
[NIFI-3069](https://issues.apache.org/jira/browse/NIFI-3069) to document and 
address this. 


> Error clearing content_repository causes app to fail when configured securely
> -
>
> Key: NIFI-3069
> URL: https://issues.apache.org/jira/browse/NIFI-3069
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Andy LoPresto
>Priority: Blocker
>  Labels: content-repository, logging
> Fix For: 1.1.0
>
>
> An issue was introduced into {{master}} (I believe commit 
> [1be08714731f01347ac1f98e18047fe7d9ab8afd|https://github.com/apache/nifi/commit/1be08714731f01347ac1f98e18047fe7d9ab8afd]
>  from [PR 1202|https://github.com/apache/nifi/pull/1202] to resolve 
> [NIFI-2854]) which causes the following line to be repeated in 
> {{logs/nifi-app.log}} infinitely. In a normal instance, the application can 
> still start up successfully, but in a secured instance, this prevents the 
> application from running. 
> {code}
> hw12203:...space/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT
>  (master) alopresto
> 🔓 100s @ 17:32:36 $ tail -f logs/nifi-app.log
> 2016-11-18 17:39:03,012 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:04,016 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:05,020 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:06,023 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
> 2016-11-18 17:39:07,027 ERROR [Cleanup Archive for default] 
> o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for 
> container default due to java.nio.file.NoSuchFileException: 
> /Users/alopresto/Workspace

[GitHub] nifi issue #1247: Introducing restricted components which require additional...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/1247
  
There is an issue in `master` where the `nifi-app.log` is spammed with the 
following lines:

```
2016-11-18 17:39:14,053 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
2016-11-18 17:39:15,055 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
```

Running an unsecured instance has this message but still loads the app 
fine, but with keystores configured, the application fails to start. 

```

hw12203:...space/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT
 (pr1247) alopresto
🔓 404s @ 17:39:21 $ ./bin/nifi.sh status

Java home: /Users/alopresto/.jenv/versions/1.8
NiFi home: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT

Bootstrap Config File: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/conf/bootstrap.conf

2016-11-18 17:40:05,146 INFO [main] org.apache.nifi.bootstrap.Command 
Apache NiFi is not running


hw12203:...space/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT
 (pr1247) alopresto
🔓 47s @ 17:40:08 $
```

At this point I can't verify the restricted processor changes because it 
requires a secured instance with multiple user accounts. The bug needs to be 
fixed in `master` first. I have filed 
[NIFI-3069](https://issues.apache.org/jira/browse/NIFI-3069) to document and 
address this. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (NIFI-3069) Error clearing content_repository causes app to fail when configured securely

[Andy LoPresto (JIRA)]

Andy LoPresto created NIFI-3069:
---

 Summary: Error clearing content_repository causes app to fail when 
configured securely
 Key: NIFI-3069
 URL: https://issues.apache.org/jira/browse/NIFI-3069
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.0.0
Reporter: Andy LoPresto
Priority: Blocker
 Fix For: 1.1.0


An issue was introduced into {{master}} (I believe commit 
[1be08714731f01347ac1f98e18047fe7d9ab8afd|https://github.com/apache/nifi/commit/1be08714731f01347ac1f98e18047fe7d9ab8afd]
 from [PR 1202|https://github.com/apache/nifi/pull/1202] to resolve 
[NIFI-2854]) which causes the following line to be repeated in 
{{logs/nifi-app.log}} infinitely. In a normal instance, the application can 
still start up successfully, but in a secured instance, this prevents the 
application from running. 

{code}
hw12203:...space/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT
 (master) alopresto
🔓 100s @ 17:32:36 $ tail -f logs/nifi-app.log
2016-11-18 17:39:03,012 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
2016-11-18 17:39:04,016 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
2016-11-18 17:39:05,020 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
2016-11-18 17:39:06,023 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
2016-11-18 17:39:07,027 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
2016-11-18 17:39:08,031 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
2016-11-18 17:39:09,035 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
2016-11-18 17:39:10,040 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
2016-11-18 17:39:11,045 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
2016-11-18 17:39:12,047 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
2016-11-18 17:39:13,050 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
2016-11-18 17:39:14,053 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
default due to java.nio.file.NoSuchFileException: 
/Users/alopresto/Workspace/nifi/nifi-assembly/target/nifi-1.1.0-SNAPSHOT-bin/nifi-1.1.0-SNAPSHOT/content_repository
2016-11-18 17:39:15,055 ERROR [Cleanup Archive for default] 
o.a.n.c.repository.FileSystemRepository Failed to cleanup archive for container 
d

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678276#comment-15678276
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88766699
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
+The following NOTICE information applies:
+   Elasticsearch
+   Copyright 2009-2015 Elasticsearch
+
+  (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2002-2016 The Apache Software Foundation
+
+  (ASLv2) Apache Lucene
+The following NOTICE information applies:
+  Apache Lucene
+  Copyright 2014 The Apache Software Foundation
+
+  Includes software from other Apache Software Foundation projects,
+  including, but not limited to:
+   - Apache Ant
+   - Apache Jakarta Regexp
+   - Apache Commons
+   - Apache Xerces
+
+  ICU4J, (under analysis/icu) is licensed under an MIT styles license
+  and Copyright (c) 1995-2008 International Business Machines 
Corporation and others
+
+  Some data files (under analysis/icu/src/data) are derived from 
Unicode data such
+  as the Unicode Character Database. See 
http://unicode.org/copyright.html for more
+  details.
+
+  Brics Automaton (under 
core/src/java/org/apache/lucene/util/automaton) is
+  BSD-licensed, created by Anders Møller. See 
http://www.brics.dk/automaton/
+
+  The levenshtein automata tables (under 
core/src/java/org/apache/lucene/util/automaton) were
+  automatically generated with the moman/finenight FSA library, 
created by
+  Jean-Philippe Barrette-LaPierre. This library is available under an 
MIT license,
+  see http://sites.google.com/site/rrettesite/moman and
+  http://bitbucket.org/jpbarrette/moman/overview/
+
+  The class org.apache.lucene.util.WeakIdentityMap was derived from
+  the Apache CXF project and is Apache License 2.0.
+
+  The Google Code Prettify is Apache License 2.0.
+  See http://code.google.com/p/google-code-prettify/
+
+  JUnit (junit-4.10) is licensed under the Common Public License v. 1.0
+  See http://junit.sourceforge.net/cpl-v10.html
+
+  This product includes code (JaspellTernarySearchTrie) from Java 
Spelling Checkin
+  g Package (jaspell): http://jaspell.sourceforge.net/
+  License: The BSD License 
(http://www.opensource.org/licenses/bsd-license.php)
+
+  The snowball stemmers in
+analysis/common/src/java/net/sf/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The snowball stopword lists in
+analysis/common/src/resources/org/apache/lucene/analysis/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The full snowball package is available from
+http://snowball.tartarus.org/
+
+  The KStem stemmer in
+analysis/common/src/org/apache/lucene/analysis/en
+  was developed by Bob Krovetz and Sergio Guzman-Lara (CIIR-UMass 
Amherst)
+  under the BSD-license.
+
+  The Arabic,Persian,Romanian,Bulgarian, and Hindi analyzers (common) 
come with a default
+  stopword list that is BSD-licensed created by Jacques Savoy.  These 
files reside in:
+  
analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/fa/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/ro/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/bg/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/hi/stopwords.txt
+  See http://members.unine.ch/jacques.savoy/clef/index.html.
+
+  The 
German,Spanish,Finnish,French,Hungarian,Italian,Portuguese,Russian and Swedish 
light stemmers
+  (common) are based on BSD-licensed reference implementations created 
by Jacques Savoy and
+  Ljiljana Dolamic. These files reside in:
+  
analysis/common/src/java/org

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r88766699
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
+The following NOTICE information applies:
+   Elasticsearch
+   Copyright 2009-2015 Elasticsearch
+
+  (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2002-2016 The Apache Software Foundation
+
+  (ASLv2) Apache Lucene
+The following NOTICE information applies:
+  Apache Lucene
+  Copyright 2014 The Apache Software Foundation
+
+  Includes software from other Apache Software Foundation projects,
+  including, but not limited to:
+   - Apache Ant
+   - Apache Jakarta Regexp
+   - Apache Commons
+   - Apache Xerces
+
+  ICU4J, (under analysis/icu) is licensed under an MIT styles license
+  and Copyright (c) 1995-2008 International Business Machines 
Corporation and others
+
+  Some data files (under analysis/icu/src/data) are derived from 
Unicode data such
+  as the Unicode Character Database. See 
http://unicode.org/copyright.html for more
+  details.
+
+  Brics Automaton (under 
core/src/java/org/apache/lucene/util/automaton) is
+  BSD-licensed, created by Anders Møller. See 
http://www.brics.dk/automaton/
+
+  The levenshtein automata tables (under 
core/src/java/org/apache/lucene/util/automaton) were
+  automatically generated with the moman/finenight FSA library, 
created by
+  Jean-Philippe Barrette-LaPierre. This library is available under an 
MIT license,
+  see http://sites.google.com/site/rrettesite/moman and
+  http://bitbucket.org/jpbarrette/moman/overview/
+
+  The class org.apache.lucene.util.WeakIdentityMap was derived from
+  the Apache CXF project and is Apache License 2.0.
+
+  The Google Code Prettify is Apache License 2.0.
+  See http://code.google.com/p/google-code-prettify/
+
+  JUnit (junit-4.10) is licensed under the Common Public License v. 1.0
+  See http://junit.sourceforge.net/cpl-v10.html
+
+  This product includes code (JaspellTernarySearchTrie) from Java 
Spelling Checkin
+  g Package (jaspell): http://jaspell.sourceforge.net/
+  License: The BSD License 
(http://www.opensource.org/licenses/bsd-license.php)
+
+  The snowball stemmers in
+analysis/common/src/java/net/sf/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The snowball stopword lists in
+analysis/common/src/resources/org/apache/lucene/analysis/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The full snowball package is available from
+http://snowball.tartarus.org/
+
+  The KStem stemmer in
+analysis/common/src/org/apache/lucene/analysis/en
+  was developed by Bob Krovetz and Sergio Guzman-Lara (CIIR-UMass 
Amherst)
+  under the BSD-license.
+
+  The Arabic,Persian,Romanian,Bulgarian, and Hindi analyzers (common) 
come with a default
+  stopword list that is BSD-licensed created by Jacques Savoy.  These 
files reside in:
+  
analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/fa/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/ro/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/bg/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/hi/stopwords.txt
+  See http://members.unine.ch/jacques.savoy/clef/index.html.
+
+  The 
German,Spanish,Finnish,French,Hungarian,Italian,Portuguese,Russian and Swedish 
light stemmers
+  (common) are based on BSD-licensed reference implementations created 
by Jacques Savoy and
+  Ljiljana Dolamic. These files reside in:
+  
analysis/common/src/java/org/apache/lucene/analysis/de/GermanLightStemmer.java
+  
analysis/common/src/java/org/apache/lucene/analysis/de/GermanMinimalStemmer.java
+  
analysis/common/src/java/org/apache/lucene/analysis/es/SpanishLightStemmer.java
+  
anal

[GitHub] nifi pull request #1233: NIFI-3011: Added Elasticsearch5 processors

[mattyb149]

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r8876
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
+The following NOTICE information applies:
+   Elasticsearch
+   Copyright 2009-2015 Elasticsearch
+
+  (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2002-2016 The Apache Software Foundation
+
+  (ASLv2) Apache Lucene
+The following NOTICE information applies:
+  Apache Lucene
+  Copyright 2014 The Apache Software Foundation
+
+  Includes software from other Apache Software Foundation projects,
+  including, but not limited to:
+   - Apache Ant
+   - Apache Jakarta Regexp
+   - Apache Commons
+   - Apache Xerces
+
+  ICU4J, (under analysis/icu) is licensed under an MIT styles license
+  and Copyright (c) 1995-2008 International Business Machines 
Corporation and others
+
+  Some data files (under analysis/icu/src/data) are derived from 
Unicode data such
+  as the Unicode Character Database. See 
http://unicode.org/copyright.html for more
+  details.
+
+  Brics Automaton (under 
core/src/java/org/apache/lucene/util/automaton) is
+  BSD-licensed, created by Anders Møller. See 
http://www.brics.dk/automaton/
+
+  The levenshtein automata tables (under 
core/src/java/org/apache/lucene/util/automaton) were
+  automatically generated with the moman/finenight FSA library, 
created by
+  Jean-Philippe Barrette-LaPierre. This library is available under an 
MIT license,
+  see http://sites.google.com/site/rrettesite/moman and
+  http://bitbucket.org/jpbarrette/moman/overview/
+
+  The class org.apache.lucene.util.WeakIdentityMap was derived from
+  the Apache CXF project and is Apache License 2.0.
+
+  The Google Code Prettify is Apache License 2.0.
+  See http://code.google.com/p/google-code-prettify/
+
+  JUnit (junit-4.10) is licensed under the Common Public License v. 1.0
+  See http://junit.sourceforge.net/cpl-v10.html
+
+  This product includes code (JaspellTernarySearchTrie) from Java 
Spelling Checkin
+  g Package (jaspell): http://jaspell.sourceforge.net/
+  License: The BSD License 
(http://www.opensource.org/licenses/bsd-license.php)
+
+  The snowball stemmers in
+analysis/common/src/java/net/sf/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The snowball stopword lists in
+analysis/common/src/resources/org/apache/lucene/analysis/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The full snowball package is available from
+http://snowball.tartarus.org/
+
+  The KStem stemmer in
+analysis/common/src/org/apache/lucene/analysis/en
+  was developed by Bob Krovetz and Sergio Guzman-Lara (CIIR-UMass 
Amherst)
+  under the BSD-license.
+
+  The Arabic,Persian,Romanian,Bulgarian, and Hindi analyzers (common) 
come with a default
+  stopword list that is BSD-licensed created by Jacques Savoy.  These 
files reside in:
+  
analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/fa/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/ro/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/bg/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/hi/stopwords.txt
+  See http://members.unine.ch/jacques.savoy/clef/index.html.
+
+  The 
German,Spanish,Finnish,French,Hungarian,Italian,Portuguese,Russian and Swedish 
light stemmers
+  (common) are based on BSD-licensed reference implementations created 
by Jacques Savoy and
+  Ljiljana Dolamic. These files reside in:
+  
analysis/common/src/java/org/apache/lucene/analysis/de/GermanLightStemmer.java
+  
analysis/common/src/java/org/apache/lucene/analysis/de/GermanMinimalStemmer.java
+  
analysis/common/src/java/org/apache/lucene/analysis/es/SpanishLightStemmer.java
+  
anal

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678274#comment-15678274
 ] 

ASF GitHub Bot commented on NIFI-3011:
--

Github user mattyb149 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1233#discussion_r8876
  
--- Diff: 
nifi-nar-bundles/nifi-elasticsearch-5-bundle/nifi-elasticsearch-5-nar/src/main/resources/META-INF/NOTICE
 ---
@@ -0,0 +1,252 @@
+nifi-elasticsearch-5-nar
+Copyright 2015-2016 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+**
+Apache Software License v2
+**
+
+The following binary components are provided under the Apache Software 
License v2
+
+  (ASLv2) Elasticsearch
+The following NOTICE information applies:
+   Elasticsearch
+   Copyright 2009-2015 Elasticsearch
+
+  (ASLv2) Apache Commons IO
+The following NOTICE information applies:
+  Apache Commons IO
+  Copyright 2002-2016 The Apache Software Foundation
+
+  (ASLv2) Apache Lucene
+The following NOTICE information applies:
+  Apache Lucene
+  Copyright 2014 The Apache Software Foundation
+
+  Includes software from other Apache Software Foundation projects,
+  including, but not limited to:
+   - Apache Ant
+   - Apache Jakarta Regexp
+   - Apache Commons
+   - Apache Xerces
+
+  ICU4J, (under analysis/icu) is licensed under an MIT styles license
+  and Copyright (c) 1995-2008 International Business Machines 
Corporation and others
+
+  Some data files (under analysis/icu/src/data) are derived from 
Unicode data such
+  as the Unicode Character Database. See 
http://unicode.org/copyright.html for more
+  details.
+
+  Brics Automaton (under 
core/src/java/org/apache/lucene/util/automaton) is
+  BSD-licensed, created by Anders Møller. See 
http://www.brics.dk/automaton/
+
+  The levenshtein automata tables (under 
core/src/java/org/apache/lucene/util/automaton) were
+  automatically generated with the moman/finenight FSA library, 
created by
+  Jean-Philippe Barrette-LaPierre. This library is available under an 
MIT license,
+  see http://sites.google.com/site/rrettesite/moman and
+  http://bitbucket.org/jpbarrette/moman/overview/
+
+  The class org.apache.lucene.util.WeakIdentityMap was derived from
+  the Apache CXF project and is Apache License 2.0.
+
+  The Google Code Prettify is Apache License 2.0.
+  See http://code.google.com/p/google-code-prettify/
+
+  JUnit (junit-4.10) is licensed under the Common Public License v. 1.0
+  See http://junit.sourceforge.net/cpl-v10.html
+
+  This product includes code (JaspellTernarySearchTrie) from Java 
Spelling Checkin
+  g Package (jaspell): http://jaspell.sourceforge.net/
+  License: The BSD License 
(http://www.opensource.org/licenses/bsd-license.php)
+
+  The snowball stemmers in
+analysis/common/src/java/net/sf/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The snowball stopword lists in
+analysis/common/src/resources/org/apache/lucene/analysis/snowball
+  were developed by Martin Porter and Richard Boulton.
+  The full snowball package is available from
+http://snowball.tartarus.org/
+
+  The KStem stemmer in
+analysis/common/src/org/apache/lucene/analysis/en
+  was developed by Bob Krovetz and Sergio Guzman-Lara (CIIR-UMass 
Amherst)
+  under the BSD-license.
+
+  The Arabic,Persian,Romanian,Bulgarian, and Hindi analyzers (common) 
come with a default
+  stopword list that is BSD-licensed created by Jacques Savoy.  These 
files reside in:
+  
analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/fa/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/ro/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/bg/stopwords.txt,
+  
analysis/common/src/resources/org/apache/lucene/analysis/hi/stopwords.txt
+  See http://members.unine.ch/jacques.savoy/clef/index.html.
+
+  The 
German,Spanish,Finnish,French,Hungarian,Italian,Portuguese,Russian and Swedish 
light stemmers
+  (common) are based on BSD-licensed reference implementations created 
by Jacques Savoy and
+  Ljiljana Dolamic. These files reside in:
+  
analysis/common/src/java/org

[jira] [Updated] (NIFI-3068) NiFi can not reliably support multiple HDFS clusters in the same flow

[Sam Hjelmfelt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Hjelmfelt updated NIFI-3068:

Summary: NiFi can not reliably support multiple HDFS clusters in the same 
flow  (was: NiFi can only be configured for one HDFS cluster per installation)

> NiFi can not reliably support multiple HDFS clusters in the same flow
> -
>
> Key: NIFI-3068
> URL: https://issues.apache.org/jira/browse/NIFI-3068
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Sam Hjelmfelt
>  Labels: HDFS
>
> The HDFS configurations in PutHDFS are not respected when two (or more) 
> PutHDFS processors exist with different configurations. The second processor 
> to run will use the configurations from the first processor. This can cause 
> data to be written to the wrong cluster.
> This appears to be caused by configuration caching in 
> AbstractHadoopProcessor, which would affect all HDFS processors.
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/AbstractHadoopProcessor.java#L144



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-3068) NiFi can only be configured for one HDFS clusters per installation

[Sam Hjelmfelt (JIRA)]

Sam Hjelmfelt created NIFI-3068:
---

 Summary: NiFi can only be configured for one HDFS clusters per 
installation
 Key: NIFI-3068
 URL: https://issues.apache.org/jira/browse/NIFI-3068
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core Framework
Affects Versions: 1.0.0
Reporter: Sam Hjelmfelt


The HDFS configurations in PutHDFS are not respected when two (or more) PutHDFS 
processors exist with different configurations. The second processor to run 
will use the configurations from the first processor. This can cause data to be 
written to the wrong cluster.

This appears to be caused by configuration caching in AbstractHadoopProcessor, 
which would affect all HDFS processors.
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/AbstractHadoopProcessor.java#L144



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-3068) NiFi can only be configured for one HDFS cluster per installation

[Sam Hjelmfelt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Hjelmfelt updated NIFI-3068:

Summary: NiFi can only be configured for one HDFS cluster per installation  
(was: NiFi can only be configured for one HDFS clusters per installation)

> NiFi can only be configured for one HDFS cluster per installation
> -
>
> Key: NIFI-3068
> URL: https://issues.apache.org/jira/browse/NIFI-3068
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Sam Hjelmfelt
>  Labels: HDFS
>
> The HDFS configurations in PutHDFS are not respected when two (or more) 
> PutHDFS processors exist with different configurations. The second processor 
> to run will use the configurations from the first processor. This can cause 
> data to be written to the wrong cluster.
> This appears to be caused by configuration caching in 
> AbstractHadoopProcessor, which would affect all HDFS processors.
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/AbstractHadoopProcessor.java#L144



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1236: LDAP - Configurable strategy to identify users

[olegz]

Github user olegz commented on the issue:

https://github.com/apache/nifi/pull/1236
  
I am +1 now. If @ijokarumawak has no issues let's merge


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1247: Introducing restricted components which require additional...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/1247
  
I'm getting some errors when running the standard test suite. Did 
configuration requirements change for this?

```

Results :

Failed tests:
  
TestPutTcpSSL>TestPutTCPCommon.testLoadTest:231->TestPutTCPCommon.sendTestData:254
 Processor has 1 validation failures:
'SSL Context Service' validated against 'ssl-context' is invalid because 
Controller Service is not valid: ['Keystore Properties' is invalid because 
Invalid KeyStore Password or Type specified for file 
src/test/resources/localhost-ks.jks, 'Truststore Properties' is invalid because 
Invalid KeyStore Password or Type specified for file 
src/test/resources/localhost-ts.jks]


Tests in error:
  TestListDatabaseTables.cleanUpAfterClass:71 » SQL Database 
'target/db_ldt' not...
  TestListDatabaseTables.testListTablesAfterRefresh:153 » Process 
getConnection ...
  TestListDatabaseTables.testListTablesMultipleRefresh:190 » Process 
getConnecti...
  TestListDatabaseTables.testListTablesNoCount:100 » Process getConnection 
faile...
  TestListDatabaseTables.testListTablesWithCount:126 » Process 
getConnection fai...
  
TestPutTcpSSL>TestPutTCPCommon.testConnectionPerFlowFile:142->TestPutTCPCommon.checkReceivedAllData:278->TestPutTCPCommon.checkReceivedAllData:285
 » TestTimedOut
  TestPutUDP.testLoadTest:176->checkReceivedAllData:236 » TestTimedOut 
test time...
  
TestPutUDP.testValidFiles:98->checkReceivedAllData:229->checkReceivedAllData:236
 » TestTimedOut


Tests run: 994, Failures: 1, Errors: 8, Skipped: 17
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1248: pr for nifi 3064

[rkarthik29]

GitHub user rkarthik29 opened a pull request:

https://github.com/apache/nifi/pull/1248

pr for nifi 3064

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rkarthik29/nifi-1 master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1248.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1248


commit 4a8d58d7b2a1989f072163b6fd670566ef5ac1f6
Author: Kathik Narayanan 
Date:   2016-11-18T22:24:54Z

fix for nifi 3064

commit 47ba71d4574727027fdb35083a780f804c11ff78
Author: Kathik Narayanan 
Date:   2016-11-18T22:27:04Z

Merge branch 'master' of https://github.com/apache/nifi

merged to apache master




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1248: pr for nifi 3064

[rkarthik29]

Github user rkarthik29 closed the pull request at:

https://github.com/apache/nifi/pull/1248


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #1247: Introducing restricted components which require additional...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/1247
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2954) Bouncycastle dependencies are duplicated throughout NARs

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2954:
--
Status: Patch Available  (was: Open)

> Bouncycastle dependencies are duplicated throughout NARs
> 
>
> Key: NIFI-2954
> URL: https://issues.apache.org/jira/browse/NIFI-2954
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Affects Versions: 1.0.0
>Reporter: Aldrin Piri
>Assignee: Joseph Witt
>Priority: Blocker
> Fix For: 1.1.0
>
>
> I was working with some of the NiFi dependencies for incorporation with/use 
> by MiNiFi and saw that several libraries were duplicated throughout all NARs.
> Of note were the bouncycastle dependencies which, together, are ~4MB in size.
> {code}
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:23]
> $ find {lib,work} -type f -name '*bcprov-jdk15on*' | wc -l
>   54
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:28]
> $ find {lib,work} -type f -name '*bcpkix*' | wc -l
>   53
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:33]
> $ ls -lash 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcprov-jdk15on-1.54.jar
> 3.2M -rw-r--r-- 1 apiri admin 3.2M Oct  4 12:24 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcprov-jdk15on-1.54.jar
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:39]
> $ ls -lash 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcpkix-jdk15on-1.54.jar
> 660K -rw-r--r-- 1 apiri admin 658K Oct  4 12:24 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcpkix-jdk15on-1.54.jar
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3058) Add documentation for ZooKeeper Migrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677834#comment-15677834
 ] 

ASF GitHub Bot commented on NIFI-3058:
--

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88744920
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
+* `-h,--help`   Prints help, displays 
available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper 
Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf `   Allows the specification 
of a JAAS configuration file to allow authentication with a ZooKeeper 
configured to use Kerberos.  This option is mutually exclusive with the 
`-a,--auth` option.
+* `-r,--receive`Receives data from 
ZooKeeper and writes to the given filename (if the `-f,--file` option is 
provided) or standard output. The data received will contain the full path to 
each node read from ZooKeeper. This option is mutually exclusive with the 
`-s,--send` option.
+* `-s,--send`   Sends data to ZooKeeper 
that is read from the given filename (if the `-f,--file` option is provided) or 
standard input. The paths for each node in the data being sent to ZooKeeper are 
absolute paths, and will be stored in ZooKeeper under the *path* portion of the 
`-z,--zookeeper` argument.  Typically, the *path* portion of the argument can 
be omitted, which will store the nodes at their absolute paths. This option is 
mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper ` The ZooKeeper server to 
use, specified by a  connection string with path, in the format of 
_host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+ Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination 
ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, 
you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data 
processing).
+* If one of the ZooKeeper clusters you are using is configured with 
Kerberos, you are running the ZooKeeper Migrator from a host that has access to 
NiFi’s ZooKeeper client jaas configuration file (see <> for 
more information).
+
+1. Collect the following information:
--- End diff --

Good suggestion


> Add documentation for ZooKeeper Migrator
> 
>
> Key: NIFI-3058
> URL: https://issues.apache.org/jira/browse/NIFI-3058
>

[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...

[andrewmlim]

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88744920
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
+* `-h,--help`   Prints help, displays 
available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper 
Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf `   Allows the specification 
of a JAAS configuration file to allow authentication with a ZooKeeper 
configured to use Kerberos.  This option is mutually exclusive with the 
`-a,--auth` option.
+* `-r,--receive`Receives data from 
ZooKeeper and writes to the given filename (if the `-f,--file` option is 
provided) or standard output. The data received will contain the full path to 
each node read from ZooKeeper. This option is mutually exclusive with the 
`-s,--send` option.
+* `-s,--send`   Sends data to ZooKeeper 
that is read from the given filename (if the `-f,--file` option is provided) or 
standard input. The paths for each node in the data being sent to ZooKeeper are 
absolute paths, and will be stored in ZooKeeper under the *path* portion of the 
`-z,--zookeeper` argument.  Typically, the *path* portion of the argument can 
be omitted, which will store the nodes at their absolute paths. This option is 
mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper ` The ZooKeeper server to 
use, specified by a  connection string with path, in the format of 
_host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+ Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination 
ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, 
you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data 
processing).
+* If one of the ZooKeeper clusters you are using is configured with 
Kerberos, you are running the ZooKeeper Migrator from a host that has access to 
NiFi’s ZooKeeper client jaas configuration file (see <> 
for more information).
+
+1. Collect the following information:
--- End diff --

Good suggestion


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3058) Add documentation for ZooKeeper Migrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677823#comment-15677823
 ] 

ASF GitHub Bot commented on NIFI-3058:
--

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88743987
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
+* `-h,--help`   Prints help, displays 
available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper 
Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf `   Allows the specification 
of a JAAS configuration file to allow authentication with a ZooKeeper 
configured to use Kerberos.  This option is mutually exclusive with the 
`-a,--auth` option.
+* `-r,--receive`Receives data from 
ZooKeeper and writes to the given filename (if the `-f,--file` option is 
provided) or standard output. The data received will contain the full path to 
each node read from ZooKeeper. This option is mutually exclusive with the 
`-s,--send` option.
+* `-s,--send`   Sends data to ZooKeeper 
that is read from the given filename (if the `-f,--file` option is provided) or 
standard input. The paths for each node in the data being sent to ZooKeeper are 
absolute paths, and will be stored in ZooKeeper under the *path* portion of the 
`-z,--zookeeper` argument.  Typically, the *path* portion of the argument can 
be omitted, which will store the nodes at their absolute paths. This option is 
mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper ` The ZooKeeper server to 
use, specified by a  connection string with path, in the format of 
_host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+ Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination 
ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, 
you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data 
processing).
+* If one of the ZooKeeper clusters you are using is configured with 
Kerberos, you are running the ZooKeeper Migrator from a host that has access to 
NiFi’s ZooKeeper client jaas configuration file (see <> for 
more information).
+
+1. Collect the following information:
+
+|
+|*Required Information*|*Description*
+|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of 
the hosts running in the ZooKeeper ensemble, which can be found in /conf/zookeeper.properties.  Any 

[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...

[andrewmlim]

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88743987
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
+* `-h,--help`   Prints help, displays 
available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper 
Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf `   Allows the specification 
of a JAAS configuration file to allow authentication with a ZooKeeper 
configured to use Kerberos.  This option is mutually exclusive with the 
`-a,--auth` option.
+* `-r,--receive`Receives data from 
ZooKeeper and writes to the given filename (if the `-f,--file` option is 
provided) or standard output. The data received will contain the full path to 
each node read from ZooKeeper. This option is mutually exclusive with the 
`-s,--send` option.
+* `-s,--send`   Sends data to ZooKeeper 
that is read from the given filename (if the `-f,--file` option is provided) or 
standard input. The paths for each node in the data being sent to ZooKeeper are 
absolute paths, and will be stored in ZooKeeper under the *path* portion of the 
`-z,--zookeeper` argument.  Typically, the *path* portion of the argument can 
be omitted, which will store the nodes at their absolute paths. This option is 
mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper ` The ZooKeeper server to 
use, specified by a  connection string with path, in the format of 
_host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+ Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination 
ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, 
you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data 
processing).
+* If one of the ZooKeeper clusters you are using is configured with 
Kerberos, you are running the ZooKeeper Migrator from a host that has access to 
NiFi’s ZooKeeper client jaas configuration file (see <> 
for more information).
+
+1. Collect the following information:
+
+|
+|*Required Information*|*Description*
+|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of 
the hosts running in the ZooKeeper ensemble, which can be found in /conf/zookeeper.properties.  Any of the hostnames declared in 
the *server.N* properties can be used.
+|Destination ZooKeeper hostname (*destinationHostname*)|The hostname must 
be one of the hosts running in the ZooKeeper ensemble, which can be found in 
/conf/zookeeper.properti

[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...

[andrewmlim]

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88743950
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
+* `-h,--help`   Prints help, displays 
available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper 
Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf `   Allows the specification 
of a JAAS configuration file to allow authentication with a ZooKeeper 
configured to use Kerberos.  This option is mutually exclusive with the 
`-a,--auth` option.
+* `-r,--receive`Receives data from 
ZooKeeper and writes to the given filename (if the `-f,--file` option is 
provided) or standard output. The data received will contain the full path to 
each node read from ZooKeeper. This option is mutually exclusive with the 
`-s,--send` option.
+* `-s,--send`   Sends data to ZooKeeper 
that is read from the given filename (if the `-f,--file` option is provided) or 
standard input. The paths for each node in the data being sent to ZooKeeper are 
absolute paths, and will be stored in ZooKeeper under the *path* portion of the 
`-z,--zookeeper` argument.  Typically, the *path* portion of the argument can 
be omitted, which will store the nodes at their absolute paths. This option is 
mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper ` The ZooKeeper server to 
use, specified by a  connection string with path, in the format of 
_host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+ Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination 
ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, 
you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data 
processing).
+* If one of the ZooKeeper clusters you are using is configured with 
Kerberos, you are running the ZooKeeper Migrator from a host that has access to 
NiFi’s ZooKeeper client jaas configuration file (see <> 
for more information).
+
+1. Collect the following information:
+
+|
+|*Required Information*|*Description*
+|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of 
the hosts running in the ZooKeeper ensemble, which can be found in /conf/zookeeper.properties.  Any of the hostnames declared in 
the *server.N* properties can be used.
+|Destination ZooKeeper hostname (*destinationHostname*)|The hostname must 
be one of the hosts running in the ZooKeeper ensemble, which can be found in 
/conf/zookeeper.properti

[jira] [Commented] (NIFI-3058) Add documentation for ZooKeeper Migrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677822#comment-15677822
 ] 

ASF GitHub Bot commented on NIFI-3058:
--

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88743950
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
+* `-h,--help`   Prints help, displays 
available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper 
Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf `   Allows the specification 
of a JAAS configuration file to allow authentication with a ZooKeeper 
configured to use Kerberos.  This option is mutually exclusive with the 
`-a,--auth` option.
+* `-r,--receive`Receives data from 
ZooKeeper and writes to the given filename (if the `-f,--file` option is 
provided) or standard output. The data received will contain the full path to 
each node read from ZooKeeper. This option is mutually exclusive with the 
`-s,--send` option.
+* `-s,--send`   Sends data to ZooKeeper 
that is read from the given filename (if the `-f,--file` option is provided) or 
standard input. The paths for each node in the data being sent to ZooKeeper are 
absolute paths, and will be stored in ZooKeeper under the *path* portion of the 
`-z,--zookeeper` argument.  Typically, the *path* portion of the argument can 
be omitted, which will store the nodes at their absolute paths. This option is 
mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper ` The ZooKeeper server to 
use, specified by a  connection string with path, in the format of 
_host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+ Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination 
ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, 
you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data 
processing).
+* If one of the ZooKeeper clusters you are using is configured with 
Kerberos, you are running the ZooKeeper Migrator from a host that has access to 
NiFi’s ZooKeeper client jaas configuration file (see <> for 
more information).
+
+1. Collect the following information:
+
+|
+|*Required Information*|*Description*
+|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of 
the hosts running in the ZooKeeper ensemble, which can be found in /conf/zookeeper.properties.  Any 

[jira] [Commented] (NIFI-3058) Add documentation for ZooKeeper Migrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677820#comment-15677820
 ] 

ASF GitHub Bot commented on NIFI-3058:
--

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88743900
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1432,7 +1432,7 @@ If Kerberos is not already setup in your environment, 
you can find information o
 
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html[_https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html_]
 . This guide assumes that Kerberos already has been installed in the 
environment in which NiFi is running.
 
-Note, the following procedures for kerberizing an Embedded Zookeeper 
server in your NiFI Node and kerberizing a zookeeper NiFI client will require 
that
+Note, the following procedures for kerberizing an Embedded ZooKeeper 
server in your NiFI Node and kerberizing a ZooKeeper NiFI client will require 
that
--- End diff --

Good catch.


> Add documentation for ZooKeeper Migrator
> 
>
> Key: NIFI-3058
> URL: https://issues.apache.org/jira/browse/NIFI-3058
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.1.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> NIFI-2957 added the ZooKeeper Migrator to the NiFi Toolkit.  Documentation 
> should be added to guide the usage of the migrator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3058) Add documentation for ZooKeeper Migrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677817#comment-15677817
 ] 

ASF GitHub Bot commented on NIFI-3058:
--

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88743855
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
--- End diff --

will do.


> Add documentation for ZooKeeper Migrator
> 
>
> Key: NIFI-3058
> URL: https://issues.apache.org/jira/browse/NIFI-3058
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.1.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> NIFI-2957 added the ZooKeeper Migrator to the NiFi Toolkit.  Documentation 
> should be added to guide the usage of the migrator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...

[andrewmlim]

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88743900
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1432,7 +1432,7 @@ If Kerberos is not already setup in your environment, 
you can find information o
 
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html[_https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html_]
 . This guide assumes that Kerberos already has been installed in the 
environment in which NiFi is running.
 
-Note, the following procedures for kerberizing an Embedded Zookeeper 
server in your NiFI Node and kerberizing a zookeeper NiFI client will require 
that
+Note, the following procedures for kerberizing an Embedded ZooKeeper 
server in your NiFI Node and kerberizing a ZooKeeper NiFI client will require 
that
--- End diff --

Good catch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...

[andrewmlim]

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88743855
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
--- End diff --

will do.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3058) Add documentation for ZooKeeper Migrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677815#comment-15677815
 ] 

ASF GitHub Bot commented on NIFI-3058:
--

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88743768
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
--- End diff --

thanks...will do.


> Add documentation for ZooKeeper Migrator
> 
>
> Key: NIFI-3058
> URL: https://issues.apache.org/jira/browse/NIFI-3058
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.1.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> NIFI-2957 added the ZooKeeper Migrator to the NiFi Toolkit.  Documentation 
> should be added to guide the usage of the migrator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...

[andrewmlim]

Github user andrewmlim commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88743768
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
--- End diff --

thanks...will do.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3050) Restrict dangerous processors to special permission

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman updated NIFI-3050:
--
Status: Patch Available  (was: Open)

> Restrict dangerous processors to special permission
> ---
>
> Key: NIFI-3050
> URL: https://issues.apache.org/jira/browse/NIFI-3050
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Andy LoPresto
>Assignee: Matt Gilman
>Priority: Blocker
>  Labels: security
> Fix For: 1.1.0
>
>
> As evidenced by [NIFI-3045] and other discoveries (e.g. using an 
> {{ExecuteScript}} processor to iterate over a {{NiFiProperties}} instance 
> after the application has already decrypted the sensitive properties from the 
> {{nifi.properties}} file on disk, using a {{GetFile}} processor to retrieve 
> {{/etc/passwd}}, etc.) NiFi is a powerful tool which can allow unauthorized 
> users to perform malicious actions. While no tool as versatile as NiFi will 
> ever be completely immune to insider threat, to further restrict the 
> potential for abuse, certain processors should be designated as 
> {{restricted}}, and these processors can only be added to the canvas or 
> modified by users who, along with the proper permission to modify the canvas, 
> have a special permission to interact with these "dangerous" processors. 
> From the [Security Feature 
> Roadmap|https://cwiki.apache.org/confluence/display/NIFI/Security+Feature+Roadmap]:
> {quote}
> Dangerous Processors
> * Processors which can directly affect behavior/configuration of NiFi/other 
> services
> - {{GetFile}}
> - {{PutFile}}
> - {{ListFile}}
> - {{FetchFile}}
> - {{ExecuteScript}}
> - {{InvokeScriptedProcessor}}
> - {{ExecuteProcess}}
> - {{ExecuteStreamCommand}}
> * These processors should only be creatable/editable by users with special 
> access control policy
> * Marked by {{@Restricted}} annotation on processor class
> * All flowfiles originating/passing through these processors have special 
> attribute/protection
> * Perhaps *File processors can access a certain location by default but 
> cannot access the root filesystem without special user permission?
> {quote}
> [~mcgilman] and I should have a PR for this tomorrow. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-3050) Restrict dangerous processors to special permission

[Matt Gilman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Gilman reassigned NIFI-3050:
-

Assignee: Matt Gilman  (was: Andy LoPresto)

> Restrict dangerous processors to special permission
> ---
>
> Key: NIFI-3050
> URL: https://issues.apache.org/jira/browse/NIFI-3050
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Andy LoPresto
>Assignee: Matt Gilman
>Priority: Blocker
>  Labels: security
> Fix For: 1.1.0
>
>
> As evidenced by [NIFI-3045] and other discoveries (e.g. using an 
> {{ExecuteScript}} processor to iterate over a {{NiFiProperties}} instance 
> after the application has already decrypted the sensitive properties from the 
> {{nifi.properties}} file on disk, using a {{GetFile}} processor to retrieve 
> {{/etc/passwd}}, etc.) NiFi is a powerful tool which can allow unauthorized 
> users to perform malicious actions. While no tool as versatile as NiFi will 
> ever be completely immune to insider threat, to further restrict the 
> potential for abuse, certain processors should be designated as 
> {{restricted}}, and these processors can only be added to the canvas or 
> modified by users who, along with the proper permission to modify the canvas, 
> have a special permission to interact with these "dangerous" processors. 
> From the [Security Feature 
> Roadmap|https://cwiki.apache.org/confluence/display/NIFI/Security+Feature+Roadmap]:
> {quote}
> Dangerous Processors
> * Processors which can directly affect behavior/configuration of NiFi/other 
> services
> - {{GetFile}}
> - {{PutFile}}
> - {{ListFile}}
> - {{FetchFile}}
> - {{ExecuteScript}}
> - {{InvokeScriptedProcessor}}
> - {{ExecuteProcess}}
> - {{ExecuteStreamCommand}}
> * These processors should only be creatable/editable by users with special 
> access control policy
> * Marked by {{@Restricted}} annotation on processor class
> * All flowfiles originating/passing through these processors have special 
> attribute/protection
> * Perhaps *File processors can access a certain location by default but 
> cannot access the root filesystem without special user permission?
> {quote}
> [~mcgilman] and I should have a PR for this tomorrow. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1236: LDAP - Configurable strategy to identify users

[mcgilman]

Github user mcgilman commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1236#discussion_r88742365
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/contextlistener/ApplicationStartupContextListener.java
 ---
@@ -121,4 +111,16 @@ public void contextDestroyed(ServletContextEvent sce) {
 }
 }
 }
+
+private void shutdown(final FlowService flowService, final 
RequestReplicator requestReplicator) {
+// ensure the flow service is terminated
+if (flowService != null && flowService.isRunning()) {
+flowService.stop(false);
+}
+
+// ensure the request replicator is shutdown
+if (requestReplicator != null) {
+requestReplicator.shutdown();
+}
--- End diff --

Good call. I'll update.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3050) Restrict dangerous processors to special permission

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677790#comment-15677790
 ] 

ASF GitHub Bot commented on NIFI-3050:
--

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/1247

Introducing restricted components which require additional access

NIFI-3050:
- Introducing a Restricted annotation for components that require elevated 
privileges to use.
- Updating the new Processor, Controller Service, and Reporting Task 
dialogs to include these details and prevent unauthorized selection.
- Including the Restricted description in the generated component 
documentation.
- Updating processor access control integration test to verify restricted 
component creation.
- Updating the developer, user, and admin guide to include the restricted 
component policy.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-3050

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1247.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1247


commit 890c42bb79d9b57ce7ed3e4d547337d76e741bc1
Author: Matt Gilman 
Date:   2016-11-18T21:19:04Z

NIFI-3050:
- Introducing a Restricted annotation for components that require elevated 
privileges to use.
- Updating the new Processor, Controller Service, and Reporting Task 
dialogs to include these details and prevent unauthorized selection.
- Including the Restricted description in the generated component 
documentation.
- Updating processor access control integration test to verify restricted 
component creation.
- Updating the developer, user, and admin guide to include the restricted 
component policy.




> Restrict dangerous processors to special permission
> ---
>
> Key: NIFI-3050
> URL: https://issues.apache.org/jira/browse/NIFI-3050
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Blocker
>  Labels: security
> Fix For: 1.1.0
>
>
> As evidenced by [NIFI-3045] and other discoveries (e.g. using an 
> {{ExecuteScript}} processor to iterate over a {{NiFiProperties}} instance 
> after the application has already decrypted the sensitive properties from the 
> {{nifi.properties}} file on disk, using a {{GetFile}} processor to retrieve 
> {{/etc/passwd}}, etc.) NiFi is a powerful tool which can allow unauthorized 
> users to perform malicious actions. While no tool as versatile as NiFi will 
> ever be completely immune to insider threat, to further restrict the 
> potential for abuse, certain processors should be designated as 
> {{restricted}}, and these processors can only be added to the canvas or 
> modified by users who, along with the proper permission to modify the canvas, 
> have a special permission to interact with these "dangerous" processors. 
> From the [Security Feature 
> Roadmap|https://cwiki.apache.org/confluence/display/NIFI/Security+Feature+Roadmap]:
> {quote}
> Dangerous Processors
> * Processors which can directly affect behavior/configuration of NiFi/other 
> services
> - {{GetFile}}
> - {{PutFile}}
> - {{ListFile}}
> - {{FetchFile}}
> - {{ExecuteScript}}
> - {{InvokeScriptedProcessor}}
> - {{ExecuteProcess}}
> - {{ExecuteStreamCommand}}
> * These processors should only be creatable/editable by users with special 
> access control policy
> * Marked by {{@Restricted}} annotation on processor class
> * All flowfiles originating/passing through these processors have special 
> attribute/protection
> * Perhaps *File processors can access a certain location by default but 
> cannot access the root filesystem without special user permission?
> {quote}
> [~mcgilman] and I should have a PR for this tomorrow. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1247: Introducing restricted components which require add...

[mcgilman]

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/1247

Introducing restricted components which require additional access

NIFI-3050:
- Introducing a Restricted annotation for components that require elevated 
privileges to use.
- Updating the new Processor, Controller Service, and Reporting Task 
dialogs to include these details and prevent unauthorized selection.
- Including the Restricted description in the generated component 
documentation.
- Updating processor access control integration test to verify restricted 
component creation.
- Updating the developer, user, and admin guide to include the restricted 
component policy.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-3050

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1247.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1247


commit 890c42bb79d9b57ce7ed3e4d547337d76e741bc1
Author: Matt Gilman 
Date:   2016-11-18T21:19:04Z

NIFI-3050:
- Introducing a Restricted annotation for components that require elevated 
privileges to use.
- Updating the new Processor, Controller Service, and Reporting Task 
dialogs to include these details and prevent unauthorized selection.
- Including the Restricted description in the generated component 
documentation.
- Updating processor access control integration test to verify restricted 
component creation.
- Updating the developer, user, and admin guide to include the restricted 
component policy.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3058) Add documentation for ZooKeeper Migrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677781#comment-15677781
 ] 

ASF GitHub Bot commented on NIFI-3058:
--

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88741590
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
+* `-h,--help`   Prints help, displays 
available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper 
Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf `   Allows the specification 
of a JAAS configuration file to allow authentication with a ZooKeeper 
configured to use Kerberos.  This option is mutually exclusive with the 
`-a,--auth` option.
+* `-r,--receive`Receives data from 
ZooKeeper and writes to the given filename (if the `-f,--file` option is 
provided) or standard output. The data received will contain the full path to 
each node read from ZooKeeper. This option is mutually exclusive with the 
`-s,--send` option.
+* `-s,--send`   Sends data to ZooKeeper 
that is read from the given filename (if the `-f,--file` option is provided) or 
standard input. The paths for each node in the data being sent to ZooKeeper are 
absolute paths, and will be stored in ZooKeeper under the *path* portion of the 
`-z,--zookeeper` argument.  Typically, the *path* portion of the argument can 
be omitted, which will store the nodes at their absolute paths. This option is 
mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper ` The ZooKeeper server to 
use, specified by a  connection string with path, in the format of 
_host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+ Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination 
ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, 
you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data 
processing).
+* If one of the ZooKeeper clusters you are using is configured with 
Kerberos, you are running the ZooKeeper Migrator from a host that has access to 
NiFi’s ZooKeeper client jaas configuration file (see <> for 
more information).
+
+1. Collect the following information:
--- End diff --

When the document is previewed (at least in Atom), the first step looks 
like it's indented along with the previouss section of the "Before you begin" 
steps.  Perhaps adding a ZooKeeper Migration Steps header for

[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...

[jtstorck]

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88741590
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
+* `-h,--help`   Prints help, displays 
available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper 
Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf `   Allows the specification 
of a JAAS configuration file to allow authentication with a ZooKeeper 
configured to use Kerberos.  This option is mutually exclusive with the 
`-a,--auth` option.
+* `-r,--receive`Receives data from 
ZooKeeper and writes to the given filename (if the `-f,--file` option is 
provided) or standard output. The data received will contain the full path to 
each node read from ZooKeeper. This option is mutually exclusive with the 
`-s,--send` option.
+* `-s,--send`   Sends data to ZooKeeper 
that is read from the given filename (if the `-f,--file` option is provided) or 
standard input. The paths for each node in the data being sent to ZooKeeper are 
absolute paths, and will be stored in ZooKeeper under the *path* portion of the 
`-z,--zookeeper` argument.  Typically, the *path* portion of the argument can 
be omitted, which will store the nodes at their absolute paths. This option is 
mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper ` The ZooKeeper server to 
use, specified by a  connection string with path, in the format of 
_host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+ Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination 
ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, 
you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data 
processing).
+* If one of the ZooKeeper clusters you are using is configured with 
Kerberos, you are running the ZooKeeper Migrator from a host that has access to 
NiFi’s ZooKeeper client jaas configuration file (see <> 
for more information).
+
+1. Collect the following information:
--- End diff --

When the document is previewed (at least in Atom), the first step looks 
like it's indented along with the previouss section of the "Before you begin" 
steps.  Perhaps adding a ZooKeeper Migration Steps header for this section?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
con

[jira] [Commented] (NIFI-3058) Add documentation for ZooKeeper Migrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1565#comment-1565
 ] 

ASF GitHub Bot commented on NIFI-3058:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88740952
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1432,7 +1432,7 @@ If Kerberos is not already setup in your environment, 
you can find information o
 
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html[_https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html_]
 . This guide assumes that Kerberos already has been installed in the 
environment in which NiFi is running.
 
-Note, the following procedures for kerberizing an Embedded Zookeeper 
server in your NiFI Node and kerberizing a zookeeper NiFI client will require 
that
+Note, the following procedures for kerberizing an Embedded ZooKeeper 
server in your NiFI Node and kerberizing a ZooKeeper NiFI client will require 
that
--- End diff --

NiFi is spelled with a capital I twice here


> Add documentation for ZooKeeper Migrator
> 
>
> Key: NIFI-3058
> URL: https://issues.apache.org/jira/browse/NIFI-3058
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.1.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> NIFI-2957 added the ZooKeeper Migrator to the NiFi Toolkit.  Documentation 
> should be added to guide the usage of the migrator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3058) Add documentation for ZooKeeper Migrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1567#comment-1567
 ] 

ASF GitHub Bot commented on NIFI-3058:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88741022
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
+* `-h,--help`   Prints help, displays 
available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper 
Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf `   Allows the specification 
of a JAAS configuration file to allow authentication with a ZooKeeper 
configured to use Kerberos.  This option is mutually exclusive with the 
`-a,--auth` option.
+* `-r,--receive`Receives data from 
ZooKeeper and writes to the given filename (if the `-f,--file` option is 
provided) or standard output. The data received will contain the full path to 
each node read from ZooKeeper. This option is mutually exclusive with the 
`-s,--send` option.
+* `-s,--send`   Sends data to ZooKeeper 
that is read from the given filename (if the `-f,--file` option is provided) or 
standard input. The paths for each node in the data being sent to ZooKeeper are 
absolute paths, and will be stored in ZooKeeper under the *path* portion of the 
`-z,--zookeeper` argument.  Typically, the *path* portion of the argument can 
be omitted, which will store the nodes at their absolute paths. This option is 
mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper ` The ZooKeeper server to 
use, specified by a  connection string with path, in the format of 
_host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+ Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination 
ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, 
you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data 
processing).
+* If one of the ZooKeeper clusters you are using is configured with 
Kerberos, you are running the ZooKeeper Migrator from a host that has access to 
NiFi’s ZooKeeper client jaas configuration file (see <> for 
more information).
+
+1. Collect the following information:
+
+|
+|*Required Information*|*Description*
+|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of 
the hosts running in the ZooKeeper ensemble, which can be found in /conf/zookeeper.properties.  Any of

[jira] [Commented] (NIFI-3058) Add documentation for ZooKeeper Migrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1566#comment-1566
 ] 

ASF GitHub Bot commented on NIFI-3058:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88741041
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
+* `-h,--help`   Prints help, displays 
available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper 
Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf `   Allows the specification 
of a JAAS configuration file to allow authentication with a ZooKeeper 
configured to use Kerberos.  This option is mutually exclusive with the 
`-a,--auth` option.
+* `-r,--receive`Receives data from 
ZooKeeper and writes to the given filename (if the `-f,--file` option is 
provided) or standard output. The data received will contain the full path to 
each node read from ZooKeeper. This option is mutually exclusive with the 
`-s,--send` option.
+* `-s,--send`   Sends data to ZooKeeper 
that is read from the given filename (if the `-f,--file` option is provided) or 
standard input. The paths for each node in the data being sent to ZooKeeper are 
absolute paths, and will be stored in ZooKeeper under the *path* portion of the 
`-z,--zookeeper` argument.  Typically, the *path* portion of the argument can 
be omitted, which will store the nodes at their absolute paths. This option is 
mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper ` The ZooKeeper server to 
use, specified by a  connection string with path, in the format of 
_host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+ Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination 
ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, 
you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data 
processing).
+* If one of the ZooKeeper clusters you are using is configured with 
Kerberos, you are running the ZooKeeper Migrator from a host that has access to 
NiFi’s ZooKeeper client jaas configuration file (see <> for 
more information).
+
+1. Collect the following information:
+
+|
+|*Required Information*|*Description*
+|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of 
the hosts running in the ZooKeeper ensemble, which can be found in /conf/zookeeper.properties.  Any of

[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88740952
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1432,7 +1432,7 @@ If Kerberos is not already setup in your environment, 
you can find information o
 
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html[_https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Configuring_a_Kerberos_5_Server.html_]
 . This guide assumes that Kerberos already has been installed in the 
environment in which NiFi is running.
 
-Note, the following procedures for kerberizing an Embedded Zookeeper 
server in your NiFI Node and kerberizing a zookeeper NiFI client will require 
that
+Note, the following procedures for kerberizing an Embedded ZooKeeper 
server in your NiFI Node and kerberizing a ZooKeeper NiFI client will require 
that
--- End diff --

NiFi is spelled with a capital I twice here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88741041
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
+* `-h,--help`   Prints help, displays 
available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper 
Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf `   Allows the specification 
of a JAAS configuration file to allow authentication with a ZooKeeper 
configured to use Kerberos.  This option is mutually exclusive with the 
`-a,--auth` option.
+* `-r,--receive`Receives data from 
ZooKeeper and writes to the given filename (if the `-f,--file` option is 
provided) or standard output. The data received will contain the full path to 
each node read from ZooKeeper. This option is mutually exclusive with the 
`-s,--send` option.
+* `-s,--send`   Sends data to ZooKeeper 
that is read from the given filename (if the `-f,--file` option is provided) or 
standard input. The paths for each node in the data being sent to ZooKeeper are 
absolute paths, and will be stored in ZooKeeper under the *path* portion of the 
`-z,--zookeeper` argument.  Typically, the *path* portion of the argument can 
be omitted, which will store the nodes at their absolute paths. This option is 
mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper ` The ZooKeeper server to 
use, specified by a  connection string with path, in the format of 
_host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+ Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination 
ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, 
you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data 
processing).
+* If one of the ZooKeeper clusters you are using is configured with 
Kerberos, you are running the ZooKeeper Migrator from a host that has access to 
NiFi’s ZooKeeper client jaas configuration file (see <> 
for more information).
+
+1. Collect the following information:
+
+|
+|*Required Information*|*Description*
+|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of 
the hosts running in the ZooKeeper ensemble, which can be found in /conf/zookeeper.properties.  Any of the hostnames declared in 
the *server.N* properties can be used.
+|Destination ZooKeeper hostname (*destinationHostname*)|The hostname must 
be one of the hosts running in the ZooKeeper ensemble, which can be found in 
/conf/zookeeper.properties

[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88741022
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
+* `-h,--help`   Prints help, displays 
available parameters with descriptions
+* `--ignore-source` Allows the ZooKeeper 
Migrator to write to the ZooKeeper and path from which the data was obtained.
+* `-k,--krb-conf `   Allows the specification 
of a JAAS configuration file to allow authentication with a ZooKeeper 
configured to use Kerberos.  This option is mutually exclusive with the 
`-a,--auth` option.
+* `-r,--receive`Receives data from 
ZooKeeper and writes to the given filename (if the `-f,--file` option is 
provided) or standard output. The data received will contain the full path to 
each node read from ZooKeeper. This option is mutually exclusive with the 
`-s,--send` option.
+* `-s,--send`   Sends data to ZooKeeper 
that is read from the given filename (if the `-f,--file` option is provided) or 
standard input. The paths for each node in the data being sent to ZooKeeper are 
absolute paths, and will be stored in ZooKeeper under the *path* portion of the 
`-z,--zookeeper` argument.  Typically, the *path* portion of the argument can 
be omitted, which will store the nodes at their absolute paths. This option is 
mutually exclusive with the `-r,--receive` option.
+* `-z,--zookeeper ` The ZooKeeper server to 
use, specified by a  connection string with path, in the format of 
_host:port/znode/path_.
+
+[[migrating_between_source_destination_zookeepers]]
+ Migrating Between Source and Destination ZooKeepers
+
+Before you begin, confirm that:
+
+* You have installed the destination ZooKeeper cluster.
+* You have installed and configured a NiFi cluster to use the destination 
ZooKeeper cluster.
+* If you are migrating ZooKeepers due to upgrading NiFi from 0.x to 1.x,, 
you have already followed appropriate NiFi upgrade steps.
+* You have configured Kerberos as needed.
+* You have not started processing any dataflow (to avoid duplicate data 
processing).
+* If one of the ZooKeeper clusters you are using is configured with 
Kerberos, you are running the ZooKeeper Migrator from a host that has access to 
NiFi’s ZooKeeper client jaas configuration file (see <> 
for more information).
+
+1. Collect the following information:
+
+|
+|*Required Information*|*Description*
+|Source ZooKeeper hostname (*sourceHostname*)|The hostname must be one of 
the hosts running in the ZooKeeper ensemble, which can be found in /conf/zookeeper.properties.  Any of the hostnames declared in 
the *server.N* properties can be used.
+|Destination ZooKeeper hostname (*destinationHostname*)|The hostname must 
be one of the hosts running in the ZooKeeper ensemble, which can be found in 
/conf/zookeeper.properties

[jira] [Commented] (NIFI-3058) Add documentation for ZooKeeper Migrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1561#comment-1561
 ] 

ASF GitHub Bot commented on NIFI-3058:
--

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88740943
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
--- End diff --

Suggest changing the first sentence to "The file used for ZooKeeper data 
serialized as JSON."


> Add documentation for ZooKeeper Migrator
> 
>
> Key: NIFI-3058
> URL: https://issues.apache.org/jira/browse/NIFI-3058
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.1.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> NIFI-2957 added the ZooKeeper Migrator to the NiFi Toolkit.  Documentation 
> should be added to guide the usage of the migrator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...

[jtstorck]

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88740943
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
+link:https://nifi.apache.org/download.html[Apache NiFi download page].
+
+[[zk_migrator_command_line_parameters]]
+ zk-migrator.sh Command Line Parameters
+
+You can use the following command line options with the ZooKeeper Migrator:
+
+* `-a,--auth `   Allows the specification 
of a username and password for authentication with ZooKeeper.  This option is 
mutually exclusive with the `-k,--krb-conf` option.
+* `-f,--file `The file to be used for 
JSON data.  When used with the `-r,--receive` option, data read from ZooKeeper 
will be stored in the given filename.  When used with the `-s,--send` option, 
the data in the file will be sent to ZooKeeper.
--- End diff --

Suggest changing the first sentence to "The file used for ZooKeeper data 
serialized as JSON."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...

[jtstorck]

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88740528
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
--- End diff --

Might want to remove the mention of zk-migrator.sh here, since the tool 
provites a .bat file for Windows as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3058) Add documentation for ZooKeeper Migrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677766#comment-15677766
 ] 

ASF GitHub Bot commented on NIFI-3058:
--

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88740528
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
+* Migrating ZooKeeper node ownership
+
+For example, you may want to use the ZooKeeper Migrator when you are:
+
+* Upgrading from NiFi 0.x to NiFi 1.x in which embedded ZooKeepers are used
+* Migrating from an embedded ZooKeeper in NiFi 0.x or 1.x to an external 
ZooKeeper
+* Upgrading from NiFi 0.x with an external ZooKeeper to NiFi 1.x with the 
same external ZooKeeper
+* Migrating from an external ZooKeeper to an embedded ZooKeeper in NiFi 1.x
+
+The NiFi ZooKeeper Migrator (`zk-migrator.sh`) is part of the NiFi Toolkit 
and is downloaded separately from the
--- End diff --

Might want to remove the mention of zk-migrator.sh here, since the tool 
provites a .bat file for Windows as well.


> Add documentation for ZooKeeper Migrator
> 
>
> Key: NIFI-3058
> URL: https://issues.apache.org/jira/browse/NIFI-3058
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.1.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> NIFI-2957 added the ZooKeeper Migrator to the NiFi Toolkit.  Documentation 
> should be added to guide the usage of the migrator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3058) Add documentation for ZooKeeper Migrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677763#comment-15677763
 ] 

ASF GitHub Bot commented on NIFI-3058:
--

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88740216
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
--- End diff --

Suggest changing to "Moving" instead of "Move"


> Add documentation for ZooKeeper Migrator
> 
>
> Key: NIFI-3058
> URL: https://issues.apache.org/jira/browse/NIFI-3058
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.1.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> NIFI-2957 added the ZooKeeper Migrator to the NiFi Toolkit.  Documentation 
> should be added to guide the usage of the migrator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...

[jtstorck]

Github user jtstorck commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1246#discussion_r88740216
  
--- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc ---
@@ -1599,7 +1599,97 @@ java.arg.16=-Dsun.security.krb5.debug=true
 This will cause the debug output to be written to the NiFi Bootstrap log 
file. By default, this is located at _$NIFI_HOME/logs/nifi-bootstrap.log_.
 This output can be rather verbose but provides extremely valuable 
information for troubleshooting Kerberos failures.
 
+[[zookeeper_migrator]]
+=== ZooKeeper Migrator
+You can use the NiFi ZooKeeper Migrator to perform the following tasks:
 
+* Move ZooKeeper information from one ZooKeeper cluster to another
--- End diff --

Suggest changing to "Moving" instead of "Move"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-3058) Add documentation for ZooKeeper Migrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677707#comment-15677707
 ] 

ASF GitHub Bot commented on NIFI-3058:
--

GitHub user andrewmlim opened a pull request:

https://github.com/apache/nifi/pull/1246

NIFI-3058 Add documentation for ZooKeeper Migrator to Admin Guide

Also, corrected some incorrectly capitalized references to ZooKeeper.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewmlim/nifi NIFI-3058

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1246.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1246


commit c3e65387650cc1d37cbf0b870fe02b66726f2d10
Author: Andrew Lim 
Date:   2016-11-18T20:41:04Z

NIFI-3058 Add documentation for ZooKeeper Migrator to Admin Guide




> Add documentation for ZooKeeper Migrator
> 
>
> Key: NIFI-3058
> URL: https://issues.apache.org/jira/browse/NIFI-3058
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.1.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> NIFI-2957 added the ZooKeeper Migrator to the NiFi Toolkit.  Documentation 
> should be added to guide the usage of the migrator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1246: NIFI-3058 Add documentation for ZooKeeper Migrator ...

[andrewmlim]

GitHub user andrewmlim opened a pull request:

https://github.com/apache/nifi/pull/1246

NIFI-3058 Add documentation for ZooKeeper Migrator to Admin Guide

Also, corrected some incorrectly capitalized references to ZooKeeper.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrewmlim/nifi NIFI-3058

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1246.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1246


commit c3e65387650cc1d37cbf0b870fe02b66726f2d10
Author: Andrew Lim 
Date:   2016-11-18T20:41:04Z

NIFI-3058 Add documentation for ZooKeeper Migrator to Admin Guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3066) MergeContent: "Cannot migrate FlowFiles from a Process Session to itself"

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-3066:
-
Assignee: Mark Payne
  Status: Patch Available  (was: Open)

> MergeContent: "Cannot migrate FlowFiles from a Process Session to itself"
> -
>
> Key: NIFI-3066
> URL: https://issues.apache.org/jira/browse/NIFI-3066
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Joseph Gresock
>Assignee: Mark Payne
>Priority: Critical
> Fix For: 1.1.0
>
>
> With the following properties in MergeContent, I get this error when I try to 
> process a flow file (which is 135MB):
> Merge Strategy = Bin-Packing Algorithm
> Merge Format = ZIP
> Attribute Strategy = Keep Only Common Attributes
> Correlation Attribute Name = bundle.identifier
> Minimum Number of Entries = 200
> Maximum Number of Entries = 200
> Minimum Group Size = 20 MB
> Maximum Group Size = *20 MB*
> Max Bin Age = 1 min
> Maximum number of Bins = 100
> Delimiter Strategy = Filename
> Keep Path = false
> {code}
> 016-11-18 18:00:13,752 ERROR [Timer-Driven Process Thread-83] 
> o.a.n.processors.standard.MergeContent
> java.lang.IllegalArgumentException: Cannot migrate FlowFiles from a Process 
> Session to itself
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1091)
>  ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142) 
> ~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at 
> org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:282) 
> ~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at 
> org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178) 
> ~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1089)
>  ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> ...
> 2016-11-18 18:00:13,753 WARN [Timer-Driven Process Thread-83] 
> o.a.n.processors.standard.MergeContent 
> MergeContent[id=a9476272-dab0-3d2f-acef-abf11f7d1b80] Processor 
> Administratively Yielded for 1 sec due to processing failure
> {code}
> However, other smaller flow files are able to successfully go through the 
> processor.  Not sure if it's related to the size of the flow file or not.  
> Perhaps there just needs to be a check around this line in Bin.java:142, to 
> make sure we don't try to migrate a flow file to its own session:
> {code}
> session.migrate(getSession(), Collections.singleton(flowFile));
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1245: Nifi 3066

[markap14]

Github user markap14 commented on the issue:

https://github.com/apache/nifi/pull/1245
  
This PR contains two commits. The first just consists of a unit test 
reproducing the issue. The second commit addresses the issue, so that the first 
commit will now pass all unit tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #1245: Nifi 3066

[markap14]

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/1245

Nifi 3066

Thank you for submitting a contribution to Apache NiFi.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced 
 in the commit message?

- [ ] Does your PR title start with NIFI- where  is the JIRA number 
you are trying to resolve? Pay particular attention to the hyphen "-" character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn 
-Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] If applicable, have you updated the LICENSE file, including the main 
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main 
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to 
.name (programmatic access) for each of the new properties?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-3066

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/1245.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1245


commit b406119dab33d8bfc6b54a5c2f38144749ce1c2f
Author: Mark Payne 
Date:   2016-11-18T20:21:56Z

NIFI-3066: Create unit test to reproduce bug that results in 
IllegalArgumentException: Cannot migrate FlowFiles from a Process Session to 
itself

commit af727b110441061dbf92ddea811a5b1e8031c44b
Author: Mark Payne 
Date:   2016-11-18T20:29:37Z

NIFI-3066: Ensure that when a Bin is created, it is always created with its 
own new session




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (NIFI-32) Swap File format should include summary at end for faster NiFi system startup

[Oleg Zhurakousky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-32?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Zhurakousky resolved NIFI-32.
--
Resolution: Resolved

> Swap File format should include summary at end for faster NiFi system startup
> -
>
> Key: NIFI-32
> URL: https://issues.apache.org/jira/browse/NIFI-32
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Minor
> Fix For: 1.1.0
>
>
> When NiFi is restarted with many hundreds or thousands of Swap Files, it can 
> take a very long time to startup.
> If when we write out the Swap Files, we append a summary of the Swap File 
> (queue identifier, number of FlowFiles, size of FlowFiles, and ContentClaim 
> info/count map), we can then read far less data for restart. This means we 
> would serialize the summary to a ByteArrayOutputStream, then get the length, 
> and then write out the summary, followed by length. On restore, we read just 
> the last 8 bytes, and that allows us to generate the offset into the file to 
> start reading the summary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[Oleg Zhurakousky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Zhurakousky updated NIFI-2854:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known to older version, if seen after a rollback, 
> will simply be ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677583#comment-15677583
 ] 

ASF subversion and git services commented on NIFI-2854:
---

Commit 1be08714731f01347ac1f98e18047fe7d9ab8afd in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=1be0871 ]

NIFI-2854: Refactor repositories and swap files to use schema-based 
serialization so that nifi can be rolled back to a previous version after an 
upgrade.

NIFI-2854: Incorporated PR review feedback

NIFI-2854: Implemented feedback from PR Review

NIFI-2854: Ensure that all resources are closed on 
CompressableRecordReader.close() even if an IOException is thrown when closing 
one of them

This closes #1202


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known to older version, if seen after a rollback, 
> will simply be ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677580#comment-15677580
 ] 

ASF subversion and git services commented on NIFI-2854:
---

Commit 1be08714731f01347ac1f98e18047fe7d9ab8afd in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=1be0871 ]

NIFI-2854: Refactor repositories and swap files to use schema-based 
serialization so that nifi can be rolled back to a previous version after an 
upgrade.

NIFI-2854: Incorporated PR review feedback

NIFI-2854: Implemented feedback from PR Review

NIFI-2854: Ensure that all resources are closed on 
CompressableRecordReader.close() even if an IOException is thrown when closing 
one of them

This closes #1202


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known to older version, if seen after a rollback, 
> will simply be ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677584#comment-15677584
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1202


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known to older version, if seen after a rollback, 
> will simply be ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-3064) ExecuteSQL convert to Avro fails when a MySQL query returns two columns with the same name even with different aliases

[Karthik Narayanan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Narayanan reassigned NIFI-3064:
---

Assignee: Karthik Narayanan  (was: Matt Burgess)

> ExecuteSQL convert to Avro fails when a MySQL query returns two columns with 
> the same name even with different aliases
> --
>
> Key: NIFI-3064
> URL: https://issues.apache.org/jira/browse/NIFI-3064
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.7.0, 0.6.1
>Reporter: Karthik Narayanan
>Assignee: Karthik Narayanan
>
> For the following query:
> select A.host as host1,B.host as host from `hosts` A  join host2 B on 
> B.hostid=A.hostid;
> A.host and B.host are columns that have same name, but come from different 
> tables, and have a different alias name. But looks like Avro conversion is 
> ignoring the alias and just looking at the column name. This causes the query 
> to fail with this error
> Duplicate Field Name in any.data.hosts ; name type UNION pos:1 and name type 
> UNION pos:0.
> I have attached a screenshot for the same. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677581#comment-15677581
 ] 

ASF subversion and git services commented on NIFI-2854:
---

Commit 1be08714731f01347ac1f98e18047fe7d9ab8afd in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=1be0871 ]

NIFI-2854: Refactor repositories and swap files to use schema-based 
serialization so that nifi can be rolled back to a previous version after an 
upgrade.

NIFI-2854: Incorporated PR review feedback

NIFI-2854: Implemented feedback from PR Review

NIFI-2854: Ensure that all resources are closed on 
CompressableRecordReader.close() even if an IOException is thrown when closing 
one of them

This closes #1202


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known to older version, if seen after a rollback, 
> will simply be ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677582#comment-15677582
 ] 

ASF subversion and git services commented on NIFI-2854:
---

Commit 1be08714731f01347ac1f98e18047fe7d9ab8afd in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=1be0871 ]

NIFI-2854: Refactor repositories and swap files to use schema-based 
serialization so that nifi can be rolled back to a previous version after an 
upgrade.

NIFI-2854: Incorporated PR review feedback

NIFI-2854: Implemented feedback from PR Review

NIFI-2854: Ensure that all resources are closed on 
CompressableRecordReader.close() even if an IOException is thrown when closing 
one of them

This closes #1202


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known to older version, if seen after a rollback, 
> will simply be ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1202


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2996) Processor/Service validation takes exponentially longer to run as the graph grows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677505#comment-15677505
 ] 

ASF GitHub Bot commented on NIFI-2996:
--

Github user mosermw commented on the issue:

https://github.com/apache/nifi/pull/1192
  
@mcgilman I updated this PR for the following:

- reverted isValid() change
- modified FlowController to avoid calling isValid() if disabled or 
running.  this matches similar logic for ports that is already in place
- modified getValidationErrors() to be all or nothing
- make getValidationErrors() act the same for Processors, Ports, Reporting 
Tasks, and Controller Services


> Processor/Service validation takes exponentially longer to run as the graph 
> grows
> -
>
> Key: NIFI-2996
> URL: https://issues.apache.org/jira/browse/NIFI-2996
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Michael Moser
>Assignee: Michael Moser
>
> As you add processors that reference controller services to your NiFi, the 
> validation that occurs during normal UI usage increases dramatically.
> When running in a cluster, I have to increase the nifi.properties 
> nifi.cluster.node.read.timeout well beyond its 5 sec default timeout in order 
> for the UI to work.  Eventually, simple operations in the UI take close to a 
> minute to happen.
> As a test, I created an SSLContextService using certs created with the 
> amazingly useful nifi-toolkit.  I created a DistributedMapCacheClientService 
> that references this SSLContextService.  Then I created 108 
> FetchDistributedMapCache processors that reference the 
> DistributedMapCacheClient service. I used ${hostname(true)} for my 
> FetchDistributedMapCache processor's Cache Entry Identifier property.
> When NiFi is up with no UI connected, during a single "NiFi status" phase I 
> noticed that the SSLContextService was validated 108 times and the EL 
> hostname(true) was evaluated 216 times.  When I connect the UI and go through 
> a normal status refresh cycle, the SSLContextService was validated 432 times 
> and the EL hostname(true) was evaluated 864 times. These validations take a 
> full second on my slowest machine.
> In NiFi 0.x, the expensive REST API call is 
> /nifi-api/controller/controller-services/node.
> In NiFi 1.x, it appears to divide the work among REST API calls to 
> /nifi-api/process-groups/UUID and 
> /nifi-api/flow/process-groups/UUID/controller-services and 
> /nifi-api/flow/status
> Rather than attempting to fix StandardSSLContextService or the EL 
> HostnameEvaluator, I wonder if there was a more generic approach to helping 
> resolve this?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1192: NIFI-2996 validate processors only when they are in STOPPE...

[mosermw]

Github user mosermw commented on the issue:

https://github.com/apache/nifi/pull/1192
  
@mcgilman I updated this PR for the following:

- reverted isValid() change
- modified FlowController to avoid calling isValid() if disabled or 
running.  this matches similar logic for ports that is already in place
- modified getValidationErrors() to be all or nothing
- make getValidationErrors() act the same for Processors, Ports, Reporting 
Tasks, and Controller Services


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3052) Update screenshots in Admin Guide and Getting Starting guide for additional colors added to UI

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim updated NIFI-3052:
-
Fix Version/s: 1.1.0

> Update screenshots in Admin Guide and Getting Starting guide for additional 
> colors added to UI
> --
>
> Key: NIFI-3052
> URL: https://issues.apache.org/jira/browse/NIFI-3052
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.1.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
> Fix For: 1.1.0
>
>
> Additional color was added in the UI via NIFI-2603.
> Some of the screenshots in the Getting Started Guide and Admin Guides need to 
> be updated for these changes, mostly around the processor Alert, Run and Stop 
> icons now having color.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2954) Bouncycastle dependencies are duplicated throughout NARs

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677484#comment-15677484
 ] 

Joseph Witt commented on NIFI-2954:
---

okyeah I agree on the package naming.

I will change the package naming back for those two classes (StandardValidators 
and FLowFileFilters).  This way there will be actually no impact to existing 
users at all since nifi-processor-utils pulls that in anyway.  They won't even 
realized it moved modules.  It helped me to change package names during the 
work so I could more easily find candidates to tighten dependencies on.  Will 
do this now.

> Bouncycastle dependencies are duplicated throughout NARs
> 
>
> Key: NIFI-2954
> URL: https://issues.apache.org/jira/browse/NIFI-2954
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Affects Versions: 1.0.0
>Reporter: Aldrin Piri
>Assignee: Joseph Witt
>Priority: Blocker
> Fix For: 1.1.0
>
>
> I was working with some of the NiFi dependencies for incorporation with/use 
> by MiNiFi and saw that several libraries were duplicated throughout all NARs.
> Of note were the bouncycastle dependencies which, together, are ~4MB in size.
> {code}
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:23]
> $ find {lib,work} -type f -name '*bcprov-jdk15on*' | wc -l
>   54
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:28]
> $ find {lib,work} -type f -name '*bcpkix*' | wc -l
>   53
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:33]
> $ ls -lash 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcprov-jdk15on-1.54.jar
> 3.2M -rw-r--r-- 1 apiri admin 3.2M Oct  4 12:24 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcprov-jdk15on-1.54.jar
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:39]
> $ ls -lash 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcpkix-jdk15on-1.54.jar
> 660K -rw-r--r-- 1 apiri admin 658K Oct  4 12:24 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcpkix-jdk15on-1.54.jar
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3052) Update screenshots in Admin Guide and Getting Starting guide for additional colors added to UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677482#comment-15677482
 ] 

ASF GitHub Bot commented on NIFI-3052:
--

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi/pull/1239
  
Thanks @andrewmlim this has been merged to master.


> Update screenshots in Admin Guide and Getting Starting guide for additional 
> colors added to UI
> --
>
> Key: NIFI-3052
> URL: https://issues.apache.org/jira/browse/NIFI-3052
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.1.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> Additional color was added in the UI via NIFI-2603.
> Some of the screenshots in the Getting Started Guide and Admin Guides need to 
> be updated for these changes, mostly around the processor Alert, Run and Stop 
> icons now having color.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-3067) Upgrade ZooKeeper dependency version

[Jeff Storck (JIRA)]

Jeff Storck created NIFI-3067:
-

 Summary: Upgrade ZooKeeper dependency version
 Key: NIFI-3067
 URL: https://issues.apache.org/jira/browse/NIFI-3067
 Project: Apache NiFi
  Issue Type: Improvement
  Components: Core Framework, Tools and Build
Affects Versions: 1.0.0, 1.1.0
Reporter: Jeff Storck
 Fix For: 1.2.0


ZooKeeper 3.4.6 has dependencies on log4j, although it uses slf4j as the 
logging API; if the log4j dependency is excluded from ZooKeeper 3.4.6 in favor 
of using logback, ZooKeeper has runtime issues.  

ZooKeeper 3.4.6's hard dependency on log4j has caused issues in NiFi Toolkit, 
which originally used logback.  When the ZooKeeper Migrator was added to NiFi 
Toolkit, the log4j dependency came with it, which broke the logging in NiFi 
Toolkit due to having both logback and log4j on the classpath.  To resolve the 
issue in NiFI Toolkit, the logback dependency has been removed, and it now 
explicitly uses log4j.

In ZooKeeper 3.5.2 and 3.6.0, the dependency on log4j has been removed.  NiFi 
(and the Toolkit) would benefit from upgrading to one of these versions of 
ZooKeeper, allowing the usage of log4j or logback without the classpath and 
runtime issues mentioned above.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1239: NIFI-3052 Update screenshots in Admin Guide and Getting St...

[scottyaslan]

Github user scottyaslan commented on the issue:

https://github.com/apache/nifi/pull/1239
  
Thanks @andrewmlim this has been merged to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (NIFI-3052) Update screenshots in Admin Guide and Getting Starting guide for additional colors added to UI

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan resolved NIFI-3052.
---
Resolution: Fixed

Please make sure to make the changes in 
https://issues.apache.org/jira/browse/NIFI-3056 for 1.1.0.

> Update screenshots in Admin Guide and Getting Starting guide for additional 
> colors added to UI
> --
>
> Key: NIFI-3052
> URL: https://issues.apache.org/jira/browse/NIFI-3052
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.1.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> Additional color was added in the UI via NIFI-2603.
> Some of the screenshots in the Getting Started Guide and Admin Guides need to 
> be updated for these changes, mostly around the processor Alert, Run and Stop 
> icons now having color.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3052) Update screenshots in Admin Guide and Getting Starting guide for additional colors added to UI

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677474#comment-15677474
 ] 

ASF GitHub Bot commented on NIFI-3052:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1239


> Update screenshots in Admin Guide and Getting Starting guide for additional 
> colors added to UI
> --
>
> Key: NIFI-3052
> URL: https://issues.apache.org/jira/browse/NIFI-3052
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.1.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> Additional color was added in the UI via NIFI-2603.
> Some of the screenshots in the Getting Started Guide and Admin Guides need to 
> be updated for these changes, mostly around the processor Alert, Run and Stop 
> icons now having color.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3052) Update screenshots in Admin Guide and Getting Starting guide for additional colors added to UI

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677472#comment-15677472
 ] 

ASF subversion and git services commented on NIFI-3052:
---

Commit 5a25884f52e92b3c73f425a1dd6c0993866b1b8f in nifi's branch 
refs/heads/master from [~andrewmlim]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=5a25884 ]

NIFI-3052 Update screenshots in Admin Guide and Getting Starting guide for 
additional colors added to UI
This closes #1239


> Update screenshots in Admin Guide and Getting Starting guide for additional 
> colors added to UI
> --
>
> Key: NIFI-3052
> URL: https://issues.apache.org/jira/browse/NIFI-3052
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Documentation & Website
>Affects Versions: 1.1.0
>Reporter: Andrew Lim
>Assignee: Andrew Lim
>Priority: Minor
>
> Additional color was added in the UI via NIFI-2603.
> Some of the screenshots in the Getting Started Guide and Admin Guides need to 
> be updated for these changes, mostly around the processor Alert, Run and Stop 
> icons now having color.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #1239: NIFI-3052 Update screenshots in Admin Guide and Get...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/1239


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2954) Bouncycastle dependencies are duplicated throughout NARs

[Mark Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677464#comment-15677464
 ] 

Mark Payne commented on NIFI-2954:
--

@joewitt I agree. I am not a fan of having to move the classes around like 
this, but we should consider the following key points here:
* Old NAR's that were previously built should still work without a problem.
* In order for a third-party NAR to use the 1.1.0 dependencies, they are 
already required to update their pom.xml (or equivalent) to point to a new 
artifact. We are simply changing which artifact that they should point at.
* I'd like to avoid changing any package names but I understand why that was 
done. I'm a +0 there.

We definitely do need to get rid of those extra bouncy castle jars, though. 
Kind of a rock & a hard place. But I think the pro's do outweigh the con's in 
this case, and I think we are operating within the scope of the Versioning 
Guide that we have, provided that the Migration Guidance has a blurb supplied 
that details exactly what must be done to depend on the validators.

Overall I'm a +1.

> Bouncycastle dependencies are duplicated throughout NARs
> 
>
> Key: NIFI-2954
> URL: https://issues.apache.org/jira/browse/NIFI-2954
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Affects Versions: 1.0.0
>Reporter: Aldrin Piri
>Assignee: Joseph Witt
>Priority: Blocker
> Fix For: 1.1.0
>
>
> I was working with some of the NiFi dependencies for incorporation with/use 
> by MiNiFi and saw that several libraries were duplicated throughout all NARs.
> Of note were the bouncycastle dependencies which, together, are ~4MB in size.
> {code}
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:23]
> $ find {lib,work} -type f -name '*bcprov-jdk15on*' | wc -l
>   54
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:28]
> $ find {lib,work} -type f -name '*bcpkix*' | wc -l
>   53
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:33]
> $ ls -lash 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcprov-jdk15on-1.54.jar
> 3.2M -rw-r--r-- 1 apiri admin 3.2M Oct  4 12:24 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcprov-jdk15on-1.54.jar
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:39]
> $ ls -lash 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcpkix-jdk15on-1.54.jar
> 660K -rw-r--r-- 1 apiri admin 658K Oct  4 12:24 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcpkix-jdk15on-1.54.jar
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677460#comment-15677460
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user olegz commented on the issue:

https://github.com/apache/nifi/pull/1202
  
LGTM, merging


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/backward compatibility and specifically this means that 
> things like the way each is serialized to disk cannot change.
> * If changes are made which impact forward or backward compatibility they 
> should be reserved for major releases only and should include a utility to 
> help users with pre-existing data convert from some older format to the newer 
> format.  It may not be feasible to have rollback on major releases.
> * The content repository should not be changed within a major release cycle 
> in any way that will harm forward or backward compatibility.
> * The flow file repository can change in that new fields can be added to 
> existing write ahead log record types but no fields can be removed nor can 
> any new types be added.  Once a field is considered required it must remain 
> required.  Changes may only be made across minor version changes - not 
> incremental.
> * Swap File storage should follow very similar rules to the flow file 
> repository.  Adding a schema to the swap file header may allow some variation 
> there but the variation should only be hints to optimize how they're 
> processed and not change their behavior otherwise. Changes are only permitted 
> during minor version releases.
> * Provenance repository changes are only permitted during minor version 
> releases.  These changes may include adding or removing fields from existing 
> event types.  If a field is considered required it must always be considered 
> required.  If a field is removed then it must not be a required field and 
> there must be a sensible default an older version could use if that value is 
> not found in new data once rolled back.  New event types may be added.  
> Fields or event types not known to older version, if seen after a rollback, 
> will simply be ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #1202: NIFI-2854: Refactor repositories and swap files to use sch...

[olegz]

Github user olegz commented on the issue:

https://github.com/apache/nifi/pull/1202
  
LGTM, merging


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[Joseph Percivall (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Percivall updated NIFI-3011:
---
Fix Version/s: 1.1.0

> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
> Fix For: 1.1.0
>
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-3011) Support Elasticsearch 5.0 for Put/FetchElasticsearch processors

[Joseph Percivall (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677430#comment-15677430
 ] 

Joseph Percivall commented on NIFI-3011:


Adding a fix version of 1.1.0 as there is minimal work left (just need to add 
L&N)

> Support Elasticsearch 5.0 for Put/FetchElasticsearch processors
> ---
>
> Key: NIFI-3011
> URL: https://issues.apache.org/jira/browse/NIFI-3011
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Matt Burgess
>Assignee: Matt Burgess
>
> Now that Elastic has released a new major version (5.0) of Elasticsearch, the 
> Put/FetchElasticsearch processors would need to be upgraded (or duplicated) 
> as the major version of the transport client needs to match the major version 
> of the Elasticsearch cluster.
> If upgrade is selected, then Put/FetchES will no longer work with 
> Elasticsearch 2.x clusters, so in that case users would want to switch to the 
> Http versions of those processors. However this might not be desirable (due 
> to performance concerns with the HTTP API vs the transport API), so care must 
> be taken when deciding whether to upgrade the existing processors or create 
> new ones.
> Creating new versions of these processors (to use the 5.0 transport client) 
> will also take some consideration, as it is unlikely the different versions 
> can coexist in the same NAR due to classloading issues (multiple versions of 
> JARs containing the same class names, e.g.). It may be necessary to create an 
> "elasticsearch-5.0" version of the NAR, containing only the new versions of 
> these processors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-3024) Encrypted configuration migrator should be able to update sensitive properties key and migrate flow.xml.gz

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto updated NIFI-3024:

Fix Version/s: 1.1.0

> Encrypted configuration migrator should be able to update sensitive 
> properties key and migrate flow.xml.gz
> --
>
> Key: NIFI-3024
> URL: https://issues.apache.org/jira/browse/NIFI-3024
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Configuration, Tools and Build
>Affects Versions: 1.0.0
>Reporter: Bryan Rosander
>Assignee: Bryan Rosander
>  Labels: config, encryption, security, serialization
> Fix For: 1.1.0
>
>
> In order to allow changing of nifi.sensitive.props.key and updating of the 
> flow.xml.gz, the ConfigEncryptionTool should be able to accept a new value 
> for that field and update encrypted values in the flow.xml.gz appropriately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-3024) Encrypted configuration migrator should be able to update sensitive properties key and migrate flow.xml.gz

[Andy LoPresto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy LoPresto reassigned NIFI-3024:
---

Assignee: Andy LoPresto  (was: Bryan Rosander)

> Encrypted configuration migrator should be able to update sensitive 
> properties key and migrate flow.xml.gz
> --
>
> Key: NIFI-3024
> URL: https://issues.apache.org/jira/browse/NIFI-3024
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Configuration, Tools and Build
>Affects Versions: 1.0.0
>Reporter: Bryan Rosander
>Assignee: Andy LoPresto
>  Labels: config, encryption, security, serialization
> Fix For: 1.1.0
>
>
> In order to allow changing of nifi.sensitive.props.key and updating of the 
> flow.xml.gz, the ConfigEncryptionTool should be able to accept a new value 
> for that field and update encrypted values in the flow.xml.gz appropriately.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-3041) Remove references to Windows start-nifi.bat tool in Admin Guide

[Andrew Lim (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Lim reassigned NIFI-3041:


Assignee: Andrew Lim

> Remove references to Windows start-nifi.bat tool in Admin Guide
> ---
>
> Key: NIFI-3041
> URL: https://issues.apache.org/jira/browse/NIFI-3041
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0, 0.7.1
>Reporter: Andy LoPresto
>Assignee: Andrew Lim
>  Labels: documentation, windows
>
> As documented by this question on HCC [1], there are references in the NiFi 
> quick start guide [2] and admin guide [3] that reference Windows shell script 
> artifacts that no longer exist for running NiFi in the foreground/background. 
> The documentation should be updated to be correct, and new documentation or a 
> new tool should be provided to allow users to run NiFi in the background on 
> Windows [4]. 
> [1] 
> https://community.hortonworks.com/questions/66408/nifis-start-nifibat-documentation.html
> [2] https://nifi.apache.org/quickstart.html
> [3] 
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#how-to-install-and-start-nifi
> [3] 
> https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#for-windows-users



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-3066) MergeContent: "Cannot migrate FlowFiles from a Process Session to itself"

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-3066:
--
Priority: Critical  (was: Major)

> MergeContent: "Cannot migrate FlowFiles from a Process Session to itself"
> -
>
> Key: NIFI-3066
> URL: https://issues.apache.org/jira/browse/NIFI-3066
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Joseph Gresock
>Priority: Critical
> Fix For: 1.1.0
>
>
> With the following properties in MergeContent, I get this error when I try to 
> process a flow file (which is 135MB):
> Merge Strategy = Bin-Packing Algorithm
> Merge Format = ZIP
> Attribute Strategy = Keep Only Common Attributes
> Correlation Attribute Name = bundle.identifier
> Minimum Number of Entries = 200
> Maximum Number of Entries = 200
> Minimum Group Size = 20 MB
> Maximum Group Size = *20 MB*
> Max Bin Age = 1 min
> Maximum number of Bins = 100
> Delimiter Strategy = Filename
> Keep Path = false
> {code}
> 016-11-18 18:00:13,752 ERROR [Timer-Driven Process Thread-83] 
> o.a.n.processors.standard.MergeContent
> java.lang.IllegalArgumentException: Cannot migrate FlowFiles from a Process 
> Session to itself
> at 
> org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1091)
>  ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142) 
> ~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at 
> org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:282) 
> ~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at 
> org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178) 
> ~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> at 
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1089)
>  ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
> ...
> 2016-11-18 18:00:13,753 WARN [Timer-Driven Process Thread-83] 
> o.a.n.processors.standard.MergeContent 
> MergeContent[id=a9476272-dab0-3d2f-acef-abf11f7d1b80] Processor 
> Administratively Yielded for 1 sec due to processing failure
> {code}
> However, other smaller flow files are able to successfully go through the 
> processor.  Not sure if it's related to the size of the flow file or not.  
> Perhaps there just needs to be a check around this line in Bin.java:142, to 
> make sure we don't try to migrate a flow file to its own session:
> {code}
> session.migrate(getSession(), Collections.singleton(flowFile));
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-3066) MergeContent: "Cannot migrate FlowFiles from a Process Session to itself"

[Joseph Gresock (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Gresock updated NIFI-3066:
-
Description: 
With the following properties in MergeContent, I get this error when I try to 
process a flow file (which is 135MB):

Merge Strategy = Bin-Packing Algorithm
Merge Format = ZIP
Attribute Strategy = Keep Only Common Attributes
Correlation Attribute Name = bundle.identifier
Minimum Number of Entries = 200
Maximum Number of Entries = 200
Minimum Group Size = 20 MB
Maximum Group Size = *20 MB*
Max Bin Age = 1 min
Maximum number of Bins = 100
Delimiter Strategy = Filename
Keep Path = false

016-11-18 18:00:13,752 ERROR [Timer-Driven Process Thread-83] 
o.a.n.processors.standard.MergeContent
java.lang.IllegalArgumentException: Cannot migrate FlowFiles from a Process 
Session to itself
at 
org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1091)
 ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142) 
~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:282) 
~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178) 
~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1089)
 ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_111]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
[na:1.8.0_111]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [na:1.8.0_111]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [na:1.8.0_111]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_111]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2016-11-18 18:00:13,753 WARN [Timer-Driven Process Thread-83] 
o.a.n.processors.standard.MergeContent 
MergeContent[id=a9476272-dab0-3d2f-acef-abf11f7d1b80] Processor 
Administratively Yielded for 1 sec due to processing failure

However, other smaller flow files are able to successfully go through the 
processor.  Not sure if it's related to the size of the flow file or not.  
Perhaps there just needs to be a check around this line in Bin.java:142, to 
make sure we don't try to migrate a flow file to its own session:

session.migrate(getSession(), Collections.singleton(flowFile));


  was:
With the following properties in MergeContent, I get this error when I try to 
process a flow file (which is 135MB):

 
  Merge Strategy
  Bin-Packing Algorithm


  Merge Format
  ZIP


  Attribute Strategy
  Keep Only Common Attributes


  Correlation Attribute Name
  bundle.identifier


  Minimum Number of Entries
  200


  Maximum Number of Entries
  200


  Minimum Group Size
  20 MB


  Maximum Group Size
  20 MB


  Max Bin Age
  1 min


  Maximum number of Bins
  100


  Delimiter Strategy
  Filename


  Header File


  Footer File


  Demarcator File


  Compression Level
  1


  Keep Path
  false


016-11-18 18:00:13,752 ERROR [Timer-Driven Process Thread-83] 
o.a.n.processors.standard.MergeContent
java.lang.IllegalArgumentException: Cannot migrate FlowFiles from a Process 
Session to itself
at 
org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1091)
 ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
  

[jira] [Updated] (NIFI-3066) MergeContent: "Cannot migrate FlowFiles from a Process Session to itself"

[Joseph Gresock (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-3066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Gresock updated NIFI-3066:
-
Description: 
With the following properties in MergeContent, I get this error when I try to 
process a flow file (which is 135MB):

Merge Strategy = Bin-Packing Algorithm
Merge Format = ZIP
Attribute Strategy = Keep Only Common Attributes
Correlation Attribute Name = bundle.identifier
Minimum Number of Entries = 200
Maximum Number of Entries = 200
Minimum Group Size = 20 MB
Maximum Group Size = *20 MB*
Max Bin Age = 1 min
Maximum number of Bins = 100
Delimiter Strategy = Filename
Keep Path = false

{code}
016-11-18 18:00:13,752 ERROR [Timer-Driven Process Thread-83] 
o.a.n.processors.standard.MergeContent
java.lang.IllegalArgumentException: Cannot migrate FlowFiles from a Process 
Session to itself
at 
org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1091)
 ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142) 
~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:282) 
~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178) 
~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1089)
 ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
...
2016-11-18 18:00:13,753 WARN [Timer-Driven Process Thread-83] 
o.a.n.processors.standard.MergeContent 
MergeContent[id=a9476272-dab0-3d2f-acef-abf11f7d1b80] Processor 
Administratively Yielded for 1 sec due to processing failure
{code}

However, other smaller flow files are able to successfully go through the 
processor.  Not sure if it's related to the size of the flow file or not.  
Perhaps there just needs to be a check around this line in Bin.java:142, to 
make sure we don't try to migrate a flow file to its own session:

{code}
session.migrate(getSession(), Collections.singleton(flowFile));
{code}


  was:
With the following properties in MergeContent, I get this error when I try to 
process a flow file (which is 135MB):

Merge Strategy = Bin-Packing Algorithm
Merge Format = ZIP
Attribute Strategy = Keep Only Common Attributes
Correlation Attribute Name = bundle.identifier
Minimum Number of Entries = 200
Maximum Number of Entries = 200
Minimum Group Size = 20 MB
Maximum Group Size = *20 MB*
Max Bin Age = 1 min
Maximum number of Bins = 100
Delimiter Strategy = Filename
Keep Path = false

016-11-18 18:00:13,752 ERROR [Timer-Driven Process Thread-83] 
o.a.n.processors.standard.MergeContent
java.lang.IllegalArgumentException: Cannot migrate FlowFiles from a Process 
Session to itself
at 
org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1091)
 ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142) 
~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:282) 
~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178) 
~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1089)
 ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_111]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
[na:1.8.0_111]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [na:1.8.0_111]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [na:1.8.0_111]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_111]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677328#comment-15677328
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r88708219
  
--- Diff: 
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/serialization/CompressableRecordReader.java
 ---
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.provenance.serialization;
+
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.zip.GZIPInputStream;
+
+import org.apache.nifi.provenance.ProvenanceEventRecord;
+import org.apache.nifi.provenance.StandardProvenanceEventRecord;
+import org.apache.nifi.provenance.StandardRecordReader;
+import org.apache.nifi.provenance.toc.TocReader;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.stream.io.ByteCountingInputStream;
+import org.apache.nifi.stream.io.LimitingInputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public abstract class CompressableRecordReader implements RecordReader {
+private static final Logger logger = 
LoggerFactory.getLogger(StandardRecordReader.class);
+
+private final ByteCountingInputStream rawInputStream;
+private final String filename;
+private final int serializationVersion;
+private final boolean compressed;
+private final TocReader tocReader;
+private final int headerLength;
+private final int maxAttributeChars;
+
+private DataInputStream dis;
+private ByteCountingInputStream byteCountingIn;
+
+public CompressableRecordReader(final InputStream in, final String 
filename, final int maxAttributeChars) throws IOException {
+this(in, filename, null, maxAttributeChars);
+}
+
+public CompressableRecordReader(final InputStream in, final String 
filename, final TocReader tocReader, final int maxAttributeChars) throws 
IOException {
+logger.trace("Creating RecordReader for {}", filename);
+
+rawInputStream = new ByteCountingInputStream(in);
+this.maxAttributeChars = maxAttributeChars;
+
+final InputStream limitedStream;
+if (tocReader == null) {
+limitedStream = rawInputStream;
+} else {
+final long offset1 = tocReader.getBlockOffset(1);
+if (offset1 < 0) {
+limitedStream = rawInputStream;
+} else {
+limitedStream = new LimitingInputStream(rawInputStream, 
offset1 - rawInputStream.getBytesConsumed());
+}
+}
+
+final InputStream readableStream;
+if (filename.endsWith(".gz")) {
+readableStream = new BufferedInputStream(new 
GZIPInputStream(limitedStream));
+compressed = true;
+} else {
+readableStream = new BufferedInputStream(limitedStream);
+compressed = false;
+}
+
+byteCountingIn = new ByteCountingInputStream(readableStream);
+dis = new DataInputStream(byteCountingIn);
+
+final String repoClassName = dis.readUTF();
+final int serializationVersion = dis.readInt();
+headerLength = 
repoClassName.getBytes(StandardCharsets.UTF_8).length + 2 + 4; // 2 bytes for 
string length, 4 for integer.
+
+this.serializationVersion = serializationVersion;
+this.filename = filename;
+this.tocReader = tocReader;
+
+readHeader(dis, serialization

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r88708219
  
--- Diff: 
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/serialization/CompressableRecordReader.java
 ---
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.provenance.serialization;
+
+import java.io.DataInputStream;
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.zip.GZIPInputStream;
+
+import org.apache.nifi.provenance.ProvenanceEventRecord;
+import org.apache.nifi.provenance.StandardProvenanceEventRecord;
+import org.apache.nifi.provenance.StandardRecordReader;
+import org.apache.nifi.provenance.toc.TocReader;
+import org.apache.nifi.stream.io.BufferedInputStream;
+import org.apache.nifi.stream.io.ByteCountingInputStream;
+import org.apache.nifi.stream.io.LimitingInputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public abstract class CompressableRecordReader implements RecordReader {
+private static final Logger logger = 
LoggerFactory.getLogger(StandardRecordReader.class);
+
+private final ByteCountingInputStream rawInputStream;
+private final String filename;
+private final int serializationVersion;
+private final boolean compressed;
+private final TocReader tocReader;
+private final int headerLength;
+private final int maxAttributeChars;
+
+private DataInputStream dis;
+private ByteCountingInputStream byteCountingIn;
+
+public CompressableRecordReader(final InputStream in, final String 
filename, final int maxAttributeChars) throws IOException {
+this(in, filename, null, maxAttributeChars);
+}
+
+public CompressableRecordReader(final InputStream in, final String 
filename, final TocReader tocReader, final int maxAttributeChars) throws 
IOException {
+logger.trace("Creating RecordReader for {}", filename);
+
+rawInputStream = new ByteCountingInputStream(in);
+this.maxAttributeChars = maxAttributeChars;
+
+final InputStream limitedStream;
+if (tocReader == null) {
+limitedStream = rawInputStream;
+} else {
+final long offset1 = tocReader.getBlockOffset(1);
+if (offset1 < 0) {
+limitedStream = rawInputStream;
+} else {
+limitedStream = new LimitingInputStream(rawInputStream, 
offset1 - rawInputStream.getBytesConsumed());
+}
+}
+
+final InputStream readableStream;
+if (filename.endsWith(".gz")) {
+readableStream = new BufferedInputStream(new 
GZIPInputStream(limitedStream));
+compressed = true;
+} else {
+readableStream = new BufferedInputStream(limitedStream);
+compressed = false;
+}
+
+byteCountingIn = new ByteCountingInputStream(readableStream);
+dis = new DataInputStream(byteCountingIn);
+
+final String repoClassName = dis.readUTF();
+final int serializationVersion = dis.readInt();
+headerLength = 
repoClassName.getBytes(StandardCharsets.UTF_8).length + 2 + 4; // 2 bytes for 
string length, 4 for integer.
+
+this.serializationVersion = serializationVersion;
+this.filename = filename;
+this.tocReader = tocReader;
+
+readHeader(dis, serializationVersion);
+}
+
+@Override
+public void skipToBlock(final int blockIndex) throws IOException {
+if (tocReader == null) {
+throw new IllegalStateException("Cannot skip to block " + 
blockIndex + " fo

[jira] [Created] (NIFI-3066) MergeContent: "Cannot migrate FlowFiles from a Process Session to itself"

[Joseph Gresock (JIRA)]

Joseph Gresock created NIFI-3066:


 Summary: MergeContent: "Cannot migrate FlowFiles from a Process 
Session to itself"
 Key: NIFI-3066
 URL: https://issues.apache.org/jira/browse/NIFI-3066
 Project: Apache NiFi
  Issue Type: Bug
Affects Versions: 1.1.0
Reporter: Joseph Gresock
 Fix For: 1.1.0


With the following properties in MergeContent, I get this error when I try to 
process a flow file (which is 135MB):

 
  Merge Strategy
  Bin-Packing Algorithm


  Merge Format
  ZIP


  Attribute Strategy
  Keep Only Common Attributes


  Correlation Attribute Name
  bundle.identifier


  Minimum Number of Entries
  200


  Maximum Number of Entries
  200


  Minimum Group Size
  20 MB


  Maximum Group Size
  20 MB


  Max Bin Age
  1 min


  Maximum number of Bins
  100


  Delimiter Strategy
  Filename


  Header File


  Footer File


  Demarcator File


  Compression Level
  1


  Keep Path
  false


016-11-18 18:00:13,752 ERROR [Timer-Driven Process Thread-83] 
o.a.n.processors.standard.MergeContent
java.lang.IllegalArgumentException: Cannot migrate FlowFiles from a Process 
Session to itself
at 
org.apache.nifi.controller.repository.StandardProcessSession.migrate(StandardProcessSession.java:1091)
 ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at org.apache.nifi.processor.util.bin.Bin.offer(Bin.java:142) 
~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.processor.util.bin.BinFiles.binFlowFiles(BinFiles.java:282) 
~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:178) 
~[nifi-processor-utils-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1089)
 ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
 [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_111]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
[na:1.8.0_111]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 [na:1.8.0_111]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 [na:1.8.0_111]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_111]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2016-11-18 18:00:13,753 WARN [Timer-Driven Process Thread-83] 
o.a.n.processors.standard.MergeContent 
MergeContent[id=a9476272-dab0-3d2f-acef-abf11f7d1b80] Processor 
Administratively Yielded for 1 sec due to processing failure

However, other smaller flow files are able to successfully go through the 
processor.  Not sure if it's related to the size of the flow file or not.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2854) Enable repositories to support upgrades and rollback in well defined scenarios

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677301#comment-15677301
 ] 

ASF GitHub Bot commented on NIFI-2854:
--

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r88706738
  
--- Diff: 
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/AbstractRecordWriter.java
 ---
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.provenance;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+import org.apache.nifi.provenance.serialization.RecordWriter;
+import org.apache.nifi.provenance.toc.TocWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public abstract class AbstractRecordWriter implements RecordWriter {
+private static final Logger logger = 
LoggerFactory.getLogger(AbstractRecordWriter.class);
+
+private final File file;
+private final TocWriter tocWriter;
+private final Lock lock = new ReentrantLock();
+
+private volatile boolean dirty = false;
+private volatile boolean closed = false;
+
+private int recordsWritten = 0;
+
+public AbstractRecordWriter(final File file, final TocWriter writer) 
throws IOException {
+logger.trace("Creating Record Writer for {}", file);
+
+this.file = file;
+this.tocWriter = writer;
+}
+
+@Override
+public synchronized void close() throws IOException {
+closed = true;
+
+logger.trace("Closing Record Writer for {}", file == null ? null : 
file.getName());
+
+lock();
--- End diff --

The 'synchronized' and the lock are there to protected two different 
things. The writer exposes an ability to lock it externally so that multiple 
operations (such as writeRecord, flush, etc) can be called atomically without 
anything else being written. The synchronized protects a few different member 
variables. Essentially, it's employing two completely disparate synchronization 
barriers in order to improve the throughput (no need to wait for a writer to 
finish writing many records and flush before returning the number of records 
written via getRecordsWritten() ).

I believe the code was more clear before I separated the writers out into 
abstract classes. As they are now, it is a bit confusing and perhaps is worth 
simply using the lock and not synchronizing for simplicity purposes. However, I 
would be very hesitant to refactor something like that now, as this ticket is 
blocking the 1.1.0 release, and I believe it is correct as-is. It is simply an 
abstraction of existing classes to produce more layers to avoid code repetition.


> Enable repositories to support upgrades and rollback in well defined scenarios
> --
>
> Key: NIFI-2854
> URL: https://issues.apache.org/jira/browse/NIFI-2854
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
> Fix For: 1.1.0
>
>
> The flowfile, swapfile, provenance, and content repositories play a very 
> important roll in NiFi's ability to be safely upgraded and rolled back.  We 
> need to have well documented behaviors, designs, and version adherence so 
> that users can safely rely on these mechanisms.
> Once this is formalized and in place we should update our versioning guidance 
> to reflect this as well.
> The following would be true from NiFi 1.2.0 onward
> * No changes to how the repositories are persisted to disk can be made which 
> will break forward/back

[GitHub] nifi pull request #1202: NIFI-2854: Refactor repositories and swap files to ...

[markap14]

Github user markap14 commented on a diff in the pull request:

https://github.com/apache/nifi/pull/1202#discussion_r88706738
  
--- Diff: 
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/AbstractRecordWriter.java
 ---
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.provenance;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+import org.apache.nifi.provenance.serialization.RecordWriter;
+import org.apache.nifi.provenance.toc.TocWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public abstract class AbstractRecordWriter implements RecordWriter {
+private static final Logger logger = 
LoggerFactory.getLogger(AbstractRecordWriter.class);
+
+private final File file;
+private final TocWriter tocWriter;
+private final Lock lock = new ReentrantLock();
+
+private volatile boolean dirty = false;
+private volatile boolean closed = false;
+
+private int recordsWritten = 0;
+
+public AbstractRecordWriter(final File file, final TocWriter writer) 
throws IOException {
+logger.trace("Creating Record Writer for {}", file);
+
+this.file = file;
+this.tocWriter = writer;
+}
+
+@Override
+public synchronized void close() throws IOException {
+closed = true;
+
+logger.trace("Closing Record Writer for {}", file == null ? null : 
file.getName());
+
+lock();
--- End diff --

The 'synchronized' and the lock are there to protected two different 
things. The writer exposes an ability to lock it externally so that multiple 
operations (such as writeRecord, flush, etc) can be called atomically without 
anything else being written. The synchronized protects a few different member 
variables. Essentially, it's employing two completely disparate synchronization 
barriers in order to improve the throughput (no need to wait for a writer to 
finish writing many records and flush before returning the number of records 
written via getRecordsWritten() ).

I believe the code was more clear before I separated the writers out into 
abstract classes. As they are now, it is a bit confusing and perhaps is worth 
simply using the lock and not synchronizing for simplicity purposes. However, I 
would be very hesitant to refactor something like that now, as this ticket is 
blocking the 1.1.0 release, and I believe it is correct as-is. It is simply an 
abstraction of existing classes to produce more layers to avoid code repetition.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2954) Bouncycastle dependencies are duplicated throughout NARs

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677004#comment-15677004
 ] 

Joseph Witt commented on NIFI-2954:
---

The version compatibility guidance is spelled out here: 
https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility

It is essential to note that this does not break existing processors.  They 
will continue to work just fine.  This is a utility library that would be 
pulled into the nars/extensions themselves.  What it does mean is that when 
someone with an existing extension wants to update to a newer version of the 
nar-bundle parent, if they even use that, or if they want to use a newer 
version of nifi-processor-utils they would have to adjust to the new package 
naming.

I already updated the arcetype.

And I flagged this JIRA as release-notes relevant and will be highlighting this 
for developers purposes on the release notes.

I added comments to the poms of the relevant nifi-*-util modules to explain 
their purpose and the thinking behind dependency management for them.

That said and as commented previously when describing the plan this is the 
least intrusive way to unwind our dependency handling which got us to this 
point.

> Bouncycastle dependencies are duplicated throughout NARs
> 
>
> Key: NIFI-2954
> URL: https://issues.apache.org/jira/browse/NIFI-2954
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Affects Versions: 1.0.0
>Reporter: Aldrin Piri
>Assignee: Joseph Witt
> Fix For: 1.1.0
>
>
> I was working with some of the NiFi dependencies for incorporation with/use 
> by MiNiFi and saw that several libraries were duplicated throughout all NARs.
> Of note were the bouncycastle dependencies which, together, are ~4MB in size.
> {code}
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:23]
> $ find {lib,work} -type f -name '*bcprov-jdk15on*' | wc -l
>   54
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:28]
> $ find {lib,work} -type f -name '*bcpkix*' | wc -l
>   53
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:33]
> $ ls -lash 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcprov-jdk15on-1.54.jar
> 3.2M -rw-r--r-- 1 apiri admin 3.2M Oct  4 12:24 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcprov-jdk15on-1.54.jar
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:39]
> $ ls -lash 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcpkix-jdk15on-1.54.jar
> 660K -rw-r--r-- 1 apiri admin 658K Oct  4 12:24 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcpkix-jdk15on-1.54.jar
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2954) Bouncycastle dependencies are duplicated throughout NARs

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15677011#comment-15677011
 ] 

Joseph Witt commented on NIFI-2954:
---

increasing the priority to blocker.  We need to ensure that all nars which 
contain the bouncycastle deps do in fact account for them and now that many 
have been cleaned up we need to ensure they're not called out in the license on 
those.  This is an important L&N requirement.  This will be an additional 
commit to my existing PR so the review of the current PR which focuses on the 
functional aspect of the change can continue.

> Bouncycastle dependencies are duplicated throughout NARs
> 
>
> Key: NIFI-2954
> URL: https://issues.apache.org/jira/browse/NIFI-2954
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Affects Versions: 1.0.0
>Reporter: Aldrin Piri
>Assignee: Joseph Witt
>Priority: Blocker
> Fix For: 1.1.0
>
>
> I was working with some of the NiFi dependencies for incorporation with/use 
> by MiNiFi and saw that several libraries were duplicated throughout all NARs.
> Of note were the bouncycastle dependencies which, together, are ~4MB in size.
> {code}
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:23]
> $ find {lib,work} -type f -name '*bcprov-jdk15on*' | wc -l
>   54
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:28]
> $ find {lib,work} -type f -name '*bcpkix*' | wc -l
>   53
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:33]
> $ ls -lash 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcprov-jdk15on-1.54.jar
> 3.2M -rw-r--r-- 1 apiri admin 3.2M Oct  4 12:24 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcprov-jdk15on-1.54.jar
> # apiri @ computer in /usr/local/opt/nifi/libexec [17:20:39]
> $ ls -lash 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcpkix-jdk15on-1.54.jar
> 660K -rw-r--r-- 1 apiri admin 658K Oct  4 12:24 
> work/nar/framework/nifi-framework-nar-1.0.0.nar-unpacked/META-INF/bundled-dependencies/bcpkix-jdk15on-1.54.jar
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)