[jira] [Commented] (NIFI-5628) Verify content length on replicated requests
[ https://issues.apache.org/jira/browse/NIFI-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629638#comment-16629638 ] ASF GitHub Bot commented on NIFI-5628: -- Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/3035 Will review... > Verify content length on replicated requests > > > Key: NIFI-5628 > URL: https://issues.apache.org/jira/browse/NIFI-5628 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.7.1 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Critical > > Verify the content-length of requests when replicated in the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3035: NIFI-5628 Added content length check to OkHttpReplicationC...
Github user mcgilman commented on the issue: https://github.com/apache/nifi/pull/3035 Will review... ---
[jira] [Updated] (NIFI-5628) Verify content length on replicated requests
[ https://issues.apache.org/jira/browse/NIFI-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy LoPresto updated NIFI-5628: Status: Patch Available (was: In Progress) > Verify content length on replicated requests > > > Key: NIFI-5628 > URL: https://issues.apache.org/jira/browse/NIFI-5628 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.7.1, 1.7.0, 1.6.0, 1.5.0 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Critical > > Verify the content-length of requests when replicated in the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5628) Verify content length on replicated requests
[ https://issues.apache.org/jira/browse/NIFI-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629631#comment-16629631 ] ASF GitHub Bot commented on NIFI-5628: -- GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/3035 NIFI-5628 Added content length check to OkHttpReplicationClient. Added unit tests. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5628 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3035.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3035 commit 1baead6f525046a613fc4fe494a0d193776ea70f Author: Andy LoPresto Date: 2018-09-27T01:18:22Z NIFI-5628 Added content length check to OkHttpReplicationClient. Added unit tests. > Verify content length on replicated requests > > > Key: NIFI-5628 > URL: https://issues.apache.org/jira/browse/NIFI-5628 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework >Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.7.1 >Reporter: Andy LoPresto >Assignee: Andy LoPresto >Priority: Critical > > Verify the content-length of requests when replicated in the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3035: NIFI-5628 Added content length check to OkHttpRepli...
GitHub user alopresto opened a pull request: https://github.com/apache/nifi/pull/3035 NIFI-5628 Added content length check to OkHttpReplicationClient. Added unit tests. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/alopresto/nifi NIFI-5628 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3035.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3035 commit 1baead6f525046a613fc4fe494a0d193776ea70f Author: Andy LoPresto Date: 2018-09-27T01:18:22Z NIFI-5628 Added content length check to OkHttpReplicationClient. Added unit tests. ---
[jira] [Updated] (NIFIREG-203) Apache Nifi Registry with Git Integration
[
https://issues.apache.org/jira/browse/NIFIREG-203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vijay Mathew updated NIFIREG-203:
-
Priority: Critical (was: Major)
> Apache Nifi Registry with Git Integration
> -
>
> Key: NIFIREG-203
> URL: https://issues.apache.org/jira/browse/NIFIREG-203
> Project: NiFi Registry
> Issue Type: Bug
>Reporter: Vijay Mathew
>Priority: Critical
>
> Please note that the below issue is occurring in the Windows environment.
> From the Nifi V1.7 UI, I am able to commit a test process group to the Git
> Repository, however, when I try to import the process group from Git, UI
> throws an exception "The Flow Registry with ID {} reports that no Flow exists
> with Bucket {}, Flow {}, Version {}".Below is the logs from Registry
> 2018-09-26 19:45:11,543 WARN [NiFi Registry Web Server-19]
> o.a.n.registry.service.RegistryService The specified bucket id
> [64df4f73-160d-4c19-b47c-563804263300] does not exist.
> 2018-09-26 19:45:11,544 INFO [NiFi Registry Web Server-19]
> o.a.n.r.w.m.ResourceNotFoundExceptionMapper
> org.apache.nifi.registry.exception.ResourceNotFoundException: The specified
> bucket ID does not exist in this registry.. Returning Not Found response.
> 2018-09-26 19:45:49,325 ERROR [NiFi Registry Web Server-19]
> o.a.n.r.web.mapper.ThrowableMapper An unexpected error has occurred:
> java.lang.NullPointerException. Returning Internal Server Error response.
> java.lang.NullPointerException: null
> at org.eclipse.jgit.lib.Repository.resolve(Repository.java:526)
> ~[org.eclipse.jgit-4.11.0.201803080745-r.jar:4.11.0.201803080745-r]
> at org.eclipse.jgit.lib.Repository.resolve(Repository.java:487)
> ~[org.eclipse.jgit-4.11.0.201803080745-r.jar:4.11.0.201803080745-r]
> at
> org.apache.nifi.registry.provider.flow.git.GitFlowMetaData.getContent(GitFlowMetaData.java:422)
> ~[nifi-registry-framework-0.3.0.jar:0.3.0]
> at
> org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider.getFlowContent(GitFlowPersistenceProvider.java:180)
> ~[nifi-registry-framework-0.3.0.jar:0.3.0]
> at org.apache.nifi.registry.service.RegistryService.g
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFIREG-203) Apache Nifi Registry with Git Integration
Vijay Mathew created NIFIREG-203:
Summary: Apache Nifi Registry with Git Integration
Key: NIFIREG-203
URL: https://issues.apache.org/jira/browse/NIFIREG-203
Project: NiFi Registry
Issue Type: Bug
Reporter: Vijay Mathew
Please note that the below issue is occurring in the Windows environment.
>From the Nifi V1.7 UI, I am able to commit a test process group to the Git
>Repository, however, when I try to import the process group from Git, UI
>throws an exception "The Flow Registry with ID {} reports that no Flow exists
>with Bucket {}, Flow {}, Version {}".Below is the logs from Registry
2018-09-26 19:45:11,543 WARN [NiFi Registry Web Server-19]
o.a.n.registry.service.RegistryService The specified bucket id
[64df4f73-160d-4c19-b47c-563804263300] does not exist.
2018-09-26 19:45:11,544 INFO [NiFi Registry Web Server-19]
o.a.n.r.w.m.ResourceNotFoundExceptionMapper
org.apache.nifi.registry.exception.ResourceNotFoundException: The specified
bucket ID does not exist in this registry.. Returning Not Found response.
2018-09-26 19:45:49,325 ERROR [NiFi Registry Web Server-19]
o.a.n.r.web.mapper.ThrowableMapper An unexpected error has occurred:
java.lang.NullPointerException. Returning Internal Server Error response.
java.lang.NullPointerException: null
at org.eclipse.jgit.lib.Repository.resolve(Repository.java:526)
~[org.eclipse.jgit-4.11.0.201803080745-r.jar:4.11.0.201803080745-r]
at org.eclipse.jgit.lib.Repository.resolve(Repository.java:487)
~[org.eclipse.jgit-4.11.0.201803080745-r.jar:4.11.0.201803080745-r]
at
org.apache.nifi.registry.provider.flow.git.GitFlowMetaData.getContent(GitFlowMetaData.java:422)
~[nifi-registry-framework-0.3.0.jar:0.3.0]
at
org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider.getFlowContent(GitFlowPersistenceProvider.java:180)
~[nifi-registry-framework-0.3.0.jar:0.3.0]
at org.apache.nifi.registry.service.RegistryService.g
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5638) AvroReader Schema Access Strategy "Use 'Schema Text' Property" is ineffiecient.
Stefan Kupstaitis-Dunkler created NIFI-5638: --- Summary: AvroReader Schema Access Strategy "Use 'Schema Text' Property" is ineffiecient. Key: NIFI-5638 URL: https://issues.apache.org/jira/browse/NIFI-5638 Project: Apache NiFi Issue Type: Bug Components: Core Framework Affects Versions: 1.7.0, 1.6.0, 1.5.0 Reporter: Stefan Kupstaitis-Dunkler AvroReader Schema Access Strategy "Use 'Schema Text' Property" is ineffiecient, because schema is getting parsed for every record, discovered by [~markap14]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5637) Swappiness recommendation
Ron Isaacson created NIFI-5637: -- Summary: Swappiness recommendation Key: NIFI-5637 URL: https://issues.apache.org/jira/browse/NIFI-5637 Project: Apache NiFi Issue Type: Task Components: Documentation Website Reporter: Ron Isaacson Looking at the [System Administrator's Guide|https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html], I see that you recommend setting vm.swappiness = 0 in /etc/sysctl.conf. This is fine in most cases, but must be avoided on RHEL 6.4 through 6.7. On RedHat kernels pre-2.6.32-642, there's a bug which can cause the OOM killer to aggressively kill processes even when RAM is still available and swap is unused. See [this RedHat article|https://access.redhat.com/solutions/1149413] for more information. (A RedHat support login is required; if you don't have one, please contact me.) I kindly recommend you add a note to your docs advising users not to apply this setting on affected kernels. Thank you! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5224) Add SolrClientService
[ https://issues.apache.org/jira/browse/NIFI-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629334#comment-16629334 ] Mike Thomsen commented on NIFI-5224: Congratulations! That certainly is a good reason to have no time these days! Unless I hear back saying otherwise, I'll unassign you from any tickets you're assigned to at some point soon. > Add SolrClientService > - > > Key: NIFI-5224 > URL: https://issues.apache.org/jira/browse/NIFI-5224 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Johannes Peter >Assignee: Mike Thomsen >Priority: Major > > The Solr CRUD functions that are currently included in SolrUtils should be > moved to a controller service. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (NIFI-5224) Add SolrClientService
[ https://issues.apache.org/jira/browse/NIFI-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629334#comment-16629334 ] Mike Thomsen edited comment on NIFI-5224 at 9/26/18 8:05 PM: - Congratulations! That certainly is a good reason to have no time these days! Unless I hear back saying otherwise, I'll unassign you from any tickets you're assigned to at some point soon. That way you won't get bothered anymore. was (Author: mike.thomsen): Congratulations! That certainly is a good reason to have no time these days! Unless I hear back saying otherwise, I'll unassign you from any tickets you're assigned to at some point soon. > Add SolrClientService > - > > Key: NIFI-5224 > URL: https://issues.apache.org/jira/browse/NIFI-5224 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Johannes Peter >Assignee: Mike Thomsen >Priority: Major > > The Solr CRUD functions that are currently included in SolrUtils should be > moved to a controller service. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-578) Validate that Content-Length is being sent for agent posts.
[ https://issues.apache.org/jira/browse/MINIFICPP-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-578: - Fix Version/s: 0.5.0 > Validate that Content-Length is being sent for agent posts. > > > Key: MINIFICPP-578 > URL: https://issues.apache.org/jira/browse/MINIFICPP-578 > Project: NiFi MiNiFi C++ > Issue Type: Question >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.5.0 > > > It would help to have the content length. I'm fairly certain that it exists, > but this ticket is to open that question when time allows. If it does not > exist this can be changed to a bug to merge it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-578) Validate that Content-Length is being sent for agent posts.
[ https://issues.apache.org/jira/browse/MINIFICPP-578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault resolved MINIFICPP-578. -- Resolution: Fixed > Validate that Content-Length is being sent for agent posts. > > > Key: MINIFICPP-578 > URL: https://issues.apache.org/jira/browse/MINIFICPP-578 > Project: NiFi MiNiFi C++ > Issue Type: Question >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.5.0 > > > It would help to have the content length. I'm fairly certain that it exists, > but this ticket is to open that question when time allows. If it does not > exist this can be changed to a bug to merge it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-527) Change Build versions.
[ https://issues.apache.org/jira/browse/MINIFICPP-527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault resolved MINIFICPP-527. -- Resolution: Fixed > Change Build versions. > --- > > Key: MINIFICPP-527 > URL: https://issues.apache.org/jira/browse/MINIFICPP-527 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-537) Fix internal UUID BUILD
[ https://issues.apache.org/jira/browse/MINIFICPP-537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault resolved MINIFICPP-537. -- Resolution: Fixed Appears to be resolved with the windows functionality > Fix internal UUID BUILD > --- > > Key: MINIFICPP-537 > URL: https://issues.apache.org/jira/browse/MINIFICPP-537 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > > cmake -DUSE_SYSTEM_UUID= .. fails to build. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-527) Change Build versions.
[ https://issues.apache.org/jira/browse/MINIFICPP-527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-527: - Fix Version/s: 0.6.0 > Change Build versions. > --- > > Key: MINIFICPP-527 > URL: https://issues.apache.org/jira/browse/MINIFICPP-527 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-537) Fix internal UUID BUILD
[ https://issues.apache.org/jira/browse/MINIFICPP-537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-537: - Fix Version/s: 0.6.0 > Fix internal UUID BUILD > --- > > Key: MINIFICPP-537 > URL: https://issues.apache.org/jira/browse/MINIFICPP-537 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.6.0 > > > cmake -DUSE_SYSTEM_UUID= .. fails to build. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-17) C++11 Structural changes
[ https://issues.apache.org/jira/browse/MINIFICPP-17?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault resolved MINIFICPP-17. - Resolution: Fixed > C++11 Structural changes > > > Key: MINIFICPP-17 > URL: https://issues.apache.org/jira/browse/MINIFICPP-17 > Project: NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Mr TheSegfault >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-347) Need Test Harness for Processors
[ https://issues.apache.org/jira/browse/MINIFICPP-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault resolved MINIFICPP-347. -- Resolution: Fixed Assignee: Mr TheSegfault Fix Version/s: 0.3.0 > Need Test Harness for Processors > > > Key: MINIFICPP-347 > URL: https://issues.apache.org/jira/browse/MINIFICPP-347 > Project: NiFi MiNiFi C++ > Issue Type: Test >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.3.0 > > > To facilitate testing we need a test harness to load, initialize, execute, > and otherwise stress test processors that are created. The tests that we do > have duplicate efforts that can be built into a clean test harness. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-288) Push flowfiles directly to a processor once persisted to avoid relying on spills to enforce state transition
[ https://issues.apache.org/jira/browse/MINIFICPP-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-288: - Labels: CAPI (was: ) > Push flowfiles directly to a processor once persisted to avoid relying on > spills to enforce state transition > > > Key: MINIFICPP-288 > URL: https://issues.apache.org/jira/browse/MINIFICPP-288 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Priority: Major > Labels: CAPI > > Currently we may rely on multiple threads that ultimately rely on state > transition through the repositories; however, we may be able to avoid this > and do some of the work serially. This should be detectable – this will also > facilitate using the C API and improve throughput and need for repositories. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-288) Push flowfiles directly to a processor once persisted to avoid relying on spills to enforce state transition
[ https://issues.apache.org/jira/browse/MINIFICPP-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-288: - Description: Currently we may rely on multiple threads that ultimately rely on state transition through the repositories; however, we may be able to avoid this and do some of the work serially. This should be detectable – this will also facilitate using the C API and improve throughput and need for repositories. (was: Currently we may rely on multiple threads that ultimately rely on state transition through the repositories; however, we may be able to avoid this and do some of the work serially. This should be detectable. ) > Push flowfiles directly to a processor once persisted to avoid relying on > spills to enforce state transition > > > Key: MINIFICPP-288 > URL: https://issues.apache.org/jira/browse/MINIFICPP-288 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Priority: Major > > Currently we may rely on multiple threads that ultimately rely on state > transition through the repositories; however, we may be able to avoid this > and do some of the work serially. This should be detectable – this will also > facilitate using the C API and improve throughput and need for repositories. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-591) Investigate batching feature in packet capture.
[ https://issues.apache.org/jira/browse/MINIFICPP-591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault resolved MINIFICPP-591. -- Resolution: Invalid > Investigate batching feature in packet capture. > > > Key: MINIFICPP-591 > URL: https://issues.apache.org/jira/browse/MINIFICPP-591 > Project: NiFi MiNiFi C++ > Issue Type: Test >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (MINIFICPP-438) Expression language test fails with json unescape procedure
[ https://issues.apache.org/jira/browse/MINIFICPP-438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault resolved MINIFICPP-438. -- Resolution: Fixed > Expression language test fails with json unescape procedure > --- > > Key: MINIFICPP-438 > URL: https://issues.apache.org/jira/browse/MINIFICPP-438 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-438) Expression language test fails with json unescape procedure
[ https://issues.apache.org/jira/browse/MINIFICPP-438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-438: - Fix Version/s: 0.5.0 > Expression language test fails with json unescape procedure > --- > > Key: MINIFICPP-438 > URL: https://issues.apache.org/jira/browse/MINIFICPP-438 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Fix For: 0.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-493) Create referenced content repository
[ https://issues.apache.org/jira/browse/MINIFICPP-493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-493: - Labels: CAPI (was: ) > Create referenced content repository > > > Key: MINIFICPP-493 > URL: https://issues.apache.org/jira/browse/MINIFICPP-493 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Labels: CAPI > > The content file system content repository moves a file into the content > repo; however, for the purposes of the C API this isn't necessary. Nor is > allocating memory when using the volatile repos. As a result, a referenced > content repo would keep the content in place that is imported ( Such as > GetFile ). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-490) Create common recipes based on the C & C2 api
[ https://issues.apache.org/jira/browse/MINIFICPP-490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-490: - Labels: CAPI (was: ) > Create common recipes based on the C & C2 api > - > > Key: MINIFICPP-490 > URL: https://issues.apache.org/jira/browse/MINIFICPP-490 > Project: NiFi MiNiFi C++ > Issue Type: Sub-task >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Labels: CAPI > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (MINIFICPP-485) Create C connectors for MiNiFi C++ C2 operations
[ https://issues.apache.org/jira/browse/MINIFICPP-485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mr TheSegfault updated MINIFICPP-485: - Labels: CAPI (was: ) > Create C connectors for MiNiFi C++ C2 operations > > > Key: MINIFICPP-485 > URL: https://issues.apache.org/jira/browse/MINIFICPP-485 > Project: NiFi MiNiFi C++ > Issue Type: Epic >Reporter: Mr TheSegfault >Assignee: Mr TheSegfault >Priority: Major > Labels: CAPI > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5479) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFI-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629210#comment-16629210 ] ASF GitHub Bot commented on NIFI-5479: -- Github user thenatog commented on the issue: https://github.com/apache/nifi/pull/3034 Let me know whether the way I've changed these dependencies is acceptable. The dependencies that were modified were causing WARN messages to be shown when NiFi started, for example: ``` 2018-09-26 14:07:54,806 WARN [NiFi Web Server-21] o.e.jetty.annotations.AnnotationParser javax.inject.Provider scanned from multiple locations: jar:file:///.../.../.../nifi/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/work/jetty/nifi-update-attribute-ui-1.8.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.inject-1.jar!/javax/inject/Provi der.class, jar:file:///.../.../.../nifi/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/work/jetty/nifi-update-attribute-ui-1.8.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.inject-2.5.0-b42.jar!/javax/inject/Provider.class ``` This was a result of the Jetty AnnotationParser scanning all classes for annotations. It would add these to a list of scanned classes, and if a new class is scanned which has a previous entry it emits a warning. We saw these warnings for a few reasons: * servlet-api was resolving as 'compile' rather than 'provided', causing it to be included in two places and then scanned twice. * Some libraries were transitive dependencies with older versions. * Some transitive dependencies were what appeared to be repackaged dependencies ie. bundled in a different package name but the same set of classes within. > Upgrade version of Jetty > > > Key: NIFI-5479 > URL: https://issues.apache.org/jira/browse/NIFI-5479 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Matt Gilman >Priority: Blocker > Fix For: 1.8.0 > > > Upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #3034: NIFI-5479 - Fixed up dependencies to remove the WARNs caus...
Github user thenatog commented on the issue: https://github.com/apache/nifi/pull/3034 Let me know whether the way I've changed these dependencies is acceptable. The dependencies that were modified were causing WARN messages to be shown when NiFi started, for example: ``` 2018-09-26 14:07:54,806 WARN [NiFi Web Server-21] o.e.jetty.annotations.AnnotationParser javax.inject.Provider scanned from multiple locations: jar:file:///.../.../.../nifi/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/work/jetty/nifi-update-attribute-ui-1.8.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.inject-1.jar!/javax/inject/Provi der.class, jar:file:///.../.../.../nifi/nifi-assembly/target/nifi-1.8.0-SNAPSHOT-bin/nifi-1.8.0-SNAPSHOT/work/jetty/nifi-update-attribute-ui-1.8.0-SNAPSHOT.war/webapp/WEB-INF/lib/javax.inject-2.5.0-b42.jar!/javax/inject/Provider.class ``` This was a result of the Jetty AnnotationParser scanning all classes for annotations. It would add these to a list of scanned classes, and if a new class is scanned which has a previous entry it emits a warning. We saw these warnings for a few reasons: * servlet-api was resolving as 'compile' rather than 'provided', causing it to be included in two places and then scanned twice. * Some libraries were transitive dependencies with older versions. * Some transitive dependencies were what appeared to be repackaged dependencies ie. bundled in a different package name but the same set of classes within. ---
[jira] [Commented] (NIFI-5479) Upgrade version of Jetty
[ https://issues.apache.org/jira/browse/NIFI-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629202#comment-16629202 ] ASF GitHub Bot commented on NIFI-5479: -- GitHub user thenatog opened a pull request: https://github.com/apache/nifi/pull/3034 NIFI-5479 - Fixed up dependencies to remove the WARNs caused by the A… …nnotationParser in Jetty. NIFI-5479 - Deleted instead of commented out. NIFI-5479 - Removed empty line. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/thenatog/nifi NIFI-5479-rebased Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3034.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3034 commit ea70a2dd6f2b8a5e45970696f85f02078bc5bc87 Author: thenatog Date: 2018-09-07T16:39:18Z NIFI-5479 - Fixed up dependencies to remove the WARNs caused by the AnnotationParser in Jetty. NIFI-5479 - Deleted instead of commented out. NIFI-5479 - Removed empty line. > Upgrade version of Jetty > > > Key: NIFI-5479 > URL: https://issues.apache.org/jira/browse/NIFI-5479 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.7.1 >Reporter: Andy LoPresto >Assignee: Matt Gilman >Priority: Blocker > Fix For: 1.8.0 > > > Upgrade to a new version of Jetty. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3034: NIFI-5479 - Fixed up dependencies to remove the WAR...
GitHub user thenatog opened a pull request: https://github.com/apache/nifi/pull/3034 NIFI-5479 - Fixed up dependencies to remove the WARNs caused by the A⦠â¦nnotationParser in Jetty. NIFI-5479 - Deleted instead of commented out. NIFI-5479 - Removed empty line. Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [x] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? - [x] Is your initial contribution a single, squashed commit? ### For code changes: - [x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/thenatog/nifi NIFI-5479-rebased Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3034.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3034 commit ea70a2dd6f2b8a5e45970696f85f02078bc5bc87 Author: thenatog Date: 2018-09-07T16:39:18Z NIFI-5479 - Fixed up dependencies to remove the WARNs caused by the AnnotationParser in Jetty. NIFI-5479 - Deleted instead of commented out. NIFI-5479 - Removed empty line. ---
[jira] [Commented] (NIFI-5629) GetFile becomes slow listing vast directories
[ https://issues.apache.org/jira/browse/NIFI-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629004#comment-16629004 ] ASF GitHub Bot commented on NIFI-5629: -- GitHub user adyoun2 opened a pull request: https://github.com/apache/nifi/pull/3033 NIFI-5629 GetFile vast listing performance See related PR. When the number of files to pickup becomes a certain size, the OS can become slow performing the listing, so re-use this listing multiple times. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adyoun2/nifi NIFI-5629-GetFile-vast-directory-re-use-listing Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3033.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3033 commit d06534647b7d2aef2069dfebebf4bea3651a8b69 Author: adyoun2 Date: 2018-09-26T16:14:55Z NIFI-5629 GetFile continues to use the same listing stream until 1 files or no files remain > GetFile becomes slow listing vast directories > - > > Key: NIFI-5629 > URL: https://issues.apache.org/jira/browse/NIFI-5629 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.6.0 >Reporter: Adam >Priority: Minor > > GetFile repeatedly lists entire directories before applying batching, meaning > for vast directories it spends a long time listing directories. > > Pull request to follow. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3033: NIFI-5629 GetFile vast listing performance
GitHub user adyoun2 opened a pull request: https://github.com/apache/nifi/pull/3033 NIFI-5629 GetFile vast listing performance See related PR. When the number of files to pickup becomes a certain size, the OS can become slow performing the listing, so re-use this listing multiple times. You can merge this pull request into a Git repository by running: $ git pull https://github.com/adyoun2/nifi NIFI-5629-GetFile-vast-directory-re-use-listing Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3033.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3033 commit d06534647b7d2aef2069dfebebf4bea3651a8b69 Author: adyoun2 Date: 2018-09-26T16:14:55Z NIFI-5629 GetFile continues to use the same listing stream until 1 files or no files remain ---
[jira] [Commented] (NIFI-4360) Add support for Azure Data Lake Store (ADLS)
[ https://issues.apache.org/jira/browse/NIFI-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628876#comment-16628876 ] ASF GitHub Bot commented on NIFI-4360: -- Github user mohitgargk commented on the issue: https://github.com/apache/nifi/pull/2158 Hi @milanchandna, was wondering if you are still working on this. I need this feature for my use case and was going the same path as you. Seems you are almost there. In case you are working on it, Can you please try syncing your fork and rebase your commits. > Add support for Azure Data Lake Store (ADLS) > > > Key: NIFI-4360 > URL: https://issues.apache.org/jira/browse/NIFI-4360 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Milan Chandna >Assignee: Milan Chandna >Priority: Major > Labels: adls, azure, hdfs > Original Estimate: 336h > Remaining Estimate: 336h > > Currently ingress and egress in ADLS account is possible only using HDFS > processors. > Opening this feature to support separate processors for interaction with ADLS > accounts directly. > Benefits are many like > - simple configuration. > - Helping users not familiar with HDFS > - Helping users who currently are accessing ADLS accounts directly. > - using the ADLS SDK rather than HDFS client, one lesser layer to go through. > Can be achieved by adding separate ADLS processors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi issue #2158: NIFI-4360 Adding support for ADLS Processors. Feature incl...
Github user mohitgargk commented on the issue: https://github.com/apache/nifi/pull/2158 Hi @milanchandna, was wondering if you are still working on this. I need this feature for my use case and was going the same path as you. Seems you are almost there. In case you are working on it, Can you please try syncing your fork and rebase your commits. ---
[jira] [Resolved] (NIFIREG-201) nifi-registry-extensions impacted by top-level dependency management
[ https://issues.apache.org/jira/browse/NIFIREG-201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Doran resolved NIFIREG-201. - Resolution: Fixed > nifi-registry-extensions impacted by top-level dependency management > > > Key: NIFIREG-201 > URL: https://issues.apache.org/jira/browse/NIFIREG-201 > Project: NiFi Registry > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Major > Fix For: 0.3.0 > > > While reviewing NIFIREG-200, I noticed that when building with the > include-ranger profile, the JARs in ext/ranger/lib ended up being affected by > the dependency management section in the root pom. > For example, the versions of Jetty and Jackson JARs were being forced to the > versions registry needs, but may not be versions that are compatible with the > ranger client. > To deal with this, I propose restructuring the the repository to something > like the following: > _nifi-registry-core (eveything that used to be at the root, except the > assembly and extensions)_ > _nifi-registry-extensions_ > _nifi-registry-assembly_ > The dependency management can then be moved to the pom of nifi-registry-core > so that it does not impact the modules under nifi-registry-extensions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFIREG-202) Release Manager - Release 0.3.0
[ https://issues.apache.org/jira/browse/NIFIREG-202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Doran resolved NIFIREG-202. - Resolution: Done > Release Manager - Release 0.3.0 > --- > > Key: NIFIREG-202 > URL: https://issues.apache.org/jira/browse/NIFIREG-202 > Project: NiFi Registry > Issue Type: Task >Reporter: Kevin Doran >Assignee: Kevin Doran >Priority: Major > Fix For: 0.3.0 > > > Perform release manager activities for 0.3.0 release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-618) Add C2 triggers for local updates
[
https://issues.apache.org/jira/browse/MINIFICPP-618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628625#comment-16628625
]
ASF GitHub Bot commented on MINIFICPP-618:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/405#discussion_r220525598
--- Diff: libminifi/include/c2/C2Trigger.h ---
@@ -0,0 +1,63 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_C2_C2TRIGGER_H_
+#define LIBMINIFI_INCLUDE_C2_C2TRIGGER_H_
+
+#include "core/Connectable.h"
+#include "c2/C2Payload.h"
+#include "properties/Configure.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace c2 {
+
+class C2Trigger : public core::Connectable{
+ public:
+
+ C2Trigger(std::string name, utils::Identifier uuid)
+: core::Connectable(name, uuid){
+
+ }
+ virtual ~C2Trigger() {
+ }
+
+
+ /**
--- End diff --
You were right. I went through and added some comments and probably would
have missed this had you not mentioned it. Thanks! adding some docs now to our
readmes.
Made an early PR to give some people a chance to review and test -- so
really appreciate all input!
> Add C2 triggers for local updates
> -
>
> Key: MINIFICPP-618
> URL: https://issues.apache.org/jira/browse/MINIFICPP-618
> Project: NiFi MiNiFi C++
> Issue Type: New Feature
>Reporter: Mr TheSegfault
>Assignee: Mr TheSegfault
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #405: MINIFICPP-618: Add C2 triggers, first of ...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/405#discussion_r220525598
--- Diff: libminifi/include/c2/C2Trigger.h ---
@@ -0,0 +1,63 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_C2_C2TRIGGER_H_
+#define LIBMINIFI_INCLUDE_C2_C2TRIGGER_H_
+
+#include "core/Connectable.h"
+#include "c2/C2Payload.h"
+#include "properties/Configure.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace c2 {
+
+class C2Trigger : public core::Connectable{
+ public:
+
+ C2Trigger(std::string name, utils::Identifier uuid)
+: core::Connectable(name, uuid){
+
+ }
+ virtual ~C2Trigger() {
+ }
+
+
+ /**
--- End diff --
You were right. I went through and added some comments and probably would
have missed this had you not mentioned it. Thanks! adding some docs now to our
readmes.
Made an early PR to give some people a chance to review and test -- so
really appreciate all input!
---
[jira] [Commented] (MINIFICPP-618) Add C2 triggers for local updates
[
https://issues.apache.org/jira/browse/MINIFICPP-618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628610#comment-16628610
]
ASF GitHub Bot commented on MINIFICPP-618:
--
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/405#discussion_r220403308
--- Diff: libminifi/include/c2/C2Trigger.h ---
@@ -0,0 +1,63 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_C2_C2TRIGGER_H_
+#define LIBMINIFI_INCLUDE_C2_C2TRIGGER_H_
+
+#include "core/Connectable.h"
+#include "c2/C2Payload.h"
+#include "properties/Configure.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace c2 {
+
+class C2Trigger : public core::Connectable{
+ public:
+
+ C2Trigger(std::string name, utils::Identifier uuid)
+: core::Connectable(name, uuid){
+
+ }
+ virtual ~C2Trigger() {
+ }
+
+
+ /**
--- End diff --
I think this comment is for a different function
> Add C2 triggers for local updates
> -
>
> Key: MINIFICPP-618
> URL: https://issues.apache.org/jira/browse/MINIFICPP-618
> Project: NiFi MiNiFi C++
> Issue Type: New Feature
>Reporter: Mr TheSegfault
>Assignee: Mr TheSegfault
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi-minifi-cpp pull request #405: MINIFICPP-618: Add C2 triggers, first of ...
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/405#discussion_r220403308
--- Diff: libminifi/include/c2/C2Trigger.h ---
@@ -0,0 +1,63 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_C2_C2TRIGGER_H_
+#define LIBMINIFI_INCLUDE_C2_C2TRIGGER_H_
+
+#include "core/Connectable.h"
+#include "c2/C2Payload.h"
+#include "properties/Configure.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace c2 {
+
+class C2Trigger : public core::Connectable{
+ public:
+
+ C2Trigger(std::string name, utils::Identifier uuid)
+: core::Connectable(name, uuid){
+
+ }
+ virtual ~C2Trigger() {
+ }
+
+
+ /**
--- End diff --
I think this comment is for a different function
---
[GitHub] nifi issue #1964: Nifi 4141 - Adding a processor for convert Json to Orc
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/1964 @bbasosuho - do you mind closing this PR now we added the PutORC processor in NiFi? Thanks. ---
[jira] [Commented] (NIFI-1763) Provide an integration with 'Schema registry for Kafka'
[ https://issues.apache.org/jira/browse/NIFI-1763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628595#comment-16628595 ] ASF GitHub Bot commented on NIFI-1763: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1661 > Provide an integration with 'Schema registry for Kafka' > --- > > Key: NIFI-1763 > URL: https://issues.apache.org/jira/browse/NIFI-1763 > Project: Apache NiFi > Issue Type: Wish > Components: Extensions >Reporter: Joseph Witt >Assignee: Mark Payne >Priority: Minor > Fix For: 1.4.0 > > > Reported on a mailing list question on 13 April 2016 > https://github.com/confluentinc/schema-registry > The registry itself is an ASLv2 licensed codebase. It offers a REST-based > Web API for interaction. It would be good to support integration with it for > users of Kafka so it would register schemas if needed when writing to Kafka > and understand how to parse data based on the indicated schema when reading > from Kafka. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #1661: NIFI-1763 [Provide an integration with 'Schema regi...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1661 ---
[jira] [Commented] (NIFI-5612) org.apache.avro.UnresolvedUnionException: Not in union ["null","int"]: 0
[
https://issues.apache.org/jira/browse/NIFI-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628463#comment-16628463
]
ASF GitHub Bot commented on NIFI-5612:
--
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/3032
Hey @colindean, I'll let @bbende & @mattyb149 have a look on the code but
as long as 1 of the 3 Travis builds is green, you're good to go. We have to fix
the intermittent issues on the builds but that's not related to your PR.
> org.apache.avro.UnresolvedUnionException: Not in union ["null","int"]: 0
>
>
> Key: NIFI-5612
> URL: https://issues.apache.org/jira/browse/NIFI-5612
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.5.0, 1.6.0, 1.7.1
> Environment: Microsoft Windows, MySQL Enterprise 5.0.80
>Reporter: Colin Dean
>Priority: Major
> Labels: ExecuteSQL, avro, nifi
>
> I'm seeing this when I execute {{SELECT * FROM }} on a few tables
> but not on dozens of others in the same database.
> {code:java}
> 2018-09-13 01:11:31,434 WARN [Timer-Driven Process Thread-8]
> o.a.n.controller.tasks.ConnectableTask Administratively Yielding
> ExecuteSQL[id=cf5c0996-eddf-3e05-25a3-c407c5edf990] due to uncaught
> Exception: org.apache.avro.file.DataFileWriter$AppendWriteException:
> org.apache.avro.UnresolvedUnionException: Not in union ["null","int"]: 0
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> org.apache.avro.UnresolvedUnionException: Not in union ["null","int"]: 0
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
> at
> org.apache.nifi.processors.standard.util.JdbcCommon.convertToAvroStream(JdbcCommon.java:462)
> at
> org.apache.nifi.processors.standard.ExecuteSQL.lambda$onTrigger$1(ExecuteSQL.java:252)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2625)
> at
> org.apache.nifi.processors.standard.ExecuteSQL.onTrigger(ExecuteSQL.java:242)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.avro.UnresolvedUnionException: Not in union
> ["null","int"]: 0
> at
> org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:709)
> at
> org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:192)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:110)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
> at
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
> ... 15 common frames omitted
> {code}
> I don't know if I can share the database schema – still working with my team
> on that – but looking at it, I think it has something to do with the
> signedness of int(1) or tinyint(1) because those two are the only numerical
> types common to all of the table.
>
> *Edit 2018-09-24, so that my update doesn't get buried:*
> I am able to reproduce the exception using
> * Vagrant 2.1.1
> * Virtualbox 5.2.18 r124319
> * Ubuntu 18.04
> * MySQL 5.0.81 (as close as I can get to the 5.0.80 Enterprise Edition in
> use on the system where I observed this failure first)
>
[GitHub] nifi issue #3032: NIFI-5612: Support JDBC drivers that return Long for unsig...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3032 Hey @colindean, I'll let @bbende & @mattyb149 have a look on the code but as long as 1 of the 3 Travis builds is green, you're good to go. We have to fix the intermittent issues on the builds but that's not related to your PR. ---
[jira] [Commented] (NIFI-5516) Allow data in a Connection to be Load-Balanced across cluster
[
https://issues.apache.org/jira/browse/NIFI-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628399#comment-16628399
]
ASF GitHub Bot commented on NIFI-5516:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r220458441
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/FlowFilePartitioner.java
---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue.clustered.partition;
+
+import org.apache.nifi.controller.repository.FlowFileRecord;
+
+public interface FlowFilePartitioner {
+
+/**
+ * Determines which partition the given FlowFile should go to
+ *
+ * @param flowFile the FlowFile to partition
+ * @param partitions the partitions to choose from
+ * @param localPartition the local partition, which is also included
in the given array of partitions
+ * @return the partition for the FlowFile
+ */
+QueuePartition getPartition(FlowFileRecord flowFile, QueuePartition[]
partitions, QueuePartition localPartition);
+
+/**
+ * @return true if a change in the size of a cluster
should result in re-balancing all FlowFiles in queue,
+ * false if a change in the size of a cluster
does not require re-balancing.
+ */
+boolean isRebalanceOnClusterResize();
+
+/**
+ * @return true if FlowFiles should be rebalanced to
another partition if they cannot be sent to the designated peer,
+ * false if a failure should result in the FlowFiles
remaining in same partition.
+ */
+boolean isRebalanceOnFailure();
--- End diff --
Similarly, `shouldRebalanceOnFailure()`?
> Allow data in a Connection to be Load-Balanced across cluster
> -
>
> Key: NIFI-5516
> URL: https://issues.apache.org/jira/browse/NIFI-5516
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> Allow user to configure a Connection to be load balanced across the cluster.
> For more information, see Feature Proposal at
> https://cwiki.apache.org/confluence/display/NIFI/Load-Balanced+Connections
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5516) Allow data in a Connection to be Load-Balanced across cluster
[
https://issues.apache.org/jira/browse/NIFI-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628400#comment-16628400
]
ASF GitHub Bot commented on NIFI-5516:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r220466208
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/server/ConnectionLoadBalanceServer.java
---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue.clustered.server;
+
+import org.apache.nifi.engine.FlowEngine;
+import org.apache.nifi.events.EventReporter;
+import org.apache.nifi.reporting.Severity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLServerSocket;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.ServerSocket;
+import java.net.Socket;
+import java.net.SocketTimeoutException;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+
+public class ConnectionLoadBalanceServer {
+private static final Logger logger =
LoggerFactory.getLogger(ConnectionLoadBalanceServer.class);
+
+private final String hostname;
+private final int port;
+private final SSLContext sslContext;
+private final ExecutorService threadPool;
+private final LoadBalanceProtocol loadBalanceProtocol;
+private final int connectionTimeoutMillis;
+private final int numThreads;
+private final EventReporter eventReporter;
+
+private volatile Set communicationActions =
Collections.emptySet();
+private final BlockingQueue connectionQueue = new
LinkedBlockingQueue<>();
+
+private volatile AcceptConnection acceptConnection;
+private volatile ServerSocket serverSocket;
+private volatile boolean stopped = true;
+
+public ConnectionLoadBalanceServer(final String hostname, final int
port, final SSLContext sslContext, final int numThreads, final
LoadBalanceProtocol loadBalanceProtocol,
+ final EventReporter eventReporter,
final int connectionTimeoutMillis) {
+this.hostname = hostname;
+this.port = port;
+this.sslContext = sslContext;
+this.loadBalanceProtocol = loadBalanceProtocol;
+this.connectionTimeoutMillis = connectionTimeoutMillis;
+this.numThreads = numThreads;
+this.eventReporter = eventReporter;
+
+threadPool = new FlowEngine(numThreads, "Load Balance Server");
+}
+
+public void start() throws IOException {
+if (!stopped) {
+return;
+}
+
+stopped = false;
+if (serverSocket != null) {
+return;
+}
+
+try {
+serverSocket = createServerSocket();
+} catch (final Exception e) {
+throw new IOException("Could not begin listening for incoming
connections in order to load balance data across the cluster. Please verify the
values of the " +
+"'nifi.cluster.load.balance.port' and
'nifi.cluster.load.balance.host' properties as well as the 'nifi.security.*'
properties", e);
+}
+
+final Set actions = new HashSet<>(numThreads);
+for (int i=0; i < numThreads; i++) {
+final CommunicateAction action = new
CommunicateAction(loadBalanceProtocol);
+actions.add(action);
+threadPool.submit(action);
+}
+
+
[jira] [Commented] (NIFI-5516) Allow data in a Connection to be Load-Balanced across cluster
[
https://issues.apache.org/jira/browse/NIFI-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628401#comment-16628401
]
ASF GitHub Bot commented on NIFI-5516:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r220458232
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/FlowFilePartitioner.java
---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue.clustered.partition;
+
+import org.apache.nifi.controller.repository.FlowFileRecord;
+
+public interface FlowFilePartitioner {
+
+/**
+ * Determines which partition the given FlowFile should go to
+ *
+ * @param flowFile the FlowFile to partition
+ * @param partitions the partitions to choose from
+ * @param localPartition the local partition, which is also included
in the given array of partitions
+ * @return the partition for the FlowFile
+ */
+QueuePartition getPartition(FlowFileRecord flowFile, QueuePartition[]
partitions, QueuePartition localPartition);
+
+/**
+ * @return true if a change in the size of a cluster
should result in re-balancing all FlowFiles in queue,
+ * false if a change in the size of a cluster
does not require re-balancing.
+ */
+boolean isRebalanceOnClusterResize();
--- End diff --
Trivial, but I prefer `shouldRebalanceOnClusterResize()` or
`shouldRebalanceOnClusterTopologyChange`.
> Allow data in a Connection to be Load-Balanced across cluster
> -
>
> Key: NIFI-5516
> URL: https://issues.apache.org/jira/browse/NIFI-5516
> Project: Apache NiFi
> Issue Type: New Feature
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
>
> Allow user to configure a Connection to be load balanced across the cluster.
> For more information, see Feature Proposal at
> https://cwiki.apache.org/confluence/display/NIFI/Load-Balanced+Connections
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #2947: [WIP] NIFI-5516: Implement Load-Balanced Connection...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r220458232
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/FlowFilePartitioner.java
---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue.clustered.partition;
+
+import org.apache.nifi.controller.repository.FlowFileRecord;
+
+public interface FlowFilePartitioner {
+
+/**
+ * Determines which partition the given FlowFile should go to
+ *
+ * @param flowFile the FlowFile to partition
+ * @param partitions the partitions to choose from
+ * @param localPartition the local partition, which is also included
in the given array of partitions
+ * @return the partition for the FlowFile
+ */
+QueuePartition getPartition(FlowFileRecord flowFile, QueuePartition[]
partitions, QueuePartition localPartition);
+
+/**
+ * @return true if a change in the size of a cluster
should result in re-balancing all FlowFiles in queue,
+ * false if a change in the size of a cluster
does not require re-balancing.
+ */
+boolean isRebalanceOnClusterResize();
--- End diff --
Trivial, but I prefer `shouldRebalanceOnClusterResize()` or
`shouldRebalanceOnClusterTopologyChange`.
---
[GitHub] nifi pull request #2947: [WIP] NIFI-5516: Implement Load-Balanced Connection...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r220458441
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/FlowFilePartitioner.java
---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue.clustered.partition;
+
+import org.apache.nifi.controller.repository.FlowFileRecord;
+
+public interface FlowFilePartitioner {
+
+/**
+ * Determines which partition the given FlowFile should go to
+ *
+ * @param flowFile the FlowFile to partition
+ * @param partitions the partitions to choose from
+ * @param localPartition the local partition, which is also included
in the given array of partitions
+ * @return the partition for the FlowFile
+ */
+QueuePartition getPartition(FlowFileRecord flowFile, QueuePartition[]
partitions, QueuePartition localPartition);
+
+/**
+ * @return true if a change in the size of a cluster
should result in re-balancing all FlowFiles in queue,
+ * false if a change in the size of a cluster
does not require re-balancing.
+ */
+boolean isRebalanceOnClusterResize();
+
+/**
+ * @return true if FlowFiles should be rebalanced to
another partition if they cannot be sent to the designated peer,
+ * false if a failure should result in the FlowFiles
remaining in same partition.
+ */
+boolean isRebalanceOnFailure();
--- End diff --
Similarly, `shouldRebalanceOnFailure()`?
---
[GitHub] nifi pull request #2947: [WIP] NIFI-5516: Implement Load-Balanced Connection...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r220466208
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/server/ConnectionLoadBalanceServer.java
---
@@ -0,0 +1,251 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue.clustered.server;
+
+import org.apache.nifi.engine.FlowEngine;
+import org.apache.nifi.events.EventReporter;
+import org.apache.nifi.reporting.Severity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLServerSocket;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.ServerSocket;
+import java.net.Socket;
+import java.net.SocketTimeoutException;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.BlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.TimeUnit;
+
+
+public class ConnectionLoadBalanceServer {
+private static final Logger logger =
LoggerFactory.getLogger(ConnectionLoadBalanceServer.class);
+
+private final String hostname;
+private final int port;
+private final SSLContext sslContext;
+private final ExecutorService threadPool;
+private final LoadBalanceProtocol loadBalanceProtocol;
+private final int connectionTimeoutMillis;
+private final int numThreads;
+private final EventReporter eventReporter;
+
+private volatile Set communicationActions =
Collections.emptySet();
+private final BlockingQueue connectionQueue = new
LinkedBlockingQueue<>();
+
+private volatile AcceptConnection acceptConnection;
+private volatile ServerSocket serverSocket;
+private volatile boolean stopped = true;
+
+public ConnectionLoadBalanceServer(final String hostname, final int
port, final SSLContext sslContext, final int numThreads, final
LoadBalanceProtocol loadBalanceProtocol,
+ final EventReporter eventReporter,
final int connectionTimeoutMillis) {
+this.hostname = hostname;
+this.port = port;
+this.sslContext = sslContext;
+this.loadBalanceProtocol = loadBalanceProtocol;
+this.connectionTimeoutMillis = connectionTimeoutMillis;
+this.numThreads = numThreads;
+this.eventReporter = eventReporter;
+
+threadPool = new FlowEngine(numThreads, "Load Balance Server");
+}
+
+public void start() throws IOException {
+if (!stopped) {
+return;
+}
+
+stopped = false;
+if (serverSocket != null) {
+return;
+}
+
+try {
+serverSocket = createServerSocket();
+} catch (final Exception e) {
+throw new IOException("Could not begin listening for incoming
connections in order to load balance data across the cluster. Please verify the
values of the " +
+"'nifi.cluster.load.balance.port' and
'nifi.cluster.load.balance.host' properties as well as the 'nifi.security.*'
properties", e);
+}
+
+final Set actions = new HashSet<>(numThreads);
+for (int i=0; i < numThreads; i++) {
+final CommunicateAction action = new
CommunicateAction(loadBalanceProtocol);
+actions.add(action);
+threadPool.submit(action);
+}
+
+this.communicationActions = actions;
+
+acceptConnection = new AcceptConnection(serverSocket);
+final Thread receiveConnectionThread = new
Thread(acceptConnection);
+receiveConnectionThread.setName("Receive Queue
[jira] [Commented] (NIFI-5224) Add SolrClientService
[ https://issues.apache.org/jira/browse/NIFI-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628391#comment-16628391 ] Johannes Peter commented on NIFI-5224: -- sorry [~mike.thomsen] for not responding for some reasons the jira notifications arrived in my spam folder. I already have started developing this, but haven't done too much, so go ahead. in the meanwhile, my second kid was born who consumes my entire open source time ;) > Add SolrClientService > - > > Key: NIFI-5224 > URL: https://issues.apache.org/jira/browse/NIFI-5224 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Johannes Peter >Assignee: Mike Thomsen >Priority: Major > > The Solr CRUD functions that are currently included in SolrUtils should be > moved to a controller service. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5516) Allow data in a Connection to be Load-Balanced across cluster
[
https://issues.apache.org/jira/browse/NIFI-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628364#comment-16628364
]
ASF GitHub Bot commented on NIFI-5516:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r220428865
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/client/async/nio/RegisteredPartition.java
---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue.clustered.client.async.nio;
+
+import org.apache.nifi.controller.queue.LoadBalanceCompression;
+import
org.apache.nifi.controller.queue.clustered.client.async.TransactionCompleteCallback;
+import
org.apache.nifi.controller.queue.clustered.client.async.TransactionFailureCallback;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+
+import java.util.function.BooleanSupplier;
+import java.util.function.Supplier;
+
+public class RegisteredPartition {
+private final String connectionId;
+private final Supplier flowFileRecordSupplier;
+private final TransactionFailureCallback failureCallback;
+private final BooleanSupplier emptySupplier;
+private final TransactionCompleteCallback successCallback;
+private final LoadBalanceCompression compression;
+
+public RegisteredPartition(final String connectionId, final
BooleanSupplier emptySupplier, final Supplier flowFileSupplier,
final TransactionFailureCallback failureCallback,
+ final TransactionCompleteCallback
successCallback, final LoadBalanceCompression compression) {
+this.connectionId = connectionId;
+this.emptySupplier = emptySupplier;
+this.flowFileRecordSupplier = flowFileSupplier;
+this.failureCallback = failureCallback;
+this.successCallback = successCallback;
+this.compression = compression;
--- End diff --
I haven't read the code enough to understand how these partitions are
managed. But it seems once a partition is registered, we won't be able to
change its `compression`. The steps for a connection to use compression by
NiFi UI is as follows:
1. Send a POST request to create a connection
2. Send a PUT request to updated load-balance configuration for a connection
But it seems that current PR only support setting compression only when a
connection is added at the sending side. Receiving side uses the latest
compression setting and resulted following Exception:
```
2018-09-26 05:08:18,439 ERROR [Load Balance Server Thread-4]
o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer
nifi0.rumawaks.com/10.1.0.5:35502
java.util.zip.ZipException: Not in GZIP format
at
java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:165)
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:79)
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:91)
at
org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFile(StandardLoadBalanceProtocol.java:428)
at
org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:267)
at
org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:151)
at
org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:176)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
[jira] [Commented] (NIFI-5516) Allow data in a Connection to be Load-Balanced across cluster
[
https://issues.apache.org/jira/browse/NIFI-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628366#comment-16628366
]
ASF GitHub Bot commented on NIFI-5516:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r220440614
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/RemoteQueuePartition.java
---
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue.clustered.partition;
+
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.controller.queue.DropFlowFileRequest;
+import org.apache.nifi.controller.queue.FlowFileQueueContents;
+import org.apache.nifi.controller.queue.LoadBalancedFlowFileQueue;
+import org.apache.nifi.controller.queue.QueueSize;
+import org.apache.nifi.controller.queue.RemoteQueuePartitionDiagnostics;
+import
org.apache.nifi.controller.queue.StandardRemoteQueuePartitionDiagnostics;
+import org.apache.nifi.controller.queue.SwappablePriorityQueue;
+import
org.apache.nifi.controller.queue.clustered.TransferFailureDestination;
+import
org.apache.nifi.controller.queue.clustered.client.async.AsyncLoadBalanceClientRegistry;
+import
org.apache.nifi.controller.queue.clustered.client.async.TransactionCompleteCallback;
+import
org.apache.nifi.controller.queue.clustered.client.async.TransactionFailureCallback;
+import org.apache.nifi.controller.repository.ContentNotFoundException;
+import org.apache.nifi.controller.repository.ContentRepository;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.FlowFileRepository;
+import org.apache.nifi.controller.repository.RepositoryRecord;
+import org.apache.nifi.controller.repository.StandardRepositoryRecord;
+import org.apache.nifi.controller.repository.SwapSummary;
+import org.apache.nifi.controller.repository.claim.ContentClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.flowfile.FlowFilePrioritizer;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.provenance.ProvenanceEventBuilder;
+import org.apache.nifi.provenance.ProvenanceEventRecord;
+import org.apache.nifi.provenance.ProvenanceEventRepository;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.provenance.StandardProvenanceEventRecord;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+/**
+ * A Queue Partition that is responsible for transferring FlowFiles to
another node in the cluster
+ */
+public class RemoteQueuePartition implements QueuePartition {
+private static final Logger logger =
LoggerFactory.getLogger(RemoteQueuePartition.class);
+
+private final NodeIdentifier nodeIdentifier;
+private final SwappablePriorityQueue priorityQueue;
+private final LoadBalancedFlowFileQueue flowFileQueue;
+private final TransferFailureDestination failureDestination;
+
+private final FlowFileRepository flowFileRepo;
+private final ProvenanceEventRepository provRepo;
+private final ContentRepository contentRepo;
+private final AsyncLoadBalanceClientRegistry clientRegistry;
+
+private boolean running = false;
+private final String description;
+
+public RemoteQueuePartition(final NodeIdentifier nodeId, final
SwappablePriorityQueue priorityQueue, final TransferFailureDestination
failureDestination,
+
[GitHub] nifi pull request #2947: [WIP] NIFI-5516: Implement Load-Balanced Connection...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r220440614
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/partition/RemoteQueuePartition.java
---
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue.clustered.partition;
+
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.controller.queue.DropFlowFileRequest;
+import org.apache.nifi.controller.queue.FlowFileQueueContents;
+import org.apache.nifi.controller.queue.LoadBalancedFlowFileQueue;
+import org.apache.nifi.controller.queue.QueueSize;
+import org.apache.nifi.controller.queue.RemoteQueuePartitionDiagnostics;
+import
org.apache.nifi.controller.queue.StandardRemoteQueuePartitionDiagnostics;
+import org.apache.nifi.controller.queue.SwappablePriorityQueue;
+import
org.apache.nifi.controller.queue.clustered.TransferFailureDestination;
+import
org.apache.nifi.controller.queue.clustered.client.async.AsyncLoadBalanceClientRegistry;
+import
org.apache.nifi.controller.queue.clustered.client.async.TransactionCompleteCallback;
+import
org.apache.nifi.controller.queue.clustered.client.async.TransactionFailureCallback;
+import org.apache.nifi.controller.repository.ContentNotFoundException;
+import org.apache.nifi.controller.repository.ContentRepository;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.FlowFileRepository;
+import org.apache.nifi.controller.repository.RepositoryRecord;
+import org.apache.nifi.controller.repository.StandardRepositoryRecord;
+import org.apache.nifi.controller.repository.SwapSummary;
+import org.apache.nifi.controller.repository.claim.ContentClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.flowfile.FlowFilePrioritizer;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.provenance.ProvenanceEventBuilder;
+import org.apache.nifi.provenance.ProvenanceEventRecord;
+import org.apache.nifi.provenance.ProvenanceEventRepository;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.provenance.StandardProvenanceEventRecord;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+/**
+ * A Queue Partition that is responsible for transferring FlowFiles to
another node in the cluster
+ */
+public class RemoteQueuePartition implements QueuePartition {
+private static final Logger logger =
LoggerFactory.getLogger(RemoteQueuePartition.class);
+
+private final NodeIdentifier nodeIdentifier;
+private final SwappablePriorityQueue priorityQueue;
+private final LoadBalancedFlowFileQueue flowFileQueue;
+private final TransferFailureDestination failureDestination;
+
+private final FlowFileRepository flowFileRepo;
+private final ProvenanceEventRepository provRepo;
+private final ContentRepository contentRepo;
+private final AsyncLoadBalanceClientRegistry clientRegistry;
+
+private boolean running = false;
+private final String description;
+
+public RemoteQueuePartition(final NodeIdentifier nodeId, final
SwappablePriorityQueue priorityQueue, final TransferFailureDestination
failureDestination,
+final FlowFileRepository flowFileRepo,
final ProvenanceEventRepository provRepo, final ContentRepository
contentRepository,
+final AsyncLoadBalanceClientRegistry
[GitHub] nifi pull request #2947: [WIP] NIFI-5516: Implement Load-Balanced Connection...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r220454819
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/client/async/nio/NioAsyncLoadBalanceClientTask.java
---
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue.clustered.client.async.nio;
+
+import org.apache.nifi.cluster.coordination.ClusterCoordinator;
+import org.apache.nifi.cluster.coordination.node.NodeConnectionState;
+import org.apache.nifi.cluster.coordination.node.NodeConnectionStatus;
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.events.EventReporter;
+import org.apache.nifi.reporting.Severity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class NioAsyncLoadBalanceClientTask implements Runnable {
+private static final Logger logger =
LoggerFactory.getLogger(NioAsyncLoadBalanceClientTask.class);
+private static final String EVENT_CATEGORY = "Load-Balanced
Connection";
+
+private final NioAsyncLoadBalanceClientRegistry clientRegistry;
+private final ClusterCoordinator clusterCoordinator;
+private final EventReporter eventReporter;
+private volatile boolean running = true;
+
+public NioAsyncLoadBalanceClientTask(final
NioAsyncLoadBalanceClientRegistry clientRegistry, final ClusterCoordinator
clusterCoordinator, final EventReporter eventReporter) {
+this.clientRegistry = clientRegistry;
+this.clusterCoordinator = clusterCoordinator;
+this.eventReporter = eventReporter;
+}
+
+@Override
+public void run() {
+while (running) {
+try {
+boolean success = false;
+for (final NioAsyncLoadBalanceClient client :
clientRegistry.getAllClients()) {
+if (!client.isRunning()) {
+logger.trace("Client {} is not running so will not
communicate with it", client);
+continue;
+}
+
+if (client.isPenalized()) {
+logger.trace("Client {} is penalized so will not
communicate with it", client);
+continue;
+}
+
+final NodeIdentifier clientNodeId =
client.getNodeIdentifier();
+final NodeConnectionStatus connectionStatus =
clusterCoordinator.getConnectionStatus(clientNodeId);
+final NodeConnectionState connectionState =
connectionStatus.getState();
+if (connectionState != NodeConnectionState.CONNECTED) {
--- End diff --
Do we want to rebalance queued FlowFiles here if strategy allows to do so?
FlowFilePartitioner.isRebalanceOnFailure is only evaluated when there is a
communication activity. After a node goes down and a remaining node updates the
connection status to DISCONNECTED, this branch stops further processing. Thus,
FlowFiles will be kept at the connection even though the strategy allows
rebalance.
Scenario, with 2 nodes cluster, using RoundRobin, FlowFiles being created
continuously and distributed evenly. Then one node goes down. 50% of generated
FlowFiles will stay in the connection.
With CorrelationAttributePartitioner, this is the expected behavior. But I
expect FlowFiles to be processed at the remaining node in this case with
RoundRobin partitioner.
---
[jira] [Commented] (NIFI-5516) Allow data in a Connection to be Load-Balanced across cluster
[
https://issues.apache.org/jira/browse/NIFI-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628365#comment-16628365
]
ASF GitHub Bot commented on NIFI-5516:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r220454819
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/client/async/nio/NioAsyncLoadBalanceClientTask.java
---
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue.clustered.client.async.nio;
+
+import org.apache.nifi.cluster.coordination.ClusterCoordinator;
+import org.apache.nifi.cluster.coordination.node.NodeConnectionState;
+import org.apache.nifi.cluster.coordination.node.NodeConnectionStatus;
+import org.apache.nifi.cluster.protocol.NodeIdentifier;
+import org.apache.nifi.events.EventReporter;
+import org.apache.nifi.reporting.Severity;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class NioAsyncLoadBalanceClientTask implements Runnable {
+private static final Logger logger =
LoggerFactory.getLogger(NioAsyncLoadBalanceClientTask.class);
+private static final String EVENT_CATEGORY = "Load-Balanced
Connection";
+
+private final NioAsyncLoadBalanceClientRegistry clientRegistry;
+private final ClusterCoordinator clusterCoordinator;
+private final EventReporter eventReporter;
+private volatile boolean running = true;
+
+public NioAsyncLoadBalanceClientTask(final
NioAsyncLoadBalanceClientRegistry clientRegistry, final ClusterCoordinator
clusterCoordinator, final EventReporter eventReporter) {
+this.clientRegistry = clientRegistry;
+this.clusterCoordinator = clusterCoordinator;
+this.eventReporter = eventReporter;
+}
+
+@Override
+public void run() {
+while (running) {
+try {
+boolean success = false;
+for (final NioAsyncLoadBalanceClient client :
clientRegistry.getAllClients()) {
+if (!client.isRunning()) {
+logger.trace("Client {} is not running so will not
communicate with it", client);
+continue;
+}
+
+if (client.isPenalized()) {
+logger.trace("Client {} is penalized so will not
communicate with it", client);
+continue;
+}
+
+final NodeIdentifier clientNodeId =
client.getNodeIdentifier();
+final NodeConnectionStatus connectionStatus =
clusterCoordinator.getConnectionStatus(clientNodeId);
+final NodeConnectionState connectionState =
connectionStatus.getState();
+if (connectionState != NodeConnectionState.CONNECTED) {
--- End diff --
Do we want to rebalance queued FlowFiles here if strategy allows to do so?
FlowFilePartitioner.isRebalanceOnFailure is only evaluated when there is a
communication activity. After a node goes down and a remaining node updates the
connection status to DISCONNECTED, this branch stops further processing. Thus,
FlowFiles will be kept at the connection even though the strategy allows
rebalance.
Scenario, with 2 nodes cluster, using RoundRobin, FlowFiles being created
continuously and distributed evenly. Then one node goes down. 50% of generated
FlowFiles will stay in the connection.
With CorrelationAttributePartitioner, this is the expected behavior. But I
expect FlowFiles to be processed at the remaining node in this case with
RoundRobin partitioner.
> Allow data in a Connection to be Load-Balanced across cluster
> -
>
> Key: NIFI-5516
>
[GitHub] nifi pull request #2947: [WIP] NIFI-5516: Implement Load-Balanced Connection...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r219366959
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/AbstractFlowFileQueue.java
---
@@ -0,0 +1,460 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue;
+
+import org.apache.nifi.controller.ProcessScheduler;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.FlowFileRepository;
+import org.apache.nifi.controller.repository.RepositoryRecord;
+import org.apache.nifi.controller.repository.claim.ContentClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.provenance.ProvenanceEventBuilder;
+import org.apache.nifi.provenance.ProvenanceEventRecord;
+import org.apache.nifi.provenance.ProvenanceEventRepository;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.util.FormatUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+public abstract class AbstractFlowFileQueue implements FlowFileQueue {
+private static final Logger logger =
LoggerFactory.getLogger(AbstractFlowFileQueue.class);
+private final String identifier;
+private final FlowFileRepository flowFileRepository;
+private final ProvenanceEventRepository provRepository;
+private final ResourceClaimManager resourceClaimManager;
+private final ProcessScheduler scheduler;
+
+private final AtomicReference expirationPeriod = new
AtomicReference<>(new TimePeriod("0 mins", 0L));
+private final AtomicReference maxQueueSize = new
AtomicReference<>(new MaxQueueSize("1 GB", 1024 * 1024 * 1024, 1));
+
+private final ConcurrentMap
listRequestMap = new ConcurrentHashMap<>();
+private final ConcurrentMap
dropRequestMap = new ConcurrentHashMap<>();
+
+private LoadBalanceStrategy loadBalanceStrategy =
LoadBalanceStrategy.DO_NOT_LOAD_BALANCE;
+private String partitioningAttribute = null;
+
+private LoadBalanceCompression compression =
LoadBalanceCompression.DO_NOT_COMPRESS;
+
+
+public AbstractFlowFileQueue(final String identifier, final
ProcessScheduler scheduler,
+final FlowFileRepository flowFileRepo, final
ProvenanceEventRepository provRepo, final ResourceClaimManager
resourceClaimManager) {
+this.identifier = identifier;
+this.scheduler = scheduler;
+this.flowFileRepository = flowFileRepo;
+this.provRepository = provRepo;
+this.resourceClaimManager = resourceClaimManager;
+}
+
+@Override
+public String getIdentifier() {
+return identifier;
+}
+
+protected ProcessScheduler getScheduler() {
+return scheduler;
+}
+
+@Override
+public String getFlowFileExpiration() {
+return expirationPeriod.get().getPeriod();
+}
+
+@Override
+public int getFlowFileExpiration(final TimeUnit timeUnit) {
+return (int) timeUnit.convert(expirationPeriod.get().getMillis(),
TimeUnit.MILLISECONDS);
+}
+
+@Override
+public void setFlowFileExpiration(final String flowExpirationPeriod) {
+final long millis =
[jira] [Commented] (NIFI-5516) Allow data in a Connection to be Load-Balanced across cluster
[
https://issues.apache.org/jira/browse/NIFI-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628367#comment-16628367
]
ASF GitHub Bot commented on NIFI-5516:
--
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r219366959
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/AbstractFlowFileQueue.java
---
@@ -0,0 +1,460 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue;
+
+import org.apache.nifi.controller.ProcessScheduler;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+import org.apache.nifi.controller.repository.FlowFileRepository;
+import org.apache.nifi.controller.repository.RepositoryRecord;
+import org.apache.nifi.controller.repository.claim.ContentClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaim;
+import org.apache.nifi.controller.repository.claim.ResourceClaimManager;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.CoreAttributes;
+import org.apache.nifi.processor.DataUnit;
+import org.apache.nifi.provenance.ProvenanceEventBuilder;
+import org.apache.nifi.provenance.ProvenanceEventRecord;
+import org.apache.nifi.provenance.ProvenanceEventRepository;
+import org.apache.nifi.provenance.ProvenanceEventType;
+import org.apache.nifi.util.FormatUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+public abstract class AbstractFlowFileQueue implements FlowFileQueue {
+private static final Logger logger =
LoggerFactory.getLogger(AbstractFlowFileQueue.class);
+private final String identifier;
+private final FlowFileRepository flowFileRepository;
+private final ProvenanceEventRepository provRepository;
+private final ResourceClaimManager resourceClaimManager;
+private final ProcessScheduler scheduler;
+
+private final AtomicReference expirationPeriod = new
AtomicReference<>(new TimePeriod("0 mins", 0L));
+private final AtomicReference maxQueueSize = new
AtomicReference<>(new MaxQueueSize("1 GB", 1024 * 1024 * 1024, 1));
+
+private final ConcurrentMap
listRequestMap = new ConcurrentHashMap<>();
+private final ConcurrentMap
dropRequestMap = new ConcurrentHashMap<>();
+
+private LoadBalanceStrategy loadBalanceStrategy =
LoadBalanceStrategy.DO_NOT_LOAD_BALANCE;
+private String partitioningAttribute = null;
+
+private LoadBalanceCompression compression =
LoadBalanceCompression.DO_NOT_COMPRESS;
+
+
+public AbstractFlowFileQueue(final String identifier, final
ProcessScheduler scheduler,
+final FlowFileRepository flowFileRepo, final
ProvenanceEventRepository provRepo, final ResourceClaimManager
resourceClaimManager) {
+this.identifier = identifier;
+this.scheduler = scheduler;
+this.flowFileRepository = flowFileRepo;
+this.provRepository = provRepo;
+this.resourceClaimManager = resourceClaimManager;
+}
+
+@Override
+public String getIdentifier() {
+return identifier;
+}
+
+protected ProcessScheduler getScheduler() {
+return scheduler;
+}
+
+@Override
+public String getFlowFileExpiration() {
+return expirationPeriod.get().getPeriod();
+}
+
+@Override
+public int getFlowFileExpiration(final TimeUnit timeUnit) {
+
[GitHub] nifi pull request #2947: [WIP] NIFI-5516: Implement Load-Balanced Connection...
Github user ijokarumawak commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2947#discussion_r220428865
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/clustered/client/async/nio/RegisteredPartition.java
---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.controller.queue.clustered.client.async.nio;
+
+import org.apache.nifi.controller.queue.LoadBalanceCompression;
+import
org.apache.nifi.controller.queue.clustered.client.async.TransactionCompleteCallback;
+import
org.apache.nifi.controller.queue.clustered.client.async.TransactionFailureCallback;
+import org.apache.nifi.controller.repository.FlowFileRecord;
+
+import java.util.function.BooleanSupplier;
+import java.util.function.Supplier;
+
+public class RegisteredPartition {
+private final String connectionId;
+private final Supplier flowFileRecordSupplier;
+private final TransactionFailureCallback failureCallback;
+private final BooleanSupplier emptySupplier;
+private final TransactionCompleteCallback successCallback;
+private final LoadBalanceCompression compression;
+
+public RegisteredPartition(final String connectionId, final
BooleanSupplier emptySupplier, final Supplier flowFileSupplier,
final TransactionFailureCallback failureCallback,
+ final TransactionCompleteCallback
successCallback, final LoadBalanceCompression compression) {
+this.connectionId = connectionId;
+this.emptySupplier = emptySupplier;
+this.flowFileRecordSupplier = flowFileSupplier;
+this.failureCallback = failureCallback;
+this.successCallback = successCallback;
+this.compression = compression;
--- End diff --
I haven't read the code enough to understand how these partitions are
managed. But it seems once a partition is registered, we won't be able to
change its `compression`. The steps for a connection to use compression by
NiFi UI is as follows:
1. Send a POST request to create a connection
2. Send a PUT request to updated load-balance configuration for a connection
But it seems that current PR only support setting compression only when a
connection is added at the sending side. Receiving side uses the latest
compression setting and resulted following Exception:
```
2018-09-26 05:08:18,439 ERROR [Load Balance Server Thread-4]
o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer
nifi0.rumawaks.com/10.1.0.5:35502
java.util.zip.ZipException: Not in GZIP format
at
java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:165)
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:79)
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:91)
at
org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFile(StandardLoadBalanceProtocol.java:428)
at
org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:267)
at
org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:151)
at
org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:176)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
